You are invited to participate in a research study to improve our understanding of the psychophysiology of students doing AI-assisted software development. The project is led by students, Nadja Brix Koch and Theis Helth Stensgaard, both from the Computer Science Master Programme at the IT University of Copenhagen.
The Research Study
With the increasing use of artificial intelligence (AI) in the software engineering industry, the amount of research on adoption and effects has increased. However, the impact of AI on development is an evolving research topic and there are still areas that remain mostly unexplored. One of such area is the cognitive effect of using AI assistants while writing code.
In recent years, research has found that prompting an AI tool in a correct way enhances the interaction with the tool. We therefore hypothesise that there is a greater cognitive similarity between writing code and writing prompts for an AI than between writing code and working with the output of an AI. Additionally, we hypothesise that working with the output of an AI is cognitively more similar to reading and understanding code than creating new code.
Our study aims to compare the neurological activity of developers while using AI assistants to complete coding tasks and those without such assistants for the same kinds of tasks.
The Procedure
Your participation in the study consists of an experiment of 4 parts, which will take approximately 90 minutes:
- Briefing: In the first part, you will be asked to sign this document and divulge any questions you may have related to the study, prior to the experiment. You will also fill out a demographics questionnaire.
- Setup: In the second part, we will set up the experiment, calibrate the measurement devices and collect a baseline for our analysis.
- Experiment: In the experiment, you will solve 2 programming tasks, one with the use of AI assistance, and one without (the order is random). You will answer a short questionnaire after each task. During the experiment, you will be monitored with an electroencephalographic (EEG) device, an eye-tracker, a mouse- and a keylogger, and the screen will be recorded.
Debriefing: We will have a short conversation about the experiment.
Risks and Adverse Effects
The monitoring devices used in this experiment are non-intrusive, painless and safe. There is a very small risk that EEG can cause seizures if you have a seizure disorder. To ensure the safety of all participants, we exclude people with such disorders from the experiment. Each participant will also receive a coupon valid for any drink sold at Analog Cafe as a token of appreciation for their help.
Outcomes
The data from the experiment will be part of an analysis that will form the basis of our report on the topic. We will happily share the final report if you are interested. Please let us know at the experiment, or by sending us an e-mail.
Participation
You are at all times a voluntary participant in the study. You may discontinue or withdraw participation at any time during the research study. This includes after you have participated in the experiment. You can do so by contacting us via e-mail, after which we will delete all data related to your participation.
Anonymity
No identifying information about yourself will be recorded during the experiment.
Confidentiality
The responses provided will be kept confidential and will be used for academic research purposes only. The data collected during the experiment will not be released or shared outside the research team in a way that allows for identifying you as a participant of the study.
Data Handling
The data from the experiment will be stored on the ITU Github server. The data will be kept until at most 30 days after the end of the study. In case of withdrawal from the study, your data will be deleted within 7 days.
Further Information About the Project
If you have any questions about the study, you are welcome to contact the authors, Nadja Brix Koch, nako@itu.dk, and Theis Helth Stensgaard, thhs@itu.dk.
Experiment Booking
By booking the experiment using the following link you declare that:
- I can write simple projects in Java with multiple classes.
- I’m familiar with Visual Studio Code.
- I’m familiar with simple UML, specifically class, activity and sequence diagrams.
- I’m comfortable writing code on a Danish keyboard.
- I have experience with AI including chat-based AI assistants (ChatGPT, Claude etc.).
- I have used AI for coding assistance.
- I do not have any vision impairment that cannot be corrected with glasses or contact lenses.
- I do not have a history of epilepsy and seizures.
- I’m comfortable sitting still and focusing for 2×20 minutes.