Passau study shows: ChatGPT writes the better school essays

The language model ChatGPT is making enormous progress. After version 3.5 had failed the Bavarian Abitur in early 2023, its successor version 4 earned a solid 2 nearly six months later.

A study by the University of Passau has now been able to demonstrate to what extent AI-generated content could revolutionise the school system. The researchers also experimented with the two language model versions. In a study entitled "A large-scale comparison of human-written versus ChatGPT-generated essays" and published in the prestigious nature journal "Scientific Report", they concluded that the machine writes the better English essays. They had evaluated machine-generated texts and essays written by secondary school students according to guidelines established by the Ministry of Education of Lower Saxony.

I was surprised by how clear the outcome was. This shows that schools shouldn't turn a blind eye on these new tools.

Professor Steffen Herbold

"I was surprised by how clear the outcome was," says Professor Steffen Herbold, who holds the Chair of AI Engineering at the University of Passau and initiated the study. Both Open AI chatbot versions scored higher than the students, with GPT-3 ranking in the middle and GPT-4 achieving the best score. "This shows that schools shouldn't turn a blind eye on these new tools."

Reflecting on AI models

The interdisciplinary study was carried out by the computer scientists in collaboration with computer linguist Professor Annette Hautli-Janisz and computer science didactician Ute Heuer. "I find it important to prepare teachers for the challenges and opportunities coming their way as artificial intelligence models become increasingly available," says computer science didactician Heuer.

She initiated a training course on "ChatGPT – Opportunity and Challenge" that the research team conducted. This event, which had taken place in March 2023, was attended by 139 teachers, most of whom teach at German gymnasiums. The teachers were first briefed on selected technological ideas behind general text generators and ChatGPT. The practical stage then specifically involved English-language texts where the training course participants were left unaware of the origin of these texts.

Using questionnaires, the teachers were asked to evaluate the essays presented to them based on grading scales established by the Ministry of Education of Lower Saxony. Content was evaluated based on the criteria topic, completeness, and logic as well as linguistic aspects like vocabulary, complexity, and language mastery. The research team from Passau defined a scale from 0 to 6 for each criterion, with 0 being the worst score and 6 the best.

The machine scores above average in language mastery

One hundred eleven teachers completed the entire questionnaire and evaluated a total of two hundred seventy English language essays. The research team found the biggest difference in language mastery where the machine scored 5.25 (GPT-4) and 5.03 points (GPT-3) respectively, whereas the students scored an average of 3.9 points. "This does not mean that students have poor English language skills. Rather, the scores achieved by the machine are exceptionally high," underscores Annette Hautli-Janisz, Junior Professor of Computational Rhetoric and Natural Language Processing at the University of Passau.

When we read more AI-generated texts going forward, we'll have to ask ourselves whether and how that affects our human language.

Professor Annette Hautli-Janisz

For Hautli-Janisz, who analysed the texts from a linguistic perspective together with doctoral student Zlata Kikteva, the study provides further exciting insights into the machine's language development. "We have seen how the models change over time and are able to demonstrate with our studies that they have improved in performing the task we give them." The researchers have also been able to identify differences between human and machine-generated language: "When we read more AI-generated texts going forward, we'll have to ask ourselves whether and how that affects our human language," says Hautli-Janisz.

About the research team

Professor Steffen Herbold holds the Chair of AI Engineering at the University of Passau. In his research, he focuses on the quality of AI models. For the present study, he collaborated with Dr Alexander Trautsch to run the statistical analysis and set up a data collection platform. He had developed the study design together with Professor Hautli-Janisz.

Annette Hautli-Janisz is Junior Professor of Computational Rhetoric and Natural Language Processing. Her research interest is in finding out how the argumentative skills of AI-powered language models develop. Besides carrying out the computer linguistic analysis in the study, she also came up with the idea of using a dataset of English language essays written by students available at the Technical University of Darmstadt. The essays were from an online homework forum where students had asked native speakers for feedback to improve their texts. This dataset has been used repeatedly in research.

Ute Heuer is Akademische Direktorin and leads the team in charge of Computer Science Education at the University of Passau. She is responsible for computer science teacher education at the University of Passau, and her research include the didactics of programming education. As part of her work, she initiates training courses for teachers in order to give them an awareness of the opportunities and challenges resulting from the availability of artificial intelligence models.