AI- or Student-Generated Papers

Submitter: Roger Thompson, Stony Brook U

——————————————————

The experiment:

In May 2023, we sent out a set of 13 essays that were either written by students or created with AI. We had 22 faculty participate in the exercise from across all ranks (lec. to full prof). The exercise was available for one week, from May 3 to May 10. Scoring was done via a scale made up of the following domains:

1. Definitely AI-Written
2. Suspected to be AI-Written
3. Suspected to be Student-Written
4. Definitely Student-Written

Results:

Scoring:
AI Paper Incorrectly Labeled a Student Paper: 15%
AI Paper Suspected to be a Student Paper: 19%
Total AI Papers that were incorrectly labeled as Student: 34%
Student Paper Incorrectly Labeled an AI Paper: 4%
Student Paper Suspected to be an AI Paper: 8%
Total Student Papers that were incorrectly labeled as AI: 11%

Accuracy:
66% of the time, we were able to CORRECTLY identify an AI paper
89% of the time, we were able to CORRECTLY identify a student paper.
30% of essays were scored in three different domains
70% of essays were scored across all four domains
2 of 8 (25%) of AI essays had a majority of readers (+50%) score them as student-written.

35% of the time, AI- generated essays were able to secure scores from faculty as being student-written, meaning 1/3 of scores represent an inability to distinguish AI-generated work from student work. Without knowing how readers made their choices, having a full grasp of the implications of this exercise is out of reach, even as the exercise suggests pretty clearly that the AI-written essays are decently effective at fooling our (well-trained and regularly normed) faculty. Interestingly, no single essay was scored 100% accurately, with 70% of the essays having at least one vote in each domain (AI- written to Student-written), and 25% of AI work being labeled by a majority of faculty as student-written.

Relevant resources: I can provide a google doc link or pdf of fuller results, including by paper. Let me know what email address to share to.

Contact: roger[DOT]thompson[AT]stonybrook[DOT]edu