Critical AI Pedagogy: Teaching Undergraduate Essay Writing Using ChatGPT

Submitter: Nathan Murray, Algoma U, and Elisa Tersigni, U of Toronto

——————————————————

The experiment:

Students experimented with using ChatGPT to write essays about whether AI is changing research and writing in the field of media studies. In class, we used ChatGPT to generate an essay on this topic, which we then read, critiqued, and finally graded. We considered whether the arguments in it were factual and substantive, and assessed whether the writing was clear and well organized.

A follow-up exercise (take-home exam) asked the students to write an essay about how AI is changing research and writing in media studies and to make an argument about whether those changes are positive, negative, or neutral. Students were permitted to use ChatGPT to generate an essay as long as their use was transparent. The purpose of assigning this essay topic was three-fold: to demonstrate competent essay construction, to demonstrate critical knowledge of AI text generators, and to prevent potential academic integrity issues by making the use of text generators integral to completion of the exam.

Results:

In the classroom critique, students were engaged but not critical of the ChatGPT output. When we asked them to assign a grade to ChatGPT’s essay, they assigned it an A, whereas we assigned it a C+. They were surprised at the difference in grades. They overvalued the clarity of ChatGPT’s language and did not observe the substantive issues with its arguments, which were vague and problematic.

We thought that the first exercise would encourage more critical use of ChatGPT. However, in the follow-up exercise, most students chose to use ChatGPT uncritically. Some students even asked ChatGPT to rewrite their essay in “more academic language,” which rendered it overly wordy and almost incoherent. Moreover, even though ChatGPT’s use was permitted with citation, several students did not cite it.

As the intended learning outcome was not met, in future, we will expand the initial exercise to include a written reflection, giving students an additional opportunity to practice critical AI literacy.

Contact: elisa.tersigni[AT]utoronto[DOT]ca