Date of Award
7-2025
Degree Type
Dissertation
Degree Name
Doctor of Education
Department
Human Services
First Advisor
Brian Uriegas
Second Advisor
Ali Hachem
Third Advisor
Luis Aguerrevere
Fourth Advisor
Amber Wagnon
Abstract
This mixed methods study investigated the capacity of customized versions of ChatGPT-4o to generate scores and feedback for student responses to AP English Literature and Composition free response questions. This study entailed creating a customized version of ChatGPT-4o using researcher-created model instructions and released materials from the College Board. The quantitative phase tested the ability of the model to reliably score 56 released essays with official College Board scores; the AI-generated scores were compared to official scores using Cohen’s kappa and quadratic weighted kappa. Next, the four public high school AP English teachers participated in a 30-minute exploration of the customized version of ChatGPT-4o before answering semi-structured interview questions on the usefulness, ease of use, and accuracy of the scores and writing feedback generated by the model. Thematic analysis of the semi-structured interview transcript revealed five major themes. The quantitative and qualitative results were convergent for the accuracy of scoring.
Repository Citation
Harrell, Aaron M., "Developing an AI Teacher Assistant: A Mixed Methods Study of Artificial Intelligence Scoring and Feedback Capabilities" (2025). Electronic Theses and Dissertations. 638.
https://scholarworks.sfasu.edu/etds/638
Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-No Derivative Works 4.0 License.
