AI-Generated Summaries for Course Selection

Abstract

Many university students use course evaluation guides to select courses. However, these guides do not present course feedback in a way conducive to the course selection process; many offer lists of written comments without providing tools for students to easily analyze these data. A simple improvement to such guides would be the inclusion of AI-generated summaries of the comments. This paper implements a summarization tool for the Harvard Course Evaluation Guide which efficiently summarizes feedback comments through few-shot prompting of ChatGPT with a focus on capturing the overall quality, instructor quality, and workload of each course. Using summaries generated in this way, ChatGPT is better able to rate the qualities of a course than random sampling, using summaries generated through zero-shot prompting, and using the verbatim first five feedback comments. A user study investigating the difference between course selection using feedback comments and using summaries of the comments generated by the summarization tool did not find statistically significant differences. However, summaries might potentially improve understanding of a course’s workload, and qualitative feedback suggested AI-generated summaries offer distinct advantages, especially in terms of cost. Therefore, AI-generated summaries cannot replace feedback comments, but tangibly improve the course selection process.