Course chatbot outreach for large gateway courses
A text-message assistant for a large introductory course that sends reminders about assignments, exams, and help sessions, answers routine questions about the course from material the instructors approve, and passes anything personal or difficult to the instructor or an adviser.
Supporting approachTechnique: Retrieval-Augmented Generation
This solution gives students in a large introductory course a text-message assistant that sends reminders about upcoming assignments, exams, and help sessions, and answers routine questions about the course from material the instructors approve.
In randomised trials in two large Georgia State courses, American Government and Microeconomics, the students who were sent messages earned an A or B four percentage points more often than those who were not.
The researchers point to more homework completed and more use of supplemental instruction as the likely reason, and report suggestive evidence that fewer students dropped or withdrew.
That chatbot was non-generative, and a modern build would usually retrieve answers from the course material with a language model, which needs its own evaluation, because a confidently wrong deadline or grading rule harms the students it is meant to help.
The assistant supports course completion in the large introductory courses where many students stumble, which is one step before the retention the other solutions target.
It is the wrong choice where the course material changes faster than anyone will update it, and for any question about a student’s own grades or circumstances, which must go to a person.
Strengths for this problem
- Reaches every student in a course of hundreds with timely reminders that an instructor could not send individually.
- Answers routine questions about deadlines, formats, and where to get help at any hour, which frees instructors' time for students who need more.
- Has been tested in randomised trials in gateway courses, with students who received the messages more likely to earn an A or B.
- Students in the trials were mostly positive, with 82% of those surveyed recommending it be kept and extended.
Limitations for this problem
- The trial evidence is for a non-generative chatbot with scripted messages and curated answers, so a generative assistant has to be evaluated again rather than assumed to perform the same.
- Effects in the trials were on course grades and suggestive on withdrawals, not yet on whether students stay to the next year or graduate.
- A generative assistant can state wrong deadlines, policies, or grading rules with confidence, which in a course with real consequences is worse than no answer.
- Depends on instructors keeping the course material and calendar current, term after term.
Data needed
- The course calendar, assignment and exam dates, grading policy, and help session times, approved by the instructor each term.
- Students' mobile numbers and consent to receive course messages, with a simple way to opt out.
- A curated set of answers to common course questions, and the questions the assistant could not answer, for instructors to review.
- Course completion, withdrawal, and grade records for students who did and did not receive messages, to evaluate it.
How it is measured
- Share of students earning a C or better, and share who withdraw or drop, against a comparison group or the previous term.
- Homework completion and use of supplemental instruction or tutoring among messaged students.
- Share of student questions answered correctly, judged by the instructor on a sample, and share handed to a person.
- Opt-out rate and student satisfaction at the end of the course.
Pitfalls
- Swapping a scripted, curated assistant for a generative one and assuming the trial results carry over.
- Letting the assistant answer questions about grades, extensions, or personal circumstances, which belong to the instructor or an adviser.
- Sending so many messages that students opt out, losing the channel for the messages that matter.
- Not telling students they are talking to an automated assistant.
Human oversight
The instructor approves the course material and messages each term, reviews a sample of answers and every question the assistant could not answer, and decides anything about a student's grade, extension, or standing. Questions about personal, financial, or wellbeing difficulties are passed to an adviser or support service the same day. The course team decides whether to continue the assistant each term based on its results.