Persistence risk score to order advisers' outreach
A model trained on past students' records that estimates each current student's chance of not returning next term, used only to order the outreach list within an adviser's caseload, built without race, measured by group, and never shown as a label on the student's record.
Supporting approachTechnique: Supervised Learning
This solution scores each current student’s chance of not returning next term from their records so far and uses it to order the outreach list in each adviser’s caseload.
Research using records of more than 330,000 Virginia community college students found that a simple linear model was only a little less accurate than a complex one, which argues for a transparent model the institution builds itself, and that removing demographic inputs left accuracy almost unchanged while students of colour were still flagged more often.
Later work by the same researchers found that such models would give at-risk Black students fewer resources than lower-risk White students, by an amount that changed with the cutoff chosen, so the bias has to be measured at the cutoff actually used, and that adding race reduced the bias in one of their models and increased it in the other, so leaving race out is a policy choice rather than a fairness fix.
The score’s value depends on what advisers do with it, and research on Georgia State’s advisers found that two of every three of their interventions in a trial were plausibly well targeted using context no model held.
This is the wrong choice for an institution whose advisers cannot reach most of their caseload anyway, since a reordered list they cannot finish changes nothing, and for any use beyond ordering outreach.
Holding back a random or rotating share of students from score-ordered outreach is the most reliable way to show that it beats the advisers’ own judgement.
Strengths for this problem
- Combines grades, credits, registration, and financial signals into one ordering, which helps an adviser with a caseload of hundreds decide whom to call first.
- Can be evaluated against what actually happened to past cohorts, so its accuracy is measurable before it is used.
- Simple models do most of the work, so an institution can build a transparent one from its own records rather than buying an opaque one.
Limitations for this problem
- Accuracy does not translate into graduates on its own, and the largest trial of analytics-enabled advising found no overall effect on graduation or persistence.
- Different models with similar accuracy flag different students, with only 60% overlap between the high-risk groups of a linear and a complex model in the Virginia community college research.
- Carries racial bias even when race is not an input, because grades and other records reflect it, and records predict Black students' success less well, especially for new students.
- Learns from past cohorts whose support, policies, and pandemic disruption may not match the current one.
Data needed
- Term-by-term records as they were known at each point: credits attempted and earned, term and cumulative GPA, registration status, major, holds, and balances owed.
- Whether each past student returned the next term and graduated, from the institution and the National Student Clearinghouse or equivalent, so transfers are not counted as dropouts.
- Race and ethnicity, sex, Pell or low-income status, and first-generation status, held apart from the model's inputs and used only to measure who it flags and misses.
- Advising contact records for flagged and unflagged students, to tell whether outreach based on the score did better than outreach without it.
How it is measured
- Share of students who did not return that the score ranked in the top of the caseload, and share of those ranked at the top who did not return, by group.
- Continuation for students contacted in score order against a randomly or rotationally held out share contacted in the adviser's usual order.
- Gap between groups in the share of at-risk students the score ranks for early contact, at the cutoff actually used.
Pitfalls
- Displaying the score as a "high risk" label on the student's record, where it shapes every adviser's first impression, as reported for EAB Navigate.
- Including race as a predictor, as The Markup found at several universities using Navigate, and as Texas A&M asked EAB to stop after the reporting.
- Showing predicted risk by major, which experts warned would steer Black students away from science and maths.
- Choosing the model on accuracy alone, when models with similar accuracy flag different students and differ in bias by the cutoff used.
- Treating the score as a reason to discourage a student from continuing, which is the opposite of its purpose.
Human oversight
The adviser decides whom to contact, in what order, and what to discuss, and can override the ordering for anything they know about the student. The score is never shown to the student or to faculty, never recorded as a label on the student's record, and never used in admission, aid, academic standing, or a choice of major. The institutional research team measures flagged and missed rates by group each term, and the student success leadership retires the score if outreach in its order does not beat the advisers' own order.