Risk-ranked referrals for caseworker review
A model trained on past review and investigation outcomes that ranks new claims or advance requests by the risk of fraud, and refers the highest-risk ones, mixed with a random control group, to a caseworker who sees neither the score nor that a model chose the case.
Supporting approachTechnique: Supervised Learning
The model scores each new claim or request on the details available at that moment and refers the highest-risk ones to a caseworker, alongside a random control group referred in exactly the same way.
The caseworker does not see the score and is not told whether the model or the random draw selected the case, so the review is an examination of evidence rather than a confirmation of the model.
DWP runs this design for Universal Credit advances and reports that its model identifies high-risk advances around two and a half to three times as often as the control group, with a median payment delay of one day for referred requests that were approved.
Its fairness assessment also found older claimants referred more often while less often confirmed, and its December 2025 transparency record says the model has been retrained and needs further testing before the new version goes live, which is the kind of finding and response this design exists to produce.
This is the wrong choice where an authoritative record can answer the question, and the wrong choice for any agency that cannot measure referral and confirmed rates for the groups it must not discriminate against.
It must never decide a refusal, a suspension, or a debt on its own, and in the EU it is a high-risk system under the AI Act even with a caseworker deciding, because it profiles the people it scores.
Strengths for this problem
- Directs scarce caseworker time at the requests most likely to be wrong, where no authoritative record can settle the question.
- Measured against a random control group referred the same way, so the agency knows how much better than chance it is.
- Can be applied before payment, as DWP does for Universal Credit advances, which prevents an overpayment rather than recovering it.
Limitations for this problem
- Learns from cases that were chosen for review, so it can reproduce whatever bias shaped those choices.
- Can refer some groups more often than their confirmed fraud rate justifies, even when nationality, sex, and other protected characteristics are excluded from its inputs, because other inputs act as proxies.
- Fairness can be measured only for characteristics the agency records, which DWP found limited it to age among the protected characteristics.
- Adds little where the largest losses can be checked directly against employer, tax, or bank records.
Data needed
- Claim or request details available at the moment of the decision, such as claim type, household composition, and reported housing costs, and nothing recorded afterwards.
- The outcome of each referred case as the caseworker decided it, including referrals found to be correct, and the outcomes of the random control group.
- Protected characteristics held apart from the model's inputs, used only for fairness analysis.
- Reconsideration and appeal outcomes, so overturned decisions are not counted as confirmed fraud.
How it is measured
- Confirmed rate among model referrals against the random control group, by count and by value.
- Referral rate and confirmed rate per group, and whether any group's referral rate rises without a matching rise in its confirmed rate.
- Payment delay for legitimate claimants whose requests were referred.
- Share of refusals overturned on reconsideration or appeal.
Pitfalls
- Showing the caseworker the risk score, or telling them the model made the referral, which invites confirmation rather than examination.
- Measuring fairness only by referral rates, so a group that is referred more often and confirmed less often looks the same as one the model is targeting correctly.
- Testing fairness before deployment and assuming it holds, when Amsterdam's model passed its pre-deployment checks for nationality and then wrongly flagged Dutch nationals and women more often in the live pilot.
- Using inputs such as language skills, neighbourhood, or caseworker impressions, which the Rotterdam model used and which act as proxies for ethnicity and other protected characteristics.
- Retraining on the outcomes of the model's own referrals without the control group, so it learns only from the cases it chose.
Human oversight
A caseworker reviews every referred case using all the evidence available and decides whether to approve or decline, without seeing the score or knowing a model made the referral. Declining a request does not stop the claimant asking again or decide their wider claim. Analysts measure referral and confirmed rates per group at least yearly, and senior leaders decide whether the model remains proportionate, publish the assessment, and order retraining or withdrawal when the rates diverge.