Automated overpayment and fraud decisions
Raising a debt, suspending a payment, or recording a fraud finding automatically from a data match or a risk score without a caseworker deciding, described here because it has repeatedly harmed claimants and is the approach this playbook recommends against.
Not recommended for this problemTechnique: Rules Engine
An automated decision system takes a data match or a risk score and acts on it directly, raising a debt, stopping a payment, or recording fraud without a caseworker deciding.
Australia’s Robodebt scheme raised debts from matches against tax office income data and required people to prove they did not owe them, until the Federal Court found the scheme unlawful and the government agreed to repay more than $1,000 million to more than 400,000 people.
Michigan’s unemployment insurance agency used its automated system to decide fraud between 2013 and 2015, and the system’s error rate reached 93% before the agency changed the process in 2015.
Both cases led to large settlements, and Australia’s Royal Commission produced recommendations for review rights, plain-language explanation, and independent scrutiny of automated decision-making that the government accepted in full.
There is no version of this problem where removing the caseworker from a decision that takes money from a claimant is the right choice.
The same data matches and models belong in the recommended solutions, where they select cases for a person to decide.
Strengths for this problem
- Processes a very large number of cases with very few staff, which is why agencies under budget pressure have adopted it.
- Produces the same output for the same inputs, so its behaviour can be audited when its rules are published.
Limitations for this problem
- A data match or a score is not evidence of what a claimant earned in a given period or whether they acted dishonestly, so automatic decisions turn uncertainty into debts.
- Errors are repeated at the scale of the whole caseload before anyone notices, as in Michigan, where the automated system's fraud findings reached an error rate of 93%.
- Puts the burden on claimants to disprove a decision they were never asked about, which Australia's Federal Court found unlawful in the Robodebt scheme.
Data needed
- The same data matches and scores used by the recommended solutions, which is why the difference lies in who decides, not in the data.
How it is measured
- Share of automated decisions overturned on reconsideration, appeal, or in court.
- Number of claimants with debts raised or payments stopped who were later found to owe nothing.
Pitfalls
- Removing the caseworker step to save cost, which was the design choice that made the Robodebt and Michigan systems harmful.
- Defending the system publicly after complaints show it is wrong, which the Royal Commission found the Australian government did with Robodebt from 2017.
- Applying penalties automatically on top of the debt, multiplying the harm of every error.
Human oversight
None by design, which is the reason it is not recommended. Every refusal, suspension, overpayment, and fraud finding should be decided by a caseworker or investigator, with the claimant told what the agency holds and able to respond before the decision.