Annual random-sample measurement of fraud and error
An in-depth review of a random sample of benefit claims each year that estimates the overpayment rate per benefit and splits it by cause into claimant fraud, claimant error, and official error, so counter-fraud effort goes where the money is lost.
Part of the recommended approachTechnique: Random Sampling and Review
The agency draws a random sample of claims in payment for each benefit and reviews each one in depth, checking the claimant’s circumstances against evidence rather than against what the claim says.
The findings give the estimated overpayment rate for each benefit and split it by the circumstance that went wrong and by whether the claimant or the agency caused it.
In Great Britain, the Department for Work and Pensions publishes these estimates each year, and they show undeclared self-employed earnings, undeclared partners, and undeclared savings as the main causes of fraud overpayments.
Those results decide what the data matching and case review programmes target, and they are the only fair benchmark for any of them.
This is the right starting point for any agency that cannot say how much it overpays and why.
It is the wrong tool for catching an individual fraud, which it finds only if it happens to be sampled.
Strengths for this problem
- Gives the agency, the auditor, and the legislature one unbiased estimate of how much is overpaid on each benefit, and why.
- Separates official error from claimant fraud and claimant error, which a referral-driven programme never sees in proportion.
- Its findings are the only labels for any later risk model that were not chosen by caseworker suspicion or an earlier model.
- Sets the hit rate that every targeted review route, data match, or model has to beat.
Limitations for this problem
- Reviews are slow and costly, because each one may need an interview, bank statements, and evidence from employers or landlords.
- The sample is too small to estimate rates for small groups of claimants or for rare kinds of fraud with useful precision.
- Measures overpayments after they happened, so it directs prevention rather than preventing anything itself.
Data needed
- The full population of claims in payment for each benefit, from which the sample is drawn, with the award as assessed at the sample date.
- For each sampled claim: the reviewer's findings, the overpayment or underpayment found, the circumstance it concerned, such as earnings, capital, or living together, and whether it was claimant fraud, claimant error, or official error.
- The evidence each finding rests on, kept so a finding can be checked by a second reviewer and by the auditor.
How it is measured
- Estimated overpayment rate and value per benefit and per cause, with confidence intervals, year on year.
- Share of overpayments that are official error, and whether it falls after process fixes.
- The random-sample hit rate, used as the baseline for every targeted review route.
Pitfalls
- Comparing a targeted review's hit rate with the random sample without using the same threshold for an incorrect payment, which DWP told the National Audit Office limits how far its two figures can be compared.
- Letting reviewers know a sampled case was suspected, which turns the random sample into another targeted review.
- Reporting the overall rate only, when the decision about where to invest depends on the split by cause.
Human oversight
Trained reviewers examine each sampled claim and decide what was incorrect and why, and a second reviewer checks a share of findings. Statisticians own the sample design and the estimates. The national auditor reviews the method. A claimant whose sampled claim is found to be overpaid has the overpayment decided by a caseworker, with the usual rights to reconsideration and appeal.