Your ROC-AUC of 0.97 means nothing
I have been prepping for a senior ML interview at a delivery marketplace recently, and the same question keeps coming back: "how do you evaluate your fraud model?" If you answer "ROC-AUC of 0.97," the interviewer will smile, write something down, and move on. You have quietly lost a piece of the round.
Here is why.
ROC-AUC compresses everything near 1.0
ROC-AUC is the probability that a randomly chosen fraud transaction scores higher than a randomly chosen legitimate one. It is a global ranking metric, averaged across every possible decision threshold from "catch nothing" to "block everyone."
The problem: at 1% fraud prevalence (which is the upper end for real marketplaces), the universe of "possible thresholds" is dominated by what you would do with the other 99% of traffic. You would never operate there. ROC-AUC averages over regions of behavior you would never deploy.
The result is that almost every reasonable fraud model scores 0.95 or higher. Which sounds great. But "0.95 vs. 0.99" is the gap between a model that is losing you money and one that is saving it.
Same two models. Look at the picture.
I generated 50,000 synthetic transactions at 1% fraud prevalence (500 fraud, 49,500 legitimate). Two models score them. Both look excellent on ROC: Model A at 0.975, Model B at 0.997. About three points apart. Sounds tiny.
Now check the right panel. PR-AUC: 0.17 vs. 0.82.
That is not "tiny." That is the difference between a model worth deploying and one you would be embarrassed to ship.
At a real operating point, precision at the top 1% of alerts, Model A is right 16% of the time. Model B is right 66% of the time. Four times the actual fraud per alert sent to a human reviewer. Same ROC range. One number on the slide. Catastrophically different model.
What to report instead
For any classification problem where one class is rare — fraud, account takeover, click fraud, intrusion detection, medical diagnosis — PR-AUC reflects the experience your stakeholder actually feels in production.
Better still, report what your business actually cares about:
Precision at top-k alerts. Your review queue has capacity for 200 cases per hour. What fraction of those will be real?
Capture rate at a fixed false-positive rate. At 0.5% legitimate-decline rate, what share of fraud do you catch?
Net dollars saved. Dollars of fraud blocked minus dollars of legitimate revenue lost to false declines minus the cost of running the review queue. The number an executive will actually approve a launch on.
None of these are ROC-AUC.
Calibration is the other half
Even with PR-AUC, a fraud model is only useful if its scores are calibrated. A score of 0.7 should mean roughly 70% probability of fraud. Without that, you cannot pick thresholds by expected value, and you cannot compare scores across surfaces.
Train on the raw imbalanced distribution (with class weights), then fit Platt scaling or isotonic regression on a held-out set. Plot a reliability diagram. Confirm the curve hugs y = x. Only then talk about your "0.5 threshold."
What the senior interviewer is actually listening for
When the interviewer asks about your metric, they are not asking for ROC-AUC trivia. They are checking whether you understand that the metric and the operating zone are the same conversation.
ROC-AUC tells you about a model. PR-AUC, capture-at-FPR, and expected dollars tell you about a deployment. The good ones live in the deployment zone all day.
If you are prepping for a fraud interview, or if you are building a fraud model yourself, kill the ROC-AUC habit. Report what your alerts will actually look like on Monday morning.
I'm building BrightLearn — an AI-personalized learning platform for K-12 — where calibrated scoring shows up in a different form: how confident is the tutor's assessment of a student's mastery, and when should it ask one more question vs. move on? The metric problem is the same, even when the surface looks different.
Comments