Your fraud model has never seen the attack that will work

The documents beside the screen look ordinary. The path across the fields does not.

Labels are an archive of fraud that already succeeded. The model that caught this ring had none of them.

In 2020 a fraud ring ran against a major unsecured payday loan company. It was not small. Across one two-month window, 123 of 8,274 applications traced back to it, sharing a handful of bank routing numbers and a narrow block of IP addresses.

Five models scored those same applications on the same days. One put 85% of the ring in its top few percent of scores. Another put 2%. The one that worked had never been shown a single example of fraud.

Your fraud model's labels are a list of attacks that already worked

A supervised fraud model trains on confirmed fraud, and confirmation is the trouble. An application becomes a labelled fraud after it is funded, after it fails, after somebody investigates and writes it down. That chain runs for months, and it is paid for with the loss. The label is a receipt.

So the training set is not a sample of fraud. It is an archive of fraud that already worked, aged by however long confirmation takes. A pattern that started last month is not in it. A pattern nobody has tried yet cannot be in it.

The gradient-boosted model in this stack learned that archive. It is a competent model. On this ring it put 32.5% of the applications into its top 2% of scores, and left 17% of the ring in the bottom 80%, scored as unremarkable.

The ring passed the credit checks

The model built on Clarity bureau attributes did worse. Clarity is the subprime bureau most of this industry runs on, and the model read the things you would expect: high-risk bank accounts, how many distinct phones and emails share an SSN, how many installment-loan inquiries in the last month, whether the surname matches the phone.

On the ring, that model put 2.4% into its top 2% of scores. Seventeen percent of the ring sat in its bottom 80%.

This is the part worth sitting with. The ring was not defeating the credit data. It was invisible to it. Whatever identities these applications were built on, they presented to a bureau as ordinary thin-file borrowers, which in this market is what almost everyone presents as. Identity data cannot separate a real weak file from a manufactured one when both look the same on paper.

It failed at typing

The model that caught it read none of that. Neuro-ID watches the application session: typing fluency, hesitation before the bank routing field, whether the routing number was edited after entry, whether fields were filled by automation rather than by hand, whether the device registers touch, how long the whole interaction took.

Nothing in that list is a credit attribute. Nothing in it is a fraud label. It is an unsupervised anomaly model over behaviour, and it put 85.4% of the ring in its top bands while leaving 1.6% in the bottom 80%.

Five models, one ring, the same applications on the same days. The behavioural model caught 85.4%; the model trained on labelled fraud caught 32.5%; the bureau-attribute model caught 2.4%.

The lesson is not that unsupervised wins

It would be neat to read that chart as unsupervised beating supervised. It does not say that. Clarity is also unsupervised, and Clarity is the worst bar on the page.

The axis that matters is what the model watches, not how it was trained. Clarity and the supervised model both read identity, and both missed most of the ring. Neuro-ID reads behaviour, and caught it. The blended model, which mixes the two, lands between them, roughly where you would expect a diluted signal to land.

Sorting by training method does not explain the result. Sorting by signal type does. Horizontal placement is a qualitative reading of each model's variable list; the vertical axis is measured.

The ring was novel in identity and ordinary in behaviour. It had bought or built identities that no bureau had a reason to flag. What it could not buy was a person to fill in each form, so it automated the filling, and the automation is what showed.

Two things this result is not

The comparison is in-period. Every model here was fitted and measured on overlapping windows, so these are not out-of-time numbers, and a prospective test would likely be less flattering to all five.

The ring was also defined by a rule, specific routing numbers combined with specific IP blocks, and several of these models carry features built from routing-number and IP-address counts. That inflates their apparent lift on this particular ring by some unknown amount. It does not touch the behavioural model, whose variables are purely session-based, and it makes the identity models look better here than they deserve, not worse. The gap in that chart is a floor.

What the two scores buy together

None of this argues for throwing out the supervised model. It argues for not asking it to do the one job it structurally cannot do. Run both, and route on the combination.

Routing on both scores together: 76% of the bad reviewed at the cost of touching 2.8% of the good. Presented in the source as a demonstration to be tuned to business need, not a recommended cut.

What it costs to find out

Four questions, and none of them needs a new model:

• How old is your newest confirmed fraud label? That number is the lower bound on how long a new pattern runs before your supervised model can learn it.

• If you rank your last fraud event by every score you run, do they agree? Disagreement is where the information is.

• Do you record anything about the session, as opposed to the applicant? Most lenders record nothing.

• What share of your confirmed fraud was caught by a rule somebody wrote after the last incident?

A stack that reads only identity is a stack that can recognise the last attack very well.

---

Take the most recent fraud ring you found, and score it with everything you run. If the only model that ranked it highly was a rule written afterward, you did not detect it. You reconstructed it.