COMPAS: Conformlex recovers the racial disparity ProPublica exposed

You can be sold a compliance tool on the strength of a demo. Or it can be proven to you on a public dataset, with an already documented bias, that anyone can re-download and check. We chose the second. Meet COMPAS.

The COMPAS case, in brief

In 2016, ProPublica’s Machine Bias investigation showed that COMPAS — software that assigns US defendants a recidivism risk score used by judges — labelled Black defendants “high risk” markedly more often than White defendants. The debate that followed became a textbook case in measuring algorithmic bias.

The investigation’s dataset is public. So it makes an ideal test bench: if an analysis engine deserves your trust, it should recover that bias on its own.

What the engine sees

We point Conformlex at the CSV, set the target to the COMPAS risk score (score_text: Low / Medium / High) and the protected attributes to race, sex, age_cat. We do not tell it which bias to look for. It computes, for each group, the share labelled “High”.

African-American: 27.7% labelled “High”. White: 11.2%. A high-risk score assigned 2.5× more often — from the data alone, with no hint.

This is the disparity ProPublica made famous: the score tilts, and it tilts against the same groups. The engine surfaces it in one command, inside a readable Article 10 dossier.

“Recover”, not “copy”

Let’s be precise, because this is what separates proof from marketing.

Our ×2.5 is a raw ratio: the share labelled “High” among African-Americans divided by that of White defendants, with no controls. ProPublica’s famous figures are different, more elaborate computations — a logistic model controlling for priors, age and sex (+45% odds of a higher score), and a false-positive rate (~2× among defendants who did not reoffend). A raw ratio is mechanically larger than an adjusted one.

So Conformlex points to the same disparity, in the same direction, without claiming to reproduce the investigation’s exact numbers. That is what a screening tool is for: to flag what deserves a deeper look — which an expert then performs. We measure; we do not claim to replace a statistical study.

Two caveats, which the dossier shows in the open rather than hiding:

The maximum raw disparity across racial categories is even higher (×4.8), but it is driven by two tiny groups (Native American n=18, Asian n=32): statistically fragile. The solid figure is the African-American vs White comparison (n=3696 vs 2454).
The COMPAS score is not actual recidivism. We measure the bias of the score, not ground truth — which is precisely the subject of the investigation.

Why it matters for Article 10

Criminal justice is one of the domains the AI Act classifies as high-risk (Annex III). And Article 10 requires the training data of such a system to be examined for bias. COMPAS is the exact illustration of what the text seeks to prevent.

If the engine recovers a documented bias on a public dataset, it will measure yours — on your data, in-place, without it leaving your perimeter, and in a defensible dossier, maintained as your sources change.

Replay it yourself

Nothing to take on faith. The Validation page gives the exact files (source.yaml + mapping.yaml), the command, and the full Article 10 dossier for COMPAS — plus three other public datasets covering employment, credit and health. Grab the CSV, run the Conformlex engine, compare the number.

Indicative guidance, not legal advice, and not a regulatory certification. Conformlex detects documented biases; the AI Act is evolving (Digital Omnibus) — to be confirmed for your case. A structured diagnostic removes the ambiguity.