Validation on public data
A governance platform makes you declare your compliance. We measure the data. To prove it without asking you to take our word for it, we ran the engine on four well-known public datasets with documented biases — and it recovered them.
These four datasets happen to cover four high-risk domains from Annex III of the AI Act.
What the engine measures
| Domain (Annex III) | Dataset | Documented bias | What Conformlex measures |
|---|---|---|---|
| Criminal justice | COMPAS (ProPublica) | the risk score labels Black defendants “high risk” more often | “High” score: African-American 27.7% vs White 11.2% (×2.5) |
| Employment & income | Adult / Census Income (UCI) | gender and race bias on income | sex ×2.8 · race ×2.9 |
| Credit / finance | German Credit — Statlog (UCI) | credit history, status/sex | credit_history ×2.2 · personal_status_sex |
| Health | Heart Disease — Cleveland (UCI) | gap by sex | sex ×2.4 |
Targeting the COMPAS risk score, the engine measures — without being told to — that African-American defendants receive a “High” score 2.5× more often than White defendants (27.7% vs 11.2%). This is the score disparity ProPublica exposed.
These are not our claims: they are biases already documented in the literature and public investigations. The engine measures them automatically, from the data alone — without being told which bias to look for. Our ratio is raw (no statistical controls): it points to the same disparity as ProPublica, without claiming to reproduce their adjusted figures (controlled model, false-positive rates).
Why it matters for Article 10
Article 10 requires your training data to be examined for bias. These four domains are the ones the AI Act classifies as high-risk. If the engine recovers the known biases of COMPAS or Adult, it will measure yours — on your data, in-place (it never leaves your perimeter).
Reproduce it yourself
True to our stance — you keep control, not us — every proof is replayable. Take COMPAS. Grab the public CSV (propublica/compas-analysis), then two readable files describe the whole analysis.
The first names the source:
# source.yaml
source: COMPAS (recidivism)
inputs:
- name: compas
kind: file
path: compas.csvThe second is the column dictionary — which column is the measured outcome (here the COMPAS risk score, not actual recidivism), which are protected attributes, which are personal data never to be analysed:
# mapping.yaml
validated: true
target: score_text # the COMPAS risk score (Low/Medium/High) — the subject of ProPublica's finding
sensitive: # protected attributes → analysed for bias
- race
- sex
- age_cat
pii: # personal data → never analysed
- name
- first
- last
- dob
ignore:
- id
# (any column not listed = "feature", excluded from bias analysis)One command, no network access, and the engine only emits a report.json of aggregates — never
a raw row:
./conformlex-engine --source source.yaml --mapping mapping.yaml --out ./out
# → ./out/report.json — "High" score rate per racial group:
# African-American 27.7% vs Caucasian 11.2% ≈ ×2.5These are exactly the files you would edit on your data — and the ones we ship with the engine’s test suite for the four datasets above. The method is domain- and geography-agnostic: these datasets are international benchmarks, not a limit.
We show aggregates; we do not redistribute the datasets (fetch them from their public sources). Conformlex detects documented biases — it is not a regulatory certification.