notes on building fair credit index
What it takes to publish a number that lenders, advocates, and regulators all have reasons to attack — and keep publishing it.
Fair Credit Index publishes a quarterly fairness score for consumer lenders. Simple to say, and the hardest thing we’ve shipped. A benchmark has no product features to hide behind — it’s a number, a methodology, and the willingness of hostile parties to keep citing it. These are working notes from the first two years, for anyone considering building a public benchmark of their own.
define fairness narrowly or not at all
“Fairness” carries a decade of academic argument, and we spent the first two months trying to honor all of it. That version of the index would have satisfied everyone in theory and no one in practice — too many dimensions, each contestable, none comparable. The index that shipped measures three things, precisely defined: approval-rate gaps after controlling for stated underwriting factors, pricing dispersion within risk bands, and adverse-action notice quality.
Narrowness was the price of comparability. We publish what the index doesn’t measure as prominently as what it does, and that page — the limitations page — has done more for our credibility than any methodological cleverness. Critics who arrive to attack the index find their objection already documented, with our reasoning. Some of them end up citing us.
A benchmark is only as strong as its limitations page.
what sits under the score
The public sees a score. Underneath it is a knowledge hub holding every input: the datasets, the cleaning decisions, the methodology versions, and the reasoning behind each change — every one of them dated and citable. When a lender disputes its score, the conversation isn’t an argument; it’s a walk through the hub. Here are your submitted figures, here’s the transformation, here’s the comparable cohort.
Most disputes end at that walk-through. The few that don’t have twice caught real errors in our pipeline — which we published, along with the correction. A benchmark that has never issued a correction is a benchmark nobody has checked.
version the methodology like software
The methodology will change — better data arrives, a blind spot surfaces, a definition improves. The trap is changing it silently, which converts every past score into a lie. We adopted software discipline: versioned releases, a public changelog, and re-stated historical scores whenever a change is material, with both series available. Q1 scores under methodology 2.1 are labeled as such, forever.
This felt like overkill until the first regulator cited the index in guidance. The citation names the version. That’s when versioning stopped being hygiene and became the thing that makes the index citable at all.
what we’d do differently
Start the lender working group earlier — the index got materially better once the scored parties could file structured objections, and we lost two quarters to unstructured ones. Budget for the dispute process as a product surface, not a support queue. And resist the scoreboard instinct: the monthly version we almost shipped would have generated more attention and less trust. Quarterly is slow enough to be right, and a benchmark that is right on schedule, quarter after quarter, is the only kind anyone ends up citing.