How we score
Every leaderboard is one of three things: the dump (HF), the crowd vote (LMSYS), or the vendor self-report. None are written like a magazine. We try to be the magazine read on top of defensible numbers. This page is how the numbers work.
What we pull
Two sources in Phase 1, both refreshed on cron:
- HuggingFace Open LLM Leaderboard v2. IFEval, BBH, MATH Lvl-5, GPQA-Diamond, MuSR, MMLU-Pro. Standardized lm-eval-harness against open-weight models. Pulled daily at 06:00 UTC.
- LMSYS Chatbot Arena. Overall Elo from crowdsourced pairwise preference. Covers open and closed models. Pulled weekly on Mondays at 06:00 UTC.
What we do not yet pull (Phase 2): vendor model-card numbers (closed-model SWE-bench / vendor-internal scores), SWE-bench Verified, Artificial Analysis price/perf, lm-eval-harness self runs.
Categories
Numbers are grouped under the task you pick a model for. Each benchmark maps to exactly one category:
- Reasoning: BBH, GPQA-Diamond, MuSR, MMLU-Pro.
- Math: MATH Lvl-5.
- Instruction-following: IFEval.
- Vibes (Arena): LMSYS overall Elo.
- Coding / Agent / Long-context: Phase 2 (HumanEval+, SWE-bench Verified, RULER).
A category needs at least two benchmarks per model to publish a rollup. Models that don't meet the threshold do not appear in that category. Not as a zero. Not at all.
The formula
For each model, in each category:
- Convert each raw benchmark score into percentile rank within the field of models we have for that benchmark. Best score in the field maps to ~100; worst to ~0.
- Weight each contribution by recency: weight = exp(-age_days / 180). After six months a score is worth roughly 37% of fresh.
- Combine contributions inside the category with a weighted geometric mean. Geometric mean punishes one-trick-pony models. A model that's great at one thing and dreadful at another lands behind a model that's solid at both.
- When (Phase 2) a vendor self-reports a score, that contribution counts at 0.5x weight and the row is chipped on the UI.
The reference implementation lives in lib/benchmarks/aggregate.ts. The unit tests in lib/benchmarks/__tests__/ assert the properties above: balanced beats spiky, old beats out by fresh, the field needs two benchmarks before it publishes.
Ties and uncertainty
Models within 1.0 points of each other on a rollup are shown as tied. We expose the min/max band of the underlying contributions on hover so you can see whether a model is consistent or spiky.
Provenance
Every row in our fact table records: the source, a deep link (when one is available), the timestamp the source attributes the score to, the timestamp we fetched it, and the raw payload. Numbers are auditable: if a vendor disputes their own score, we can show what we saw and when we saw it.
Changelog
If the formula changes, this list grows.
- 2026-05-28
- Phase 1 ship. Two sources (HuggingFace Open LLM Leaderboard v2, LMSYS Chatbot Arena). Geometric mean of recency-weighted percentile rank per task category. Need >=2 benchmarks per category to publish a rollup. Vendor self-reports not yet ingested (Phase 2).
Spot a mistake or a number that disagrees with what a vendor published? Email editor@ft-news.com. If we got it wrong we'll fix the formula in code, update this page, and add a changelog entry above.