Benchmarks
AGI Ranker Audit: Every LLM Score Dropped 6-15 Points
A self-audit of the AGI Ranker leaderboard exposed scoring bias that inflated every model by 6-15 points. Here's what the correction actually revealed.
July 31, 2026
A Show HN post landed this week that most leaderboard maintainers would rather bury. The creator of AGI Ranker audited their own scoring scale, shipped a v2.0 methodology change, and republished the whole board with every model dropping between 6 and 15 points.
Not a single score went up. That's the part worth sitting with.
Every score dropped 6-15 points. Nobody's model actually got worse. The measurement just got more honest.
How often does a leaderboard maintainer voluntarily wipe points off every model on their own board? Almost never. Most quietly patch discrepancies and hope nobody diffs the JSON. The AGI Ranker audit is unusual because the author walked through the methodology error in the open, which gives the rest of us something concrete to talk about.
What the AGI Ranker audit actually found
This caught our eye. The short version: the previous methodology normalized scores against assumed human-performance ceilings that, on closer look, were mostly unmeasured guesses. Of the ten benchmarks on the board, only one (GPQA Diamond, at 0.81) has a verified human ceiling. The other nine had been scored against ad-hoc human anchors, which quietly inflated every composite.
Photo by Frelo Design on Unsplash
So the models were being credited for closing a gap toward a target that didn't really exist. Reset the ceiling to the benchmark maximum and the credit disappears.
After recalibration, the leaderboard also rebuilt its Agency component around three independent evaluators instead of leaning on a single source. Every model lost ground. The relative ordering barely moved, which is a useful signal that the audit didn't just reshuffle winners for drama.
The methodology change in plain terms
The core aggregation formula, a weighted mean across five domain components, didn't change. What changed was the inputs.
The recalibrated version does two things:
- Resets the human-ceiling normalization for nine of the ten live benchmarks to the benchmark maximum, since only GPQA Diamond has a measured human anchor
- Reconstructs the Agency component across three independent evaluators (SWE-bench Verified, Terminal-Bench 2.1, τ³-Banking, LiveBench Agentic Coding) to reduce single-source dependence
The ceiling reset alone accounts for roughly six points of the composite drop. The agency rebuild pushes several models further down depending on how much of their previous total leaned on a single evaluator.
And yes, that's a lot of knobs. But the audit post shows the math, which is more transparency than most public leaderboards offer.
Which benchmarks AGI Ranker actually aggregates
Worth pinning down, since this drives everything else. The v2.0 board uses ten live benchmarks:
- GPQA Diamond
- Humanity's Last Exam (HLE)
- ARC-AGI-2
- AIME 2025
- MMMU-Pro
- LiveBench
- SWE-bench Verified
- Terminal-Bench 2.1
- τ³-Banking
- LiveBench Agentic Coding
Notably absent: MMLU, GSM8K, HumanEval, and MATH. Those got retired earlier because the top-of-frontier spread had collapsed on all of them. The v2.0 audit is a separate correction on top of that earlier pruning.
Why saturation matters more than most people think
Benchmark saturation is the quiet crisis in LLM evaluation. When LMSYS Chatbot Arena started, Elo spreads at the top were much wider. Now the top cluster is tightly packed, and the same story played out on MMLU, HumanEval, and GSM8K before AGI Ranker dropped them.
So what does that leave a composite to measure? Composite leaderboards that keep saturated benchmarks at equal weight are effectively measuring noise. The AGI Ranker author's framing on the v2.0 change hits the same nerve, just from the ceiling side rather than the spread side.
The models that dropped hardest
Public commentary around the v2.0 change flagged that models heavily reliant on the old Agency evaluator lost the most composite ground, since the rebuild spread that component across three independent sources. Models that were already strong across GPQA Diamond, HLE, and ARC-AGI-2 held their relative position best.
The current top of the leaderboard rewards models that were doing the hardest work all along. That's exactly what a composite should do.
What this means if you actually use leaderboards for decisions
Most people who look at LLM rankings aren't writing an eval paper — they're picking a model for a task. So the practical takeaway is straightforward.
Stop trusting composite scores in isolation. If you're choosing a model for coding, the SWE-bench Verified, Terminal-Bench, and LiveBench Agentic Coding numbers matter far more than the composite. Our own 2026 coding-LLM benchmark ranking leans on these three for the same reason. If you're picking for research reasoning, GPQA Diamond, HLE, and ARC-AGI-2 tell you more than any aggregate.
Weight by your actual workload. A composite is a rough proxy at best. The AGI Ranker audit works because it acknowledges this, but even the recalibrated composite is still a compression of wildly different capabilities.
Watch for saturation and ceiling assumptions in whatever board you trust. Any benchmark with a sub-3-point spread at the top won't distinguish next year's models, and any board that normalizes against an unmeasured human ceiling is silently inflating everyone.
Should other leaderboards follow suit?
Probably. The Open LLM Leaderboard already handles some of this by rotating benchmarks in and out as they saturate, but the composite math still treats each active benchmark equally. LMSYS Arena sidesteps the problem entirely by using human preference Elo, which is a different can of worms: small sample sizes on niche prompts, formatting bias, model recognition.
Most private eval benchmarks that VCs and hyperscalers rely on have moved to workload-weighted composites years ago (we walked through a public version in our 2026 LLM benchmark showdown). The public leaderboards have been slower, mostly because rotating weights and ceilings breaks longitudinal comparisons and everyone likes clean historical charts.
The AGI Ranker author basically decided accurate rankings today matter more than clean charts back to launch. That's the right call, even if it means the whole board looks worse.
The bigger takeaway
Public self-audits are the exception, not the norm. When they happen, the interesting thing is rarely which model won. It's what the audit reveals about how confident anyone should be in the number underneath the ranking.
Every score dropped 6-15 points. Nobody's model actually got worse. The measurement just got more honest, and that gap between the old and new score is a decent estimate of how much slack every other public leaderboard is still carrying.
So the next time you see a composite score cited in a launch tweet, ask what benchmarks are in the mix, whether they're saturated, whether the human-ceiling assumption is measured, and whether the composite is doing anything more sophisticated than a straight average. Usually the answer is no.
