Real-SWE tests AI models on private enterprise codebases instead of public GitHub repos. The top frontier model scores 38.8%, exposing how far benchmark hype is from production reality.
Public benchmarks make AI coding models look almost superhuman. Frontier models routinely score above 90% on SWE-bench Verified. And yet, if you've actually shipped one of these agents into a real enterprise repo, you already know the truth: the scores don't survive contact with production code.
That gap is what the new Real-SWE benchmark from Specific is trying to measure. Instead of testing models on scrubbed, public GitHub issues that have almost certainly leaked into training data, Real-SWE runs frontier models against private, real-world enterprise codebases licensed from partner companies. The result is the first benchmark that reflects what engineering leaders have been saying quietly for a year now.
The scores drop. A lot.
The Real-SWE benchmark is a coding evaluation that runs AI models against private, production-grade enterprise repositories rather than public open-source projects. It uses real bug tickets, real pull request histories, and real reviewers from participating companies, which makes contamination from pretraining data effectively impossible.
Each task is drawn from an actual closed PR at a partner company. The model gets the ticket description, access to the repo, and a validation suite that mirrors the company's own CI. A task counts as "resolved" only if the patch passes the exact test the human engineer's patch had to pass. No shortcuts.
That's a much harder bar than SWE-bench Verified, which pulls from a set of well-known Python repos like django, sympy, and astropy. Those repos have been sitting on GitHub for a decade. Anything you've read about, the model has probably read too.
Specific's setup differs from SWE-bench in several ways worth understanding before you read the numbers.
First, the codebases are polyglot. Real enterprise stacks are TypeScript + Go + a legacy Java service + some Python glue. Public benchmarks are almost entirely Python. So a model that memorized every Python idiom on GitHub loses that advantage immediately.
Second, the tasks are cross-cutting. Reference solutions on Real-SWE edit a median of 11 files per task, compared with 6 files for FrontierCode and DeepSWE, according to Specific's published comparison. A single change routinely spans multiple parts of the application.
Third, there's no cached test suite. The model has to figure out how to run the tests, which in a real repo often means reading an internal Makefile, a Bazel target, or a badly documented Docker script. This alone kills a lot of agents.
Based on the leaderboard published by Specific in September 2026, here's how the top frontier model-and-harness combinations scored on Real-SWE.
| Rank | Model | Harness | Resolution Rate |
|---|---|---|---|
| 1 | Fable 5.1 | Claude Code | 38.8% |
| 2 | GPT-6 Astra | Codex CLI | 33.8% |
| 3 | Gemini 3.8 Flash | Gemini CLI | 31.2% |
| 4 | GLM 5.3 | Claude Code | 28.8% |
| =5 | Grok 4.6 | Grok Build | 23.8% |
| =5 | Muse Spark 1.3 | Muse Code | 23.8% |
| 7 | Kimi K3 | Kimi Code | 18.8% |
| 8 | GPT-5.6 Sol | Codex CLI | 16.2% |
Resolution rate is equivalent to pass@1, averaged over eight independent runs per task, according to Specific.
The top score is under 40%. Compare that to the 90%+ numbers these same model families post on SWE-bench Verified and the gap is stark. Whatever the public scores are measuring, it isn't quite the same skill.
And look at GPT-5.6 Sol, the wooden-spoon finisher at 16.2%. The very lineage that dominates on synthetic benchmarks lands at the bottom on real code. Model families do not translate cleanly from public leaderboards to private production.
Specific published a failure taxonomy alongside the scores, and this is the part worth reading twice. The failure modes aren't what you'd expect from a benchmark leaderboard.
Specific groups every failed rollout into one of five categories:
None of these are "the model couldn't solve the problem." They're operational failures rooted in poor context grounding. That's a big deal.
A few results genuinely didn't line up with expectations.
Fable 5.1's lead is surprisingly narrow. The top model tops out at 38.8%, and the second-place model (GPT-6 Astra) is only five points behind. On public benchmarks that are asymptotically pinned near 96%, the margins look enormous. On real code, the frontier is bunched.
Cost doesn't buy accuracy. Specific reports estimated per-rollout costs ranging from $2.50 (Gemini 3.8 Flash) to $6.96 (Fable 5.1). Gemini 3.8 Flash lands third overall at $2.50 per rollout, ahead of models costing twice as much. Not gonna lie, this changes the price-per-fix math a lot compared to public benchmark leaderboards. Engineers on Claude Code and other harnesses have been reporting similar patterns for months.
Some tasks nobody solved. On the "Analytics stream reducer" sample task, every model went 0/8. On "Tax jurisdiction," resolution rate across all models was 3.1%. Six of ten sample tasks resolved below 15%. Frontier models remain very weak on multi-service, business-logic-heavy work.
There are basically three reasons the public numbers are inflated, and Real-SWE isolates each one.
Contamination is the obvious one. If a model saw the fix for a django issue during pretraining, resolving that issue at inference is closer to retrieval than reasoning. Independent researchers have documented contamination issues in SWE-bench since 2024.
The second reason is that public benchmarks reward narrow file-level fixes. Real-SWE requires reference-quality patches, and Specific classifies "wrong file" and "missed requirement" outcomes as failures even when a naive test might have passed.
The third reason is tooling. Every serious agent framework, Claude Code, Devin, Aider, has been tuned against SWE-bench Verified. That tuning doesn't generalize to unfamiliar codebases the frameworks never saw during optimization.
If you're an engineering lead deciding whether to put an AI coding agent on your production repo, the practical takeaway is pretty clear. Frontier models are useful, but the SWE-bench numbers you saw in the marketing deck are lying to you. Budget accordingly.
The gap between benchmark performance and production performance is now the single most important metric in AI coding, and almost no one publishes it.
A few concrete implications:
justfile or Makefile conventions won't rescue every task, but a codebase where the model can't even run the tests is one where no agent will succeed.Specific plans to update the leaderboard as new models ship and to add participating companies over time. If the methodology holds up, this becomes the benchmark that matters for anyone actually shipping AI-authored code. The public ones will still get the marketing screenshots. But engineering decisions should start weighting Real-SWE more heavily than SWE-bench Verified.
And for what it's worth, the honest scores are more useful anyway. A benchmark that says "the best model gets 38.8%" is planning-actionable. A benchmark that says "96%" just makes your Slack full of hype threads.
Specific licenses private codebases from partner companies for Real-SWE tasks. There is no public application form listed on the benchmark page; interested companies typically reach out to Specific directly to discuss participation. Task selection prioritizes real production workloads with substantial engineering teams and business consequences.
Yes. Specific's task environments include Go, Python, Node.js, TypeScript, and multiple databases (PostgreSQL, MySQL, MongoDB, Redis). This is one of the main methodological differences from SWE-bench, which is almost entirely Python-based and does not reflect polyglot enterprise stacks.
Based on the September 2026 Real-SWE leaderboard, Fable 5.1 driven through Claude Code was the top performer at 38.8% resolution rate, followed by GPT-6 Astra on Codex CLI at 33.8%. If cost is a factor, Gemini 3.8 Flash on Gemini CLI landed third at 31.2% for the lowest per-rollout price on the board. Real-SWE evaluates model-and-harness combinations rather than models in isolation, so the harness you choose matters.
Specific has not published a fixed update cadence. The September 2026 release is the initial leaderboard; the benchmark page indicates additional companies and models will be added over time. Watch the Specific benchmarks page for updates.
Not fully, because the underlying repos are private. But Specific publishes per-task pass/fail rates, per-model failure taxonomies, cost estimates, and sample task descriptions, which is enough to validate the leaderboard ordering. For open reproduction, the closest public analogs are polyglot benchmarks like Terminal-Bench and FrontierSWE, though they still tend to overstate model performance relative to real enterprise work.
