Claude Opus 4.6 reaches 81.4% on SWE-bench Verified per Anthropic, but raw HumanEval scores tell a different story. A data-driven look at which LLM actually writes the best code right now.
Picking the best AI coding LLM used to be easy. You just looked at HumanEval and called it a day. But that benchmark is now so saturated that a 90% score barely tells you anything useful about real software engineering work.
So the question gets harder every quarter. Which model actually writes code that compiles, passes tests, and survives a pull request review? The 2026 benchmark data finally gives us a sharper answer, and the rankings flip depending on which test you trust.
Claude Opus 4.6 led the best AI coding LLM rankings for most of early 2026, with Anthropic releasing Opus 4.7 in April 2026 as its new flagship. Per Anthropic's reported results, Opus 4.6 hits 81.4% on SWE-bench Verified with scaffolding and posts the highest reported HumanEval score at 93.7%. OpenAI's o3 reportedly comes second on agentic coding tasks (69.1% on SWE-bench, lab-reported), and DeepSeek V3 is the strongest open-weight option at 89.8% on HumanEval (self-reported).
That's the headline. The interesting part is what the numbers hide.
There are basically three families of coding tests, and they measure very different things.
HumanEval (Papers with Code) is the classic. 164 hand-written Python problems with unit tests. A model gets a function signature and a docstring, and has to fill in the body. Pretty solid for measuring raw code synthesis. Useless for measuring whether a model can figure out a 50k-line repo.
SWE-bench Verified (SWE-bench) is the one that actually matters in 2026. It pulls real GitHub issues from 12 popular Python projects and asks the model to produce a patch that passes the hidden test suite. The "Verified" subset was hand-checked by OpenAI to remove ambiguous or broken tasks. This is the benchmark every frontier lab now optimizes for.
LiveCodeBench rotates problems monthly from LeetCode, Codeforces, and AtCoder to fight contamination. Less commonly reported, but worth watching.
The gap between these tests is huge. A model can crush HumanEval with 93% accuracy and still flop on SWE-bench at 50%, because writing one function from a clean spec is nothing like fixing a bug across three files you've never seen.
Here's how the major models stack up across the benchmarks that matter for coding. Most SWE-bench numbers below are self-reported by the labs — the public SWE-bench Verified leaderboard currently has no submissions for Claude Opus 4.6, Claude Sonnet 4.6, or GPT-4.1, so treat those rows as vendor-reported rather than independently audited. Anthropic has since released Claude Opus 4.7 as its new flagship for agentic coding, but most public third-party benchmarking still references Opus 4.6.
| Model | HumanEval | SWE-bench Verified | MMLU | Context |
|---|---|---|---|---|
| Claude Opus 4.6 | 93.7% | 81.4% (with scaffold, Anthropic-reported) | 92.3% | 1M |
| OpenAI o3 | N/A | 69.1% (with scaffold, OpenAI-reported) | N/A | 200K |
| GPT-4.1 | N/A | 54.6% (OpenAI-reported) | N/A | 1M |
| Claude Sonnet 4.6 | 88% | 55.3% (lab-reported) | 89.5% | 1M |
| GPT-4o | 90.2% | N/A | 88.7% | 128K |
| DeepSeek V3 | 89.8% | N/A | N/A | 128K |
| DeepSeek R1 | N/A | 49.2% (self-reported) | 90.8% | 128K |
| Gemini 2.5 Pro | N/A | N/A | N/A | 1M |
A few things jump out immediately.
Claude Opus 4.6 is the only model reported to break 80% on SWE-bench Verified. That's a meaningful lead on paper, though note these are lab-reported numbers — Opus 4.6 has not been submitted to the public SWE-bench leaderboard. SWE-bench results below 50% mean the model is fixing fewer than half of real bugs, even with scaffolding help. Above 70% means it's genuinely useful as an autonomous agent on real repos.
GPT-4o, the model that everyone still uses by default in ChatGPT, doesn't even crack the top tier on SWE-bench. Its HumanEval score of 90.2% looks great in isolation. Put it on a real GitHub issue and the wheels come off faster than you'd expect.
And DeepSeek deserves a closer look. The open-weight V3 model hits 89.8% on HumanEval, which is within a hair of GPT-4o and only 4 points behind Claude Opus 4.6. For a model you can download and run on your own infrastructure, that's wild.
Benchmark scores are easy to game and easier to misread. Let me walk through what these results actually predict about day-to-day coding work.
If you're using an LLM to write isolated functions (think: "give me a Python function that validates an IPv6 address"), every model in the top tier is going to do fine. Claude Opus 4.6 at 93.7% on HumanEval is technically best, but the practical gap between 88% and 94% on this benchmark is small. You'd struggle to notice the difference on most tasks.
This is the regime where GitHub Copilot, Codeium, and Tabnine all feel roughly equivalent. The inline suggestion gets it right most of the time. When it doesn't, you fix it in two seconds.
This is where the rankings split hard. SWE-bench Verified is the closest proxy we've for "can this model do my job?" and the numbers say:
If your workflow involves an agent picking up a ticket and producing a PR with minimal hand-holding, you basically have two choices right now (Opus 4.6 or o3) and a long tail of "helpful but not autonomous."
For algorithm-heavy work, look at the MATH and ARC-AGI numbers. o3 is dominant here, scoring 96.7% on MATH and 87.5% on ARC-AGI with high compute. Claude Opus 4.6 sits at 85.1% on MATH, which is solid but a real step down.
So if you're writing a complex algorithm from scratch (graph algorithms, dynamic programming, cryptography), o3 is probably your model. If you're doing application code (web servers, data pipelines, infrastructure tooling), Claude Opus 4.6 is the better default.
A few things in the 2026 data are genuinely surprising.
GPT-4o is no longer competitive for serious coding. It still ranks first on LMSYS Chatbot Arena at 1287 Elo, which measures vibes-based user preference. But on the benchmarks that test actual coding capability, it's been overtaken by Claude Opus 4.6, o3, and even some open-weight models. The Elo ranking tells you GPT-4o is pleasant to chat with. SWE-bench tells you it's not the model you want shipping production code.
Open-source is closer than people think. DeepSeek V3 at 89.8% on HumanEval and DeepSeek R1 at 49.2% on SWE-bench are remarkable for free, downloadable models. For another open-weight contender, the NousCoder-14B vs Claude Code benchmark is worth a look. Two years ago, the gap between open and closed weights was a chasm. Now it's a manageable ditch. For teams with serious privacy or cost constraints, DeepSeek is a real option, not a compromise.
Scaffolding matters as much as the model. Anthropic's reported 81.4% Opus 4.6 score on SWE-bench is "with scaffold" and a prompt modification, meaning their agentic harness for dealing with repos, running tests, and iterating on fixes. Without that scaffolding, the raw model scores meaningfully lower. This is why Claude Code and Cursor feel different even when both are powered by the same underlying model. The scaffolding is half the product.
Gemini is missing from coding leaderboards. Google reports plenty of MMLU and reasoning scores, but conspicuously avoids SWE-bench Verified head-to-head numbers. Read into that what you will.
So what should you actually use? It depends on the work.
For autonomous coding agents: Claude Opus 4.6 via Claude Code, or o3 through OpenAI's API. These are the only two models clearing 65% on SWE-bench Verified. Everything else needs a human in the loop more often than not.
For IDE pair programming: Cursor with Claude Sonnet 4.6 or Opus 4.6 is the current sweet spot. GitHub Copilot is fine if you're already in the Microsoft ecosystem, but the model behind the suggestions matters more than the UI wrapper. If you want a free alternative, the Goose vs Claude Code comparison covers an open-source contender.
For open-source / self-hosted: DeepSeek V3 for general coding, DeepSeek R1 when you need reasoning. Both genuinely competitive with closed models from 18 months ago, and you keep all your code on your own hardware.
For simple completions in any IDE: Honestly, free tools like Codeium are good enough. You don't need a frontier model to autocomplete a for-loop.
The top of the coding leaderboard has consolidated. In 2026 there are two models worth running autonomous agents on, four models worth using in your editor, and everything else is a rounding error.
Claude Opus 4.6 isn't free. At $5/$25 per million tokens (input/output), running an agentic workflow on a non-trivial codebase gets expensive fast. A single SWE-bench attempt with full scaffolding can burn through $5 to $20 in tokens depending on how many iterations the agent needs.
o3 pricing is similar in the premium tier. GPT-4o stays cheaper at $2.50 input and $10 output per million tokens, which is why it's still the default for high-volume use cases where capability isn't the bottleneck.
If you're choosing a coding LLM and budget matters, the question becomes: how much is a successful PR worth to you? At $20 per successful autonomous fix, Opus 4.6 is a bargain compared to an engineer-hour. For autocomplete in your IDE, it's overkill.
The interesting frontier isn't HumanEval anymore. It's:
Expect Anthropic, OpenAI, and DeepSeek to keep trading the SWE-bench crown every few months. The Google coding story is still TBD. And the open-weight gap is closing faster than most people predicted — see GLM-5.1's SWE-bench performance for a recent example.
Sources
Expect $5 to $20 per non-trivial agentic task at the current $5 input / $25 output per million tokens. A simple bug fix might cost under a dollar, but multi-file refactors with iterative test runs add up quickly because every loop reads the codebase context again. Most teams set a daily token budget per developer rather than per-task limits.
Yes, with caveats. The 89.8% HumanEval score is real, and the model handles standard CRUD app work, API integrations, and most debugging tasks competently. The gap shows up on complex multi-file refactors and tasks requiring deep repo understanding, where it lags Claude Opus 4.6 by roughly 20 percentage points on SWE-bench equivalents. For self-hosted setups or cost-sensitive teams, it's the strongest open option available.
Chatbot Arena measures user preference in blind comparisons, which heavily weights response style, friendliness, and explanation quality. Coding benchmarks measure whether the output actually compiles and passes tests. GPT-4o is genuinely pleasant to chat with and explains code well, but its raw code generation accuracy has been surpassed by Claude Opus 4.6 and o3 on every objective programming test.
Probably yes, especially for autonomous tasks. Claude Code uses Anthropic's own agentic harness, which is what produced the 81.4% SWE-bench Verified score Anthropic reports for Opus 4.6 with their own harness. Cursor uses its own scaffolding, which is optimized more for interactive IDE use than autonomous multi-step work. For chat-and-edit workflows the difference is small; for fire-and-forget agent tasks, Claude Code typically completes more of them successfully.
Based on the 2025-2026 release cadence, expect frontier model updates every 3 to 5 months from Anthropic and OpenAI, with DeepSeek releasing every 4 to 6 months. The SWE-bench Verified leaderboard typically sees a new top score within weeks of each major release. If you're making a tooling decision today, the ranking will likely shift by Q3 2026, but the top two or three names will probably stay similar.
