Snorkel AI's new Senior SWE-Bench flips the script on coding agents by testing them as senior engineers. Top models land in the 40-55% range on basic solves and fall into the teens or twenties once quality-graded.
Regular SWE-bench measures whether an agent can fix a bug. Senior SWE-Bench asks a harder question: can it act like the engineer who reviews that fix, catches the missed edge case, and pushes back on the architecture?
Snorkel AI dropped Senior SWE-Bench as an open-source evaluation aimed squarely at the ceiling of current agent capabilities. And based on the initial numbers, that ceiling is a lot lower than the leaderboard-topping SWE-bench Verified scores would suggest.
So what changes when you evaluate agents at the senior level instead of the junior one? A lot. Let's break down the methodology, the results, and what this means if you're actually shipping code with AI in the loop.
Senior SWE-Bench is an open-source benchmark from Snorkel AI that evaluates coding agents on senior-engineer tasks: reviewing pull requests, planning multi-file refactors, and reasoning across long-lived codebases. Instead of measuring whether an agent can patch a single failing test, it tests judgment, taste, and cross-cutting reasoning.
The benchmark was announced in mid-2026 and quickly picked up traction on Hacker News. The framing is deliberate: SWE-bench Verified rewards agents that can localize a bug and produce a passing patch. That's a useful skill. But it's also a junior-level skill. Real senior engineers spend more time preventing bugs, shaping designs, and reviewing other people's code than they do writing patches themselves.
The existing SWE-bench Verified leaderboard is getting crowded at the top. Claude Opus 4.6 leads at 75.6%. Gemini 3 Pro Preview lands at 74.2%. GPT-5.2 with high reasoning hits 71.8%. When top scores compress into a few points of each other, the signal-to-noise ratio collapses. You can't tell if one model is genuinely better or if it just got lucky on a handful of instances.
And honestly, patching Django bugs from 2023 isn't a great proxy for what senior engineers actually do in 2026. The task selection was designed for a specific evaluation window that agents have now outgrown.
Senior SWE-Bench splits evaluation into three task categories that mirror senior IC responsibilities:
Each task is graded against a rubric produced by actual senior engineers, not just a passing unit test. This is the big methodological shift. SWE-bench Verified uses pytest results as ground truth. Senior SWE-Bench uses rubric-based grading with human-authored criteria, similar to what you'd see in a real code review.
Unit tests catch regressions on paths you thought to check. Senior engineers catch bugs on paths nobody thought about yet. A rubric can encode "did the agent notice this race condition" in a way that a test suite structurally can't.
The tradeoff is obvious: rubric grading is more expensive to construct and harder to scale. Snorkel has been in the data-labeling business since 2019, so they're arguably the right team to attempt it (their expertise here isn't accidental).
Here's where things get interesting. The initial leaderboard shows a much wider spread than SWE-bench Verified, and the ordering isn't quite what you'd expect.
| Model | Senior SWE-Bench (basic solve) | Senior SWE-Bench (tasteful pass@1) |
|---|---|---|
| GPT-5.5 | 55% | 16% |
| GPT-5.4 | 49% | 14% |
| Claude Sonnet 5 | 44.8% | 19.4% |
| Claude Opus 4.8 | 42% | 24% |
| Claude Opus 4.7 | 40.4% | 14.1% |
| Claude Sonnet 4.6 | 31.6% | 8.2% |
| GLM-5.2 | 31.2% | 12.5% |
| Gemini 3.1 Pro | 26.3% | 6.1% |
| Kimi K2.6 | 23.7% | 8.2% |
| Gemini 3.5 Flash | 19% | 3% |
Scores from the current Mini-SWE-Agent harness runs on the Senior SWE-Bench leaderboard. "Basic solve" measures runtime correctness; "tasteful" combines correctness with rubric-graded quality metrics. Note that most of these models do not have direct entries on the official swebench.com SWE-bench Verified leaderboard, which makes strict head-to-head gap comparisons hard.
Even so, the overall picture is clear: top models land in the 40-55% range on basic solves and drop into the teens or low twenties once quality-graded. Whatever these models are good at when writing patches for SWE-bench Verified, they're systematically weaker once the grading rewards senior-level judgment as well as passing tests.
Breaking down by category tells you why:
The official Snorkel writeup has more granular category breakdowns and per-task examples.
Let's talk about the implications, because "model X scored 62%" doesn't tell you much on its own.
First, the Claude Opus 4.8 vs GPT-5.5 split is notable. On basic (correctness-only) scoring, GPT-5.5 leads Opus 4.8 by 55% to 42%. But once you add rubric-graded quality metrics under the "tasteful" score, Opus 4.8 pulls ahead 24% to 16%. That flip isn't noise. Anthropic's models have a reputation for being stronger at taste-driven judgment tasks, and the tasteful metric appears to reward exactly that.
Second, scaffolding is a live variable. The leaderboard runs listed above all use the same Mini-SWE-Agent harness with different underlying models, which isolates model capability from scaffold quality. Snorkel's how-it-works writeup notes that harness choice materially moves scores, so if you're building agent products, the wrapper you build around the model may matter more than which model you picked.
And third, the gap between best and worst is much wider than on SWE-bench Verified. That gives the benchmark room to breathe as models improve. It won't saturate in six months the way SWE-bench Verified is about to.
Cognition Labs' Devin isn't on the initial leaderboard, which is a notable absence given how much marketing it does around autonomous engineering. Snorkel hasn't published official Devin results and Cognition hasn't submitted a harness run, so where it would actually land relative to the entries listed above is speculation for now.
A few things jumped out on close reading of the benchmark data.
Reasoning models don't dominate. You'd expect o3-style reasoning models to crush a benchmark built around judgment. But the initial numbers suggest chain-of-thought reasoning helps less than context management does. Long-horizon coding tasks reward staying grounded in the codebase, not thinking longer in isolation.
Small model scaffolds do surprisingly well. Some open-weight models with proper agent scaffolding land within 15 points of the frontier. This suggests we might be overpaying for closed-model API access on senior tasks if we're willing to invest in the use.
Rubric consistency is a stated priority. One worry with human-graded benchmarks is inter-rater disagreement. Snorkel describes a rubric quality control process where multiple senior engineers author and review task definitions, though the how-it-works writeup does not publish specific inter-annotator agreement numbers. Treating rubric quality as a first-class concern is still a credibility signal even without a published kappa.
"If your agent can't reason like a senior engineer, calling it an autonomous SWE is marketing, not engineering."
That's the tension the benchmark forces the industry to confront. And it's overdue.
So what should you actually do with this information?
If you're evaluating coding agents for your team, stop treating SWE-bench Verified as the last word. A 90% score there tells you the agent is a capable junior. It tells you nothing about whether it'll catch design bugs or handle a multi-repo refactor. Look at Senior SWE-Bench numbers when comparing agent products for review or planning workflows.
If you're building an agent product, the scaffold results should reshape your priorities. Prompt engineering, context management, and code navigation tooling appear to matter more than model choice above a certain baseline. Aider, Claude Code, and Cursor have been quietly demonstrating this for a year, and Senior SWE-Bench numbers back it up.
If you're a senior engineer wondering whether your job is safe, the answer for right now is: yes. Agents are genuinely useful for junior tasks. They're mediocre at senior tasks. That gap is closing, but it's not closing fast. Tasteful-solve rates stuck in the mid-teens to mid-twenties aren't disappearing in the next model release.
And if you're evaluating agent adoption ROI, the practical implication is to deploy agents for well-scoped patches, not open-ended reviews. Match the tool to the task the benchmarks say it can actually do.
Senior SWE-Bench matters because it targets the right problem at the right time. The AI coding narrative in 2026 has been dominated by claims about "replacing engineers." This benchmark provides the first credible open-source data on where the actual capability ceiling sits, and it's meaningfully below the marketing.
That's healthy for the field. Benchmarks that saturate at 90%+ create incentives to optimize for the benchmark rather than the underlying skill. A harder benchmark with more headroom keeps model developers honest and gives buyers real signal.
The open-source release is also significant. Snorkel could have kept this proprietary and sold access. Instead, the code, task definitions, and rubrics are public. That means academic researchers, independent labs, and product teams can all run their own evaluations. Compare that to some vendor-controlled benchmarks that are effectively pay-to-play.
Bottom line: if you care about where AI coding actually stands in 2026, Senior SWE-Bench is now the benchmark to watch. SWE-bench Verified isn't going away, but its era as the definitive coding evaluation is ending. And that's exactly how benchmark evolution is supposed to work.
Sources
The benchmark is hosted at senior-swe-bench.snorkel.ai with links to the GitHub repository containing task definitions, rubrics, and evaluation harness code. Snorkel released it under an open license, so you can run local evaluations against your own agent scaffolds without submitting to the public leaderboard.
A full run against a frontier model can cost several hundred dollars in API fees depending on scaffold verbosity and reasoning effort. The initial release contains 100 total tasks (50 public, 50 kept private to mitigate contamination), and Design-and-Build tasks in particular consume long context windows. Budget more if you use extended thinking or agentic scaffolds with many tool calls.
Yes. Snorkel accepts community submissions with a required methodology writeup explaining the scaffold, tools, and any prompt engineering. Submissions go through a light verification process to prevent gaming, and results appear on the public leaderboard within a few weeks of acceptance.
The initial 100-task release spans multiple languages across the sampled repositories, including Python, TypeScript, Go, Elixir, and Rust — a broader footprint than the original SWE-bench, which was Python-only. Language coverage tracks the real-world stacks of the source repositories rather than being restricted to a single ecosystem.
Snorkel describes a rubric quality control process where multiple senior engineers author and review task definitions to catch ambiguity before tasks land in the benchmark. The how-it-works writeup emphasizes this validation loop but does not publish specific inter-annotator agreement numbers, so external validation of consistency will have to come as researchers run their own studies against the open data.
