REAP mines production coding agent sessions to build execution-based benchmarks. On the Harvest benchmark it produced, frontier models solve 42.9%-58.2% — well below SWE-bench Verified scores.
Static coding benchmarks have a shelf-life problem. SWE-bench Verified came out, models climbed toward the top of the leaderboard, and now the ranking tells you a lot less than it used to about how a coding agent behaves when a real developer asks it to refactor a legacy Django app at 2 a.m.
That's the gap REAP is trying to close. Posted to arXiv in April 2026 by researchers at Meta, REAP (Relevance and Execution-Audited Pipeline) mines interactive production coding agent sessions and turns them into an execution-based benchmark called Harvest. No hand-picked GitHub issues. No frozen snapshot from 2023. Just what developers were really doing with a production coding assistant.
And honestly? It's about time.
REAP is an automatic curation pipeline that harvests coding tasks from live agent sessions in production, filters them with prompt-quality, test-relevance, and multi-run test-validation checks, and packages the survivors into an executable evaluation set. Instead of researchers writing tasks by hand, REAP lets real user behavior define what "realistic" means. The benchmark it produced for the paper is called Harvest.
Photo by Flipsnack on Unsplash
That's the core pitch. The interesting part is how the pipeline handles the messy reality of production traces.
Production agent sessions are noisy. Half of them are people asking "why is my import broken?" and half are multi-hour refactors across a monorepo. The paper describes filtering that stream in stages:
Every surviving task ships with a repo state, the verbatim developer prompt, and a fail-to-pass test suite retrieved from the same repository — grading is execution-based, not rubric-based.
Look at the SWE-bench Verified leaderboard and the top of the pile is bunched up: Sonar Foundation Agent + Claude 4.5 Opus sits at 79.2%, TRAE + Doubao-Seed-Code at 78.8%, live-SWE-agent + Gemini 3 Pro Preview at 77.4%, and Atlassian Rovo Dev at 76.8% (source: swebench.com). Multiple frontier scaffolds cluster inside a five-point band at the top, and the incremental gains are getting smaller with every release.
That's not because the models are all equally good at coding. It's that SWE-bench Verified has been so heavily optimized against that the remaining headroom is compressed. The original SWE-bench paper framed itself as a "realistic" test because it used real GitHub issues. But those issues are now several years old, patterns leak into training data, and the tasks reflect what open-source maintainers filed publicly, not what developers actually do inside proprietary codebases day-to-day.
REAP's implicit argument is that any benchmark you can memorize is a benchmark you've already killed.
The paper reports numbers from the Harvest benchmark it curated. A few things stand out.
Harvest is much smaller and shaped differently than the SWE-bench Verified pool. The paper reports 132 verified single-turn tasks after filtering, with a median of 2 files modified, 39 lines changed, 1 fail-to-pass test, and 3 total tests per task. Language coverage is dominated by Hack (Meta's PHP dialect), with Python, C++, Kotlin, and others in the tail — a very different distribution from SWE-bench Verified's Python-heavy Django/Sympy composition.
That distribution shift matters. A benchmark drawn from real Meta developer traffic looks nothing like the open-source Python ecosystem most static coding benchmarks live in.
When five frontier models are evaluated on Harvest, solve rates range from 42.9% (GPT-5.4) to 58.2% (Claude Opus 4.6). The other three models in the evaluation are Claude Sonnet 4.5, Claude Haiku 4.5, and Gemini 3.1 Pro; the paper reports the range across all five without necessarily publishing each individual score in every table.
That's meaningfully below the mid-70s scores same-generation frontier scaffolds are posting on SWE-bench Verified. The absolute-score gap is roughly 20 points, and it's the exact gap developers have been complaining about for a year: the leaderboard says the agent is nearly superhuman, but the actual PR keeps introducing subtle bugs.
The paper compares a "Basic Agent" against an "Advanced Agent" harness on the same underlying Claude Opus 4.6 model. Tool-call distributions shift meaningfully between the two — the Basic Agent leans heavily on Read File (~37%) and Search Files (~31%) calls, while the Advanced Agent adds Todo Write and Str Replace Edit tools that change how tokens get spent. Adding context files to the Basic Agent lifts solve rate by about 6.4 percentage points, at the cost of roughly 32% higher latency.
That's the "scaffolding matters" story from prior benchmarks, quantified on a production-derived task set.
The five frontier models the paper evaluates on Harvest:
| Model | Harvest solve rate |
|---|---|
| Claude Opus 4.6 | 58.2% (top of range) |
| Claude Sonnet 4.5 | Within 42.9-58.2% range (exact value N/A here) |
| Claude Haiku 4.5 | Within 42.9-58.2% range (exact value N/A here) |
| Gemini 3.1 Pro | Within 42.9-58.2% range (exact value N/A here) |
| GPT-5.4 | 42.9% (bottom of range) |
Most of these models don't have direct entries on the public SWE-bench Verified leaderboard — the mapping to third-party harness runs on Verified is not clean, so cross-benchmark deltas per model are speculative. What is clear is that the Harvest range (roughly 43-58%) is substantially below the top-of-Verified band (mid-to-high 70s).
Three takeaways matter here.
First, coding agent evaluation is entering a new phase. Static benchmarks gave us useful signals from 2023 to early 2026, but the models have gotten close to solving the current bar. Dynamic, production-derived benchmarks like Harvest are the natural next step, and similar efforts (SWE-bench Live, SWE-Lancer, Senior SWE-Bench) are pointing the same direction.
Second, the gap between benchmark scores and real-world usefulness isn't a myth developers made up. It's measurable. If your team has been frustrated that your top-of-leaderboard agent still ships broken PRs, this is the quantitative version of that feeling.
Third, scaffolding matters more than ever. Harness choice moves scores by several points on Harvest, which matches what the top of the SWE-bench Verified leaderboard has been showing: the wrapper around the model is often worth as much as the model choice.
The moment your benchmark stops discriminating between frontier models, it's not a benchmark anymore. It's a leaderboard for training data leakage.
A few things stand out from the paper.
Task classification is reliable enough to be trusted. REAP's classifiers report 87% agreement (κ=0.74) on task classification and 86% agreement (κ=0.72) on test relevance in the paper's evaluation. That's a load-bearing detail — the pipeline only works if these filters are reasonably accurate, and the numbers are competitive with human annotator agreement.
The relevance-first ordering keeps more tasks. The paper compares a relevance-first ordering to an execution-first ordering and reports that relevance-first retained 88% of the F2P test pairs while admitting 22% more net tasks (132 vs 108). That's the kind of detail that only matters if you're building a similar pipeline, but it's a useful design finding.
Language distribution reflects the deployment, not the ecosystem. Because Harvest is drawn from Meta traffic, Hack dominates. That's a feature, not a bug: any org running REAP on its own agent traces will get a benchmark shaped like its own stack.
So what do you actually do with this?
If you're picking a coding agent for your team, treat SWE-bench Verified numbers as directional. The ranking is still informative, but the absolute scores are inflated relative to messier real work — a gap of roughly 20 points at the top of the range is what Harvest surfaces.
Budget for iteration. Real production tasks span more tool calls, more search, and more retries than tidy benchmark tasks do. At current API pricing, that meaningfully changes cost per completed task, and prompt caching is doing a lot of work on the economics.
Stop optimizing your internal evals against SWE-bench Verified clones. If you're building your own benchmark to compare vendors, the REAP recipe is the interesting one: log real sessions, retrieve fail-to-pass tests, filter for reproducibility, grade against actual execution. Made-up tasks will mislead you.
And expect the benchmark meta to shift within the next six months. Every serious lab has been noticing the same saturation problem, and REAP is one of several efforts pushing toward evaluation that actually moves as models improve. The full paper is on arXiv if you want the pipeline details.
The short version: static coding benchmarks are getting compressed at the top, dynamic production-curated ones are coming, and the gap between marketing claims and real agent behavior is finally something we can measure.
Sources
REAP was published as an arXiv paper (2604.01527) by researchers at Meta in April 2026. The paper describes the pipeline in detail, but the code and Harvest dataset release status was not confirmed at the time of writing. If you want to run a REAP-style pipeline internally, the paper is the reference implementation blueprint.
SWE-bench Live refreshes issues from the same GitHub-based methodology on a rolling basis, and SWE-Lancer focuses on Upwork-style freelance tasks. REAP's differentiator is that it sources tasks from live coding agent sessions inside a production coding assistant, capturing the verbatim developer prompt and grading against fail-to-pass tests retrieved from the same repository.
Yes, and it's a live question. Any pipeline that mines real user sessions needs consent, scrubbing of secrets and proprietary code, and careful handling of PII. The REAP paper draws on Meta internal traffic, which sidesteps most of this because the data stays inside one org — public releases from other deployments would need clear opt-in mechanisms.
Harvest tasks are drawn from a Hack-heavy production codebase, come with real developer prompts rather than curated issue text, and are graded against fail-to-pass tests specific to that repo. The distribution shift alone punishes models that are strong on Python OSS issues but weaker on unfamiliar language and codebase conventions.
No, but weight them carefully. The ranking (Anthropic Claude Opus > OpenAI GPT-5 line > Google Gemini for coding, at least on the current public leaderboard) still holds directionally. The absolute scores appear inflated by roughly 20 points versus messier production-derived work, so treat top-of-leaderboard percentages as ceiling estimates, not what you'll see in production.
