Benchmarks
Company Knowledge Bench: Why RAG Still Fails at Work
Kapa.ai's new benchmark tests retrieval on actual messy enterprise data, and the results expose how badly most RAG systems break outside clean academic datasets.
October 3, 2026
Most RAG benchmarks are a lie. Not deliberately, but functionally. They test clean Wikipedia passages, synthetic Q&A pairs, or hand-curated technical docs that look nothing like the ticket graveyards, duplicated Confluence pages, and half-dead Slack threads real employees actually search through.
That's the gap Kapa.ai just tried to close with the Company Knowledge Bench, a retrieval benchmark built from genuinely messy real-world company knowledge. And the numbers it produces are, not gonna lie, a bit of a reality check for anyone shipping agents into the enterprise.
What the Company Knowledge Bench actually tests
The Company Knowledge Bench is a retrieval benchmark for AI agents that evaluates how well retrieval systems surface the right chunks from a company's actual indexed corpus. It uses 1,000 eval cases built from real production queries across Kapa's deployed base, each paired with a snapshot of the corpus and a retrieval criterion that specifies which chunks are valid to return. The point is to measure performance where it hurts: duplicates, outdated pages, conflicting sources, and ambiguous queries.
That's a very different exercise from running MTEB or BEIR against polished academic datasets. Those are useful for component-level comparisons. They're not useful for predicting whether your support agent will cite the right runbook.
Why existing retrieval benchmarks mislead teams
If you've shipped a RAG system in production, you already know the drill. You pick an embedding model with a strong BEIR score. You wire up a reranker. Evaluation on your clean test set looks great. Then real users ask real questions and the system confidently cites a 2023 doc that was deprecated six months ago.
The problem isn't the models. It's the evaluation. Public benchmarks almost always assume:
- One correct document per query
- No duplicates, no near-duplicates, no stale versions
- Clean, grammatical queries
- Topically distinct corpora without overlapping concepts
Real company knowledge violates every one of those assumptions. A single question about "rate limits" might have fifteen plausible sources, three of which contradict each other, and only one of which reflects the current API version. Kapa's write-up argues that public retrieval benchmarks are mostly too narrow (built around one domain like law or medicine) or too artificial (built from synthetic documents and questions), and that this is exactly why they built the benchmark in-house against real production data.
The messiness problem, in categories
The benchmark's labelling handbook calls out specific situations that trip retrieval systems. Kapa doesn't publish per-category score drops, but the categories themselves describe what the benchmark is probing for:
| Failure mode | What it looks like |
|---|---|
| Duplicate / near-duplicate docs | Multiple versions of the same guide |
| Outdated but still indexed content | Deprecated API docs resurfacing |
| Conflicting sources | Support ticket contradicts official docs |
| Ambiguous queries | "How do I set this up?" with no context |
| Multi-source answers | Answer requires combining or choosing between docs |
The benchmark's definition of a correct retrieval is a minimal set of chunks that completely answers the query using the highest-authority, most-current sources. Returning a stale page or a lower-authority duplicate counts against you even if it technically contains the answer.
How the benchmark is constructed
The methodology is more interesting than the usual "embed, retrieve, compute nDCG" loop. Each eval case is a real production query, a snapshot of the corpus as it was when the query was asked, and a boolean retrieval criterion specifying which chunks are valid to return.
Kapa built the dataset through:
- Real production queries pulled from customer deployments, spanning developer tools, sales, and support use cases
- Agent-labelled criteria generated by candidate agents (that scan the corpus for possibly relevant chunks) and criteria agents (that turn those into boolean retrieval criteria), both built according to a written labelling handbook
- Human validation on 170 hand-labelled cases that acted as a benchmark-for-the-benchmark, used to iterate on the agents until their labels sufficiently agreed with human labellers
- Source preference rules encoded into the criteria, so a reference page outranks a tutorial repeating the same fact, and a recent Slack thread beats a three-year-old Confluence page
Grading rewards completeness (did you return enough chunks to answer?), minimality (did you avoid returning extras?), and source preference (did you pick the authoritative, current source over duplicates and outdated versions?).
The benchmark itself is not public, since it's built from real production data. The methodology is described in enough detail that teams can reproduce it on their own corpus.
Results breakdown
Kapa evaluated seven retrievers on the same ingested data, with 1,000 eval cases. Score is on a 0-1 scale where higher is better. These are the numbers Kapa reported:
| Retrieval approach | Retrieval score | Latency | Precision |
|---|---|---|---|
| Hybrid search | 0.41 | 0.4 s | 8% |
| Hybrid search + reranker | 0.50 | 0.7 s | 9% |
| Query decomposition + hybrid + rerank | 0.56 | 1.8 s | 10% |
| Kapa Default (optimized fixed pipeline) | 0.61 | 3.3 s | 12% |
| Agent + grep (small model, Luna) | 0.54 | 13 s | 4% |
| Agent + grep (large model, Sol) | 0.61 | 17 s | 4% |
| Kapa Deep (agentic retrieval) | 0.65 | 5.1 s | 26% |
Hybrid search uses gemini-embedding-001 with BM25. The reranker is Voyage AI's rerank-2. Agentic grep uses OpenAI's gpt-6.1-sol or gpt-6.1-luna with full-text and regex search as its only tool.
Two takeaways jump out, both of which Kapa calls out directly. First, adding a reranker is the cheapest big win in traditional RAG. Going from plain hybrid search to hybrid + reranker lifts the score from 0.41 to 0.50 for a third of a second more latency. Second, agentic retrieval wins on quality but at a steep cost. The grep agent with a strong model matches an optimized fixed pipeline on score (0.61) but takes five times as long. Kapa's own Deep mode (optimized agentic retriever) scores 0.65 in about five seconds, and the pipeline bias here is explicit — all seven retrievers were built by Kapa and run on their ingestion.
What "agentic retrieval" means in practice
Agentic retrieval is one of those phrases that gets thrown around without much substance. In Kapa's benchmark it's concrete. The agent:
- Decides what to search for next instead of running a fixed pipeline
- Reads intermediate results and can continue searching, read further in a found document, or prune irrelevant chunks
- Returns as many chunks as the query actually needs, rather than a fixed top-k
It's slower. It costs more tokens. The grep agents in the benchmark return around 40,000 tokens per query, roughly four times a fixed pipeline, and the large-model grep agent spends about $0.17 in orchestrator model calls per query. An optimized agentic retriever (Kapa Deep) sidesteps most of that, returning about 5,000 tokens per query at the highest precision in the benchmark.
The surprises
A few findings pushed back against conventional RAG advice.
A reranker is the cheapest big win. Going from hybrid search alone to hybrid plus a reranker lifted the score from 0.41 to 0.50 with only 0.3 seconds of added latency. If you change one thing in a basic RAG pipeline, this is it.
Query decomposition helps, but at a latency and token price. Splitting the query first into sub-queries added 0.06 in score over hybrid + rerank, but more than doubled the latency to 1.8 seconds and added an orchestrator model call per query.
An agentic loop is not automatically better. The small-model grep agent (0.54) actually lost to query decomposition (0.56) in a seventh of the time. For agentic retrieval, the orchestrator model's strength matters a lot — swapping the large model for the smaller one cost 0.07 in score.
Precision varies more than score. Kapa Deep returned about 5,000 tokens per query at 26% precision (share of chunks actually needed). The grep agents returned ~40,000 tokens at just 4% precision. Even when end score is similar, token efficiency affects what the downstream model pays to read the result.
What this means if you're shipping an agent
The honest implication is that if you've evaluated your RAG system only on a clean internal eval set, you probably have no idea how it performs. Enterprise retrieval evaluation needs to look like enterprise data, which means messy, redundant, and partially contradictory. If you want numbers on how production RAG on open models actually behaves at scale, that's a different piece of the same puzzle.
A few concrete moves worth considering:
- Build an eval set from real user queries, not LLM-generated ones. Fifty real queries beat five hundred synthetic ones.
- Include multi-answer queries where more than one doc could be correct. Grading as if there's one gold doc hides real behavior.
- Add deliberate noise: duplicates, outdated versions, conflicting sources. If your test corpus doesn't look like your prod corpus, your scores are fiction.
- Measure the full stack end-to-end, not just embedding similarity. The thing that ships is the whole pipeline.
- If you only change one thing in a fixed pipeline, add a reranker. Query decomposition and agentic loops cost more and only pay off if your use case can absorb the latency.
And for anyone building retrieval infrastructure, the Company Knowledge Bench is a useful target. According to the Kapa.ai team, the dataset itself isn't public since it's built from real production data, but the methodology (eval cases as query + corpus + boolean retrieval criterion, graded on completeness, minimality, and source preference) is reproducible on your own corpus. That's probably the more useful move anyway, since the whole point is that generic benchmarks don't predict your performance.
The bigger picture
Retrieval is having its "evaluation gap" moment, the same one that hit LLM evaluation two years ago when everyone realized MMLU scores didn't predict real-world helpfulness. Benchmarks like SWE-bench Verified solved that for coding agents by using real GitHub issues. The Company Knowledge Bench is trying to do the same for enterprise retrieval.
If it catches on, expect a flurry of "we top the Company Knowledge Bench" marketing from retrieval vendors over the next six months. We've already seen this pattern with framework-level comparisons like LangChain vs LlamaIndex vs Haystack, where vendor-run numbers rarely survive contact with your corpus. Take those claims with the same skepticism you'd apply to any self-reported benchmark — including Kapa's, where all seven retrievers in the published results were built by Kapa and run on their ingestion. But the underlying shift (toward real, messy, enterprise-grade retrieval evaluation) is overdue and genuinely useful.
The best retrieval system is the one that works on your actual data. Everything else is a leaderboard.
