Benchmarks
LLM Agents Flop at Coordination: Inside the ALEM Benchmark
A new open-ended coordination benchmark tests 13 LLMs across communication, trading, crafting, and combat. Most agents average just 6% normalised return.
July 19, 2026
So the big claim from the new ALEM benchmark is a little uncomfortable if you're bullish on agent swarms: 13 frontier LLMs, put in a Crafter-style open world together, average around 6% normalised return. That's not 60. That's six.
The paper, dropped on r/MachineLearning by user ktessera on July 18, 2026, benchmarks open-ended multi-agent coordination in language agents. It's the first serious attempt to measure whether modern LLMs can actually work as a team over long horizons, not just solo tool-using assistants. And the results are pretty sobering.
But there's a twist buried in the numbers. On the hardest setting, zero-shot Gemini 3.1 Pro performs comparably to the best MARL agent trained for 1 billion environment steps. So LLMs are terrible at coordination, but they're still weirdly competitive with purpose-built reinforcement learning baselines. Weird flex, but ok.
Let's dig into what this LLM multi-agent coordination benchmark actually measures and why it matters if you're building anything more ambitious than a single-agent RAG bot.
What ALEM Actually Tests
Interesting wrinkle: aLEM (that's what the project page at alem-world.github.io calls it) drops multiple language agents into a shared, procedurally generated world. The agents have to explore, communicate with each other, trade resources, craft tools, build structures, and fight mobs. It's Minecraft-brain in a research put to work.
Photo by Annie Spratt on Unsplash
The critical word here is open-ended. Unlike SWE-bench or HumanEval where success is a binary pass/fail on a specific goal, ALEM measures normalised return across long-horizon tasks with no single "win state." Agents can pursue different sub-goals. They have to negotiate. They can screw each other over.
According to the paper on arXiv, the authors ablate three coordination primitives: communication, resource sharing, and role specialisation. Communication had the largest effect. Turn off inter-agent messaging and even the best models collapse toward random-policy performance.
That's actually the most important finding here. It's not that LLMs can't plan. It's that they can't talk to each other productively over hundreds of steps without hallucinating shared state that doesn't exist.
The Leaderboard: Who Actually Coordinates
Based on the numbers reported on the ALEM leaderboard, average performance across the 13 evaluated models sits at 6.2% total normalised return on the Hard setting. Gemini 3.1 Pro leads the zero-shot pack at 15.4% Total% (17.5% Coord.%), essentially matching the MARL baseline trained on a billion environment steps.
Here's how the evaluated models actually stack up on the Hard coordination setting (Total% normalised return, from the official leaderboard):
| Model | Total% (Hard) | Coord.% (Hard) |
|---|---|---|
| Gemini 3.1 Pro (zero-shot) | 15.4% | 17.5% |
| GPT-5.4 (high reasoning) | 8.6% | 4.2% |
| Gemma 4 31B | 8.0% | 8.8% |
| Qwen 3.6 27B | 7.4% | 7.2% |
| Gemma 4 26B-A4B | 7.4% | 9.5% |
| Qwen 3.6 35B-A3B | 7.3% | 8.7% |
| Qwen 3.5 27B | 6.4% | 8.1% |
| Llama 3.3 70B | 2.6% | 4.2% |
| Llama 3.1 8B | 0.9% | 0.3% |
| MARL baseline (1B steps) | 15.3% | 17.6% |
| LLM average (Hard) | 6.2% | 6.9% |
(Exact per-model scores across Easy/Medium/Hard are on the ALEM leaderboard if you want the receipts.)
The fact that a purpose-built reinforcement learning system trained on a billion steps only ties with a zero-shot LLM is... actually kind of a win for LLMs? It just also shows how far the ceiling still is from anything you'd call "solved."
Why Coordination Is a Distinct Bottleneck
And this is where the paper gets genuinely interesting. The authors argue coordination is a distinct capability beyond long-horizon task competence. GPT-5.4 is a clean example: it posts the second-highest Base% (base-task competence) across difficulties, yet its Coord.% (4.2% on Hard) lags well behind smaller models like Gemma 4 26B-A4B (9.5%). Being good at solo tasks does not imply being good at coordinating with peers.
Why? A few reasons the ablations expose:
- Message drift: Agents send messages, then later act as if they'd received different messages. Basically confabulation about their own chat history.
- Goal collision: Two agents both decide to gather wood. Neither realises the other did until they're standing on the same tree.
- Credit assignment: When an agent trades a pickaxe for iron, both agents often forget the trade happened a few turns later.
- Role instability: Agents drift out of assigned roles under weak prompting.
Communication ablations dropped Gemini 3.1 Pro's Coord.% from 17.5 to 5.3, and Gemma 4 31B from 8.8 to 3.8. That's a 60-70% relative drop, larger than any other ablation the paper reports. Coordination isn't a bonus feature on top of solo reasoning. It's its own thing.
Zero-Shot LLMs vs. Trained MARL
This is the finding that'll get quoted in every AI newsletter this week. On the hardest setting, Gemini 3.1 Pro with zero fine-tuning essentially ties a Multi-Agent Reinforcement Learning system trained on 1 billion environment steps: 15.4% vs 15.3% Total%, and 17.5% vs 17.6% Coord.%.
A billion. That's a serious amount of compute burned on a specialised policy. And a general-purpose language model, prompted well, matches it out of the box.
But don't get too hyped. Both are in the mid-teens, not remotely close to solving the benchmark. Aggregate LLM Total% stays nearly flat at around 6% across Easy, Medium, and Hard — most models don't coordinate reliably enough for the difficulty parameter to bite.
Still, the implication is real. If you're deciding whether to fine-tune a specialist multi-agent policy or just prompt a frontier LLM, the frontier LLM is at least competitive, at a fraction of the engineering cost.
Surprises From the Ablation Study
Worth flagging: a few things jumped out that weren't in the TL;DR:
Reasoning helps, but conditionally. The paper's ablations show reasoning supports both base and coordination reward, while scratchpad memory mainly helps when it's used to plan ahead — not just to log past state.
Task structure matters more than raw difficulty. Handover tasks (one agent starts, another finishes with temporal slack) are comparatively easy. Synchronous-Hard tasks — where agents must identify a shared target, converge on compatible positions, communicate intent, and act in the same timestep — are the killer. Gemini 3.1 Pro leads LLMs there with just 18% coverage; the next-best sit around 9%.
Base competence does not guarantee coordination. GPT-5.4 has strong Base% (single-agent task reward) but low Coord.%. Smaller models like Gemma 4 26B-A4B beat it on the coordination reward with non-overlapping 95% CIs. Solo capability doesn't transfer.
Mixed-family teams don't magically help. The paper's heterogeneous team runs (Gemini 3.1 Pro + GPT-5.4 + Gemma 4 31B, and a Gemma-only family team) both landed within ±0.1 of the average of their homogeneous baselines. Mixed teams do not collapse to the weakest member, but they don't inherit the strongest either.
That last one is a useful reality check if you're designing production agent systems: you can't paper over coordination weaknesses by throwing different providers in a room together.
What This Means If You're Building Agents
If you're building a multi-agent product right now, the practical takeaways are pretty concrete:
Communication needs structure. Free-form chat between agents will drift. Use structured message schemas (JSON with required fields), explicit acknowledgments, and shared state stored outside the agents themselves. Don't rely on agents remembering what they told each other.
Reasoning-and-memory needs a plan. ALEM ablations show scratchpad memory helps mainly when it's used to hold a multi-step plan. Storing observations without a plan doesn't rescue coordination.
Small teams beat scaling up naively. The ALEM protocol runs 3-agent teams. Larger homogeneous groups aren't what the paper studies — but the coordination overhead is already the bottleneck at three, so adding more agents is unlikely to help without harness changes.
Don't over-index on solo benchmarks. GPT-5.4's strong Base% but weak Coord.% is exactly the failure mode: models that crush single-agent benchmarks can still fumble two- or three-agent negotiation. Solo reasoning and coordination are different capabilities.
Zero-shot is legit. Fine-tuning a specialist for coordination is expensive. If a frontier LLM zero-shot ties a 1B-step MARL policy, prompt engineering and harness design are probably a better ROI in most product contexts than training a specialist.
The most honest read: we're at the point where LLM agents can look coordinated for a demo but consistently fall apart over long horizons. That's fine for research; it's not fine for production.
The Elephant in the Bench: Anthropic Models Aren't On It
One mild caveat. The 13 evaluated models include Gemini 3.1 Pro, GPT-5.4, and open-weight Gemma/Qwen/Llama families. No Claude models are on the current leaderboard, and neither are DeepSeek or Mistral. If Anthropic's coding-agent variants generalise their tool-use skills to multi-agent settings, the leaderboard could shift once someone runs them.
The GitHub repo is open source, so anyone can run their own model against it via the SUBMISSION.md protocol. Expect a lot of "my agent beat theirs" posts on r/LocalLLaMA over the next few months.
The Bottom Line
ALEM is the first LLM multi-agent coordination benchmark that actually tries to measure the messy, long-horizon stuff. And the punchline is uncomfortable: even the best frontier models sit around 15% Total% on Hard, with the field averaging only ~6%. Communication is the biggest lever. Bigger context and reasoning traces don't automatically save you. Mixed-family teams don't beat their homogeneous average.
But the flip side matters too. Zero-shot LLMs tie purpose-built MARL systems trained on a billion steps. That's not nothing. It means the coordination gap is closable with better prompting, better harnesses, and probably better post-training rather than throwing more RL compute at specialist policies.
If you're building agent products, treat this benchmark like a wake-up call. Solo agent demos lie. Coordination is a separate skill, and right now, nobody's crushing it.
