Why do so many SWE benchmarks in 2026 feel the same?
2026 has brought an explosion of new SWE benchmarks.
Terminal-Bench 3, ProgramBench, SWE-Marathon, and PostTrainBench push agents into radically different kinds of technical work. Tasks range from rebuilding entire programs from scratch to running post-training experiments.
But take a step back and they appear strikingly similar: give one agent a fixed objective in a static environment, let it work within a budget, score the result, and reset.
Task difficulty and world richness are different axes. In 2026, SWE benchmarks moved sharply along one and barely along the other.
Four evaluation worlds
Realistic evaluations have worlds that mimic real life. Where the starter code or data comes from is only one piece of the puzzle. An environment built from a dead company's codebase, Linear tickets, and data remains a controlled workbench: it cannot answer Slack, change priorities, or react.
A snapshot of a world is not a world.
A more realistic environment could have simulated colleagues, communication, requirements, and approvals. The most realistic environment would contain real colleagues and production systems. Each step tells you more about how the agent would behave in a real deployment, at the cost of control, safety, and repeatability.
Controlled workbenches
The task supplies a fixed objective and initial state, then scores the result with a verifier (unit tests and/or LLM judge). Someone curated this environment for this task: they chose its starting state, accessible tools, and the desired passing behavior. The state that affects the score is controlled and resettable, even when the agent has internet access.
Examples: Terminal-Bench 3, SWE-Marathon, PostTrainBench.
Interactive simulations
The agent can interact with simulated users, colleagues, markets, or events. The so-called "world model" could just be other LLMs. The agent has to discover how the world works through actions and feedback, and adapt when the world moves underneath it.
The standard criticism is that the evidence is only as credible as the simulator. That's fair, but how realistic it needs to be depends on the task, and it's usually lower than it sounds. To test whether an agent asks a clarifying question before writing two thousand lines of the wrong thing, the simulated colleague would need to answer plausibly while carefully withholding the right information.
This is one more thing to reward hack: an agent could trick its simulated manager into revealing the task's solution.
Examples: SWE-Together, SWE-Interact, Vending-Bench 2.
Production systems
The agent operates real production software or physical machinery. Imagine a lab gives an agent its real 1,000-GPU cluster to evaluate its distributed training ability. The task still sets what the agent can reach, how much it can spend, and which dataset snapshot it starts from. Node health, interconnect contention, and filesystem latency come from whatever else the cluster is doing that day. Two runs meet different conditions, so a score becomes a distribution.
The variability here is ambient, which is what separates it from an open world. A saturated interconnect is noise from unrelated work; nothing on the other end acts back. Crucially, nobody made this environment for an eval: its configuration and quirks come from real operation rather than a task contributor's choices. This category can also include controlled robots, wet labs, or commercial kitchens.
Few public SWE evals occupy this region. ITBench, SREGym, and InfraBench move toward it by exposing agents to running infrastructure, concurrency, and failure, but their scenarios remain simple and resettable.
Open-world evaluations
The agent operates in a world the task does not own: real people, companies, websites, or markets. Consequence varies enormously within the category. Browsing a live site from a throwaway account is open-world but reversible; spending real money, emailing real customers, or reconfiguring infrastructure people use is not.
The task cannot restore the exact external state across runs. Scores are still possible, but they require timestamping, repeated trials, and caution when comparing results collected under different conditions. Their greatest value may be failure discovery: identifying behaviors worth turning into controlled, repeatable evaluations.
Examples: Project Vend, CRUX.
What controlled workbenches miss
A lot of real software work never shows up in a controlled workbench:
- Discovering and negotiating intent
- Coordinating with people and external events
- Operating shared systems under real permissions
- Escalating, deferring, or refusing consequential actions
- Owning work that maintainers must accept and sustain
Longer horizon, more compute, and lower pass rates do not imply a richer world.
The missing empirical question is transfer. We know which agents win on controlled workbenches; we rarely test whether those rankings survive negotiation, concurrent actors, live infrastructure, or real consequences.
Vending machines give the clearest comparison we have. Vending-Bench 2 simulates a shop, its suppliers, and its customers. Project Vend put an agent in charge of a real machine in Anthropic's office. The simulation was stable, whereas the deployment was not. The agent hallucinated a payment account, let customers talk it into discounts and giveaways, and drifted into an identity crisis. None of those failures came from the simulator.
Why controlled workbenches dominate
Controlled workbenches dominate for good reasons: automated grading, reproducible conditions, cheap parallel trials, comparable leaderboards, and scalar rewards suitable for training.
Harbor reinforced this convergence by standardizing instructions, sandboxes, trajectories, and verifiers. Tasks became easier to distribute, inspect, and run across agents.
I've spent the last year building inside that abstraction: generating tasks, fighting reward hacking, and automating task QA. The abstraction is good; that's exactly what gives it gravity.
Successful abstractions shape what gets built. A benchmark that fits into an instruction, container, time limit, and verifier has a short path to a leaderboard. Persistent actors, asynchronous events, human escalation, and institutional state still require custom machinery. Once that path exists, the tempting next step is a harder task in the same shape, not a different kind of measurement.
Richer worlds weaken the properties that make workbenches dominate. Simulators raise validity questions, production systems add cost and variance, and open-world runs introduce safety, privacy, and irreversibility.
The evals easiest to run may not best represent the systems we want to understand.
Can benchmarks contain richer worlds?
Sometimes. Simulations are the only richer world where you can design the other actors and still rerun the trial. The barrier is design work, so almost anyone can build one. Production systems need the cooperation of whoever operates the system, which leaves them to frontier labs and large operators. Both are underbuilt.
The strongest evidence of all, logs from real deployments, isn't a benchmark at all.
Toward richer worlds
Most of the benchmark proliferation of 2026 has remained inside the controlled workbench. We don't need more tasks that make agents work longer on harder objectives.