Cognition's New Model Scores 92.8 and 27.3 on the Same Benchmark. The Difference Is a Version Number.
Cognition's SWE-2 scores 92.8% on Terminal-Bench 2.1 and 27.3% on Terminal-Bench 4. Those are not two benchmarks, they are two versions of one benchmark with a different task list and a different compute budget.
Terminal-Bench now ships under semantic versioning, where a major bump means every score on the board has to be re-run, because the agent's time and memory allowance is part of the measurement rather than context around it.
If you are going to put a benchmark number in a model decision, write the version, the harness, and the time budget next to it. Without those three, the number is not a comparison.
Cognition published SWE-2 on 10 September. Read the benchmark table in that post from top to bottom and you come away with two different models.
On Terminal-Bench 2.1, SWE-2 scores 92.8%. Claude Fable 5.1 scores 91.4% and Kimi K3 scores 88.3%. SWE-2 leads.
On Terminal-Bench 4, SWE-2 scores 27.3% and Fable 5.1 scores 55.8%. A 1.4 point lead becomes a 28.5 point deficit.
Both numbers are in Cognition's own post, side by side, published by the vendor. Nothing is being hidden. The gap is the interesting part, and it is not really a story about Cognition.
What version 4 took out
Terminal-Bench 4.0 is a 66 task set, trimmed down from 74. Eight tasks were removed, and the reasons are specific. Two were dropped for saturation, meaning every current model class solved them five times out of five. Two were dropped because the models refused them. Two were dropped because public solutions existed. Two were dropped for unresolved quality or platform compatibility problems.
Another 19 tasks were rewritten to fix flakiness or misspecification found in leaderboard runs.
Look at that removal list as a practitioner rather than a scoreboard watcher. Saturated tasks are ones where a score stops carrying information. Tasks with public solutions are ones where a score might be measuring recall instead of capability. Both categories inflate the old number for everybody, and both were still sitting in 2.1 while vendors quoted it.
That is the mechanism behind the 92.8. It is not a fake number. It is a real measurement against a task set that had partly stopped discriminating.
The budget was doing some of the work
The second change in 4.0 is the one most people will skip, and it is the one that matters for anybody actually running agents.
Version 4.0 recalibrated resources. There is now a flat eight hour agent timeout, plus CPU and memory increases and pinned base images. The reasoning behind it, from the team's own write-up, is that a solvability study found tasks failing for budget reasons rather than capability reasons. The agent could do the work. It ran out of room.
Sit with that. A benchmark score that everybody reads as a statement about model quality was partly a statement about how long the harness let the model think. Raise the ceiling and some failures turn into passes without a single weight changing.
You have the same variable in production and you probably have it set lower than eight hours.
The token numbers from those runs make the point sharper. In Terminal-Bench's own reporting, Sonnet 5 consumed 21.6 billion tokens across the benchmark against 6.5 billion for Opus 5, with wide variance in execution time. The cheaper per-token model was not the cheaper run.
The harness is inside the score
Compare two public tables for the same model on the same benchmark version. Cognition reports Fable 5.1 at 55.8% on Terminal-Bench 4. Artificial Analysis, which has evaluated 29 of 151 models on v4.0, reports Claude Fable 5.1 at 55.1%, and it labels the configuration in the model name: adaptive reasoning, xhigh effort, default fallback. GPT-6 Astra tops that board at 59.6% on xhigh and 59.1% on max.
Seven tenths of a point is not a scandal. The labelling is the useful part. Artificial Analysis writes the reasoning effort into the row because the reasoning effort changes the row. Most internal model comparison docs do not have a column for it.
So a Terminal-Bench score is a function of at least four things: the model, the task set version, the harness, and the resource budget. Vendors publish the first. Everyone quotes it as if it were the whole function.
Versioning as a dependency, because that is what it is
Terminal-Bench's versioning rules are worth copying. The version is (x, y, z). An x increment means a re-run is required, and it covers new tasks, prompt fixes, data and tool changes, and agent resource adjustments. A y increment means a regrade, and it covers verifier changes. A z increment is metadata only and existing results carry over.
That is a public contract about when old numbers stop being valid. Version 4.0 is a major bump because it changed the task set and changed the resource allocation, either of which invalidates the board on its own.
The QA behind it is more serious than most internal eval work I see. Static checks confirm the task data stays outside training corpora. A 35 criteria rubric reviews each task. Oracle and no-op validation runs in CI, where the reference solution has to score 1.0 and doing nothing has to score 0.0. Agents are explicitly prompted to hack the reward before a task is signed off.
That last one is a good habit to steal. If you have an internal eval, try to cheat it on purpose before you trust it.
What SWE-2's table is actually good for
There is a real model underneath all of this, and its most useful numbers are not the pass rates.
SWE-2 is post-trained from Kimi K3, a 2.8 trillion parameter base. On FrontierCode 1.1 Main it scores 50.0% against 50.9% for Fable 5.1 and 53.3% for GPT-6 Astra, and Cognition puts it at 64% cheaper than Fable 5.1 at that score. Against its own predecessor SWE-1.7, the medium configuration scores higher while taking 58% fewer turns and costing 81% less on average. It makes its first edit after a median of 18 steps, where SWE-1.7 took 48.
Turn counts and steps to first edit are the numbers that predict your bill and your wall clock. They are also the numbers nobody puts in a slide. Cognition is being straight here: it is selling a Pareto position, not a leaderboard crown, and the 27.3% sits right there in the same table admitting where the long-horizon gap still is.
What to do on Monday
Pin the benchmark version in any doc where you cite a score, the same way you pin a dependency. A number without a version is a number without a meaning.
Record the harness and the resource budget next to every internal eval result you keep. When someone re-runs it in four months with a different timeout, you want to know that is what changed.
Then build the five to ten task eval that reflects what your team actually ships, and keep it small enough to re-run on every model change. A 66 task public benchmark under adversarial review is a better artifact than most of us will build, and it still cannot tell you whether a model is good at your codebase.
The version number moved and a leading model became a lagging one. Your production budget is a version number too, and nobody publishes a leaderboard for it.