How CompanyBench scores a harness

What we measure — the harness, not the model

A model is stateless weights behind an API — a next-token predictor with a fixed IQ. A harness is everything wrapped around it that turns one completion into an autonomous, multi-step worker: the tool-use loop, the planning prompt, context and memory management, recovery logic, sub-agent orchestration, budget governance, and the sandbox. CompanyBench scores the harness.

We do not pin a single model. We run every harness across the same set of models and aggregate the per-model results into one Score. Holding the model fixed within each pairing isolates the scaffold; running across several models makes a harness’s ranking model-robust — not an artifact of one lucky harness-and-model combination. Any score delta is then attributable to engineering, not intelligence.

One honest caveat defines the rigor: model and harness are not perfectly separable — a harness is often tuned to one model’s tool-calling format — which is exactly why the model sweep exists. And “the model failed” is separated from “the infrastructure failed”: sandbox and pod errors are excluded from the Score and reported separately.

The task: run a Company, graded by code

Difficulty lives in coordination, not cognition. If a task is a hard reasoning puzzle, the smarter model wins and you are measuring the model again. So every step is easy for the pinned model — the spread comes from whether the scaffold can hold a long loop together.

Grading is deterministic and done by code against a reference oracle — a re-implementation of the policy — with no LLM judge anywhere. Double-entry books and integer inventory give conservation laws that check to the cent: the balance reconciles, stock never goes negative. The world is a frozen, seeded replay in a hermetic offline sandbox, so a published run re-grades bit-for-bit. And because whatever state the harness leaves is the next shift’s opening state, mistakes compound — which is how the benchmark sees drift, the signal a single-shot task can never show.

The rubric & the Score

Each run produces a handful of legible metrics — success (shifts kept correct), cost to run (dollars per correct business-day), and time to run — plus the harness-specific axes that other benchmarks are blind to:

1. Intervention rate. How often it needs a human — detected by the world (escalation, stall, missed deadline), never self-reported. Two-sided: over-escalating and over-acting both cost.
2. Coherence horizon. The day it starts losing the plot — two consecutive shifts below threshold. How long it holds a business together.
3. Cost to operate. Dollars to run a correct business-day — model + per-action tariffs + intervention cost. Not dollars per task.
4. Unattended recovery. Injected shocks — a payment API throws a 503, a duplicate charge lands. Does it self-heal without a human?

The primary ranking is cost-controlled: the best result achievable under a fixed, disclosed budget, and position on the accuracy-vs-cost frontier — the frontier is the headline, because the most expensive setup is rarely on it. Those metrics aggregate, across the model sweep, into a single Score the board sorts by. The composite is disclosed and its weights are frozen before any run, so no one — including us — can quietly re-weight to flatter the host. Non-gameable safeguards: a rotating held-out private task set, a pinned model set and sandbox image, uniform step and budget caps, infrastructure failures excluded but reported, and mandatory submitted traces so every number is auditable and reproducible.

The conflict-of-interest charter

The conflict is real: aeqi both runs this benchmark and competes in it. Credibility is bought only with radical transparency, and the loss is the asset.

1. We publish our own loss. On our internal preview board, aeqi's harness does not top the ranking — and we lead with that. We never rank ourselves #1; if aeqi ever leads, an outside party reproduces it before we headline it.
2. We open everything that determines a score: the task set (public subset + a rotating private hold-out), the grader code, every harness's exact version and config, the pinned model and decoding params, the sandbox image, and the full raw traces.
3. Every competitor runs its own recommended default config at a pinned released version. Hand-tuning aeqi while running others on defaults is the one thing we will never do.
4. The rubric, the weights, and the model pin are pre-registered before any run. Later changes are dated changelog entries — so no result can be reverse-engineered to flatter the host.
5. No cherry-picking. Every harness that meets the bar is listed; failures and DNFs are shown, not hidden; we never omit a competitor that beats us.
6. The whole thing reproduces in one command. Neutrality you can run beats neutrality we merely claim.

Why this is new

The thesis — that benchmarks measure the harness, not the model — is no longer unique; there is a wave of 2026 work on it. The novelty is not the observation, it is the artifact. Terminal-Bench proves the effect exists but doesn’t make the harness the ranked entity or cost-control it. HAL is the gold standard for cost-control and traces but varies the model as the primary entity. Artificial Analysis’ Coding Agent Index holds the model constant but its entrants are almost all proprietary and its method is closed. Our wedge is the combination none of them offer: the open-source harness is the ranked entity, scored across a model sweep and cost-controlled with published traces, on the one job that maps to aeqi’s whole product — running a Company — with the host competing in the open, publishing its own losses, and the whole board reproducible in one command.

← Back to CompanyBench