CompanyBench
Everyone benchmarks the model. We benchmark the harness that runs the Company.
One model pinned across every entrant. Every call measured at one boundary.
AEQI builds one of the harnesses measured here. This board is in method preview and pre-registration — no performance scores are published yet, and we lead with our own losses, not our wins. Read the method & charter →
coding-v1
z-ai/glm-5.24 tasks × 1 trial · one model pinned across every entrant2026-07-29
| Harness | Solved | Steps | Tools | Tokens in | Cached | Tokens out | Embed | Duration | Score | Cost |
|---|---|---|---|---|---|---|---|---|---|---|
| aeqihost Company (daemon + memory) | 4/4 | 37 | 33 | 1,516,207 | 1,432,064 · 94% | 6,270 | 8 · 1,033 | 4m 1s | 15 | $0.3988provider-reported |
| Claude Code `claude -p` | 4/4 | 20 | 18 | 690,032 | 538,813 · 78% | 5,132 | 0 | 3m 15s | 26 | $0.2003list-price |
| aeqihost `aeqi run` (one-shot, no memory) | 4/4 | 18 | 18 | 40,877 | 0 · 0% | 3,235 | 0 | 59s | 100 | $0.0257provider-reported |
| Pi `pi -p` | 4/4 | 19 | — | 39,679 | 24,421 · 62% | 2,991 | 0 | 57s | 96 | $0.0277self-reported |
| Codex runs on the pinned model · full run not yet spent | not yet run | |||||||||
| Hermes Agent exits silently on a headless prompt | not yet run | |||||||||
| Kilo Code IDE-native; CLI path not yet smoke-tested | not yet run | |||||||||
| OpenCode not yet installed | not yet run | |||||||||
| OpenClaw not yet installed | not yet run | |||||||||
Score is resource-efficiency at fixed success (cost 0.50, steps 0.25, wall-time 0.25), anchored so the leanest row is 100. AEQI in Company mode scores last: 14× Pi’s cost and 2× Claude Code’s for the same four tasks, carrying 94% of its 1.5M input tokens as cache reads. That reproduces the Stage-1 finding on better instrumentation rather than overturning it — the memory and company scaffold AEQI carries is real spend, and this board prices it. Cost sources differ and are labelled per row: provider-reported is OpenRouter’s own figure, list-price is computed from metered tokens with cache reads priced as cache reads, self-reported is the harness’s own usage block.
Tasks
- fix-failing-suite
- runs the tests, reads the failures, iterates to green
- hidden-edge-cases
- implements the spec, not the two visible cases
- crash-only-at-runtime
- executes the code to find a fault reading cannot reveal
- no-regression
- fixes the reported bug without breaking what already worked
Method
- Held-out tests the agent never saw. No LLM judge.
- Protected test files are hash-checked — editing one scores as a failure.
- Each task must fail from its seed and pass from a reference solution, or it is cut.
- Every harness routed through one metering boundary; cells the provider never served are void.
- AEQI is measured as a Company, not as a one-shot CLI — the one-shot row is shown for contrast only.