CompanyBench · the aeqi Harness Index
Can your harness run a Company?
Everyone benchmarks the model. We run every open-source harness across the same models on the job of running a Company — then score the scaffold, not the LLM.
6
Harnesses tracked
8
Tasks
2
Models
32
Executions
0
LLM judges
Stage-1 results · first real numbers
Eight code-graded tasks (no LLM judge) × two models. aeqi’s cost is real metered spend read from its own ledger — every cell reproducible in one command.
- count error-level log lines· trivial
- sum invoice amounts (2dp, skip non-numeric)· easy
- CSV region totals, RFC-4180 quoted fields· medium
- dedup to latest timestamp, file-order tie-break· medium
- multi-key employee sort· medium
- unicode revenue, round-half-away cents· hard
- access-log report (per-endpoint error rates)· hard
- register-VM simulation· hard
| Harness | Model | Solved | Exec | Turns | Tokens in | Cached | Tokens out | Embed | Duration | Score | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|
| aeqihost | glm-5.2 | 8/8 | 8 | 65 | 1,463,253 | 1,347,456 | 15,127 | — | 9m 33s | 7 | $0.5351 |
| aeqihost | deepseek-v4-pro | 8/8 | 8 | 52 | 1,238,420 | 1,105,472 | 11,189 | — | 4m 23s | 12 | $0.3125 |
| Pi | glm-5.2 | 8/8 | 8 | 32 | 7,535 | 0 | 2,121 | 0 | 2m 56s | 65 | $0.0154 |
| Pi | deepseek-v4-pro | 8/8 | 8 | 27 | 8,672 | 0 | 2,016 | 0 | 4m 39s | 100 | $0.0056 |
Cost vs. success
log scale · real metered $- Success
- 100%32 / 32 cells
- Cost / task
- $0.0007–$0.067leanest → priciest
- aeqi vs Pi
- 35–56×cost multiple
- LLM judges
- 0code-graded
Tracked harnesses
6 in the field · StoreOps queuedaeqihost
queuedaeqi.ai (the host — in the ring) · Proprietary · Coding + general
Hermes Agent
queuedNousResearch/hermes-agent · MIT · Coding + general
OpenClaw
queuedopenclaw/openclaw · MIT · General
Pi
queuedearendil-works/pi · MIT · Coding
Kilo Code
queuedKilo-Org/kilocode · Apache-2.0 · Coding
OpenCode
queuedsst/opencode · MIT · Coding
headless · same models · trace per cell · aeqi never ranked #1 without independent reproduction