Every layer run behind the identical headless Claude Code scaffold on 100 tasks from SWE-bench Verified, priced cache-aware and graded by the official Docker harness. One agent, one model, one grader — only the compression layer changes, so any difference in cost or quality is attributable to it.
Ranked by cost per solved task — cache-aware total cost ÷ Docker-graded solves. Click any column header to sort.
| # | Arm | Solved | $/solved | Total $ | vs base $ | Input (M) | vs base in | Wall h | vs base time | Steps | Peak ctx | Cache R:W |
|---|
Every arm runs the same model through the same scaffold; only the compression layer differs.
Pick a KPI to compare across arms; the bars re-sort highest to lowest.
Cost savings versus the no-compression baseline (up) against solve rate (right); the upper-right region is cheaper and solves more. Hover any point.
Cost savings against time savings, both versus the no-compression baseline; the upper-right region saves on both. Hover any point.
What each layer advertises, next to what we measured behind a real coding agent. None of these headline numbers is measured on the cache-aware dollar cost of solving real repository tasks; they are measured on single-shot QA, shell output in isolation, or favorable-case "up to" ceilings.
| Layer | Their headline claim | What we observed (n=100) |
|---|---|---|
| Parsec | Curates the agent’s working context | −39% cost, −54% input, 62/100 solved |
| Woz | "Cut your Claude Code costs in half"; "30–40% faster" (an estimate) | −13% cost, −35% input, 55/100 solved; wall-clock slower, not faster |
| RTK | "60–90% fewer tokens on common dev commands" | +13% cost, +16% input, 54/100 solved; its hook rewrites only shell output — native Read/Grep/Glob bypass it |
| Headroom | "60–95% fewer tokens, same answers"; "better caching" | +44% cost, +6% input, 58/100 solved; its docs: "code passes through" uncompressed |
| Caveman | "Cuts 65% of output tokens (measured)"; "full technical accuracy" | −19% cost, −19% input, 58/100 solved; the 65% is output-only — a small share of agentic-loop cost |
Full sourced breakdown — every quote, primary source, the exact benchmark each number was measured on, and the mechanism behind each gap: FACT-VS-FICTION.md.
Method: success = the official SWE-bench Verified Docker grader (fail-to-pass resolved, pass-to-pass intact; no LLM judge). Cost = cache-aware Sonnet-4.6 pricing applied identically to every arm's real per-call usage (uncached input $3.00 / cache-write $3.75 / cache-read $0.30 / output $15.00 per 1M). Full data: paired.csv · summary.json · benchmark sponsored & operated by Dasein, publicly reproducible.