Octocode Research Benchmark

Same answers, a fraction of the context.

A blind, head-to-head test of GitHub research tools on 30 real questions. Every tool reaches near-identical correctness — so the real question is how many characters each one pushes through the model to get there. Fewer characters = a cleaner context window and sharper model attention.

Characters per question

Each line is one tool. Y = characters that tool pushed through the model to answer that question (model-in tool output + model-out commands & final answer). Lower = leaner. Log scale by default — a few questions are far larger than the rest.
Scale

Correctness per question

Blind judge score (0–10) for each tool's answer, per question. The lines cluster near the top and overlap — that's the near-parity correctness that makes characters the deciding metric.

Quick stats (per tool)

Central-tendency summaries across the 30 questions. Median = the robust “typical question”; mean is pulled up by a few large questions.

Total context budget (all 30 questions)

Sum of every character each tool delivered across the whole set. Useful as a raw budget, but not the headline: it's dominated by a few huge questions, so the multiples look bigger than the typical-question figure above.

How this is measured

1 · The 30 questions

One shared set (compare/github-questions/Q1…Q30.md) — title, id, and a ## Question only; no hints, no answer key. Real GitHub research from single-hit lookups → deep multi-hop reads. Same questions, same frozen refs for every tool — only the CLI differs.

2 · The arms

Octocode is the anchor (npx octocode tools …). Baselines: gh (plain), gh+RTK (rtk gh …), gh+Headroom (gh → Headroom compressor). Each is a separate pairwise matchup.

3 · The flow

0 Preflight verify + pin tools. 1 Answer a fresh isolated agent per (question, arm, pass), leanest legal path. 2 Judge grade blind. 3 Summarize validate + aggregate. ≥3 passes.

4 · Characters

Wrappers log each call: total = model-in + model-out (Unicode code points, both directions; primer excluded, failed calls counted). Never self-reported — recomputed & re-hashed by sumlog.py --strict. Chars ≈ tokens (not a direct token/latency/cost measure).

5 · The judge

Blind X/Y (randomized, seed=42), tool identity hidden. Establishes ground truth, then scores correctness 0–10, depth 1–5, workflow 1–5. Different model family (gpt-5.5) from runners. Correctness-first: leaner never beats more-correct.

6 · Aggregation

Question is the unit. Headline = per-question ratio geometric mean; median = robust central; pooled total shown only with top-contributor share + leave-one-out. ≥3 passes + 95% bootstrap CI. Public suite = orientation, not a shipping gate.

Questions

The 30 shared GitHub research questions — browse all on GitHub. Click a number to open it.