Lab · Built in the open · In progress
Small models,better systems.
Ask a deep-research agent a real question and it has to do what an analyst does: find what exists, read it, work out what it means, and write the answer down with its sources attached. Almost all of the difficulty sits in that last part.
Why the category is hard
A research report fails in two ways, and neither looks like failure on the page. The first is a citation that does not say what the sentence claims it says - the marker is there, the link resolves, and nobody checks. The second is coverage: the report reads as complete, and the thing it never mentions is the thing you needed.
Both are invisible to the system that produced them. A model asked whether it was thorough will say yes. A model asked whether its quote is really on the page will say yes. That is the problem in one sentence: the parts of the job that most need checking are exactly the parts a language model cannot check about itself.
The usual answer, and the bet against it
The usual answer is a bigger model. It works, it is expensive, and it teaches you nothing you can carry to the next problem: when the bill arrives you still do not know which part of the quality you were paying for.
Beane takes the other bet, which is Moneyball's - put the money where the measurement says it matters, and let code do what code can check. It runs on mid-tier models only, the largest model used nowhere, so any gain has to come from how the work is split, checked and merged. That is a falsifiable claim, which is why it is measured in the open against a public benchmark and every score goes in a ledger.
So the work gets split
Two sibling runs go through this independently, then get merged. Any stage can be replayed over a saved batch without paying for the research again.
Set up, once
- Intake
- Scout
- Checklist
- Study roster
- Plan
Each run ×2 siblings
- Research
- Ground
- Compute
- Write
- Render
Then
- Merge
Sub-section by sub-section, tables unioned in code
What each stage does
- 01IntakeFixes the language, layout, deliverable and reader before anything is searched.
- 02ScoutFinds what exists, then sweeps for the members of any set the request named in the plural.
- 03PlanA checklist and a study roster aim both the plan and the coverage judge.
- 04ResearchWorkers run in parallel: search, open sources whole, take structured reads.
- 05GroundEvery quote is located on its page in Python. Claims are judged one at a time.
- 06ComputeDerived figures are worked out in a sandboxed interpreter and filed with their code.
- 07WriteLayout, a table writer with row repair, a coverage pass, an analysis pass, a gap fill.
- 08MergeTwo sibling runs merged sub-section by sub-section, tables unioned in code.
And three rules hold it together
Each was paid for by a bad run.
01
The model judges, the code counts
Which page to open and what a finding means are judgement calls, so a model makes them. How many sources, rows or figures actually reached a section is a count, so code makes it.
Ruled out: asking a prompt to report its own coverage. A model asked how thorough it was will answer, and the answer is not a measurement.
02
A citation is a promise the source says this
Every quote is located on its page in Python before any model is asked to judge it. Figures and quoted words are used as the evidence gives them, and a computed number never wears a citation.
Ruled out: trusting a model's own reference list. A citation that was never checked against the page is decoration, and it fails exactly when someone follows it.
03
Never gather more than you can write up and check
Searching stops while there is still enough budget left to write and ground what is already held. A check that did not run is recorded as not run - every judge logs done against total.
Ruled out: gather-everything-then-summarise. Zero faults after checking nothing is the worst output the system can produce, and it looks identical to a clean pass.
Which is what the score is for
63.0
Mean score, DeepResearch Bench II
The share of the benchmark's rubrics satisfied, judged on the raw report by the benchmark's own evaluator, averaged across the tasks run. Pre-submission: a slice of the benchmark's 132 tasks, each run twice and merged. A full submission is still ahead.
By rubric
- Recall62.5
- Analysis61.6
- Presentation85.5
The share of all checks of each kind that the run satisfied. Recall is what decides rank on this benchmark, and it is what moved most in the last batch - up nine and a half points. The benchmark ↗
Measured against the field
Pre-submission scores on both generations of the benchmark - not a full submission on either. On both, every deep-research product shipped by a frontier lab sits below it.
DeepResearch Bench II · rubric score
- AI2164.4
- Beane63.0
- nvidia-aiq54.5
- OpenAI o3 Deep Research45.4
- Gemini 3 Pro Deep Research44.6
Second, 1.4 off the leader. The gap to the frontier labs' products is 17 to 18 points.
DeepResearch Bench I · RACE
- Beane54.6
- Gemini 2.5 Pro50.0
- OpenAI47.8
- Perplexity43.1
- Grok41.2
Fifth of fourteen on the older generation, ahead of Gemini by 4.6, OpenAI by 6.7, Perplexity by 11.5 and Grok by 13.4.
The number I would defend is not 63.0. It is that the benchmark runs against a public evaluator, that the full ledger of every scored label lives in the repo, and that a regression shows up as a number rather than a feeling. A change to a prompt or a check is expected to arrive with a replay over a saved batch and the score it moved. That is the only reason the last jump is worth anything: the same harness, scored the same way, on the same tasks as the run before it.
Status
Private repository today, intended for open source under Apache 2.0. Next: the full benchmark submission, then a model-provider abstraction and a local-model tier. When it opens, the ledger and the replay tooling open with it.