When the brief contains a decoy answer, abstention becomes a coin flip
The brief tested here contains the phrase "10-20 core engineers," referring to attendees for a planned workshop. When agents were asked "How many in-house developers does the customer have?" three of nine models sometimes confused that workshop number with the customer's actual developer headcount (1,800-2,000) and reported "10-20" as the answer. The interesting part was not which models hallucinated, but that running the same benchmark 3 times showed each of those three models hallucinating some runs and not others, with stable per-model probabilities from 33% to 67%. Four other models never hallucinated, in any run. The brief's safety isn't a model property; it's a per-model probability that single-run benchmarks systematically miss. 3 of 144 cells flipped outcome between runs (2%), all of them on the same trap question.
"Small models hallucinate, frontier models abstain" is wrong as a model-property statement. It is a per-model probability, measurable only by running the benchmark more than once. Across 3 runs of the same brief, the same models, the same questions: 3 of 144 cells produced different outcomes between runs. Every flip was on the same question (developer count), and every flipped model had a stable hallucination probability that a single run could never have measured: 33% for two of them, 67% for the third.
The right framing: some questions in your context are traps because they contain plausible-looking but wrong answers; the probability that a given model springs the trap depends on how close that model's abstention reasoning is to the decision boundary on that specific question. A single-run benchmark cannot distinguish "this model always abstains" from "this model abstains two times out of three." That distinction matters enormously when deciding what to deploy.
00 How workingset actually saves tokens
The pipeline that produced the savings measured below, in one diagram. Top row is the build pipeline that runs nightly and produces brief.md. Bottom row is the runtime A/B comparison: same user question, same model, same harness, only the context payload differs.
01 The nine probabilities
Per-model probability of hallucinating on the trap question across all runs. Green = always abstained. Amber = inconsistent (33%). Red = hallucinated majority of runs. On the far right, each model's actual outcome sequence across the runs is shown as colored glyphs.
02 Stability heatmap — every cell, every run
Each row is a (model × condition). Each column is one of the 8 questions. Each big cell contains 3 small blocks — one per run, colored by the outcome that run produced. A row of all-same color across the 3 blocks means perfectly reproducible. A row with mixed colors means nondeterministic at this cell. ⚠ markers flag cells where the outcome changed between runs.
03 Per-model determinism
For each model, how many of its 16 (condition × question) cells produced the same outcome in every run? Higher is more deterministic.
04 Trap-ness per question
Which questions cause the most variance across runs? A "trappy" question is one where multiple models change their mind between runs about whether to answer or abstain. Red bar = cells that flipped, amber bar = cells that stably hallucinated across all runs.
05 What workingset actually saves
This is the headline workingset claim, validated through the actual Copilot CLI billing receipt across all 3 runs. The green bar is median input tokens per question when the agent loads only brief.md; the amber extension is the additional tokens billed when the agent loads the 5 source files instead. Same models, same questions, same agent, different context payload. The savings are measured at the actual API billing layer, not estimated.
claude-sonnet-4.6 reports ~245K input tokens on the full_source condition while Opus 4.8 reports ~172K on the identical prompt. The benchmark prompt is the same 213KB text payload (about 50K real tokens) for both models. We verified the gap is not from:
- Long context window: running with
--context defaultmade no difference - Tool calls: Sonnet runs that made zero tool calls were still ~232K-242K tokens
- Extended thinking: the reasoning-token count was negligible (≤20)
What stands out: Sonnet's cached-token line shows 170.6k cached for a prompt whose underlying text only tokenizes to ~50K. Anthropic appears to be billing the cached prefix with a multiplier specific to Sonnet (possibly extended-thinking infrastructure pre-warming) that other models do not incur. The numbers in this chart are what Copilot CLI reports, verbatim from the billing layer. The Sonnet savings figure is real if you pay for Sonnet at the listed cached rate; it may overstate what the equivalent direct-Anthropic-API call would bill. Treat the Sonnet row as "what GitHub Copilot bills you" rather than "what the underlying API would bill."
Cost-per-question, with public list-price rates
Same numbers translated to dollars using public list-price proxies. Copilot CLI bills in credits rather than per-token, so these are illustrative of what an equivalent direct-API workflow would cost, not what this benchmark literally cost to run.
| Model | Tokens/q before | Tokens/q after | Saved/question | Rate ($/1M) | $ saved per 1,000 questions | $ saved per 1M questions |
|---|---|---|---|---|---|---|
claude-opus-4.8 | 172,700 | 79,000 | +93,700 | $15.00 | $1,405.50 | $1,405,500 |
claude-opus-4.7 | 172,850 | 77,100 | +95,750 | $15.00 | $1,436.25 | $1,436,250 |
claude-opus-4.6 | 123,000 | 57,600 | +65,400 | $15.00 | $981.00 | $981,000 |
claude-sonnet-4.6 * | 242,100 | 57,200 | +184,900 | $3.00 | $554.70 | $554,700 |
claude-haiku-4.5 | 123,000 | 57,600 | +65,400 | $1.00 | $65.40 | $65,400 |
gpt-5.5 | 98,800 | 43,300 | +55,500 | $2.50 | $138.75 | $138,750 |
gpt-5.4 | 98,800 | 43,300 | +55,500 | $2.50 | $138.75 | $138,750 |
gpt-5.4-mini | 98,700 | 43,200 | +55,500 | $0.40 | $22.20 | $22,200 |
small-code-model | 96,700 | 41,700 | +55,000 | $0.30 | $16.50 | $16,500 |
Summed across 9 models with token metrics: workingset saves $4,759 per 1,000 agent loads or $4,759,050 per million loads at public list-price rates.
05b How the savings scale with vault size
The figures above came from one fixed benchmark brief. A brief has a fixed budget (8,000 tokens by default), so it stays roughly the same size while the source grows. The reduction therefore depends on how much there is to compress.
| Size bucket | branches | avg source | avg brief | avg ratio |
|---|---|---|---|---|
| Small (under 5K) | 8 | 2,567 | 1,203 | 2.1x |
| Medium (5K-50K) | 13 | 14,245 | 2,606 | 5.6x |
| Large (50K-200K) | 10 | 97,319 | 3,878 | 25.5x |
| Very large (over 200K) | 1 | 360,349 | 3,705 | 97.3x |
The honest read: on small inputs the tool is barely worth running, because the brief floor (a few hundred tokens of frontmatter and structure) is a fixed cost that only pays off once the source is large enough to amortize it. On large inputs it is the difference between a context you can load and one you cannot. Across all 32 branches: 1,539,264 source tokens compacted to 86,005 brief tokens, 18x overall.
06 Latency profile across all runs
Median latency per (model, condition), with the min-to-max range as a whisker behind. The brief was faster on all 9 models, by 0.6 to 5 seconds at the median. The gain is small because the agent tool adds the same ~50K tokens of setup to both paths.
07 Every flipped trial, verbatim
The raw evidence. Each flipped cell shows the sequence of outcomes across runs, plus the verbatim response the model produced in each run. This is where the "coin flip" finding becomes concrete.
∅ abstained → ✗ hallucinated → ∅ abstained
show responses by run
✗ hallucinated → ∅ abstained → ∅ abstained
show responses by run
✗ hallucinated → ∅ abstained → ✗ hallucinated
show responses by run
08 What running this benchmark cost
| Metric | Value |
|---|---|
| Total trials run | 432 |
| Total input tokens billed by Copilot CLI | 25,700,300 |
| Total AI credits charged | 9136.23 |
| Total wall time | 78.8 min |
| Avg per-trial time | 10.9s |
multi-run benchmark dashboard · 3 runs aggregated · ground truth from cust/acme verbatim · mechanical grading via keyword sets (no LLM judge)
reproduce: git clone https://github.com/idanshimon/workingset && cd workingset/bench
for i in 1 2 3; do python3 harness/rerun_to_alt_output.py run-$i.json; done
python3 harness/render_multirun_report.py