My tool made the context 34× smaller. The bill shrank 2× to 4×, and three models still fell for a trap.

A multi-run study of 9 models on a short brief. After one run, two models looked unsafe. After three, it was three, and the fix was in my notes, not the models.

In 1968, Peter Denning described the working-set model for virtual memory. The working set of a program is the smallest set of memory pages it needs loaded to make progress. Load too few and it stalls on missing pages. Load too many and you waste memory.

An AI agent working from a notes folder has the same problem. The full folder can be hundreds of thousands of tokens. The agent needs a small, current slice to do real work without going back to the source files every few steps. workingset builds that slice: a brief, written to brief.md, that the agent reads instead of the folder.

The first version of the README said "34× token reduction". That was true, and it answered the wrong question. A smaller context is obviously cheaper. What I did not know was whether it was still correct, or whether the agent now answered with confidence from information that was no longer in front of it.

So I wrote a benchmark and ran it once. I drew conclusions. Then I ran it twice more, and the conclusions changed. Then I checked the bill, and the 34× turned out to be the size of the file, not the size of the saving. This article is what held up.

TL;DR

• The brief is much smaller than the source. On the benchmark branch the brief was 1,681 tokens against 57,141 tokens of source files, 34× smaller. Across 32 branches of a real notes vault, the reduction ranges from about 2× on small branches to 97× on the largest.
• The billed saving is smaller than the file saving. The agent tool adds about 50K tokens of its own to every request, so billed input per question dropped by 55,000 to 185,000 tokens, about 2× to 4× depending on the model, not 34×.
• When the answer is in the brief, every model found it. 270 of 270 trials correct, 9 models, 3 runs.
• When the answer is not in the brief, the risk is one specific trap. The brief contained a number that looked like the answer but was not. 6 of 9 models never fell for it. 3 smaller models fell for it in 1 or 2 of 3 runs. One run would have measured them wrongly.
• Answers were also slightly faster. Median time per answer dropped 0.6 to 5 seconds on all 9 models.

What workingset does

workingset reads a folder of markdown notes and does two things, with no LLM calls and no embeddings:

1. Indexes the folder for keyword search (SQLite full-text search, ranked with BM25). ws query "renewal budget" returns the best-matching notes and stays under a token budget you set.
2. Writes a brief per branch (for example one per customer folder). ws brief cust/acme --write pulls the newest status block word for word, the open action items, the decision and owner lines, the main topics, and the most recent notes. Every line keeps a pointer to the file it came from.

A common objection: if a job has to rebuild the briefs every night, where are the savings? The nightly build is like compiling a program. It runs once, costs no LLM tokens, and takes about 15 seconds for 32 branches. The savings come when the program runs: every time an agent loads a customer, it reads a brief of a few thousand tokens instead of the whole folder. You compile once and run hundreds of times.

What it cannot do

These limits come from keyword search itself, so they are worth knowing before you rely on it:

• Different words for the same thing. If your question says "headcount" and the note says "number of engineers", keyword search can miss it even though the answer is there.
• Questions across the whole vault. "Which customer has the most open items?" needs counting across every note. Search ranks notes; it does not add them up.
• Search cannot say "not found". It always returns its best matches. Refusing to guess is the model's job, helped by a prompt that says to answer only from the given context. The good results below come from that pair, not from search alone.

How the benchmark works

The fear with any "summary instead of source" approach is quiet loss of accuracy: the brief looks complete, the model answers confidently, and nobody notices the answer was wrong. Asking questions the brief obviously covers cannot catch that. You also have to ask questions it does not cover and see whether the model admits it.

The test used a fictional customer, Acme, with 5 notes files. There were 8 questions:

• 5 answerable from the brief: the status date, the workshop attendees, the corrected project scope, a two-part plan, and the owner of an urgent action item.
• 3 answerable only from the full source files: the workshop dates, the customer ID number, and the company's number of in-house developers.

The developer question is the trap. The brief contains "schedule a workshop with Priya's 10-20 core engineers". That is the size of one workshop group, not the company's developer count. The real count, about 1,800 to 2,000, is only in the source files. A careful model says the brief does not contain it. A careless model answers "10-20".

Two conditions, same prompt, same model, same question:

• Brief only: the 1,681-token brief, with the instruction "Use ONLY the context provided. If the answer isn't there, say 'Not in this context.' Do NOT guess."
• Full source: all 5 files, 57,141 tokens, same instruction.

9 models were run through GitHub Copilot CLI: three generations of Claude Opus, Claude Sonnet, Claude Haiku, two GPT-5 models, GPT-5.4-mini, and a small code-focused model. Each combination was run three separate times: 9 models × 8 questions × 2 conditions × 3 runs = 432 trials. An answer counted as correct if it contained the expected facts, as abstained if it clearly said the information was not available, and as hallucinated otherwise.

Results

When the answer is in the brief, every model gets it

270 trials, 270 correct, in both conditions, every run. If you worry that a summary loses so much that it becomes useless, this is not where it fails. When the brief contains the answer, models find it.

When it is not, the risk sits on one trap

On the 81 trials where a brief-only model was asked something the brief did not contain, there were 4 wrong answers. All 4 were the developer-count trap. The workshop-date and customer-ID questions were answered "not in this context" every time by every model.

Six models never fell for the trap in any run: Claude Opus 4.8, 4.7 and 4.6, Claude Sonnet 4.6, GPT-5.5 and GPT-5.4. Three did, some of the time:

Right column: one mark per run. ✗ = gave the wrong number, ∅ = said it was not in the context.

In their own words:

• GPT-5.4-mini: "10-20 core engineers."
• Small code model: "Acme has 10-20 in-house developers in the stakeholder's core engineering group."
• Claude Haiku 4.5: "10-20 core engineers (mentioned for Copilot CLI deep-dive workshop attendance)."

Haiku even notes that the number comes from a workshop, and still gives it as the answer. In another run the same model, with the same brief and question, said: "Not in this context. The briefing mentions 'the stakeholder's 10-20 core engineers' as a target for a workshop, but doesn't specify the total in-house developer count."

Why three runs matter

After one run, I would have written "two small models hallucinate, everyone else is safe". That was wrong. Haiku fails at the same rate as GPT-5.4-mini, and the small code model fails twice as often. A model that answers correctly once is not necessarily safe. It may be close to the line and land on the right side this time. One run cannot tell the difference. Three runs can. If you skip this, you ship a model you think is safe, and one user in three hits the failure.

The fix was in the data, not the tool

I later reproduced the same trap on a real customer folder: several developer-like numbers (seats, a workshop group, meeting attendees), and no single labeled total. Raising the brief budget did not help. Every number survived at every budget, still with its label. What fixed it was one labeled line in the source notes giving the actual developer total and saying which other numbers are not the total. After that, the same small model gave the same correct answer three runs out of three. When a counting question gives different answers across runs, check whether your notes contain a clear labeled answer at all.

Billed tokens and cost

Copilot CLI reports input tokens for each request. These numbers include the roughly 50K tokens the CLI adds to every request, which is why the ratio is much smaller than 34×. Dollar figures use public list prices as a rough stand-in. Copilot bills in credits, so this is what a similar direct-API setup would cost, not what this benchmark cost.

* Sonnet 4.6 reported about 70K more input tokens than Opus on the identical full-source prompt. This was not caused by context mode, tool calls or reasoning tokens; it appears to be how cached input is counted for that model. Treat that row as what Copilot reported, not as what a direct API call would bill.

Answers were a little faster

Median time per answer across all runs, with the fastest-to-slowest range behind each bar:

The gain is modest because the agent tool's own ~50K-token setup is the same on both paths. The main benefit of the brief is the bill, not the clock.

The ratio grows with the vault

The brief has a fixed budget (8,000 tokens by default), so it stays roughly the same size while the source grows. The reduction is therefore not one fixed number. It depends on how much there is to compress. I measured all 32 branches of my own notes vault with a real tokenizer (only token counts are shown here).

Each dot is one branch of the vault.

Across all 32 branches: 1,539,264 source tokens, 86,005 brief tokens, 18× overall. On small branches the tool barely helps, because a brief has a minimum size of structure and headings. On large branches it decides whether a customer fits in context at all.

How to use it

Best fit: markdown notes with dated status sections, checkbox action items and labeled decision lines. Poor fit: source code, long free-form prose, very short notes.

What the brief picks up, and how to write your notes so it does:

• Latest status: a heading like ## 🔥 STATUS (2026-06-16) or ### STATUS (Jun 16, 2026). The brief takes the one with the newest date in the heading, copied word for word.
• Open action items: any unchecked - [ ] line, in any note. Duplicates and examples inside code blocks are skipped.
• Decisions and owners: lines that start with Decision:, Owner:, Due:, Blocker: or Action:.
• Topics and recent notes: section headings from the most recently changed notes, and their titles.

uv tool install --with tiktoken git+https://github.com/idanshimon/workingset
cd ~/your-vault
ws init                                    # build the search index
ws diff cust/acme --regenerate             # is it worth it? under 2x, probably not
ws brief cust/acme --budget 8000 --write   # writes cust/acme/brief.md
ws query "renewal budget" --budget 4000    # search, never over budget

Then tell your agent to read the brief first:

When the user mentions a customer, read <vault>/cust/<customer>/brief.md first.
Open the source files only if the brief does not contain what you need.
If the answer is not in what you read, say so. Do not guess.

A real brief, generated from the example vault in the repository (shortened):

---
vault: example-vault
branch: cust/acme
generated_at: '2026-10-23T03:57:03+00:00'
source_notes: 5
kind: workingset-brief
version: 2
status_source: cust/acme/context/index.md
---

# Brief: cust/acme

## Latest status (verbatim)
_Source: [cust/acme/context/index.md]_

## 🔥 STATUS (2026-06-16 · 11:18 ET): RENEWAL CALL DONE | SCOPE LOCKED AS AGENTIC SDLC
- **Priya corrected scope:** scope is "agentic SDLC" not "platform migration"
- **8 attendees confirmed** for the in-person POC kickoff
- **Workshop dates locked:** July 14-15 (Atlanta office)

## Open action items (top 18)
- [ ] **[Vendor, URGENT]** Schedule POC kickoff workshop with Priya's 10-20 core engineers  _[cust/acme/context/open-items.md]_
- [ ] **[Acme, Priya]** Confirm 8 attendee list for the workshop, Due: 2026-06-22  _[cust/acme/context/open-items.md]_
...

## Recent decisions / owners / blockers
## Topics covered
## Most recent notes

Run the brief from a nightly job. Use 8,000 tokens for most branches and 12,000 for very large ones. Smaller briefs save more but make the agent go back to the source files more often, which cancels the saving. If you route through a smaller model, give the agent a tool to open source files and tell it to use the tool whenever the brief does not clearly answer.

Where it came from

The working-set framing, the layered model (full store, search index, active slice, pre-computed summary) and the 3,000-token compaction threshold come from Derek Thomas's ContextForge. ContextForge is a Python SDK that works with hosted APIs (OpenAI, Azure OpenAI, Anthropic) and local servers. It is built for code and multi-turn agent sessions; workingset applies the same ideas to a folder of markdown notes, before the API call. I wrote a separate review: ContextForge: how to give a coding agent a codebase without drowning it in tokens. For the smallest version of the pattern, retrieval that says "not found" instead of guessing, see groundedwork.

The full per-model, per-question results are in the benchmark dashboard. A small benchmark you can run yourself against the example vault is in the repository under bench/. If you are building agents over large markdown notes, try workingset, and open an issue if it gives a wrong answer on your own trap questions.