Saving tokens in your custom agent

A practical playbook for engineers building their own agent stacks who want to cut input-token cost without ripping out their existing infra. Covers two complementary approaches, Derek Thomas's ContextForge (model-server layer) and workingset (markdown-vault layer). You probably want one, the other, or both depending on what you control.

Where the tokens actually go in your agent

Before integrating anything, profile your existing token spend. The cheapest optimization is the one targeting your biggest line item. A typical custom agent's per-request token breakdown looks like this:

For most agent stacks, the "user-provided context" line dominates if you load docs/files. workingset addresses this line specifically. ContextForge addresses the history that grows with every turn of a long session.

Path A: integrate workingset (markdown layer)

Use this if your agent reads files from disk as context (Obsidian vault, customer-notes repo, project tracker, knowledge base, internal wiki, anything markdown-shaped).

Step 1: install + verify there's value on your data

$ uv tool install git+https://github.com/idanshimon/workingset
$ cd ~/path/to/your/vault
$ ws init
$ ws diff some/branch --regenerate

Look at the ratio. Anything under 2× means workingset isn't a fit for that vault shape. 5× or more is clearly worth it. On the notes vault this tool was built for, branches range from about 2× to 97×, 18× overall.

Step 2: generate briefs

$ ws brief some/branch --budget 8000 --write
# Writes /brief.md (deterministic shape, ~8K tokens)

The 8K default is the floor where an agent can "work from the brief without re-reading." Smaller saves more but forces re-reads, which negate the savings.

Step 3: swap your "load context" function

The integration is literally one function swap. The pattern across most custom agent stacks looks something like:

def load_customer(slug: str) -> str:
    """Load all source files for a customer branch."""
    base = Path(f"~/vault/cust/{slug}").expanduser()
    parts = []
    for f in sorted((base / "context").glob("*.md")):
        parts.append(f.read_text())
    return "\n\n---\n\n".join(parts)
    # ~57,000 tokens of notes per load in the benchmark

def load_customer(slug: str) -> str:
    """Load the workingset brief; fall back to source on demand."""
    brief = Path(f"~/vault/cust/{slug}/brief.md").expanduser()
    if brief.exists():
        return brief.read_text()
    # Cold path: vault hasn't been workingset-indexed yet
    import subprocess
    subprocess.run(["ws", "brief", f"cust/{slug}", "--budget", "8000", "--write"])
    return brief.read_text()
    # ~1,700 tokens of notes per load, same model, same harness

Step 4: fall-back to source when needed

The brief covers ~80% of "load context" cases. The remaining 20% are when the user asks for verbatim quotes, deep historical context, or something the brief explicitly says "not in this context" for. Wire a tool that lets the agent escalate:

def load_customer_source_file(slug: str, filename: str) -> str:
    """Tool the agent can call when brief is insufficient."""
    p = Path(f"~/vault/cust/{slug}/context/{filename}").expanduser()
    if not p.exists():
        return f"File not found: {filename}"
    return p.read_text()

In your tool definitions, mark this as a fall-back: "Use only when the brief returns 'not in this context' or the user explicitly asks for verbatim historical content."

Step 5: automate refresh

$ cat > ~/.local/bin/refresh-briefs.sh << 'EOF'
#!/bin/bash
set -e
WS=$(which ws)
cd ~/path/to/your/vault
"$WS" reindex
for branch in cust/*; do
    [ -d "$branch" ] && "$WS" brief "$branch" --budget 8000 --write
done
EOF
$ chmod +x ~/.local/bin/refresh-briefs.sh
$ (crontab -l; echo "0 4 * * *  ~/.local/bin/refresh-briefs.sh > /tmp/ws.log 2>&1") | crontab -

4am daily is a sane default. The brief stays useful for ~24 hours unless your vault is extremely volatile.

Path B: use ContextForge (coding agents and long sessions)

Use this if your agent generates code across many files, or runs long multi-turn sessions where history keeps growing. ContextForge is an Apache-2.0 Python SDK by Derek Thomas. It is alpha software (v0.1.0), so pin the version.

What it does

• Stores your documents and session history in SQLite as a tree, with a keyword index (BM25, no embeddings).
• Before each request, loads only the branches that match the question, plus a rolling summary of the conversation. When that summary passes 3,000 tokens it is compacted again.
• When generating a project file by file, keeps only the signatures (imports, classes, method declarations) of files already written, so file 100 costs about the same context as file 1.

Step 1: install and try it

$ pip install "contextforge[all]"

import asyncio, os
from contextforge import ContextForge

async def main():
    forge = ContextForge(provider="openai", api_key=os.environ["OPENAI_API_KEY"], db_path="./memory.db")
    await forge.ingest("./docs/", category="docs")
    print(await forge.chat("What did we decide about the Q3 launch?"))
    forge.close()

asyncio.run(main())

Providers: openai, azure, anthropic, or local with a base_url for llama-server, vLLM, Ollama or LM Studio.

Step 2: use the flat path for long sessions

forge.chat() keeps the full conversation history, so it grows every turn. For long sessions, the engine at forge.infinite is the path that keeps context size roughly flat: await forge.infinite.query(message, provider) loads matching branches plus the compacted summary instead of the whole history. For code, forge.infinite.generate_project(spec, files, provider) does the file-by-file signature compaction.

Step 3: measure

Log input tokens per turn from your provider's usage field, not from the SDK's own estimates. The number to watch is whether input tokens stay flat as the session gets longer. Its permanent-prefix option benefits from prompt caching: fully on a server you run, partly on hosted APIs, where caching is managed by the provider.

A deeper review, with the claims checked against the source and tests: ContextForge: how to give a coding agent a codebase without drowning it in tokens.

Path C: use both

If your agent both reads markdown notes and runs long sessions, use workingset to produce the brief for each folder, and load that brief into ContextForge as the knowledge it retrieves from. workingset makes each document small; ContextForge decides which ones to load on each turn and keeps the history from growing.

What NOT to do

If your stack is...

Where to start

1. This guide (you're reading it).
2. The glossary, glossary.html, so terms line up.
3. The benchmark dashboard, proof the savings claims are measured, not estimated.
4. workingset README, github.com/idanshimon/workingset
5. ContextForge README, github.com/Betanu701/ContextForge

The smallest first step: uv tool install git+https://github.com/idanshimon/workingset, point it at one folder, run ws diff. If the ratio is interesting, continue. If not, no time wasted.

Open an issue at github.com/idanshimon/workingset/issues if integration with your stack runs into something this guide doesn't cover, adoption notes go into docs/adoption-guide.md over time so the next person has less work.