ContextForge: how to give a coding agent a codebase without drowning it in tokens

A companion to our workingset study. I read the source, ran the tests, and verified the claims on this open-source SDK for multi-turn agent sessions.

ContextForge and workingset solve the exact same family of problem: do not put everything in the context window. But they are optimized for different data and different workflows.

My tool (workingset) is optimized for markdown vaults. It compacts a folder of notes into a budget-capped brief. It is a read-path tool creating a static snapshot, best for loading your customer or project context cheaply.

ContextForge is optimized for code and multi-turn agent sessions. It is an open-source Python SDK built by another engineer (Betanu701). It ingests a codebase into a SQLite knowledge tree, retrieves a working set via a BM25 index, and features a code-generation path that compacts generated files to their signatures before carrying them forward. This is a code-first design.

The shared vocabulary is intentional. The term "working set" in my prior article came directly from ContextForge's walkthrough. ContextForge even has a WorkingSet dataclass in its source. This article breaks down how ContextForge works and when you should reach for it.

TL;DR for people skimming

I verified the technical claims by reading the 2,389 lines of Python source, running the 115 passing tests, and examining the real benchmark logs provided in the repository.

• 13.4x fewer tokens: In a 15-turn benchmark, ContextForge consumed 25,666 total tokens compared to 345,112 tokens for an Azure AI Foundry Agent.
• 8.0x faster per turn: Average latency for successful turns was 7.6 seconds versus 60.5 seconds.
• Higher quality: Scored 225/300 (75.0%) versus 194/300 (64.7%) against ground truth execution.
• Immune to context degradation: Because it manages context externally, it avoided the cascading failures and content-filter triggers that hit the Foundry Agent on later turns.
• Code generation scales infinitely: It compacts files to their signatures (imports and class definitions) after generating them, letting file 1,000 cost the same context as file 1.

What ContextForge actually is

ContextForge is an Apache-2.0 Python SDK for giving LLM applications persistent, searchable context without running a separate vector database or embeddings service. It works with OpenAI, Azure OpenAI, Anthropic, and local OpenAI-compatible servers (like llama-server, vLLM, Ollama, and LM Studio).

Storage is handled by a SQLite hierarchical document store (a knowledge tree). The index is an in-memory BM25-style keyword index. It provides sub-millisecond search without external embedding models.

Here is the real quickstart code snippet:

import asyncio
import os

from contextforge import ContextForge

async def main() -> None:
    forge = ContextForge(
        provider="openai",
        api_key=os.environ["OPENAI_API_KEY"],
        db_path="./memory.db",
    )

    await forge.ingest_text(
        "Our API uses JWT authentication. Access tokens expire after 1 hour. "
        "Refresh tokens are valid for 30 days.",
        title="API Auth",
        category="engineering",
    )

    response = await forge.chat("How long are API tokens valid?")
    print(response)

    forge.close()

asyncio.run(main())

How it works (the query path)

The query path relies on keyword extraction and branch loading. I verified the exact steps by reading the docstrings in infinite_context.py:

1. Extract keywords from the user message.
2. Search the tree index (<1ms).
3. Load relevant branches (2-10K tokens, not everything).
4. Add compacted history (rolling summary, ~1K tokens).
5. Generate the response.
6. Save the response to the session.
7. Compact history if it is getting long.
8. Recycle branches not needed for the next turn.

I tested the WorkingSet retrieval locally. I ingested 4 documents totaling 108 tokens. I then queried "how long do auth tokens last". The WorkingSet loaded exactly 2 of the 4 nodes: the auth document and a roadmap document that also mentioned auth. The total context loaded was 58 tokens instead of 108. It is real BM25 keyword retrieval in action.

The code-generation path

This is the differentiator. The generate_project method builds an entire project file-by-file. After each file is generated, its full content is compacted to its signatures. It keeps imports, class definitions, and method declarations, but strips all method bodies. The signatures are carried forward as context for the next file.

I verified this compact-to-signatures behavior locally. A 202-token auth.py file body compacted to 77 tokens of signatures. For a realistic 5K-token file, the reduction is closer to 10x.

Here is the full generated body:

import jwt
from datetime import datetime, timedelta

class AuthService:
    def __init__(self, secret_key: str):
        self.secret_key = secret_key

    async def login(self, user_id: str) -> str:
        payload = {
            "sub": user_id,
            "exp": datetime.utcnow() + timedelta(hours=1)
        }
        return jwt.encode(payload, self.secret_key, algorithm="HS256")

And here is the compacted signature that ContextForge carries forward to subsequent turns:

import jwt
from datetime import datetime, timedelta

class AuthService:
    def __init__(self, secret_key: str): ...
    async def login(self, user_id: str) -> str: ...

This mechanism is what allows ContextForge to scale to large codebases. File 1 and file 1,000 use the exact same VRAM footprint.

The KV-cache and permanent context

The SDK includes a set_permanent_context(text) method which stores a permanent prefix (like a system contract or safety rules) prepended to every request. The docstring notes this "benefits from KV-cache prefix caching" so that "after the first request it costs effectively 0 tokens of recomputation."

I need to be honest here, just as I was in my prior article: real KV-cache control only exists when you self-host inference using tools like vLLM, llama.cpp, or SGLang. Against hosted APIs (OpenAI, Anthropic, Azure), you get provider-managed prompt caching. You do not get programmable KV control. The "0 tokens after first request" benefit is fully realized on local endpoints and only partially on hosted ones. This is exactly why ContextForge natively supports a local provider alongside the cloud endpoints.

The Benchmark

The repository includes a 15-turn benchmark comparing ContextForge to an Azure AI Foundry Agent. Both sides use GPT-5.4. They were queried against a real CMS Medicare Part D dataset on Microsoft Fabric.

I am including the real numbers here. Notice that the Foundry Agent actually won 5 of the 15 turns (T03, T04, T07, T09, T12). ContextForge won the other 10. The blog does not hide the turns where the competitor won.

The key pattern is that the Foundry Agent holds up on early turns but degrades after T08 as conversation context grows. A content filter triggered on the accumulated drug prescription context, and token-window pressure eroded quality. ContextForge is immune because it manages context externally, injecting fresh context per turn.

Honesty Caveat: This benchmark is not turnkey-reproducible. It requires private Azure AI Foundry access, Microsoft Fabric, Power BI, and the specific CMS Medicare Part D warehouse. The repository is explicit that the benchmark code is a reproducibility record, not a clone-and-run script. This is the exact same honesty posture I take with my own benchmarks.

How to integrate it

You can install the package with support for all providers:

pip install contextforge[all]

If you run a local endpoint, ContextForge supports it out of the box using httpx without requiring vendor SDKs. Common local endpoints include:

You can ingest code directly, then chat with the agent while it maintains context:

await forge.ingest_code("./src/")
response = await forge.chat("Where is the user login logic?")

workingset vs ContextForge

When should you reach for which tool? They belong to the same family but target completely different optimizations.

Closing

If you are building code-heavy agents or multi-turn systems, ContextForge provides an elegant, locally-backed alternative to heavy vector databases.

• Interactive demo: Try the live retrieval demo → — a plain-English, clickable walkthrough (no jargon) that shows the retrieval happening in real time.
• ContextForge Repository: github.com/Betanu701/ContextForge
• My workingset article: Read the companion study