A field guide to the words people throw around when they talk about LLM context, token cost, agents, and why "the context window is full" is a real problem with real solutions. Written for people who use AI coding tools but haven't sat through the academic version. Pin this somewhere; refer back when a podcast tosses one of these terms at you.
The five terms underneath everything else. If you only learn these, you can follow most LLM conversations.
The unit LLMs read in and bill by. A token is roughly 3-4 characters of English text.
Models don't see letters; they see numeric IDs that represent chunks of text. The word "context" might be one token. The word "antidisestablishmentarianism" might be six. Pricing is per token in and per token out. When someone says "this prompt is 50K tokens," they mean about 200 KB of text.
Like counting syllables to charge for a phone call. The platform doesn't care what you said, just how many "things" passed through.
The maximum number of tokens a model can read in a single request.
GPT-4o: 128K. Claude Sonnet 4.6: 200K (extends to 1M in "long_context" mode). Gemini 2.5: 2M. This is a hard ceiling, go over and the API rejects your request. Different models have different ceilings and you pay per token even on requests well under the limit.
Like the desk in front of you. Anything not on the desk doesn't exist to the model. Bigger desks are nicer but cost more per item placed on them.
Everything you send to the model in a request: system prompt + tools + history + your question.
When you ask "summarize this customer," the payload includes all the text the agent reads to answer. The model sees one big blob; what fraction is YOUR content vs framework overhead matters for cost.
Like a courier package. You pay by the package weight, not by the importance of what's inside.
A hidden instruction set the agent harness prepends to every request.
"You are a helpful coding assistant. Always think step by step. Available tools: …" That's the system prompt. It's usually 5K-50K tokens long depending on the harness. You pay for it on every single request, even if it never changes between your turns.
Like the standing instructions a new hire gets on day one. The boss has to re-read them out loud at the start of every conversation, every time, billed by the word.
Input tokens are what you send; output tokens are what the model writes back. Priced differently.
Output is typically 3-5× more expensive per token than input. But agents read enormous context (lots of input) and emit short answers (few output tokens), so input cost dominates. Reducing input tokens is where the actual money is.
Like a paid tutor who charges $1/page for what you give them to read and $5/page for what they write back. You'll usually hand them way more pages than they hand back.
Internal model memory of "I already processed this prefix; here's the state."
When a model reads your 50K-token prompt, it computes intermediate state for every token (the "keys" and "values" of attention). If your next request shares the first 49K tokens, the model could re-use that computation instead of redoing it. That re-use is what KV caching means.
Critical fact: most API users don't have access to KV cache. Anthropic, OpenAI, and GitHub Copilot CLI all handle it server-side and may or may not give you the cache hit. You can't reach in and control it.
Like a chef pre-chopping mise en place. If you order the same dish again, they don't re-chop. If you walk into a different restaurant, they re-chop from scratch.
A KV cache trick where the provider charges less for the cached portion of your prompt.
Anthropic's "cache_control," OpenAI's "cache_read," and other providers all bill cached input tokens at 10-90% off the full rate. If your prompt has a stable prefix that you reuse (a long system prompt, a static document), you can save real money by structuring the request so the prefix gets cached.
It doesn't reduce token COUNT (you still send the tokens), just per-token COST.
Like a frequent-flyer discount. Same flight, same plane, cheaper if you can prove you've flown it before.
The function that converts text into tokens.
Different models use different tokenizers. tiktoken for GPT-4o uses the o200k_base tokenizer. Claude uses its own. The same text becomes different token counts on different models, which is why bills can vary by 20-30% for identical prompts. For rough estimation, "chars ÷ 4" works for English markdown.
Like translating to different alphabets. The phrase "hello world" might be 11 characters, 2 words, or 8 ideograms depending on which alphabet you're counting in.
Crafting the text you send to a model to get better answers.
Older industry skill, "say 'think step by step' before the question," "show examples," "specify the output format." Less hot in 2026 than in 2023 because frontier models reason better unprompted. Still matters for cheap models and for high-stakes structured outputs.
Fetch relevant docs from a database, paste them into the prompt, then ask.
The classic "give the model context it doesn't have" pattern. Almost every "chat with your docs" product is RAG underneath. The retrieval step typically uses embeddings or BM25 to find which docs are relevant.
Like an open-book exam. The model is the student; RAG is "before answering, go look up the relevant chapter and have it in front of you."
Numbers that represent the "meaning" of text, used for similarity search.
Text → vector of ~1,500 floats. Cosine distance between vectors approximates semantic similarity. Powers most "search by meaning" features. Two main downsides: needs a separate embedding service or model (extra latency, extra cost), and embeddings are great at fuzzy similarity but poor at exact-entity match (names, IDs, codes).
Like ZIP codes for ideas. Things with nearby ZIP codes are usually similar; things with far-apart ZIP codes are usually different. Useful, but doesn't tell you the exact address.
A keyword-search ranking algorithm from 1994. Still beats embeddings on entity-heavy queries.
Stands for "Best Matching 25." Ranks documents by how well their keywords match your query, weighted by rarity. Used by Elasticsearch, SQLite FTS5, and most production search systems. Free, fast, no external service, deterministic. Beats embeddings when your query contains names, IDs, slugs, or specific terms, exactly what most customer-notes / project-tracker queries look like.
Like an old-school library card catalog with a smart librarian. They don't understand the vibe of your question, but if you mention "the book about the white whale" they find Moby-Dick instantly because both your query and the book mention rare words.
The built-in full-text search engine in SQLite. Implements BM25 with no external dependencies.
Ships with Python's stdlib. No server to run, no Docker, no API key. Will hold a few million documents on a single laptop without breaking a sweat. The reason workingset can claim "no runtime services."
Like a Swiss Army knife that happens to have a working full-text-search blade. Always there, no setup, surprisingly capable.
An LLM that can call tools and decide what to do next, in a loop.
Distinguished from a chatbot by autonomy. A chatbot answers your question and waits. An agent reads your question, calls tools, reads the output, decides whether to call more tools, eventually answers. Most "AI coding tools" you've used in 2026 are agents, Claude Code, GitHub Copilot CLI, Codex, Cursor agent mode, Hermes.
A model emitting a structured "please call this function" request that the harness executes.
The model says "I want to call read_file(path='/Users/x/foo.md')." The harness runs it, captures the output, feeds it back as the next turn. Each tool call adds tokens (the tool's definition + the result) to the running context. Lots of tool calls = lots of context inflation.
The cycle of: model thinks → model calls tool → harness runs tool → result feeds back → model thinks again.
The reason agents can do useful work but cost a lot. Every iteration sends the full conversation history back to the model, so a 10-step task pays for the same context 10 times.
Like asking a contractor to fix your kitchen by phone. Each step they have to re-describe the whole kitchen to you. By step 10 you've heard about the kitchen 10 times, charged each time.
The software layer between you and the model that runs the agentic loop.
Hermes, Claude Code CLI, GitHub Copilot CLI, Codex CLI, Cursor, these are all harnesses. They provide the system prompt, the tool catalog, the conversation memory, the loop logic. You write a question; the harness handles the model calls.
A markdown file that documents a recurring task pattern, loaded into the agent on demand.
Hermes's "skills" system; Claude Code's CLAUDE.md; Codex's AGENTS.md; Cursor's .cursorrules. Same idea: capture "how do we do X here" once, in a file the agent reliably reads, so you don't have to re-teach it every session.
When the model confidently states something false.
Usually because the answer wasn't in the context but the model pattern-matched on something that LOOKED like the answer. The most dangerous mode is "near-miss confabulation", the model takes a number from a nearby part of the context and confidently puts it in a totally unrelated answer. ("The brief mentions 10-20 core engineers" → "Acme has 10-20 in-house developers." Wrong context.)
The model correctly saying "I don't know" when the answer isn't in its context.
The opposite of hallucinating. Hard to measure in standard benchmarks because most benchmarks only ask questions whose answers ARE in the context. Production reality is the reverse: agents fail because they don't abstain when they should.
The top-tier model from a major lab. As of mid-2026: Claude Opus 4.8 / Sonnet 4.6, GPT-5.5 / 5.4, Gemini 2.5.
Distinguished from "small" or "mini" models by capability AND by behavior. Frontier models are more likely to abstain correctly, less likely to hallucinate on trap questions, and dramatically more likely to follow nuanced instructions. They're also 5-30× more expensive per token, so production stacks usually mix tiers.
A question whose context contains a plausible-but-wrong answer hiding in plain sight.
The phrase "10-20 core engineers" appears in the brief, referring to workshop attendees. Someone asks "how many engineers does the company have?" The model can either recognize the context mismatch and abstain, or grab the "10-20" and run with it. Whether it does is essentially a coin flip near the abstention threshold, and varies per model.
As context fills with stale tool output, model attention degrades on what actually matters.
If your agent makes 20 tool calls and the last 19 are just file listings, the conversation history is 90% noise by the time the real question comes up. Even within the context window, the model performs measurably worse on the question of interest because attention is diluted across irrelevant tokens.
The cluster of ideas that workingset and ContextForge address.
Replacing a big chunk of context with a smaller chunk that captures the same information.
Different from RAG. RAG fetches docs you'd otherwise miss; compaction shrinks docs you already have. Compaction can be intra-conversation ("summarize the last 50 turns") or pre-computed ("here's a brief of the whole customer file"). Both reduce input tokens, both risk losing fidelity.
Derek Thomas's framework for KV-cache-aware context management on self-hosted models.
An open-source Python SDK by Derek Thomas for coding agents and long multi-turn sessions. It stores knowledge and history in SQLite, loads only the matching branches into each request with a keyword index, compacts conversation history past 3,000 tokens, and keeps only the signatures of files it has already generated. Works with OpenAI, Azure OpenAI, Anthropic and local servers. github.com/Betanu701/ContextForge.
A markdown-layer take on the ContextForge ideas. Works on your files, before the API call.
Walk a folder of .md files; build a SQLite FTS5 index; produce a per-folder brief.md any agent can load instead of the source. About 2× smaller on small folders and up to 97× on large ones, measured on a real notes vault. Works with any model API because it runs before the API call.
The name comes from Peter Denning's 1968 "working set" concept, the minimum subset of pages that has to be resident in physical RAM for a process to make progress. The brief is the working set for an LLM agent. github.com/idanshimon/workingset.
The output artifact workingset produces: a ~8K-token compact summary of a folder, with structured sections.
Every brief has: YAML frontmatter (version, source, generated_at), the most recent STATUS block verbatim, all open action items, recent decisions/owners/blockers, topic headings round-robined across recent notes, and the 8 most-recently-modified note titles. Every line has source-file provenance. Deterministic shape, predictable for downstream consumers.
L−1 (LoRA), L0 (residual / brief), L1 (working set / query result), L2 (index), L3 (store / filesystem).
Derek's mental model. Information moves UP the layers offline (build index from store, compute brief from index). Information moves DOWN at query time (brief + relevant working-set chunks fed to the model). The brief is the L0 because it captures "what stays in memory between agent calls", the residual state.
Anthropic's open standard for letting agents talk to external tools/services via a uniform protocol.
Before MCP, every harness had its own tool-definition format. With MCP, you write an MCP server once (exposes your database, your filesystem, your API) and any MCP-compatible harness can talk to it. Adopted by Claude Code, Hermes, Codex, etc.
A discount the provider gives you for re-sending the same prefix.
Same concept as prefix caching but emphasizing the BILLING angle. Anthropic charges cached tokens at 10% of full rate (90% off). OpenAI similar. You can ENGINEER your requests to land in the cache by keeping the system prompt + tool definitions + static docs stable across calls and varying only the user-question portion.
Workingset benefits from this indirectly. The brief.md changes once a day (cron job at 4am), so during the day every "load this customer" call uses the same brief content → high cache hit rate → cheap.
The big-picture story: everybody is trying to put less stuff in front of the model, more cheaply, without losing accuracy. The token-pricing model makes this a real-money problem. Every architectural pattern in this glossary is a partial answer to "how do we do that?"
Source: this glossary distills concepts surfaced in the workingset benchmark (432 trials, 9 LLMs) and the article walkthrough. Repo: github.com/idanshimon/workingset. ContextForge: github.com/Betanu701/ContextForge. If a definition here looks wrong, open an issue.