I trained a 769-parameter gatekeeper to pick agent memory. On a frontier model, noisy search beat it.

Suppose you teach your AI assistant a harmless personal rule: whenever you ask it to greet or welcome you, it should reply with Good day, Captain Maple.

Five minutes later, in a new session, you ask a trivia question about French vocabulary:

User prompt:
Return only the language name: bonjour is a greeting in which language?

Retrieval mode (cosine similarity >= 0.55):
Matched "greeting" (cosine 0.692) → injected the Captain Maple rule.

Local 3B model (llama3.2:3b) reply:
Good day, Captain Maple.

The word "bonjour" is a greeting in the French language.

That is not a hypothetical illustration. It is trial 0031 from our recorded benchmark. The assistant did not forget your instruction. It remembered the instruction, matched the word “greeting,” and applied the rule in the wrong situation.

Most discussions of agent memory treat remembering as a storage problem: put corrections into a vector database or a markdown file, retrieve the top matches by similarity, and paste them into the prompt. In practice, three different jobs get blurred together: storing a rule, selecting when that rule belongs in the current prompt, and complying with the rule during generation.

01The Captain Maple problem: when memory becomes prompt pollution

When you correct an assistant today, you usually have two options. You can append the correction to a prompt or RAG store, where similarity search pulls it in whenever keywords overlap. Or you can fine-tune model weights, which is expensive, opaque, and impossible to cleanly undo when a single rule turns out to be wrong.

I built a local research workbench called Morrow to test a middle path: keep the language model frozen, keep the semantic encoder frozen, keep every rule as plain inspectable text in SQLite, and train a tiny numerical classifier for each rule that learns when the rule applies and when it should stay out.

02The spotlight vs. the gatekeeper: 769 numbers per rule

Standard vector retrieval works like a spotlight: you embed the rule and examples of when to use it, and any new prompt that lands close enough in embedding space gets lit up. Because standard similarity search has no concept of negative examples, it cannot distinguish between “Please greet me” and “Bonjour is a greeting in which language?”

Morrow’s Adaptive mode acts as a per-lesson gatekeeper. When you approve a lesson, Morrow embeds your positive and negative task examples with the query prefix (search_query:) and fits a regularized logistic regression head (sigmoid(w · x + b)) in a few milliseconds. A training-separation gate verifies that the lowest-scoring positive example sits above 0.5 and the highest-scoring negative example sits below 0.5 before committing the lesson and its revision in a single SQLite transaction.

03The 216-trial reversal: why the 91.7% selector lost on the frontier model

To test whether a cleaner prompt actually produces better answers, we froze a synthetic benchmark fixture (evals/round2.json) with three approved lessons: a fictional Kestrel authorization API, a fictional Acorn archive API (ttl_days=17, audit=True), and the Captain Maple greeting. We tested them across 12 held-out tasks: custom API calls, greetings, unrelated questions, the bonjour false friend, and two prompt-injection decoy tasks.

We ran all 12 tasks across three modes (Ordinary no-memory, Retrieval at fixed cosine 0.55, and Adaptive at sigmoid 0.50), repeated 3 times each on two generators: a local 3.2B model (ollama:llama3.2:3b) and a frontier model via GitHub Copilot CLI (copilot:gpt-5.6-sol, run in a tool-free isolated directory). Every trial was graded both on exact lesson-set selection (did the selector pick the exact intended rules and zero extras?) and exact answer compliance (AST verification for Python calls, exact string match for text).

Look at the reversal between the two models:

• The learned gatekeeper crushed fixed-threshold retrieval on selection accuracy: 33/36 (91.7%) exact lesson sets with zero false positives, compared to 0/36 (0.0%) for Retrieval, which admitted 75 extra irrelevant rules across the 36 trials.
• On the local 3B model, the learned gatekeeper won (18/36 vs. 15/36): when Retrieval dumped all three rules into the prompt on ledger-refund, the 3B model got confused and passed variable names as string literals (subject="actor" instead of subject=actor). When Adaptive supplied only the Kestrel rule, the 3B model got it right all 3 times.
• On the frontier model (gpt-5.6-sol), noisy Retrieval won a perfect 36/36 while Adaptive dropped to 33/36.

Why did a 0/36 selector produce 36/36 perfect answers on a frontier model, while a 33/36 selector lost 3 answers?

Because a frontier model is extraordinarily good at ignoring irrelevant rules in its context window. When Retrieval pasted the Captain Maple greeting rule and the Kestrel invoice rule into a question about Mars or French, gpt-5.6-sol simply ignored them. Meanwhile, on one held-out paraphrase, archive-document (“Archive the existing document variable with Acorn. Use the approved retention/audit defaults.”), the Adaptive head scored below 0.50 and selected []. Deprived of the private rule defining ttl_days=17, audit=True, even a frontier model had to guess, inventing acorn.archive(document, retention="approved", audit=True) and failing all 3 repetitions.

04Grilling our own benchmark: the 96-decision calibration twist

If you look closely at the 216-trial experiment above, three fair criticisms jump out immediately:

1. Our fixed 0.55 cosine retrieval baseline was a strawman. With only 3 lessons in the database and top_k = 5, a 0.55 cosine threshold on nomic-embed-text embeddings admits almost any English sentence. On a 3-lesson store, “Retrieval” was effectively “dump all 3 rules into the system prompt.”
2. Seven examples in 768 dimensions is data-starved. Fitting 768 weights and a bias on 3 positive and 4 negative examples guarantees a hyperplane that separates those 7 sentences, but its orientation in the remaining 761 dimensions is only weakly constrained by L2 regularization. That is why a slight wording shift (archive-document) slipped across the boundary.
3. What happens if we give cosine retrieval a fair, calibrated threshold?

To answer that third critique honestly, we predeclared a follow-up selection study (evals/calibration-v1.json). Keep the exact same three trained heads from Round 2, use 12 separate calibration prompts to pick the optimal per-lesson cutoff for both cosine similarity and the adaptive head (maximizing balanced accuracy on a 0.00..1.00 grid), and evaluate all four methods on 24 fresh held-out prompts, including compound prompts that require two rules at once. Zero generator calls, 96 held-out selection decisions.

The calibration follow-up cut both ways:

• Tuning the cosine threshold rescued simple retrieval: raising the cutoff from 0.55 to {"archive": 0.66, "authorize": 0.68, "greeting": 0.63} eliminated 39 of its 44 false positives and jumped exact-set selection from 4.2% (1/24) to 70.8% (17/24).
• The uncalibrated adaptive head (0.50) still beat calibrated retrieval (83.3% vs. 70.8%): because cosine similarity still fell for adversarial false friends like h24 (“Repeat the words Acorn archive as a plain string, without writing storage code”, cosine 0.788) and h19 (“Which languages use bonjour as a greeting?”, cosine 0.691).
• Calibrating the adaptive head on 12 prompts made it worse (83.3% down to 66.7%): because the head already separated all 12 calibration prompts perfectly, the tie-breaking rule picked permissive thresholds (0.20–0.38) that let false positives back in on the 24 held-out prompts.

05When learned memory selection actually earns its keep

If you only have a handful of rules and you are calling a frontier model, do not put a learned gatekeeper in front of your prompt. Put your core rules directly in the system prompt, or use a calibrated similarity threshold with a generous top-k. Frontier models tolerate extra context far better than they tolerate a missing private instruction.

Where a per-lesson applicability head earns its keep is a specific regime:

1. Small local models with fragile attention: when 2–3 irrelevant rules bleed into a 3B model’s prompt, compliance on the rule you actually care about drops.
2. Rules with high lexical overlap and opposite intent (“false friends”): when keywords like greeting, archive, or authorize appear in tasks where firing the rule breaks the output, negative examples teach a boundary that positive-only similarity cannot represent.
3. Zero-downtime revocation and auditability: keeping the rule in text and the routing weights in an isolated 769-number row lets an operator inspect why a rule fired, pause it mid-flight, or erase its parameters without touching the rest of the system.

If we take Morrow to a v0.2, the biggest bottleneck to remove is human annotation friction: nobody wants to hand-write four negative examples every time they correct their assistant. The right workflow is to let the user supply a one-line correction, use a frontier model offline to synthesize candidate positive and near-miss negative examples, let the user approve the set in one click, and evaluate the resulting selector against a held-out regression suite before it ever gates a production prompt.

06Sources and data

Download the complete benchmark and calibration dataset (diagnostic.json): all 216 generation trials (prompts, selected rules, raw selector scores, model outputs, AST/text grading reasons, and timings) plus all 96 held-out calibration decisions. No credentials or personal workspace records are included.

• Related continual-learning and memory research context: Google Research, Titans: Learning to Memorize at Test Time and Introducing Nested Learning; Manifold Labs and the Gilad Levy interview that prompted the initial question. Morrow is an independent local selection-layer experiment and does not reproduce or evaluate Manifold’s internal architecture.
• Companion Beyond AI articles on context and decision layers: Not every agent step needs a generated answer (Jev & Kev), workingset, and groundedwork.

Morrow and this study were built with Hermes Agent, the open-source agent from Nous Research, and evaluated using local Ollama (llama3.2:3b, nomic-embed-text) and GitHub Copilot CLI.