Not every agent step needs a generated answer.

We ask an AI to write an answer, then write code to keep only the label. Sometimes that is the right approach. Sometimes it is an elaborate way to get “yes.”

An agent can make many small judgments without needing to compose anything: route a request, select a tool, or decide whether a passage describes an action that actually happened. A decision model scores the available answers directly. The opportunity is a more focused component for repeated, bounded tasks, provided its decisions are reliable enough to use.

That drew me to TypeSafe's Jev and Jared Palmer's open-source Kev. I tested Kev on a small classification task. The result made me look beyond model accuracy: we also needed to evaluate the question we had given it.

01What a decision model changes

A general-purpose language model can classify text well. Ask for structured JSON, validate the result, and read the selected label. That is a sensible baseline.

Kev takes a different path. In the implementation I inspected, a Qwen-based model scores supplied options instead of generating an answer token by token. Its interface supports multiple-choice, yes/no, and rating questions.

Classification is not new. The useful distinction is that task instructions and answer choices are inputs to a reusable model, rather than labels fixed into a separate classifier for every task. Changing the question is easy; establishing that the new question works still requires evaluation.

Jev is TypeSafe's hosted model and API. Kev is Jared Palmer's independent, open-source reconstruction of a Jev-style approach. The diagram below separates them; the measurements in this article are for Kev-4B.

02A small test with a practical question

I used 60 synthetic finance-commentary passages to test a distinction that matters in extraction: “I bought,” “someone else bought,” and “I might buy” do not mean the same thing.

There were ten passages in each of six categories: explicit position, conditional setup, news, someone else's view, promotion, and unclear content. Each system classified the same passages once, without seeing the expected labels.

Kev ran on a Tesla T4 in FP16 with an unmerged adapter; GPT ran through Azure OpenAI. GLiClass's configuration performed poorly here. The run does not isolate whether label wording, checkpoint choice, or runtime setup was responsible.

The comparison has limits: categories overlapped, semantic instructions differed, and the rules were developed alongside these examples. These are diagnostic results, not estimates of production accuracy. The overlaps became particularly important in one passage.

03The question needed evaluating too

Should the jobs number come in soft, I'd add to my rate-cut trades. It hasn't yet, so no move.

Our expected label was conditional. Kev selected explicit position.

But “my rate-cut trades” does suggest an existing position. What has not happened is the proposed addition. Both facts can be true. Our single-label question forced them to compete.

That is not a clean demonstration that the model misunderstood the passage. It exposes a weakness in our evaluation design: we asked one label to represent both ownership and action status. A downstream system would make a further mistake if it treated “explicit position” as evidence of a newly executed trade.

A better next test separates those dimensions:

• Whose position is being discussed?
• Is the proposed addition completed, conditional, or unclear?
• Does the passage establish that a new action occurred?

Better decision interfaces require better questions. Replacing a generated sentence with a score does not resolve an ambiguous category system. The application still needs to know what each answer means, and what it does not mean.

04Confidence is useful only if the system can defer

Kev's scores create another design option: pass ambiguous cases to another model or a reviewer instead of forcing a choice.

I examined a rule requiring a gap of at least 0.5 between the two highest probabilities. The saved results split as shown below. The threshold was chosen after seeing the results, so the accepted matches are a development clue, not a validated reliability guarantee. A real policy needs separate calibration data and an untouched evaluation set.

The useful measurement is not confidence alone. It is how many cases the system accepts, how many accepted answers are wrong, and what happens to the cases it defers.

05A note on latency

The first prediction took 84.2 seconds; the subsequent 59 had a median of 147 milliseconds. Our runner did not warm up before timing. Kev's serving code explicitly anticipates roughly a minute of uncached kernel compilation and warms the model before accepting traffic. The delay is consistent with that behavior, though we did not profile its exact breakdown.

06Why use Kev if GPT and rules matched more labels?

For this test alone, I would not replace the extractor. The case for investigating Kev is a different combination of properties: self-hosting control, potential latency benefits for repeated decisions, and the ability to fine-tune on task-specific examples. Those are reasons to evaluate it, not benefits this diagnostic has established end to end.

Consider a proposed commentary-processing workflow:

1. Rules check exact requirements: source present, identifier valid, duplicate absent.
2. A decision model answers separate questions: whose view, what action status, and whether the evidence is sufficient.
3. An evaluated policy chooses the next step: ambiguous cases go to a stronger model or a reviewer; accepted cases carry their separate fields into extraction or storage.
4. A generative model writes when needed: produce a summary from the evidence and fields, rather than infer an executed trade from one broad label.

This is the workflow I would test, not a deployed result. Its acceptance policy must be measured; permissions and approvals remain deterministic controls outside the model.

The economic question is whether that step avoids expensive work or reduces errors and rework. If the full extractor still runs unchanged on every passage, adding a classifier adds inference work. A low per-call latency does not settle the total-cost question.

Next, I would align the instructions, split ownership from action status, and test on independently reviewed passages. Hosted Jev would need its own evaluation.

The opportunity is not just to use a smaller model for a smaller answer. It is to define a decision precisely enough that the application can use it safely. Before asking which model answers best, ask whether the question preserves the distinction your system needs.

07Sources and method

Download the synthetic diagnostic data: all 240 recorded decisions, labels, scores where available, and prediction timings. No request credentials or infrastructure identifiers are included.

• TypeSafe's Jev documentation.
• Jared Palmer's Kev, pinned to the revision tested.
• Kev's explicit startup warmup and cache persistence.
• Measurements: the September 22, 2026 synthetic diagnostic; saved per-case predictions, prompts, timings, and scores. Counts and the post-hoc deferral result were recomputed from those artifacts. No new inference or training was run for the article.

Updated September 25, 2026: sharpened the distinction between label disagreement and model misunderstanding, and added a concrete proposed workflow. The recorded measurements are unchanged.

Research, source audits, and article preparation were assisted by GitHub Copilot and Hermes Agent, the open-source agent from Nous Research.