Adding Jev: small decisions, made fast

In Part 2 each reply cost about sixteen model calls, and the most common tool call by far was reading the whole page to find one element. Those are narrow questions: which of these is the reply box? did my reply post? is a popup in the way? A general model is a very expensive way to answer them.

What Jev is

Jev is TypeSafe's decision model. It doesn't write or plan. You give it some state (a page, a pending action) and ask typed questions:

• choice: pick one of up to 255 named options, with a probability for each;
• noul: the probability that a statement is true;
• score: a rating on a scale you define.

Answers are calibrated probabilities rather than text, so you can decide in code how sure is sure enough. It charges only for input, $0.042 per million tokens, and the same request body works through TypeSafe directly, Vercel AI Gateway or OpenRouter:

jev:
  enabled: true
  provider: typesafe        # or: vercel, openrouter
  api_key: ${TYPESAFE_API_KEY}

It's an accelerator, never a requirement. If Jev is misconfigured, unreachable or unsure, Slapify falls back to the main model and carries on.

Where Slapify uses it

The benchmark

We took 58 decisions straight out of the Twitter agent's real runs, using snapshots and page text saved at the moment the agent had to decide (all handles and third-party content redacted). Then we asked the same question two ways: through Slapify's own Jev code path, and by giving Claude Sonnet 5 the same page and asking it directly, the way the agent decides without Jev.

• 30 element decisions: find the reply box on a tweet page (15), and find the main tweet's Like button among a page of Like buttons (15).
• 16 page checks: is this page a single tweet's own page with a box to reply to it?
• 12 popup checks: is something covering the page?

Finding elements

On the reply box Jev was right 15 times out of 15, and confident on 12 of them. The Like button is the harder case: a tweet page has a Like button on every reply, and "the main tweet's" is a positional judgement. Jev got 10 of 15, but it knew it: it was confident on only 4, and 3 of those were right. Sonnet 5, reading the whole snapshot, got 13 of 15.

That's the design working as intended. The question is not "is Jev as accurate as a frontier model", it's "when Jev says it's sure, is it right, and how often is it sure?" Drag the threshold to see the trade:

At Slapify's default of 0.7, Jev takes about half the element decisions with 94% accuracy on the ones it takes, and the combined system lands at 90% against 93% for the frontier model alone, while spending a fraction of the tokens. On the reply box, the decision the agent makes on every single reply, it's perfect.

Checking the page

This result surprised us. We asked both models whether each page was a single tweet's own page with a reply box. Sonnet 5 said no every time, which made it right on the eight pages that weren't and wrong on the eight that were. In fairness, the question is hard from page text alone, which doesn't include the reply box itself.

Jev didn't answer yes on those eight either. But it didn't say no: it gave them 0.39 to 0.59, and every page that really wasn't a tweet page got 0.03 or 0.04. It knew what it didn't know. Slapify treats anything between 0.15 and 0.85 as "look for yourself", so Jev's answers there hand the question back to the agent instead of being wrong. Move the band:

Popups were the easy case: both models were right on all twelve. That's the flow runner's gate, where Jev now skips the popup-checking model call whenever it's clearly unnecessary.

Method. Jev: Slapify's own JevClient and decision helpers, TypeSafe endpoint, default settings. Sonnet 5: the full snapshot or page text plus the question, one short answer. Cost: Jev's reported input tokens at $0.042/M; Sonnet 5 input estimated at about four characters per token at $2/M, output at $10/M. The estimate slightly undercounts Sonnet tokens, so if anything it flatters the frontier model on cost.

A note on reliability

The first time we ran the benchmark through a gateway, 36 of 58 requests came back 429 or 503: the upstream was under heavy demand. Slapify's Jev client now honours retry-after, retries twice, and falls back cleanly; with that, the gateway run still lost 14 of 58 at the time of testing, while TypeSafe direct answered all 58. Your mileage will vary with load. The point is that because every Jev feature has a fallback, a flaky upstream costs speed, never correctness.

In action

Back to a demo site, this time Chirp, a small fictional social feed (posting only updates the page, nothing leaves the machine). The agent's job: reply to the two posts about TypeScript with a useful tip each, and confirm each reply appears. Jev, action review and the em dash filter were all on.

Watch for three moments:

• Step 7. The model's first reply contained an em dash. The text filter replaced it before it reached the page, and told the model so.
• Steps 9 and 16. Before each Reply click, the review runs. Jev judged both replies clear under the policy (confidence 0.83 and 0.67) in about 1.1 seconds each.
• Steps 11 onward. check confirmed each reply at 0.91 and 0.94. Then the agent checked again, and again. Verification is the agent's comfort zone. After the same result three times on the same page, the harness added guidance to the result: nothing is changing, stop re-checking. It stopped, saved a note to memory, and finished.

That last one is honest footage of a weakness. The rules say to verify once. The model still over-checks when it's cheap to, and cheap is exactly what Jev makes it. Admission guidance caught it, but it's the kind of thing we keep tuning.

What it's worth on the Twitter agent

Jev can't write replies or plan a cycle, so the model calls themselves don't go away. What goes away is the reading. Each get_page_state the agent takes to locate an element puts about five thousand tokens into its context; a find_element call puts in about a hundred. The slider below estimates the effect on the 26.8-hour run, using its measured shape (5.1 snapshots per reply) and the benchmark's confidence rate.

At half, that's roughly a sixth of the bill. Not dramatic, because caching already makes repeated reading cheap, but it compounds with the other thing Jev buys: review. Checking every post against a policy with a frontier model would add a full model call per reply. With Jev it's about a second and a fraction of a cent, which is what makes reviewing every irreversible action a reasonable default instead of a luxury.