The 10 concepts of a cheap judge

Tap any concept to jump to its 5-card cluster.

• 1What Jev IsA model that decides instead of writes
• 2The Three Question Typesnoul, choice, score — and what the numbers mean
• 3Typed ≠ CorrectWell-formed answers can still be wrong
• 4Where Jev Sits in the LoopAfter the deterministic checks, before policy
• 5Four ArchitecturesThe same judge, four costs of being wrong
• 6The Published NumbersWhat the benchmarks do — and don’t — show
• 7Measure the SystemRecall, cost and misses — end to end
• 8When Jev FitsNarrow, repeated, measurable, reversible
• 9Shadow-Mode RolloutEarn authority one region at a time
• 10Jev Inside Claude CodeAn advisory hook that can never block

A model that answers typed questions, not prompts

Jev is TypeSafe AI’s first “System One” model, released in early access on 15 September 2026. You send it a state (the evidence) and a few typed questions; it sends back one typed answer per question, each with probabilities. It never writes prose.

Because an answer is a category or a number, not a sentence, TypeSafe says Jev samples all answers in one parallel pass. The published result: 70–500 ms per call and $0.042 per million input tokens, with output tokens not charged.

The triage nurse and the attending physician

BEFORE: An emergency department has one senior physician, and every patient waits for them — the broken leg and the paper cut alike.

PAIN: The physician spends the night on paper cuts while the chest-pain patient waits. And any triage protocol has a known failure: the patient who looks fine and isn’t.

MAPPING: Jev is the triage nurse: ninety seconds, fixed questions, a category. Claude is the physician who diagnoses and treats. Good hospitals never let triage send anyone home — triage orders the queue, it does not empty it.

What goes in, what comes out

1. Send stateThe evidence as a string or JSON: test output, a diff hunk, run history.
2. Ask typed questionsEach has an id, a type, instructions, and criteria describing each option.
3. Get typed answersOne per question, with probabilities attached — never free text.
4. Let code decideYour policy reads the numbers and chooses the action. Jev decides nothing on its own.

Jev or Claude for this job?

# Which model should own this step?

IF the output is a category, yes/no, or score
   AND the question is narrow and repeated:
  USE Jev  # ms, fractions of a cent

ELIF the step needs reasoning, code,
      an explanation, or a fix:
  USE Claude  # the physician, not the nurse

ELIF a deterministic tool can decide it
      (type error, lint, failing assertion):
  USE the tool  # no model at all

Price: $0.042 per 1M input tokens vs $2 for Sonnet 5 — roughly 48×. Useful, but not the number to optimise.

Misconceptions

Three shapes of question, two kinds of number

Jev has three question types. A noul is yes/no and returns the probability of yes. A choice picks one of 2–255 options and returns every option’s probability plus a confidence. A score rates against 2–10 ordered levels and returns a probability-weighted score.

Two numbers are easy to confuse: probability is about one outcome (“flaky: 0.84”); confidence is about how concentrated the whole distribution is. Good policies check both.

Questions are evaluated independently and in parallel, so adding one never changes another’s answer. Vercel’s guide adds three rules: ask atomic questions, describe each option instead of just naming it, and send only the relevant state.

A multiple-choice form, not an essay

BEFORE: You ask a colleague “what went wrong with this build?” and get three paragraphs back.

PAIN: Your script can’t branch on three paragraphs. You end up parsing prose, and the day the wording shifts, your if statement silently stops matching.

MAPPING: Jev is a form with the boxes already printed: tick one of four categories, and say how sure you are about each. Your code reads the ticks, never the prose.

Reading an answer

1. noulnoul: 0.14 — a 14% chance the answer is yes.
2. choicechoice: flaky, with probabilities for every option and one confidence.
3. scorescore: 0.55 on a 0–2 scale — between Low and Medium. Read probabilities if you need a level.
4. RoundingProbabilities can sum to 0.99. Never check for exactly 1.

Three questions about one failure

ASK Jev about failing_test:
  is_regression: noul
    "Did this diff probably cause it?"
  failure_type: choice of
    regression | flaky | environment | unknown
  severity: score over
    Low | Medium | High

READ p_reg     = is_regression.noul
READ type      = failure_type.choice
READ type_prob = failure_type.probabilities[type]
READ conf      = failure_type.confidence

Same model, two vocabularies: the Vercel AI SDK calls the yes/no type boolean and its answer probability. The native API says noul.

Misconceptions

A schema guarantees the shape, not the truth

TypeSafe markets Jev as a model that “can’t hallucinate”. That’s true in a narrow way: with four allowed categories, Jev can’t invent a fifth or return JSON your parser chokes on.

What typing can’t do is make the chosen category correct. Jev can pick flaky — a perfectly valid option — for a test that fails because the diff removed a lock. Vercel and Arize both make the same point: a schema-correct answer can still be wrong.

The form that’s always filled in neatly

BEFORE: Your clerk used to hand back forms with scribbles in the margins and boxes left empty. You redesigned the form so every box must be ticked.

PAIN: Now every form comes back perfectly filled in — and you stopped checking them. Neat handwriting started to feel like accuracy.

MAPPING: Jev is the neat form. Typing removes the scribbles, which is a real win. It doesn’t mean the right box was ticked, and your pipeline has to be designed for the day it isn’t.

What typing does and doesn’t fix

1. Fixes: malformed outputNo free text, no extra categories, nothing your parser can’t read.
2. Fixes: brittle parsingNo more if "flaky" in reply breaking when the wording changes.
3. Doesn’t fix: wrong picksA real regression can be labelled flaky.
4. So: design for wrong answersWhat happens next — rerun, escalate, or drop — decides whether a wrong pick is harmless.

How much should you trust a typed answer?

IF Jev’s answer is one signal among several:
  a wrong answer is weak evidence
  # other checks can outvote it

IF Jev’s answer decides whether Claude
   ever sees the failure:
  a wrong answer is invisible
  # nothing downstream can catch it

ALWAYS: treat low P(regression) as
  "leans away from regression",
  NEVER as "the code is fine"

Absence of a signal is not proof of correctness.

Misconceptions

Deterministic tools first, Jev second, policy decides

A coding agent works in a loop: edit, check, read the failure, fix, check again. The order of stages is the most important design decision here.

Compilers, linters, type checkers, test runners and scanners come first and stay authoritative wherever they give a certain answer. Jev works on the evidence they produce, answering what they can’t — “is this failure caused by the diff, or by a slow CI runner?” — and policy code, not Jev, chooses the action.

Lab results before the specialist’s opinion

BEFORE: A clinic sends every patient straight to a specialist’s intuition, skipping the blood test.

PAIN: The blood test is cheap and never wrong about what it measures. Skipping it trades a certain answer for a probable one and saves nothing.

MAPPING: Deterministic checks are the blood test: run them first and believe them. Jev is the quick read on what the tests can’t settle, and the policy is the clinic’s protocol for what to do with that read.

One failing test, eight stages

1. Build & checkClaude edits; tests, types and lint run. One test fails.
2. EvidenceOutput, the diff hunk, the run history.
3. Jev screensThree typed signals in milliseconds.
4. Policy routesCode picks: investigate, rerun, or review.
5. Claude fixesClaude investigates, fixes, and the loop re-verifies.

The screening policy

signals = ASK Jev about evidence
ON Jev error or timeout:
  RETURN review  # fail toward scrutiny

IF p_reg ≥ 0.75 AND conf ≥ 0.60
   AND type_prob ≥ 0.70:
  RETURN investigate-with-claude
IF p_reg ≤ 0.20 AND type in (flaky, environment)
   AND type_prob ≥ 0.75:
  RETURN rerun-and-collect-evidence
RETURN review-ambiguous-case

# thresholds are EXAMPLES -- tune on your data

This is the desktop module’s runnable screening layer, condensed. The key line is the first one: an outage routes to review, never to “fine”.

Misconceptions

Where Jev sits decides what its mistakes cost

“Put Jev in front of Claude” describes four different systems. A hard gate lets Claude see only what Jev flags. A soft prioritizer sends everything but lets Jev set the order. An independent signal is one input to wider policy. An uncertainty router automates clear cases and escalates unsure ones.

The hard gate is the most tempting and the most dangerous: its recall becomes a ceiling on the whole pipeline. 90% gate recall × 97% agent recall = 87.3% end to end — the agent didn’t get worse, it just never saw what the gate dropped.

The bouncer versus the host

BEFORE: A restaurant hires a bouncer to turn away anyone who “doesn’t look like a guest”, so the chef only cooks for real diners.

PAIN: Every guest the bouncer misjudges is gone for good — the chef never gets the chance to serve them, however good the chef is.

MAPPING: The hard gate is the bouncer. The uncertainty router is a host who seats the obvious guests fast and walks anyone unclear to the maître d’. Same judgement, but a mistake costs a short wait instead of a lost guest.

The lab’s 24 failures, two placements

1. The setup24 CI failures, 10 real regressions. One, F10, is a release-blocking race that looks flaky.
2. Hard gateClaude sees 9 failures. The pipeline catches 7 of 10: 70%. F10 is never seen.
3. Uncertainty routerUnsure cases escalate; confident “flaky” cases are rerun, and a failed rerun escalates. F10’s rerun fails, so Claude sees it.
4. ResultThe router catches 9 of 10 with 14 Claude calls instead of 24.

Optimise expected total cost

expected total cost =
    model calls
  + retries and review
  + false-positive work
  + expected cost of false negatives

# lab, illustrative costs:
#   missed HIGH regression   = $5,000
#   missed MEDIUM regression = $500
hard gate SAVES  $0.035 in API calls
hard gate ADDS   $5,500 in expected losses

A soft prioritizer only helps if someone drains the queue. A queue nobody drains is a gate with extra steps.

Misconceptions

Strong results, on one bounded task each

Ben Greenberg ran 306 decisions on one policy task. Jev, with a choice question plus four diagnostic nouls, scored 100.0%; Claude Sonnet 5 on high reasoning scored 99.0%. Median latency: 378 ms vs 3,554 ms. Cost per 10,000: about $2.27 vs $129.74.

Arize reports Jev matched Claude Opus 5 at 87% on hallucination detection after tuning thresholds, at about 1/300 of the cost. Both are judge-versus-judge results. Neither measured a coding loop with Jev in front of Claude.

Arize also names the biggest loss: Jev gives no explanation. Its suggested pattern is to run Jev on every trace, then send a sample of the failures through an LLM judge when you need the reasoning.

A sprint time isn’t a relay result

BEFORE: A runner posts the fastest 100 m time on the team, so the coach puts them on the relay.

PAIN: The relay is lost at the baton handover. The sprint time measured the runner, not the team they were placed in.

MAPPING: Benchmarks measure the judge. Your pipeline is the relay — what happens at each handover (drop, rerun, escalate) decides the result. You have to time the relay.

Reading the evidence correctly

1. AccuracyExcellent on a narrow, well-specified classification.
2. CalibrationGreenberg’s ECE: 0.037 for Jev vs 0.058 for Sonnet. Sonnet’s few errors sat at 0.9–1.0 confidence; Jev’s at 0.2–0.3.
3. Why that mattersA low-confidence error can be caught by a threshold and escalated. A 0.95-confidence error goes straight through.
4. The gapNo published result measures a Jev-gated coding loop.

What a benchmark lets you conclude

BENCHMARK SHOWS:
  price and latency are attractive
  Jev can match a big model on one task

BENCHMARK DOES NOT SHOW:
  accuracy on your question
  recall of your pipeline
  the effect of your thresholds

SO: re-establish accuracy per question,
    dataset, threshold and position

A surprising detail: rebuilding Greenberg’s policy by hand from the four nouls scored worst, 94.1%. Hand-built policy needs measuring too.

Misconceptions

Test the pipeline you’ll deploy, not the judge

Before changing the loop, freeze a set of historical cases with known outcomes and run complete variants against it: Claude-only, Jev-only, hard gate, soft priority, uncertainty router, and a cheaper Claude tier (such as Haiku 4.5) in the same position.

For each, capture end-to-end recall and precision, false negatives by severity, the share escalated to Claude, total tokens, median and p95 latency, retries and outages, review volume — and the estimated cost of a missed defect.

In the lab’s live run with real Claude verdicts, sending everything to Claude and the uncertainty router both caught 100% of regressions, while the hard gate stopped at 80%. Only a whole-system measurement shows that.

Crash-testing the car, not the airbag

BEFORE: A supplier proves its airbag deploys in 30 ms, every time, on a test bench.

PAIN: Fitted to a car, the airbag fires at the wrong moment because the crash sensor upstream lags. The bench test was true and irrelevant.

MAPPING: Jev’s accuracy is the bench test. End-to-end recall on your frozen cases is the crash test — the only one that tells you whether the whole car protects the passenger.

The measurement plan

1. Freeze casesHistorical failures with known final outcomes. Never send the answer key to a model.
2. Run every variantSame cases, same signals — only the placement changes.
3. Score end to endRecall, precision, misses by severity, calls, tokens, latency.
4. Price the missesAdd the expected cost of false negatives, not just the API bill.

Comparing architectures

FOR each architecture IN
    [claude-only, hard-gate, soft, router]:
  FOR each frozen case:
    signals = ASK Jev(evidence)
    route   = architecture(signals)
    IF route reaches Claude:
      verdict = Claude(evidence)
  recall    = caught / real_regressions
  precision = caught / flagged
  REPORT recall, precision, calls,
         misses BY severity, total cost

In the lab, the gate predicted 0.80 × 0.90 = 0.72 and observed 0.70. They differ because errors correlate — which is why you measure instead of just multiplying.

Misconceptions

Four words decide the fit

Jev is most compelling when a decision is narrow, repeated, measurable and reversible: classifying a failure, prioritising a queue, routing clear cases while escalating the rest, scoring against a rubric, or running an eval on every trace.

It’s a weak fit for repository-wide causal reasoning, arithmetic, multi-step investigation, weighing contradictory evidence, explaining itself, or writing a fix. Those remain jobs for deterministic tools, Claude, or a person.

Reversible is the word teams skip. Rerunning a test Jev wrongly called flaky costs a few CI minutes; auto-merging a change Jev wrongly scored low-risk ships the regression before anyone looks. The same accuracy can be fine for one and unacceptable for the other.

The sorting machine at the post office

BEFORE: A post office buys a machine that reads postcodes and sorts letters into bins at thousands per minute.

PAIN: Someone suggests it also decide which letters are suspicious and should be destroyed. Sorting is fast and reversible; destroying isn’t.

MAPPING: Jev is the sorter. Let it sort — a mis-sorted letter just takes a detour. Don’t hand it decisions that can’t be undone.

Checking a candidate decision

1. NarrowCould a person answer it in seconds from the evidence?
2. RepeatedDoes it happen often enough for speed and cost to matter?
3. MeasurableDo you have known outcomes to check it against?
4. ReversibleIf Jev is wrong, can the mistake be undone cheaply?

Good fit or poor fit?

GOOD FIT:
  classify: regression | flaky | env
  prioritise a large failure queue
  route clear cases, escalate unsure
  score evidence against a rubric
  eval every trace, not a sample

POOR FIT:
  repo-wide causal reasoning
  arithmetic, multi-step investigation
  explaining why, or writing a fix
  any irreversible action

Rerunning a test Jev called flaky is reversible. Auto-merging a change Jev scored low-risk is not.

Misconceptions

Run it beside the loop before it runs the loop

Don’t start with a gate. Start in shadow mode: Jev screens every failure, its decisions are logged, and no route changes. Later you join the log with what actually happened.

The disagreement data then tells you, region by region, where Jev has earned authority: shadow → study disagreements → soft priority → uncertainty routing → automate only a narrow region, with a rollback at every rung.

It’s less dramatic than dropping a cheap judge into every step of the loop. It’s also the only way to find out whether Jev saves money in your system, rather than merely having a low list price.

The trainee pilot in the right-hand seat

BEFORE: A new pilot joins the airline. On day one they could take the controls of a full flight.

PAIN: Nobody would know how they handle a crosswind until the plane is already in one.

MAPPING: Airlines put them in the simulator and then the right-hand seat, logging every call they’d have made. Authority comes in steps, each earned with evidence. Shadow mode is the right-hand seat.

The promotion ladder

1. ShadowLog Jev’s decisions; change nothing.
2. Study disagreementsExplain every real regression Jev would have sent away.
3. Soft priorityJev orders the queue; nothing is suppressed.
4. Uncertainty routingRerun confident cases, escalate unsure ones and failed reruns.
5. Narrow automationOnly where misses are rare, cheap and reversible.

Turning a shadow log into a decision

FOR each region IN [investigate, review, rerun]:
  cases = shadow_log WHERE jev_action = region
  regs  = cases WHERE outcome = regression

  IF region = rerun AND regs has HIGH:
    ALLOW rerun-then-escalate
    NEVER suppress
  ELIF region = review:
    keep escalating
  ELIF region = investigate:
    prioritise for Claude

In the lab, the rerun region hid one high-severity regression (F10) in 12 cases — enough to rule out suppression.

Misconceptions

Advise the agent; never gate it

A Claude Code PostToolUse hook runs after each tool call. The lab’s hook watches for failing test runs, asks Jev about the failure, and adds one line of additionalContext for Claude, phrased as a hint.

It deliberately never blocks (decision: block), never rewrites the tool output, and never fails shut — if Jev is down, it prints nothing and exits 0. That’s the whole design: a wrong hint costs one sentence.

Gates that should block belong to deterministic checks — a linter, a secret scanner, a failing test — which are certain about what they measure. A probabilistic judge only ever adds a hint.

The sticky note on the chart

BEFORE: A nurse could lock a patient’s file until a triage flag is resolved.

PAIN: When the flag is wrong, the doctor can’t see the file at all — the flag has become a gate.

MAPPING: The advisory hook is a sticky note on the chart: “looks flaky (p=0.84)”. The doctor reads it, then reads the chart. In the lab the note is wrong about F10, and Claude still finds the removed lock.

What the hook does

1. FilterOnly Bash calls that ran tests, and only when something failed.
2. ScreenSend the failure and the current diff to Jev.
3. AdviseReturn additionalContext: Jev’s read, labelled “NOT a verdict”.
4. Fail openAny Jev error: print nothing to stdout, log to stderr, exit 0.

The advisory hook

ON PostToolUse(tool = Bash):
  IF command is not a test run: EXIT 0
  failure = first FAILED test IN tool_response
  IF none: EXIT 0

  TRY:
    signals = ASK Jev(failure, git diff)
  ON error: LOG to stderr; EXIT 0

  PRINT additionalContext:
    "Jev advisory (NOT a verdict): looks flaky..."
  # never: decision=block, updatedToolOutput

Three silent hook failures: reading tool_result instead of tool_response, echoing the payload to stdout, and a flat {matcher, command} in settings.

Misconceptions

Test the core insight

5 questions on the module’s sharpest insights. Tap to reveal the answer.