#Take Away the Pen: Reviewing 94 Contracts for 4 Cents With an AI That Can’t Write

Jev, lawyers’ labels, and the difference between being right and knowing when you’re right. Nothing here is legal advice.

#Table of Contents

1. A Model That Can’t Write
2. Meet Jev
3. How You Call It
4. The Mailroom
5. Two Scoreboards
6. The Experiment
7. Results
8. What Surprised Me
9. The Verdict
10. Try It Yourself
11. Caveats

#A Model That Can’t Write

Contract review is a quiz, not an essay. Is there a non-compete? A cap on liability? Can either side walk away? A lawyer runs through dozens of these, and every answer is yes or no.

We hand that quiz to LLMs anyway. They read 30 pages, write their answers out word by word, and we parse the text back into booleans.

Jev does something different. It cannot write at all. You give it the contract and the questions, and it returns the odds for every answer in a single pass.

So I tested it on 94 real contracts that lawyers had already labeled, with 37 questions each. Jev took 0.22 seconds per contract. The whole run cost under 4 cents.

Then Claude Sonnet walked in, and the story got more interesting.

#Meet Jev

TypeSafe released Jev in September 2026 and calls it a System One model. The name comes from Kahneman’s Thinking, Fast and Slow: System 1 is fast intuition, System 2 is slow deliberation. LLMs think by writing, which makes them System 2. Jev is built for System 1: quick, structured judgments your code can act on. “Jev” nods to the Jevons paradox: make something cheap enough, and people use far more of it.

Three things set it apart:

1. No token-by-token generation. Every answer comes out of one parallel pass. That is the cost I wrote about in my EAGLE post, and Jev skips it entirely.
2. The whole distribution, not one sample. An LLM samples a single answer (more on that here). Jev gives you the probability of every option.
3. Trained to be honest. RLHF rewards what people like. TypeSafe trains Jev with RLCD, Reinforcement Learning for Calibrated Decisions, which rewards probabilities that match reality. When Jev says 0.9, it should be right 9 times out of 10.

One caveat up front: TypeSafe’s “zero hallucinations” means no invalid answers. The output always fits your schema. It does not mean the answer is right.

#How You Call It

You send a state (the document) and some questions. There are three question types: noul (yes or no), choice (pick an option) and score (a point on a scale). Here is the quickstart, through OpenRouter, which hosts Jev with no waitlist:

resp = httpx.post(
    "https://openrouter.ai/api/v1/systemone",
    headers={"Authorization": f"Bearer {os.environ['OPENROUTER_API_KEY']}"},
    json={
        "model": "typesafe/jev-1.13",
        "state": "I was charged twice for my annual plan. Refund the duplicate today, please.",
        "questions": {
            "refund": {"type": "noul", "instructions": "The customer is asking for money back."},
            "intent": {
                "type": "choice",
                "instructions": "What does the customer want?",
                "criteria": {"billing": "A billing problem", "technical": "Something is broken", "other": "Else"},
            },
            "frustration": {"type": "score", "instructions": "How frustrated is the customer?",
                            "criteria": ["calm", "annoyed", "furious"]},
        },
    },
    timeout=30,
).json()

answers = resp["answers"]
# The if-statement is where calibration earns its keep.
if answers["refund"]["noul"] >= 0.87:
    print("route to refunds automatically")
else:
    print("ask a human")

No parsing. No JSON repair. No retry loop. TypeSafe calls this a “smart if-statement,” and the last four lines show why.

The LLM version of the same thing (llm_structured_equivalent.py) needs a schema, a prompt that begs for JSON, fence stripping and a retry loop. And its if runs on a confidence number the model chose to write. Whether that number means anything is one of the things we’ll measure.

Now scale it up: one Jev call, one real sponsorship agreement from the benchmark, 37 questions. This is the output of paddles review, a small tool I built for this post:

Review card: cuad_sponsorship_agreement.txt
threshold 0.87 | 37 questions in one call | 269 ms | $0.00014

[HUMAN ] third_party_beneficiary            yes                      P(yes) 0.59
[HUMAN ] insurance                          no                       P(yes) 0.39
[HUMAN ] license_grant                      yes                      P(yes) 0.62
[HUMAN ] post_termination_services          yes                      P(yes) 0.63
[HUMAN ] change_of_control                  no                       P(yes) 0.35
[HUMAN ] uncapped_liability                 yes                      P(yes) 0.69
[HUMAN ] minimum_commitment                 no                       P(yes) 0.18
[  ok  ] covenant_not_to_sue                no                       P(yes) 0.10
...
29 answered automatically, 7 sent to a human.

The lawyers found two clauses in this contract: anti-assignment and insurance. Jev approved anti-assignment on its own at 0.99. It wasn’t sure about insurance (0.39), so it flagged it for a human instead of quietly missing it. Every one of the 29 automatic answers matched the lawyers.

That card is the whole idea of this post.

#The Mailroom

Picture a mailroom that needs quick answers about every document that comes in. You try a few hires:

• The brilliant novelist (base LLM): writes a lovely paragraph and sometimes answers the question.
• The novelist after finishing school (instruction tuning): polite and on topic, but still writes one word at a time, and finishing school rewarded sounding sure.
• The novelist with a pre-printed form (structured outputs): fills every box correctly, letter by letter. The form shows what they wrote, never how sure they were.
• The novelist with a work-order pad (tool calling): writes requests instead of answers. On a bad day, a work order for a queue that doesn’t exist.
• The rulebook clerk (classic ML): counts words, fast, and gets confused by “the parties shall NOT be restricted from competing.”
• The flashcard apprentice (SetFit): learns from a handful of examples but reads one sentence at a time. Hand them a contract and they’re lost.
• The sorting machine (fine-tuned encoder): fast and accurate for fixed chutes. A new question means rebuilding the machine.
• The panel of honest bookmakers (Jev): you hand over the whole contract and today’s questionnaire, and every judge raises a paddle at once. They’re paid like bookmakers, so their odds have to be honest. Low odds go to the lawyer’s desk.

Instruction tuning taught the model manners, tool calling gave it a work-order pad, structured outputs gave it a form, and Jev took away the pen and handed it betting paddles.

Classic ML gave the clerk a rulebook, SetFit gave the apprentice flashcards, fine-tuning built a sorting machine with fixed chutes, and Jev hired a panel of bookmakers you can hand a brand new questionnaire every morning.

Hence the project name: PADDLES.

#Two Scoreboards

There are two ways to grade a reviewer, and they tell very different stories.

Scoreboard 1: accuracy. Count the right answers. This is the number everyone quotes.

Scoreboard 2: safe automation. This is the one a business runs on. Sort every answer from most to least confident. Accept from the top automatically, and stop right before the accepted pile drops below your target, say 99% correct. Everything below goes to a lawyer. The share you accepted is your coverage at 99%.

Scoreboard 2 only works if confidence tells right answers from wrong ones. That property is calibration. I measure it with ECE, the average gap between stated confidence and actual accuracy (lower is better), and with reliability diagrams. Neural networks are famously overconfident (Guo et al., 2017), and a threshold on an overconfident model is a threshold on noise. (Want the probability background? See my Bayesics notes.)

In contract review, the costly mistake is the confident miss: “no non-compete, 99% sure,” nobody checks, and the clause was there.

#The Experiment

The contracts. CUAD, from The Atticus Project (Hendrycks et al., 2021, CC BY 4.0): 510 commercial contracts from SEC filings, labeled by lawyers. I used the official test split. 94 of its 102 contracts fit in Jev’s 32,000-token limit, so those are the test set.

The questions. 35 clause checks (non-compete, cap on liability, anti-assignment and more), written from CUAD’s own lawyer definitions, plus governing law and contract type. Every model got the exact same wording. The right answer for a clause is simple: did the lawyers highlight it? Free-text fields like party names and dates are out, because Jev can’t write strings. Extraction stays with an LLM.

Why lawyers, not LLMs. TypeSafe’s own evals compare Jev with the average answer of two frontier LLMs, which measures agreement with big models. They say so openly. I wanted a different question answered: does Jev agree with lawyers?

The contestants.

The trained models learned from 347 CUAD contracts. Jev and Claude saw none, so the deck is stacked against them. SetFit and other sentence-embedding classifiers sat this one out: they’re built for short text, not 30-page contracts.

One scoring note. CUAD’s contract-type labels come from folder names, not the lawyers, and they turned out to be noisy (more below). So the headline numbers use the 36 questions with lawyer labels.

Related work. The awesome-jev list collects plenty of good independent testing, including calibration benchmarks, Jev vs Haiku on phishing, a pre-registered test, a 16,000-call study, JevBench and a legal-document classifier. What’s new here is the combination: long real documents, 37 questions per call, lawyer labels, a trained model built on the same labels, and an open replica in the same harness.

#Results

#How much can each model automate?

Two rows carry most of the story.

Haiku’s 100% trap. Haiku beats Jev on accuracy, yet it can automate nothing at 99%. It claimed 100% confidence on 2,311 answers, 68.3% of everything it said, and only 96.3% of those were right. A threshold can’t split a pile where every answer claims the same certainty. You take all of it at 96.3%, or none of it. That’s the square on the chart above.

Jev’s 80 answers at 100% confidence were all correct. Its mistakes sit at odds like 0.6, exactly where a threshold catches them.

Sonnet is the best reviewer here. It wins accuracy, ranking and automation, and its calibration (ECE 0.029) matches Jev’s. I expected the big LLM to be overconfident. It wasn’t, at least on this task. One caution: 94 contracts is a small sample for a 99% target. Sonnet’s coverage at 99% could plausibly be anywhere from 4.8% to 50.7%, and Jev’s from 12.3% to 38.1%. At 95%, where the numbers are steadier, Sonnet leads clearly.

#Many questions, one call

This is Jev’s signature claim, and it holds up completely:

• 1 question: 169 ms. 37 questions: 179 ms. Same call, nearly the same time.
• Together or one at a time, the answers match 99.6% of the time. That’s the same as Jev’s agreement with itself on an identical rerun. Batching costs it nothing.
• Claude Haiku changed 7.4% of its answers depending on whether it saw one question or all of them. Asking one at a time also cost it $0.21937 and 55.22 s per contract. Jev: $0.00805 and 7.09 s.

#Trained models: strong on common clauses, blind to rare ones

The trick to training ModernBERT well: CUAD marks the exact text of every clause, so I trained on 512-token chunks labeled by those highlights, not on whole contracts. That turned 347 contracts into 10,727 training examples. It trained in 6 minutes on one GPU.

Each cell: AUROC / F1.

On rare clauses, the trained models almost never say yes (F1 of 0.00 and 0.04). The zero-shot models still find them. On common clauses, ModernBERT nearly catches up. And temperature scaling on 61 validation contracts cut its calibration error from 0.044 to 0.012 and its confident missed clauses from 119 to 46. You can bolt calibration on afterwards, if you have labels.

#Is it the interface or the training?

local-jev copies Jev’s API exactly, on an open Qwen model. It ran out of GPU memory on the longest contracts, so this comparison uses the 85 it could read:

The replica is a bit more accurate and four times worse calibrated. The interface is easy to copy. Honest odds are not.

#Long contracts

Jev’s accuracy holds on long contracts, but its calibration slips: ECE more than doubles, from 0.030 at best to 0.067 on the longest quarter. TypeSafe’s own notes warn about this: unrelated text in the input hurts. Sonnet stayed steady. With Jev, send only the sections your questions need.

#Cost and speed

Jev: $0.00039 and 0.22 s per contract. Haiku: $0.01514 and 5.17 s. Sonnet: $0.05986 and 23.11 s. All three were called through OpenRouter from the US East coast, so latencies include a gateway hop.

#What Surprised Me

The humble word counter is a real rival. TF-IDF with logistic regression trained in 20 seconds on my laptop. It automates 29.6% at 99% accuracy and has the fewest confident missed clauses of anyone (17). Many clauses come with signature phrases (“right of first refusal,” “audit the books”), and logistic regression gives fairly honest probabilities by design. It breaks on rare clauses and new questions.

Some of Jev’s “mistakes” were the labels. All five of its wrong answers at 99% confidence or higher were on contract type, and every one was a labeling problem. An insurance “Endorsement” (a policy amendment) was filed under celebrity endorsements. A “Distribution and Development Agreement” was filed under development. That’s why the headline numbers stick to the lawyers’ clause labels.

The obvious cascade saves nothing. My plan: Jev answers first, and only the questions it’s unsure about go to Sonnet. But every one of the 94 contracts had at least one unsure question, so Sonnet still had to read all 94. An LLM pays to read the document, not per question. The cascade cost $5.66; Sonnet alone cost $5.63. On long documents, send Jev’s unsure answers to a human. A cascade to an LLM only pays off on short documents, or when Jev is confident about the whole thing.

Use the labels you’re given. The gap between a useless fine-tune and a strong one wasn’t the optimizer. It was training on the lawyers’ highlighted spans instead of whole-contract labels. If your dataset has spans, use them.

#The Verdict

Jev isn’t the best contract reviewer. It’s the cheapest one you can trust with a threshold.

It automated 25.7% of decisions at 99% accuracy, about 57% of what Sonnet managed, for roughly 1/150th of the cost at about 100 times the speed. Haiku, the “cheap” LLM, cost 39 times more than Jev and couldn’t safely automate anything at 99%.

When to use what:

• Jev: your questions change often, you have no labels, you need many questions per document, or the decision sits in a request path. Send low-confidence answers to a human.
• A small trained model: your questions are stable, you have labels and volume, or the data can’t leave your network. In finance, lending, legal and healthcare, that last point can outweigh any benchmark. Temperature-scale it.
• A strong LLM: a human will read the output, you need free text (dates, names, amounts), or accuracy matters more than cost.
• All three together: Jev decides and flags, the LLM extracts and writes, and a human reviews whatever falls below the line.

This is the idea from The Leash and the Wolf: agents need a leash, and Jev makes the leash smart without making it slow. LangChain’s Jev harness post shows two versions: ModelRouterMiddleware, where Jev picks a cheap or expensive model per request, and AutoModeMiddleware, where Jev blocks risky tool calls. It’s the same instinct as Hexr, where a cheap specialist beat an expensive generalist, and as dynamic tool selection.

It also connects to Virtual DA, my legal AI that refuses to hallucinate through retrieval and a self-harness. Jev can’t return an invalid answer by construction. Those are different guarantees, neither one means “correct,” and both still need calibration.

#Try It Yourself

Everything is at gitlab.com/sugix/paddle. The piece worth stealing is paddles review: point it at any text file and a YAML list of questions, and it asks Jev everything in one call.

uv sync
export OPENROUTER_API_KEY=...
uv run paddles review my_contract.txt                       # the 36 CUAD clause checks
uv run paddles review ticket.txt -q my_questions.yaml --target 0.99

--target picks a threshold measured in this benchmark: 0.87 for 95% accuracy, 0.97 for 99%, on contracts like CUAD’s. For your own documents, label 50 to 100 examples and measure your own.

Want to find “smart if” candidates in your code? Look for any LLM call whose output immediately becomes a bool, an enum or a number. Those are decisions dressed up as text.

uv run paddles report rebuilds every table and chart from the saved results for free. The whole project cost $13.50: $11.85 in API calls and $1.65 of GPU time.

#Caveats

• One model version, one date: typesafe/jev-1.13 via OpenRouter, September 23, 2026. Jev isn’t deterministic; identical requests can differ by a few hundredths.
• Claude’s confidence is self-reported. The API doesn’t expose token probabilities, so it’s a number Claude wrote.
• 94 contracts is a small sample for 99% targets. Read those numbers as ranges.
• Public commercial contracts only. Other document types may behave differently.
• Excluded: the 8 longest test contracts (over Jev’s limit), CUAD’s free-text fields, and 9 more contracts in the replica comparison.
• TypeSafe’s claims are theirs; these results are mine, and they measure agreement with lawyers on one task.
• Not legal advice. No model here should review a contract without a human in the loop.

Some parts of an AI product need language. Others just need an outcome. For those, stop asking a novelist to fill in a form. Take away the pen, hand over the paddles, and read the odds.