Jev by TypeSafe AI: What We Found Testing It on Real Work

Jev read 275 internal documents in 30 seconds. That's 2.6 million tokens of reading, and the bill was 11 cents.

Then we checked its work. That's the part the launch posts skip.

We got access to Jev three days ago and ran it on two real jobs: a live app we're building, and 275 documents from our own email-marketing knowledge base. Both times it was fast, cheap, and never once errored. Both times, we still can't tell you whether it was right. This post is the real numbers, the parts that surprised us, and the skill we built so you can measure it on your own data.

What Jev actually is

Jev is an AI model that never writes a word. It reads language the way a chatbot does, then hands back a decision and how sure it is about that decision. TypeSafe calls it a "System One" model: the fast, instinctive kind of thinking, not the slow, step-by-step kind. It's trained with a method TypeSafe calls Reinforcement Learning for Calibrated Decisions, and it answers in one of three forms:

• Noul: a true or false question, answered with the probability that the statement is true.
• Choice: one pick from a list you define.
• Score: a rating on a scale you define.

"Noul" is TypeSafe's word for a true or false question, and yes, that's how it's spelled in the API. The part that matters most is the confidence. It means your code can decide when to trust an answer and when to hand the case to a human or a bigger model.

Because Jev never generates text, there's no JSON to coax out of a chatbot and no output to pay for. TypeSafe lists Jev at $0.042 per million input tokens, with output free.

TypeSafe's numbers, and ours

Here's what TypeSafe's own workflow evals show. Jev scored 67.8 percent at $0.0004 and 0.4 seconds a case. Claude Sonnet 5 scored the same 67.8 percent at $0.1174 and 78.1 seconds. Opus 5 scored higher, 73.1 percent, at $0.1761 and 37.8 seconds. That's roughly where the headline of 200x faster and 445x cheaper comes from: Jev's 0.4 seconds against Sonnet 5's 78.1, and its $0.0004 against Opus 5's $0.1761.

Read the fine print, because it's the same trap we hit later in this post. TypeSafe's own page says the reference answers in those evals come from two large reasoning models, not from people. So those percentages measure how often each model agrees with two other AIs. That's a useful comparison, and it isn't accuracy. They're TypeSafe's figures from TypeSafe's testing. We haven't verified them, and a few days of use can't. TypeSafe has also said early pricing may be subsidized, so check the live rate before you budget.

What we can do is tell you what we measured on our own work. The speed and the cost held up. The accuracy is a question mark.

What everyone else is building

Jev is days old and people are already piling in. A public showcase at madewithjev.com lists more than 300 builds as we write this, including well over 100 tools and repos, and the count grows daily. The pattern across them is consistent: point Jev at a pile of things to sort, or a stream of small choices to make.

Sorting a pile. One builder broke down 724 competitor ads from 37 brands in about 40 seconds for about 9 cents. Another sorted 1,018 research papers by topic for 8 cents. Another sorted 500 emails for 3.5 cents.

Making choices inside an agent. Many of the tools pair a chat model that plans with Jev picking each next click, which is how browser agents get to roughly 100 to 300 milliseconds a step. Others put Jev on every tool call as a gate, use it as a router that picks the cheapest model for a task, or have it trim long tool output before a coding agent reads it.

Real time. Someone has Jev playing Doom at about 10 queries a second, for roughly $7 an hour. Another has it playing Tetris at about 0.3 seconds a move. Speed is the point of those demos.

Two cautions before you copy any of it. These are builders' own posts, and we haven't reproduced a single one. And most of the headline numbers in that gallery are speed or cost, the easy half to measure, which is the same half of the scoreboard we ended up with after three days. A few tools in the showcase go after the other half, for example by turning human labels into calibrated classifiers. Those are worth a look.

Test one: a live app, in shadow mode

We're building a real-time assistant that listens to a live conversation and decides, moment by moment, whether to surface a saved note. It's a good fit for Jev because every decision is small and closed-ended. Does any retrieved note apply right now? Which one? Is this even a good moment to interrupt? Nobody needs a paragraph for that. They need a fast yes, no, or pick.

For now, Jev runs in shadow mode. It sees everything the real system sees and makes its calls, but nothing acts on them. Its verdicts go into a log, next to what the real system actually did.

That's the only sane way to meet a model you've known for three days. You get its speed, cost, and reliability under real load without letting it near the wheel.

We have one real call to show for it, on September 17. Jev handled 130 decisions with zero failures. Median latency was 173 milliseconds, and 95 percent of calls finished within about 400 milliseconds. The whole call, 150,854 input tokens, cost about six-tenths of a cent. That latency sits inside the 70 to 500 millisecond range TypeSafe reports, and the cost is TypeSafe's published rate applied to our tokens.

The plumbing is solid. The judgment is a blank, and here's why. On every one of those 130 rows, zero notes were retrieved, because of a separate connection problem on our side. So the two questions the product exists to answer, does any note apply and which one is best, never had a note to judge.

Test two: 275 internal documents

Next we pointed Jev at something with an answer key. We keep an internal knowledge base of email-marketing notes: 275 documents covering summaries of what we've studied, core concepts, and the tools and companies in the space. Each one was already tagged four ways: what kind of document it is, which stage of the customer journey it belongs to, who it's for (ecommerce, service business, or both), and whether it carries a compliance warning.

We hid the tags, sent Jev each document (the title plus the first 60,000 characters), and asked four narrow questions per document in a single call.

One catch, and it matters: those existing tags were written by an AI when the knowledge base was built. So this compares Jev with another model's work, not with a human's. Everything below is agreement, not accuracy.

The speed was the headline. All 275 documents took 30 seconds of wall clock. Median time was 357 milliseconds per document and 715 at p95. The total was 2,624,189 input tokens for $0.11. All 275 came back answered, with zero errors.

Then the checking, one question at a time.

Content type worked the way you'd hope. Jev agreed with our existing tags on 78 percent of documents overall, and on 96 percent of the answers it was confident about. The top confidence bin agreed 96 times out of 97. That's what a confidence score is supposed to do.

Lifecycle stage was more interesting. Agreement was 76 percent. We had tagged 229 documents "cross-lifecycle," our catch-all for advice that applies at every stage. Jev kept 171 there and moved 58 to a specific stage. That suggests we over-use that tag, but a human has to make the call, and we haven't yet.

Audience turned out to be a tagging habit, not a judgment. Agreement was 62.5 percent. Jev never once chose "both," a tag we'd used on 97 documents, and it was extremely sure of itself, with 209 documents at 95 percent confidence or higher. Very confident and mostly disagreeing is a tell, so we went looking for the reason. Our first two guesses, document length and the wording of the question, were both wrong.

The real answer was in our own tags. When we split the results by document type, our tag followed the type of document, not what was in it. Every one of the 155 source summaries was tagged "ecommerce," and every one of the 75 concept documents was tagged "both." Jev agreed on 99 percent of the summaries and none of the concepts. Nearly every document has a section for service businesses, so no reading of the content can reproduce that split. That's a finding about our tags, not a mistake by Jev.

The compliance flag changed when we changed the question. The first version looked fine at 87.5 percent agreement, until we saw that Jev said "true" on 98 percent of documents. So we rewrote it to be stricter (performance claims, promised results, testimonials, or advice on health, money or law, and not ordinary email consent) and reran it on the 155 source summaries, where our own tags actually vary. The rerun took 21 seconds and cost about four cents. The answers moved from 152 true, 3 uncertain and 0 false to 109 true, 37 uncertain and 9 false.

Agreement with our tag on the documents where Jev made a call was 95.8 percent, but discount that: always answering "true" would have scored 88.1 percent on the same documents. The better signal is ranking. Against our tags, Jev's probabilities ranked the "true" documents above the "false" ones with a score of 0.93, up only slightly from 0.91 with the first wording. So Jev could already tell the documents apart. The first wording just put nearly everything above our confidence line.

That's the lesson we didn't expect: a confidence threshold doesn't carry over when you reword the question. Set it per question, on labeled data.

One more lesson from the same run: confidence is not a substitute for labels. On lifecycle stage, answers in the 0.85 to 0.95 confidence bin agreed less often than the 0.70 to 0.85 bin.

What we can't tell you

Neither test produced an accuracy number, and we'd rather say so than dress up agreement as proof.

On the documents, we have agreement with another AI's tags and no human labels, and even our stricter compliance question is scored against a tag whose exact meaning nobody wrote down. On the app, we did run the comparison we'd skipped, and it joined cleanly, all 130 decisions matched up. Then it couldn't answer the question either. The real system's own answer was the same word on every row that had one, 38 out of 38. Agreeing with a constant only measures how often Jev says the same word. And on that one question, only 8 percent of Jev's answers were confident enough to count.

So after three days, here's the honest scoreboard. Speed: measured. Cost: measured. Reliability: measured. Whether Jev's decisions are right: still open. Fast and cheap is also a description of a coin flip.

The only fix is a human labeling a sample by hand, blind, without seeing Jev's answer. That's our next step.

What the build taught us

Getting Jev into the app took three designs in one day. The mistakes are the useful part.

The plan was wrong about the shape. Our integration plan assumed Jev would slot into the same text-in, text-out interface as our other models. It doesn't. Jev returns structured decisions, so it got its own interface.

A same-day review found six defects. We had the wrong model id, jev where the API wants jev-latest. We had the wrong answer keys, yes and no where the docs say true and false, so every real call would have been rejected. There was a missing audit-log entry, a request not tied to its session, and a self-test that never drove a real call and looked green anyway. And we were sending the wrong data as the transcript, which would have contradicted our own privacy disclosure. All six were caught before the first live call, and none were reasoning problems. They were look-it-up-in-the-docs problems, and we hadn't looked.

The cookbooks pushed us somewhere else. Then we read all 18 of TypeSafe's cookbooks. We had folded everything into one big holistic question, and the cookbooks steer you toward several narrow questions asked together in a single call. TypeSafe's parallel-questions cookbook puts a number on it: batching 13 questions over one long document came out 12.2x cheaper and 10x faster than 13 separate calls, with no change in the answers. Most of that saving comes from sending the long input once, so your mileage depends on how much text rides along. We also stopped sending note cards as titles only, which gave Jev nothing to judge, and rebuilt the whole thing as a nine-question battery.

And the key stays off the device. No user should be able to pull the Jev key out of a downloaded app. It should have been our default on day one. We now route through a hosted relay with access codes instead of shipping a key.

We didn't track build hours or tokens for any of this, so we can't put a number on what the rework cost.

The skill we built, free to download

Every lesson above went into a skill for Claude Code. jev-decisions runs repeated judgments through Jev the careful way:

• Shadow mode. Jev's verdict is logged next to your real decision, and nothing acts on it.
• Narrow questions, one call, with a confidence tier on every answer: auto, review, or abstain.
• A golden set. A blind, human-labeled sample with a frozen holdout, so accuracy is measured and reported separately from agreement.
• A comparator that warns you when your existing decision never varies, which is the trap we fell into on the app.
• Hard stops. It won't run a paid batch until you approve the exact printed run and a cost cap, and it never acts on a verdict.

It's plain Python 3.10 or newer with nothing to install, and you bring your own key. It's MIT licensed, so you're free to use it, change it, and share it, including in client work. Just keep the license file with it.

Download jev-decisions.zip (49 KB)

To set it up, unzip the folder into ~/.claude/skills/, set your key in your environment, and run python3 execution/selftest.py. Its 29 offline checks need no key, no network, and no money. We ran them ourselves from a fresh copy of the zip.

Two honest notes. The direct TypeSafe route is verified against the live API. The default OpenRouter route is not live-tested yet, and the README says so, so run one probe before you rely on it. And the confidence thresholds it ships are starting guesses, not measured values. Tune them on your own data.

What we'd tell you to do if you're starting

1. Start in shadow mode. Let Jev decide alongside your real system before it decides anything.
2. Log its verdict next to what your system actually did, and check that your existing decision actually varies. If it's constant, the comparison can't tell you anything.
3. Read the cookbooks before you design, not after your second rewrite. Skim the community showcase for a build like yours too, and treat its numbers as ideas, not benchmarks.
4. Ask narrow questions together in one call, and send Jev real content, not titles.
5. Treat confidence as tiers, and count coverage next to precision. High, act. Middle, review. Low, don't act. Set the lines per question on labeled data, because they don't carry over when you reword.
6. Build a blind golden set of about 100 real cases with known-correct answers, and score Jev on accuracy and cost before you trust it.
7. When Jev is very confident and often disagrees, suspect your tags or your question wording before you suspect the model.
8. Keep the key on the server. Never ship it inside a downloadable app.

Jev fits high-volume decisions with a closed set of answers: routing, triage, tagging, filtering, and deciding whether something is worth a human's attention. It doesn't fit anything that needs words.

What happens next

Next we decide what "audience" should even mean for our documents, label a sample by hand, and get a real accuracy number. Then we do the same for the app. If Jev holds up, we'll say so. If it doesn't, we'll say that too.

Fast and cheap got Jev through the door. Right is what gets it the job.