Jev: an AI model for classifying text

A support ticket may need a team, a priority, and a decision about whether to escalate. None of those outputs needs a paragraph. Jev, a model from TypeSafe AI, can classify the ticket, rate its urgency, and estimate whether it needs escalation. Your application defines the categories and questions; Jev returns answers and probabilities.

The useful question is whether those decisions are accurate enough for the next step. Jev makes the response easier to consume, but a valid team name can still send a ticket to the wrong team. Its low price is useful only when the resulting mistakes and fallbacks are acceptable.

TL;DR Jev classifies and scores text and estimates probabilities for yes-or-no questions. It provides probabilities, but its separate confidence field is a summary of the distribution, not a probability of correctness. An independent 200-item test found Jev cheaper and faster than the compared models, with lower overall accuracy. Use labeled examples to choose thresholds and keep a fallback for uncertain or consequential decisions.

How a request becomes a decision

TypeSafe's System One documentation describes two inputs: a state, containing the material being evaluated, and questions defining the judgments to make. The state can be a string, a JSON object, or an array of text.

For a ticket saying a payment integration has failed for three days, the state might include the message and relevant account facts. One question selects the responsible team. Another judges urgency. A third checks whether the message requests a refund. Application code then combines those answers with permissions and business rules.

Jev evaluates questions in parallel against that shared state. This avoids generating one answer as context for the next. It does not make the answers statistically unrelated: questions about urgency and frustration may draw on the same words in the ticket.

An LLM can also answer several questions and use a structured-output API to constrain its response. It is misleading to assume every LLM classifier needs to repair JSON extracted from prose. The comparison should use the actual APIs, prompts, and validation available for each model.

The three question types

The TypeSafe quick start demonstrates three output shapes:

• Choice: select a named option, such as billing, technical, or sales, and return a probability for each option.
• Score: judge the state against ordered levels, such as calm, frustrated, and very frustrated, returning a numeric position and a distribution over those levels.
• Noul: evaluate a yes-or-no proposition and return its probability in the noul field.

Here is a small request using the documented Python SDK. It requires typesafe-sdk and a TYPESAFE_API_KEY in the environment:

from typesafe_sdk import Noul, TypeSafeClient
 
with TypeSafeClient() as client:
    response = client.system_one(
        model="jev-1.13.0",
        state="The integration has failed for three days. Please help ASAP.",
        questions={
            "urgent": Noul(
                instructions="Does the message express urgency or time sensitivity?"
            )
        },
    )
 
print(response.answers["urgent"].noul)

The example asks for a judgment; it does not demonstrate a measured accuracy or predict the value the API will return. Pinning a version makes it easier to compare behavior over time. TypeSafe's jev-latest alias can change without an application code change.

Jev uses a fixed answer format for each question type. Applications still need to handle transport errors, rate limits, missing results, and incorrect judgments. Constrained answers are not a guarantee that every request succeeds.

Probability and confidence are different

TypeSafe says its training method, reinforcement learning for calibrated decisions (RLCD), optimizes probabilities against outcomes. For a well-calibrated set of predictions, events assigned 0.8 probability should occur about 80% of the time. That is a property measured over many cases, not a guarantee for one answer.

There is also a separate API detail that matters when writing a threshold. In TypeSafe's confidence documentation, the confidence field summarizes how concentrated a Choice or Score distribution is. It is not interchangeable with the probability assigned to an option. Noul returns a probability without that additional field.

For example, an application might send low-confidence team assignments to a review queue. It should choose that cutoff from held-out tickets, measuring how often the selected team was correct. Treating confidence = 0.9 as “90% of these decisions will be right” would skip that check.

Language-model probabilities also need evaluation, but they are not inherently unusable. In the independent comparison below, some LLMs had lower calibration error than Jev. The important difference is the measured behavior on the task, not whether the interface calls a number “confidence.”

What an independent comparison found

jev-frontier-bench tested Jev 1.13 and five language models on the same 200 items, with 50 each from BANKING77, BoolQ, Yelp, and ChaosNLI. The September 19, 2026 run used OpenRouter, enabled medium reasoning effort for the LLMs, and measured latency from India with network time included.

Expected calibration error (ECE) compares reported confidence with observed accuracy across groups of predictions. Lower is better under the same measurement procedure. For Jev, this benchmark used the top option probability, not the API's separate confidence field.

Jev was the fastest and cheapest model in this run, and had the lowest overall accuracy. The repository's paired analysis found Claude Fable and Astra clearly ahead on these items; the smaller accuracy differences were within the study's uncertainty. Fifty examples per task are not enough to establish a universal ranking.

The author also computed a fallback policy from the same logs: accept Jev when its top probability is at least 0.9, otherwise use Fable. It reached 82.5% accuracy at $4.41 per 1,000 decisions. The threshold was selected on the evaluated data, so this is an exploratory result. A deployment needs separate data to test whether the policy transfers.

What happens when people disagree

The ChaosNLI subset tests something different: whether the returned probabilities resemble the distribution of labels from 100 human annotators per item.

Jev's Jensen-Shannon divergence was 0.149. A uniform distribution across the three options scored 0.127, and Claude Fable scored 0.043. On that measure, Jev matched human disagreement less closely than the uniform baseline.

This is not ECE, and the numbers should not be compared directly with the table above. It shows why a probability trained for decisions cannot automatically be read as the fraction of people who would agree. A moderation task involving ambiguous language needs its own evaluation.

Why the benchmark conditions matter

This was one run through one gateway. The language models used structured outputs, and their reported reasoning-token use was small on these short tasks. A different prompt, model version, host, or task distribution can change the tradeoff. Keep those conditions attached when using the chart to choose a model.

Cost, context, and practical limits

TypeSafe's current model page, checked September 28, lists Jev 1.13 at $0.042 per million input tokens, with output tokens free. The state is ingested once, but each question contributes input tokens. Parallel evaluation does not make an unlimited number of questions free.

The documented limits are 64K tokens for the entire request and 32K for the state plus the longest question. Jev accepts text, not images, audio, or video. It does not write prose or code, and TypeSafe documents weaknesses in tasks such as exact counting and arithmetic. Compute exact amounts in code before asking the model to judge their meaning.

A routing decision also does not authorize an action. If a ticket appears to request a refund, the application still needs to check account ownership, transaction records, and refund policy. A model judgment belongs inside those controls.

Where to try it first

A useful first test is a router or batch classifier whose decisions can be compared with existing labels. Run it alongside the current process, record the model version and probabilities, and inspect the cases where the two disagree.

Separate choosing the threshold from evaluating it. Use one set of examples to decide when to accept, escalate, or fall back, then measure the complete policy on another set. Track the cost of both model calls and corrections.

Jev offers a convenient interface for narrow judgments. Its independent results make a case for testing that interface where speed and cost matter, while keeping accuracy and uncertainty visible in the decision.

Sources

• TypeSafe AI: System One concepts
• TypeSafe AI: Quick start
• TypeSafe AI: Confidence
• TypeSafe AI: AI primer and RLCD
• TypeSafe AI: Models and pricing
• TypeSafe AI: Jev 1.13 limitations
• GitHub: jev-frontier-bench, methods and results