Three weeks after TypeSafe opened access to Jev, OpenAI put its own decision model into public beta. The Decisions API was announced on 2026-10-06 1. It takes text or images plus typed questions and returns probabilities over answers you defined, the same contract decision models as judges describes for Jev 2. So a team that wants a cheap typed judge now has two vendors to choose from. This page covers how they differ, the score field that misleads on both if you read it as a level, and how to choose between them on your own data. Figures are as of 2026-10-17. The OpenAI API is in beta, so check prices and limits again before you budget.
Same three questions, two dialects
Both APIs ask the same three kinds of question under different names. OpenAI calls them predicate, choice and score. You send them to POST /v1/decisions with model, input and a questions array, and each question carries a unique name and its instructions 2. Jev calls the same three Noul, Choice and Score 3.
Figure: The question types map one to one and mean the same thing on both APIs. On both, a score is the probability-weighted average of the levels, not a level you picked. The field names and response layout differ, so a rubric ported between vendors needs a new parser and new thresholds.
On the request side, a choice question carries choices, a list of {value, description}, and a score question carries levels, a list of {label, description} ordered from lowest to highest. Each answer echoes the question's name. A predicate answer has a probability. A choice answer has the picked choice, a probabilities list and a confidence. A score answer has a score, its own probabilities and a confidence. There is a fourth answer type, refusal, and OpenAI's own sample code checks for it before reading anything else 2. Treat a refusal as its own outcome in your eval tables. Do not count it as a pass, and do not count it as a fail.
The layouts differ in small ways that break a parser. OpenAI returns each answer's probabilities as a list of objects, {value, probability} for a choice and {value, label, probability} for a score 2. Jev returns a score's probabilities as a map keyed by level number, plus a legend that maps each number back to its description 4.
TypeSafe publishes Jev's confidence formula. For a Choice it measures how far the top probability sits above an even split. For a Score it measures how tightly the probability clusters around the most likely level. A Noul has no confidence field 5. OpenAI's guide does not say how its confidence is computed 2. Do not assume a 0.9 means the same thing on both. Tune each threshold on its own vendor's output.
The score field is an average, not a level
On both APIs, score is the probability-weighted average of the 0-based level numbers. OpenAI's guide uses probabilities of 0.1, 0.7 and 0.2 over levels 0, 1 and 2, which give a score of 1.1 2. TypeSafe's docs show a Jev score of 1.43 from 0, 0.57 and 0.43 4. That is an expected value. It is not a level from your rubric, and a team that rounds it, or reads it as "the level the judge picked", is measuring something else.
Figure: The example from OpenAI's guide, with severity labels added for illustration. Jev's score works the same way. The most likely level is 1, but the reported score is 1.1. A distribution of 0.5, 0 and 0.5 would report 1.0, the same as a confident "minor defect".
An average hides the shape of the distribution. A judge split evenly between "no defect" and "blocking defect" reports 1.0, the same number as a judge that is sure the defect is minor. For a pass/fail gate, read the probabilities list and sum the mass at or above your cut level. Better still, ask the gate as a predicate ("Does this response contain a blocking defect?") and keep the score for dashboards. The same advice applies to graded scales on Jev, where exact-level accuracy was the weakest result in the independent studies 6.
| OpenAI Decisions | Jev |
|---|
| Status | Public beta since 2026-10-06 | Early access since 2026-09-15 |
| Model IDs | gpt-6-luna only | jev-1.13.0 (alias jev-latest), jev-1.14.0 (alias jev-preview) |
| Input price | $0.10 per million tokens | $0.042 per million tokens |
| Output | Not billed | Not billed |
| Inputs | Text, and images as inline base64 | Text only |
| Context | Not stated in the guide | 64k per request, 32k per question by default |
| Data controls | Zero Data Retention, HIPAA, and US or Europe (EEA plus Switzerland) processing for eligible customers | Not covered in the docs cited here |
Sources: OpenAI's guide and announcement 2 1; TypeSafe's models and System One pages 7 3. OpenAI adds multipliers for regional processing and long-context input 2. TypeSafe lets you raise context to 262k tokens with an opt-in header and warns that this mode is slower and less accurate. It now sets rate limits per organization in its console rather than publishing one number 7.
At list price OpenAI costs about 2.4 times as much. With the distilled judges assumptions (1,200 input tokens per judgment, 100,000 traces a day), that works out to 5. Either is small next to a frontier judge, so price alone should not decide this. The real questions are whether you need images, whether your compliance team needs the data controls, and which model is right on your criteria.
For speed, OpenAI says the Decisions API answers about ten times faster than the Responses API 2. That compares OpenAI to itself. There is no published head-to-head latency test yet, so measure latency on your own payloads.
What the early evidence says
The evidence is thin, and the two vendors are not equally tested. Jev has two independent arXiv studies behind it 8 6. As of 2026-10-17, the OpenAI API has forum reports from developers and nothing more. Read the following as hypotheses to test, not results.
In the announcement thread, one developer compared the two on a few hundred questions from word games and one conversational game. They reported that Luna gave about three times as many confident wrong answers on nuanced judgment calls, came out roughly even on simple yes/no checks, and cost two to three times as much. Its median turn in a multi-step conversation was about 1.6 s, against Jev's 0.3 s 1. That is one person's workload, and it is not an eval workload.
A second test from the same thread is more useful because it isolates one mechanism. A developer asked about a coin weighted to land heads 70 percent of the time, over 1,000 trials. Asked as a predicate, the answers matched the expected 70 percent. Asked as a choice between heads and tails, the model picked heads 98 percent of the time. Reordering the options also moved the choice probabilities 1.
Figure: A single developer's 1,000-trial test from the announcement thread, not a study. The point it illustrates holds anyway: a choice returns the most likely option, and how often that option wins is not a probability you can read as a rate.
Our reading is that this is expected behavior, not a bug. A choice reports the option it rates most likely, so repeated runs mostly return the favored side, and the share of picks overstates the true rate. The thread gives no figures for the probabilities field, only that reordering the options changed it. The lesson for evals is about which field you read. If you need a number you can treat as a rate, such as "what share of traces have this defect", ask a predicate and average its probability. Do not count how often a choice picked an option. Run every choice question in at least two option orders. Li et al. found the same order sensitivity in Jev on pairwise inputs 8.
A bake-off you can run this week
You do not need a vendor's benchmark. You need your own gold set and four numbers per vendor.
flowchart LR
A[Gold set<br/>100+ labeled traces<br/>per slice] --> B[Write each criterion once<br/>predicate or choice]
B --> C[Jev, pinned version]
B --> D[OpenAI Decisions,<br/>run date recorded]
C --> E[Run choices in<br/>two option orders]
D --> E
E --> F[Per slice: TPR, TNR,<br/>order-flip rate,<br/>confident-wrong count,<br/>cost per decision]
F --> G[Pick per criterion,<br/>not per vendor]
- Same instructions, both vendors. Write each criterion once as a predicate or choice. Follow OpenAI's own guidance: write observable criteria, put one concern in each question, give every choice a distinct meaning plus an "other" fallback, and send dependent decisions as separate requests 2. Pin Jev to a versioned ID 7. OpenAI lists one model, so record the run date for every result.
- Measure TPR and TNR per slice, not overall agreement, exactly as in calibrating your judge against humans.
- Measure the order-flip rate. Run each choice question with the options in two orders and count how often the pick changes. If it changes on more than a few percent of items, ask that criterion as a predicate instead.
- Count confident wrong answers at your threshold. This number decides whether a judge cascade is safe, because escalation never sees those errors.
- Compute cost per completed decision. Include refusals and retries, since a refused answer still costs input tokens and still needs a fallback.
If your evals already run in MLflow, version 3.17.1 (2026-10-13) added Decision API support to make_judge() for yes/no and fixed-choice feedback 9, so for predicate and choice criteria the bake-off is a change of judge backend rather than new plumbing; the MLflow GenAI review has the details. DeepEval reaches Jev through its eval modes, covered on the DeepEval page.
Then choose per criterion. Nothing stops you from running image checks on OpenAI and text rubrics on Jev, as long as each metric stays on one backend across runs. Switching backends changes the number, the same way switching DeepEval's eval mode does.
Security rules carry over unchanged
OpenAI's guide does not address prompt injection 2. TypeSafe says plainly that text arguing for its own classification can move Jev's answer 10. Assume the same holds for any model that reads untrusted text, and apply the three rules from the Jev chapter to both vendors. Never let one model be both the gate and the judge of that gate. Keep raw tool output and retrieved documents out of the gate's input. Keep a human approval on irreversible actions.