Jev confidence in production: thresholds, review, and model pinning

A Jev answer can tell code both what the model chose and how clearly the distribution favored that answer. That second signal is useful, but it is easy to overread.

For Choice and Score, confidence is derived from the returned probability distribution. A concentrated distribution produces higher confidence. A flat distribution produces lower confidence. It does not certify that the winning answer is correct.

Probability and confidence are different fields

A Choice response contains a probability for every allowed option. If billing receives 0.82 and the remaining probability is spread across three alternatives, the answer has a clear leader.

A Score response contains probabilities across ordered levels. Its numeric score is the expected value of those levels, while confidence summarizes the shape of the distribution.

A Noul is different. It returns the probability of yes, which already defines the binary distribution. The confidence documentation says Noul does not include a separate confidence property.

Keep the full distribution even when the convenience value is enough for the live branch. It can reveal adjacent categories, split extremes, and recurring ambiguity that one number hides.

High confidence is not the same as correctness

A model can be confidently wrong. The supplied state may omit a necessary fact, the criteria may overlap, or the model may interpret the question too literally.

This is why a production evaluation needs known outcomes. Group predictions by confidence band and measure how often each band is correct for the actual use case. Calibration is a property observed over groups of predictions, not a promise attached to one result.

Also inspect errors by consequence. A wrong help-center route is not equivalent to a wrong fraud block or payment action.

Set thresholds by action risk

One global threshold is rarely enough. The same classification can lead to actions with different stakes.

Action	Example behavior
Reversible and low risk	act at a tested moderate threshold
Visible but recoverable	ask for confirmation in the middle band
Financial, destructive, or account-changing	require a higher threshold and explicit confirmation or review
Low confidence	do not guess; gather context or escalate

answer = response.choices["action"]

if answer.confidence < 0.55:
    route_to_human()
elif answer.choice == "show_help_article":
    show_article()
elif answer.choice == "close_account":
    require_explicit_confirmation()
else:
    continue_workflow()

The numbers are placeholders. They become policy only after a team measures false positives, false negatives, review capacity, and customer impact.

TypeSafe’s confidence-gated routing pattern treats confidence as a second axis. The selected answer describes what to do; confidence helps code decide whether it is safe to do it automatically.

Build an evaluation set before choosing a threshold

Use representative production-like examples with reviewed labels. Include easy cases, ambiguous cases, missing context, edge conditions, and inputs that have caused expensive mistakes.

For each example, record:

• expected semantic result
• model result and full distribution
• confidence where available
• action the current policy would take
• cost of an incorrect automatic action
• whether a human review would have corrected it

Then plot automation rate against error rate. A threshold that looks accurate may send nearly everything to review. A threshold that automates most cases may concentrate errors in the cases the business cares about.

The operating target is not the highest confidence number. It is a tolerable error profile with a review queue the team can actually service.

Treat uncertainty as product behavior

A low-confidence result should lead somewhere deliberate:

• Ask the user for the missing detail.
• Retrieve a narrower piece of state and try again.
• Use deterministic fallback rules.
• Send the case to a specialist model.
• Place the case in a human queue.

Do not hide uncertainty by repeatedly rewriting criteria until the model sounds certain on a small test set. That can overfit examples while making the real boundary less clear.

A product manager should define what the user sees during review, how long the case can wait, and what happens when reviewers disagree. Confidence routing is not only an engineering switch. It changes the customer journey.

Version questions and models

TypeSafe’s models documentation distinguishes moving aliases such as jev-latest from versioned model IDs. If an alias moves, behavior can change without an application release.

Once thresholds are tuned:

1. Pin the versioned model ID.
2. Log the model reported in every response.
3. Version the question definitions with the application code.
4. Run the labeled evaluation before changing either one.
5. Roll out gradually and compare review and error rates.

A threshold tuned for one model and one set of criteria should not be assumed valid for another.

Plan for API failure separately

Confidence only exists when a request succeeds. Timeouts, rate limits, invalid requests, and service overload need a different fallback.

The API reference documents 429 for rate limits and 529 for overload, among other errors. SDK retries can handle short interruptions. After retries are exhausted, the application still needs a safe path.

Do not convert an unavailable model into a high-confidence default. Mark the reason, use a deterministic fallback where appropriate, or queue the case for later processing.

Review the question when confidence stays low

Persistent low confidence can point to a product-design problem:

• The categories overlap.
• The state lacks the fact needed to decide.
• The question combines multiple dimensions.
• The answer space omits a legitimate outcome.
• The task requires arithmetic, chronology, or generation rather than a semantic judgment.

The fix may be a better taxonomy or another system, not a lower threshold.

For the primitive-level design, read Jev Choice, Score, and Noul. For implementation examples, use the TypeScript quickstart or Python tutorial.

Sources

• Confidence
• Confidence-gated routing
• Models
• API reference
• Jev 1.13 jaggedness

FAQ

What does Jev confidence mean?

For Choice and Score, confidence is a single statistic derived from how concentrated or spread out the returned probability distribution is.

Does high Jev confidence guarantee a correct answer?

No. A concentrated distribution can still favor the wrong option. Confidence is useful for routing, but correctness must be measured on representative labeled data.

Why does Noul not return confidence?

A Noul result is already the probability of yes in a binary distribution, so the API does not add a separate confidence value.