I Found a Use for Jev Worth Trying: Checking Whether Your AI Got the Facts Right

Your AI finished the task. Did it get the facts right?

An assistant says it has saved your plans. The request succeeded, the output is valid, and nothing looks broken. Then you read the details: it has filled in a time you never gave it.

Consider this synthetic example from our experiment, translated from Chinese:

“There is a book club on Friday. I bought three tickets. The start time has not been announced.”

Candidate fact: “The book club starts at 19:00.”

19:00 sounds plausible. It has no support in the source. Another easy mistake is to turn someone’s arrival time into an agreed meeting time, even when the conversation says the meeting time is still undecided.

These are the errors that send you back to the original conversation. You wanted less to remember; now you have to check the assistant’s work, field by field. For a developer, a successful API response offers little reassurance about this kind of factual mistake.

Could a small, inexpensive check help us find the records that deserve a closer look?

Our first attempt tried to do too much

In our previous scheduling experiment, improving the original model’s context produced 46 exact matches in 48 attempted cases. Adding our multi-field Jev layer reduced that to 38. It corrected none of the baseline’s errors in that test.

That integration asked Jev to judge several aspects of a task and then adopted its decisions. More model calls did not automatically mean more reliable results.

For this follow-up, we narrowed the question: given the relevant source and one candidate fact, does the source support it, contradict it, or leave it unestablished? Jev would flag a possible problem rather than rewrite the task or change a calendar entry.

Give it one fact to check

Source:
There is a book club on Friday. I bought three tickets.
The start time has not been announced.

Candidate fact:
The book club starts at 19:00.

Question:
How does the source relate to the candidate fact?

Choices:
SUPPORTED / CONTRADICTED / NOT_ESTABLISHED

Both Jev and the comparison model returned NOT_ESTABLISHED for the recorded Chinese example. The point is to check evidence, not whether 19:00 would be a reasonable guess.

The candidate facts in this experiment were supplied by hand. We have not yet shown that an automated pipeline can extract every important claim from a long assistant reply. That upstream step could miss a fact or add another model call.

An October 10 probe tested 24 candidate facts twice, with 21 matches in each pass and the same three errors. That was enough to motivate a comparison, not enough to establish an improvement over an existing reviewer.

A narrow task produced a more encouraging result

On October 12, we compared selected code checks, GPT-5.5 with low reasoning effort, and Jev pinned to jev-1.13.0. The model arms received the same evidence and criteria without seeing each other’s answers. Criteria were in English; evidence was in Chinese, without a translation stage.

The held-out semantic set contained 32 checks from eight scenario families: 16 problematic facts and 16 normal ones. These were synthetic examples, not a sample of production traffic.

Why 30 exact matches if all problems were flagged? Some judgments crossed the boundary between “contradicted” and “not established.” For example, both models flagged an arrival time being treated as a meeting agreement, but chose a different category from the expected label. Useful screening and perfect classification are different claims.

The full third round comprised 80 checks across 20 families: 64 semantic checks and 16 structured controls. Each model made 80 requests, including 16 preselected repeats. Repeats are not additional independent cases.

In this batch, Jev made the extra review step substantially cheaper and quicker. These timings exclude fact extraction, context preparation and human review. They are recorded call durations, not production end-to-end latency.

Costs use the experiment’s historical rate assumptions: GPT input $5, cached input $0.50 and output $30 per million tokens; Jev input $0.042 per million tokens with free output. GPT figures are subscription-usage equivalents, not a paid API invoice. These are not current pricing promises or a claim that an entire product became proportionally cheaper.

The miss that keeps this from being an automatic approval system

One calibration example matters just as much as the encouraging table:

User: “The rehearsal is on Wednesday evening.”
Assistant: “What time?”
User: “Eight forty.”

Candidate fact: “The rehearsal starts at 08:40.”

The comparison model returned CONTRADICTED. Jev returned SUPPORTED. It missed the evening context.

Jev’s confidence for that answer was 0.43. A threshold of 0.5 selected during calibration routed it to review. But an error is not guaranteed to have low confidence, and this threshold is not a general reliability boundary. No held-out check triggered fallback, so that set did not independently demonstrate that fallback rescues mistakes.

Across all 64 semantic checks, including calibration, the comparison model flagged 32 of 32 problematic facts; raw Jev flagged 31. The small held-out result does not establish that Jev is universally as accurate as the larger model.

A passed check is not proof that a fact is true. The useful hypothesis is that Jev can help prioritize review, while missed problems are measured through independent sampling.

Where I would put it in an AI application

I would start with a background workflow that leaves the original task intact:

1. The assistant completes its existing workflow.
2. Collect a few important candidate facts and their relevant source text.
3. Use code for explicit states, identifiers, ranges and time arithmetic.
4. Use Jev for the remaining checks that need language interpretation.
5. Review flagged records and independently sample records it passed.
6. Turn confirmed defects into regression cases and test the fixes.

This is a proposed production workflow. Our experiment tested offline checks and replayed a possible fallback; it did not deploy a background reviewer, alerts or automatic corrections.

Some checks need no model at all. If a tool receipt says “awaiting confirmation” and zero changes applied, code can reject a claim that two changes were completed. Permissions, ownership, transactions and reliable execution still belong in the application.

Jev is more interesting where the question is small but semantic: does this sentence establish a meeting time, or merely mention when someone arrives?

The value to measure is less checking, not more AI

The hoped-for benefit is simple: spend less time rereading every record and more time on records with a plausible defect.

That benefit has not yet been measured. A reviewer that generates too many false alarms can create work. A cheap reviewer that misses important mistakes may not be worth integrating.

In a real trial, I would measure false alarms per hundred normal records, missed serious defects, total cost per confirmed problem, and minutes of human review. The last metric matters: did this actually reduce the attention the application demands?

Our synthetic set deliberately contains many problems. Real defects may be rare, so its screening performance cannot be carried straight into a production workload. Both flagged and unflagged records need independent review.

Try it on one thing you already have to double-check

Choose one field in your own AI workflow. Prepare a small set of non-private examples: facts the source explicitly supports, facts it contradicts, and plausible details it never states. Include ambiguous cases and corrections, not just easy positives.

Give the same evidence and candidate facts to Jev and your existing review method. Record judgments, call durations, and how much checking you still need to do. Our Jev API guide covers integration; the example above describes the task, not a ready-to-run API payload.

You can also explore the scheduling question in the playground. That preset illustrates the arrival-versus-agreement distinction; it does not reproduce this fact-checking evaluation. Check the Live or Demo label before interpreting its result.

Scheduling is the use case we tested. Applying the same idea to other assistants is an experiment for their own data, not a capability this study has already established.

Adding Jev was not the value in our first attempt. This time, we found a specific job worth investigating: quickly checking whether an assistant’s recorded facts have evidence, so a person has a better chance of finding the quiet mistakes.

It does not remove our responsibility to check. The next test is whether it makes that responsibility less exhausting.

Evidence, limits and the next installment

This article draws on our saved October 10 and 12 offline experiments. Fixtures and labels were authored by the same person without independent dual annotation. Some label boundaries are disputed. No customer conversations, calendar writes or production outcomes were involved.

The previous article proposed at least 300 new independent scenarios for a larger validation. This 20-family follow-up is an exploratory test of a narrower use case, not fulfillment of that production adoption gate. The tasks across our three rounds differ; their accuracy figures should not be combined into an improvement curve.

Recorded counts and archive checksums were verified locally before publication. The full archive is not published with this page. Aggregate tables are evidence about these saved runs, not a publicly reproducible benchmark or a guarantee of repeated results.

Start with our hands-on review of Choice, Noul and Score, then read what went wrong when we added a decision layer. This third installment asks where a smaller check might earn its place. Production usefulness still has to be measured.