[Research Preview] Where Does a Decision Model Break? A Needle-in-a-Haystack Test for Jev

• authors: Wang Yang

TL;DR

• We built a controlled benchmark for Jev-class decision models (context + candidates → pick one) with three axes: how far the answer is, how many candidates there are, and how many history updates must be combined.
• Jev 1.13.0 is perfect at retrieval and selection all the way to its 32k limit: 100% at 20k-token contexts, 64 candidates, any needle position.
• It degrades only when it has to integrate several updates about the same entity into a current state: from 100% with 1 update to 53–63% with 16 updates. In 92% of its errors it picks a stale, earlier state.
• Two follow-ups on that third case: give Jev a correct one-line state summary and it is back to 100%; give it a stale one and it follows the summary over the log 61–100% of the time. Asked whether a summary is correct, it catches a stale one only 62% of the time (chance 50%).
• Practical takeaway for agent harnesses: compile the current state before calling Jev; do not hand it a raw history — and the correctness of that compiled state is the system’s ceiling, because Jev neither repairs nor reliably checks it.

Why

Jev is a “System One” model: it reads a state and a bounded set of options and returns a choice with probabilities, in ~200 ms. In an agent loop it picks the next action every step. We wanted to know where its choices start to go wrong: when the context gets long, when there are many candidates, or when the relevant information is spread over several updates.

Existing benchmarks (JevBench) use hand-written scenarios up to ~4k tokens, so the three factors cannot be separated. We generate the data programmatically and vary one factor at a time.

Setup

Every item is the same shape: a context, a question, and 2–64 options; the model returns one option id. Answers are computed by a program, never by a model. Three heatmaps share the context-length axis (1k → 20k tokens, 28k for natural text; lengths measured with o200k_base, Jev’s own count is ~1.45× for these records).

Heatmap ③ asks two questions on the same log: event lookup (“which project did user_17824 join on day 188?”, one line to find) and final state (“after the last entry, which projects does user_17824 belong to?”, k lines to find and merge). Wrong options for final state are the user’s earlier states, states with one update skipped, or another user’s state. Only legal operations are generated; every user starts with no projects.

Each cell has 10 independent scenarios × 3 option shuffles. The same scenario is reused across all cells with nested contexts (a shorter context is a prefix of a longer one), so cells differ only in the tested factor. Wrong options are always values that occur in the context; ids are random 5-digit numbers with no near-duplicates of the target. 7,650 requests including the two follow-ups, ~95M Jev input tokens, about $4.

Code and data: github.com/uservan/jevneedlebench.

Results

① Retrieval and ② selection: no degradation

① Retrieval — one record hidden in a haystack of the same kind of records (kv) or in essay text; 4 options.

The access code of user_37874 is code_18405.
The access code of user_52462 is code_22626.
...                                              ← 70–1,400 records; the target sits at 10% / 50% / 90%
The access code of user_28195 is code_31340.     ← target
...
Q: According to the context, what is the access code of user_28195?
   option_0 code_20709   option_1 code_31340 ✓   option_2 code_59847   option_3 code_18500

In the essay version the same four records are inserted as sentences between Paul Graham paragraphs:

The world then was divided into two groups, grownups and kids. Grownups, like some kind of ...
The access code of user_86359 is code_20709.          ← one of the wrong options' records
[6] So if you make it clear you're going to succeed no matter what, and the only reason ...
The access code of user_28195 is code_31340.          ← target, at the 50% mark
Symbols differ from strings in that you can test equality by comparing a pointer ...

② Selection — 64 key records (target + 63 candidate users) scattered through the haystack; 2 to 64 of their codes are offered as options. The context is identical for every option count.

The access code of user_28391 is code_84078.     ← target
The access code of user_25247 is code_15539.     ← candidate user
...
Q: According to the context, what is the access code of user_28391?
   option_0 code_84078 ✓   option_1 code_20405   option_2 code_68467   ...   option_7 code_54839

All 75 cells are at 100% with the correct option at probability 1.0, including 20k-token contexts (≈30k Jev tokens, close to its limit), 64 options, needles in the middle, and both haystack types. Natural-text haystacks (essays) reach 28k tokens, also at 100%.

③ State tracking: finding is fine, merging is not

③ State tracking — a day-ordered log of many users; the target user (user_17824) has k lines spread through it. Two questions on the same log.

Day 1: user_69288 joined project_K.
Day 5: user_00434 joined project_E.
...
Day 38: user_17824 joined project_H.     ← target, update 1 of 4
...
Day 112: user_17824 joined project_E.    ← update 2
...
Day 188: user_17824 joined project_A.    ← update 3
...
Day 262: user_17824 left project_H.      ← update 4
...
Day 299: user_22671 joined project_A.

Event lookup:  Which project did user_17824 join on day 188?
   project_F   project_H   project_A ✓   project_G

Final state:   Rules: joining adds a project and keeps the others; leaving removes only that project; ...
               Every user starts with no projects. After the last entry, which projects does user_17824 belong to?
   project_E   project_A   project_A, project_E ✓   (none)
               (earlier state)  (one update skipped)                  (another user's state)

Event lookup (one line) is 100% in every cell. Final state (merge k lines) falls with both axes:

The confidence drops with the accuracy: mean probability on the correct option goes from 1.00 (1 update) to about 0.5 (16 updates).

Two follow-ups on the third case

Both reuse the 900 final-state items and append one line to each log, written by the generator (not by a model): Summary as of the last entry: user_17824 currently belongs to: project_A, project_E. The correct version states the true final set; the stale version states the set before the last update, the most common way a summarizer fails.

...
Day 299: user_22671 joined project_A.
Summary as of the last entry: user_17824 currently belongs to: project_A, project_E.   ← correct
Summary as of the last entry: user_17824 currently belongs to: project_A.              ← stale (last update missing)

Does Jev use a compiled state? Same final-state question, summary line appended.

Can Jev check a summary? Same log and line, but the question is is that summary correct? with two options.

Q: ... The last line of the log is a summary of user_17824's projects.
      According to the log entries and the rules, is that summary correct?
   Correct: the summary matches the user's projects after the last log entry.
   Incorrect: the summary does not match the user's projects after the last log entry.

So on the state-tracking side Jev is limited in all three roles: it does not merge history, it trusts a summary over the raw log, and it cannot reliably verify that summary. It is a precise selector over a state someone else has compiled.

Observations

1. Retrieval is not the bottleneck. Distance, position, candidate count and haystack type make no difference within Jev’s context window. A single fact is found and selected every time.
2. Integration is. With one update the final state is the one line, and Jev is perfect. From two updates on it must combine lines, and accuracy drops with the number of updates faster than with context length.
3. The errors are stale states. Of 145 wrong final-state answers, 134 (92%) are an earlier state of the same user, and 137 are smaller than the correct set: Jev reads the first updates and misses later ones. It almost never confuses users (3 cases).
4. This is a division-of-labor result, not a defect. Jev is built for immediate judgment. The test says: whatever needs history to be merged must be merged before Jev sees it — and checked by something other than Jev.

What this means for agent harnesses

In a loop like propose → Jev picks → execute → summarize → memory, the memory handed to Jev should be a state snapshot (current page, logged in, cart has 2 items, pending steps, recent failures), overwritten each step — not an appended list of what happened. A list of events about the same object is exactly the final-state task above, and it starts failing at two updates. A snapshot is the one-update case, which is at 100% up to the context limit.

Limits

• 10 scenarios per cell; 95% intervals on the final-state cells are ±0.2, so trends are reliable and individual numbers are not.
• The summary lines are program-generated. How often a real summarizer LLM produces a stale state is a separate measurement, not done here.
• Only legal, non-conflicting operations; only set membership as the state; no similar-looking ids as distractors. These are the next things to add.
• Retrieval and selection hit the ceiling everywhere, so this data does not locate Jev’s retrieval limit — only shows it is beyond 20k tokens and 64 options.
• This measures controlled retrieval and state tracking, not real multi-step execution.