arXiv：cs.AI· Xupeng Chen, Chenqian Le·· 16 小时前
Jev 记忆退役门控评估：二值判定不应直接触发删除
Suggested Is Not Superseded: A Pre-Registered Evaluation of a System-One Model as a Memory-Retirement Gate
AI 导读
一项预注册评估研究了助手长期记忆中的"退役门控"问题，并提出区分 SUPERSEDED、NEEDS-CONFIRMATION 与 COMPATIBLE 的任务契约，用三个 LLM 标注者和两名人类裁决者标注 500 个测试对，评测类型化"System One"决策模型 Jev。
正文
Abstract:Long-term memory for assistants needs a gate that decides when a new utterance retires an old stored claim. Benchmark labels for this step mark a pair as changed or not, so they cannot tell a gate that acts on established change from one that acts on a hint. We write a task contract that separates the two (SUPERSEDED vs. NEEDS-CONFIRMATION vs. COMPATIBLE), label 500 test pairs built from STALE (change pairs plus same-user controls) with three LLM annotators and two human adjudicators (one author and a second annotator; provenance reported), and evaluate Jev, a typed "System One" decision model, under pre-registered hypotheses. Of the 179 test pairs STALE marks as changed, 72 meet the contract's bar for an established change. Jev never retired a claim on 141 same-conversation non-changes. Its failure is premature retirement: the binary gate retires 39.2% of claims whose change is only suggested. A three-way decision cuts this to 15.5% at a 15.3-point detection cost, which misses our pre-registered 10-point budget, and it lands on the binary gate's own threshold frontier; these verdicts hold under three readings of the gold labels. On a held-out vault scored against a second annotator's labels, a cascade that escalated uncertain cases to a reasoning model and looked best on development data was indistinguishable from simply raising the threshold (premature retirement 26.8% vs. 6.5% on dev). Part of the gap comes from the gold labels: relative to that annotator's reading, LLM-majority gold shares most of its errors with a same-family escalation model, and an identity for binary labels says how much such shared errors hide. The rest is small-sample selection on development data. For builders this means a binary score should not be wired to delete: route suggested changes to pending evidence, and do not trust a cascade selected on development data of this size.
| Comments: | 11 pages, 3 figures, 4 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15710 [cs.AI] |
| (or arXiv:2610.15710v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15710 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chenqian Le [view email]
[v1]
Sun, 11 Oct 2026 15:48:16 UTC (236 KB)
来源：arXiv：cs.AI · arxiv.org