arXiv：cs.AI· Christopher J. Chanhnourack·· 4 小时前AI 评分45
长期记忆评估审计：重复评判、读者差异与负对照
Auditing Long-Term Memory Evaluation: Repeated Judging, Reader Variation, and Negative Controls
AI 导读
一项针对 LongMemEval-S 全部 500 道开发题的长期记忆检索链评估审计发现，最强历史 reader lane 在适配 GPT-4o 评分标准下得分为 479 和 475，对同一批 pass-1 答案重新评判后三个标签发生变化，得分变为 478。
正文
Abstract:This report audits evaluation of a long-term-memory retrieval chain on the 500 LongMemEval-S development questions. Its strongest historical reader lane scores 479 and 475 under an adapted GPT-4o rubric; re-judging the same pass-1 answers changes three labels and yields 478. Fixed-answer knowledge-update re-scoring gives 70/72 under the upstream template and 69/72 under the modified template. Reader lanes span 93 to 479 on fixed packets; paired tests between the two strongest historical lanes establish neither superiority nor equivalence. A different-family reader, configured without client tools or operator files, scores 474, 1.0 percentage point below the headline pass (paired 95% interval [-3.0,+1.0]). Live reader request bodies were not retained. With the same requested reader label, route and judge snapshot, the full package scores 474 versus 454 for baseline sessions, a difference of +4.0 percentage points [95% interval +2.2,+6.0]. Eighteen of the 23 gains, and no losses, occur where baseline packets lacked listed evidence; this post-hoc split does not identify a component effect. In recovered LoCoMo data, token-F1 gains do not survive answer-line extraction. A negative control rejects a verifier that repairs three wrong drafts but breaks eleven correct ones. All questions were used to develop the components; no untouched holdout was evaluated. These findings do not establish a new leaderboard leader or transferable memory advantage. The A/D comparison has one pass per arm, including six reused identical-prompt outcomes, with no pinned reader snapshot; B/C and repeats remain unrun. Original headline requests cannot be reconstructed and stages 1--4 remain closed. Released artifacts support packet inspection and saved-verdict recounting and re-scoring; they do not reconstruct the method.
| Comments: | 23 pages. Evaluation-audit revision; adds fixed-answer KU re-scoring, a one-pass full-package versus baseline reader comparison, and post-hoc evidence coverage. Includes ancillary data and an offline recount script. Method sources remain held; all 500 questions were used for development |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2609.38021 [cs.CL] |
| (or arXiv:2609.38021v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.38021 arXiv-issued DOI via DataCite |
Submission history
From: Christopher Chanhnourack [view email]
[v1]
Tue, 29 Sep 2026 17:04:18 UTC (21 KB)
[v2]
Fri, 2 Oct 2026 02:24:36 UTC (836 KB)
来源：arXiv：cs.AI · arxiv.org