arXiv：cs.CL· Jinlong Liu, Mohammed Bahja, Mark Lee·· 16 小时前
推理监督为何适得其反：基于 TTCW 的长篇文学评论生成研究
When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation
AI 导读
研究构建了 263,911 篇长篇故事的数据集，每篇配 14 个 TTCW 维度的评分与合成评论，用于测试推理轨迹能否提升 Qwen3-4B、Qwen3-8B 和 Nemotron-8B 的结构化文学评论生成。三个模型上无轨迹监督的评分模仿保真度更高且报告生成近乎完整，而有轨迹监督将完成率降至 85.9%–90.9%，失败案例中 88.6% 的推理轨迹存在语义重复循环。
正文
Abstract:Automatic evaluation of long-form literary writing remains challenging, particularly for creativity-related dimensions such as originality and flexibility. We address the lack of large-scale supervision for structured Torrance Test of Creative Writing (TTCW)-based evaluation by constructing a dataset of 263{,}911 long-form stories, each paired with scalar scores and meta-synthesised review comments across 14 TTCW-based dimensions. Using this dataset, we study whether reviewer-style reasoning traces improve structured literary review generation across Qwen3-4B, Qwen3-8B, and Nemotron-8B. Across all three models, no-trace supervision yields higher score imitation fidelity and near-complete report generation, whereas trace supervision reduces completion to $85.9$--$90.9\%$. Failure analysis shows that $88.6\%$ of failed reasoning generations contain semantic repetition loops inside the trace. These loops concentrate in particular story--criterion interactions and progressively harden after onset, while comparable repetition is rare in the supervision data. Our results suggest that, for long-form tasks requiring strictly structured outputs, explicit reasoning traces can introduce generation instability that outweighs their overall benefit.
| Comments: | Submit to NAACL 2027 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.20364 [cs.CL] |
| (or arXiv:2605.20364v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.20364 arXiv-issued DOI via DataCite |
Submission history
From: Jinlong Liu [view email]
[v1]
Tue, 19 May 2026 18:16:58 UTC (222 KB)
[v2]
Wed, 14 Oct 2026 11:05:55 UTC (321 KB)
来源：arXiv：cs.CL · arxiv.org