arXiv：cs.CL· Anastasia Voznyuk, Andrey Grabovoy·· 13 小时前
机器生成文本检测器依赖格式伪影：arXiv 基准上的高 F1 与格式反事实检验
Measuring Artifact Reliance in Machine-Generated Text Detectors
AI 导读
对六个资源、十个科学文档子集的研究发现，机器生成文本检测器的高分部分来自 PDF 抽取格式伪影：人类文本保留原始换行，生成文本被抽成单一文本块。仅用 27 个表层特征的逻辑回归在三个 arXiv 子集上取得 0.951-0.997 macro-F1，超过所评估的所有检测器；该线索与三个格式特征以 0.996 AUC 区分两类文本。
正文
Abstract:Detectors of machine-generated text score almost perfectly on scientific-literature benchmarks, implying we can reliably distinguish with these detectors human and machine-generated documents. We looked at ten scientific-document subsets from six popular resources, and on several of them that apparent ease is a side effect of how the documents were put together. A logistic regression using 27 superficial features reached 0.833-0.997 macro-F1, and on the three arXiv subsets it outperformed every detector we evaluate with F1-scores 0.951-0.997. The strongest cue to the detector is the PDF extraction artifact: human texts retain their original line breaks, whereas generated texts are extracted as a single block of prose. This cue, together with three other formatting features, tell the classes apart at 0.996 AUC. To distinguish benchmarks that contain such a signal from detectors that rely on one, we introduce format counterfactuals: just re-wrapping a machine-written abstract without changing a single word reverses 20-27% of a public detector's correct decisions. Normalizing formatting before training reduces the generalization gap from 0.32 to 0.22. On format-clean benchmarks, it also raises out-of-domain macro-F1 from 0.33 to 0.65 for detectors trained on artifact-containing subsets.
| Comments: | Accepted to DocInsights workshop at EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13583 [cs.CL] |
| (or arXiv:2610.13583v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13583 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Anastasia Voznyuk [view email]
[v1]
Fri, 9 Oct 2026 13:34:45 UTC (59 KB)
来源：arXiv：cs.CL · arxiv.org