arXiv：cs.CL· Philipp E. Glass, Alina Miron·· 3 小时前
CompOrca：面向指令微调数据的语料级合规标注
CompOrca: Corpus-Scale Compliance Labelling of Instruction-Tuning Data
AI 导读
CompOrca 为 OpenOrca 全部 4,233,923 条样本提供合规标注，由 LongCat-2.0（1.6T 参数）进行五次独立判定。语料按一致合规（94.75%）、一致不合规（1.28%）和判定不一致（3.97%）发布，后者可用于过滤歧义样本。
正文
Abstract:Studying how fine-tuning shapes refusal and noncompliance behaviour requires knowing which training examples refuse or otherwise fail to fulfil the request. Existing annotations cover evaluation sets, which are far smaller than training corpora. We present CompOrca, compliance labels for all 4,233,923 examples of the OpenOrca corpus. Every example was classified as compliant or noncompliant by five passes of an open-weight LLM judge (LongCat-2.0, 1.6T parameters). The corpus is released as unanimous compliance (94.75%), unanimous noncompliance (1.28%), and nonunanimous rows (3.97%), with the raw vote counts. A single pass flags 2.7-3.2% of the corpus as noncompliant, while only 1.28% is flagged by all five, so the most ambiguous rows can be filtered out. Against 450 human-annotated examples (150 annotated twice; human-human $\kappa=0.93$), the unanimous compliance and noncompliance labels are 97.3% and 86.7% precise. The noncompliance label is a high-precision subset of the corpus's noncompliance. Published refusal-detection methods recall between 0.4% and 94.1% of human-labelled noncompliance. We release the full corpus with its per-row labels and vote counts at this https URL
| Comments: | Accepted to PlurVA-LLM Workshop @ AACL-IJCNLP 2026. Dataset available on HuggingFace |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.37807 [cs.CL] |
| (or arXiv:2609.37807v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.37807 arXiv-issued DOI via DataCite |
Submission history
From: Philipp E. Glass [view email]
[v1]
Tue, 29 Sep 2026 15:25:07 UTC (20 KB)
[v2]
Thu, 8 Oct 2026 00:51:13 UTC (135 KB)
来源：arXiv：cs.CL · arxiv.org