强 harness 缩小模型差距：立法文档转 XML 的对比评测

View PDF HTML (experimental)

Abstract:Legal-content reproduction requires matching the source exactly in section hierarchy, verbatim text, citations, and amendment spans; a dropped citation or mislabeled amendment carries into the products lawyers rely on. We report an industrial pipeline converting U.S. state legislation into strict, schema-constrained XML, and hypothesize that its accuracy comes from harness engineering, the deterministic tooling built around a model, more than the model itself. Our controlled ablation crosses three harnesses (zero-shot, a single tool-using agent, and a decomposed multi-agent pipeline) with three Claude models on 153 gold bills from two jurisdictions. The harness narrows the model gap: the weakest model in the full pipeline beats the strongest model alone (0.948 versus 0.806, paired bootstrap p less than 0.001). Because the two agentic harnesses share identical tools, their gap comes from orchestration: decomposition plus the persisted state and per-agent context it brings. Across four scored dimensions, the model spread at zero-shot (0.68 to 0.81) narrows to 0.95 at the full pipeline; the residual sits almost entirely in citation normalization, where the canonical identifier is retrieved from an external authority rather than generated from the weights, so a deterministic lookup tool supplies the exactness that a larger model does not.

Comments:	Accepted to the Natural Legal Language Processing Workshop (NLLP) at EMNLP 2026
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2610.19999 [cs.CL]
	(or arXiv:2610.19999v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.19999 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Palvika Bansal [view email]
[v1] Wed, 14 Oct 2026 11:15:25 UTC (795 KB)