通过知识图谱引导的多任务预训练增强预训练语言模型的法律推理能力

View PDF HTML (experimental)

Abstract:Legal language understanding requires reasoning over semi-structured regulatory documents, yet standard pre-training objectives do not explicitly target this structural capability. Because current legal language models rely solely on masked language modeling, they treat statutory text as flat token sequences and ignore the underlying hierarchical constraints. To bridge this gap, we present a multi-task pre-training approach for German traffic law (StVO). Our approach combines three complementary objectives: Next Sentence Prediction (NSP), Knowledge Graph (KG) Guided Clustering (CLU), and Knowledge Graph (KG) Aware Paragraph Classification (CLS). This approach explicitly encodes structured legal knowledge directly into the model weights. This encoding enables zero-shot legal reasoning at inference time and removes the need for discrete graph traversal. We evaluate our approach across nine Transformer-based encoders and 324 experimental runs. On synthetic StVO question answering, our models demonstrate consistent relative improvements between 10.3% and 155.0% over baselines. Furthermore, relative to the established LexGLUE baselines (Chalkidis et al., 2022), our models improve over prior results on four of seven legal benchmarks. It also achieves consistent zero-shot transfer gains between 3.4% and 15.8% on Austrian driving regulations. These experimental results indicate that our training approach generalizes successfully beyond the initial training legal domain.

Comments:	Accepted at the NLLP 2026 Workshop, co-located with EMNLP 2026. 9 pages, 5 figures
Subjects:	Computation and Language (cs.CL)
ACM classes:	I.2.7
Cite as:	arXiv:2610.13722 [cs.CL]
	(or arXiv:2610.13722v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.13722 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Ibrahim Siddig [view email]
[v1] Fri, 9 Oct 2026 15:21:12 UTC (724 KB)