arXiv：cs.LG· Shivam Dubey, Mohamed Bouadi, Nassim Bouarour, Varun Kulkarni, Aditya Tanna, Vinay Kumar Sankarapu·· 3 小时前AI 评分29
合成关系数据何时教会模型使用关系？从预训练数据到模型行为的预测结构追踪
When Does Synthetic Relational Data Teach Models to Use Relations? Tracing Predictive Structure from Pretraining Data to Model Behavior
AI 导读
研究将"哪种合成预训练数据能诱导关系计算"作为数据归因问题，用相同架构、初始化、目标和算力预算训练的四个 Relational Transformer 检查点，对比四种关系数据生成器产出的语料。
正文
Abstract:Relational foundation models are increasingly pretrained on synthetic databases, yet downstream benchmarks reveal little about why one synthetic corpus produces a better model than another. In particular, strong performance may arise from realistic row-level statistics without the model ever learning to use relational structure. We study this as a data-attribution problem: which property of synthetic pretraining data induces relational computation? Using four Relational Transformer checkpoints trained with the same architecture, initialization, objective, and compute budget on corpora produced by four relational data generators, we trace a measurable property of the data to learned computation and downstream behavior. We hypothesize that relational mechanisms emerge when cross-table information is predictively necessary for the masked-cell pretraining objective. RelDiff exhibits by far the largest predictive gain from foreign-key-linked parents, and its corresponding model is uniquely sensitive to foreign-key interventions on unseen databases. This dependence survives a random-initialization control, grows monotonically with the fraction of corrupted links, and localizes to a serial cross-table pathway. Finally, disrupting the same mechanism during downstream inference removes RelDiff's advantage on relational tasks while leaving structure-insensitive models nearly unchanged. These results connect a property of synthetic training data to a learned mechanism and, through intervention, to downstream behavior.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.03057 [cs.LG] |
| (or arXiv:2610.03057v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.03057 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Nassim Bouarour [view email]
[v1]
Fri, 2 Oct 2026 09:37:48 UTC (440 KB)
来源：arXiv：cs.LG · arxiv.org