arXiv：cs.CL· Donggyu Lee, Hyeok Yun, Meeyoung Cha, Sungwon Park, Sangyoon Park, Jihee Kim·· 6 小时前
EconCausal：面向大语言模型的上下文感知经济推理基准
EconCausal: A Context-Aware Economic Reasoning Benchmark for Large Language Models
AI 导读
研究团队推出 EconCausal，一个从 2,595 篇顶级经济与金融期刊实证研究中提取的 10,490 条上下文标注因果三元组基准，用于检验 LLM 能否在指定情境下推断因果效应方向并随情境变化修正判断。在 19 个模型上测试发现，当源情境与目标情境的效应符号相反时，平均准确率从整体 74.75% 降至 44.27%，模型对研究报告中不显著效应的预测准确率仅 19.16%。数据集与基准已公开。
正文
Abstract:Socio-economic causal effects depend heavily on their institutional and environmental contexts. The same intervention can produce different, even opposite, effects across regulatory regimes, market conditions, time periods, or populations. This poses a challenge for large language models (LLMs) in decision-support roles: can they infer the direction of a causal effect under a specified context, and revise that judgment when the context changes?
To address this, we introduce EconCausal, a large-scale benchmark of 10,490 context-annotated causal triplets extracted from 2,595 high-quality empirical studies in top-tier economics and finance journals, constructed through a rigorous four-stage pipeline with multi-run consensus, context refinement, and multi-critic filtering.
Across 19 models, we observe source-sign over-transfer in cross-context prediction. While top models reach ~84% accuracy in single-context prediction, mean accuracy is 74.75% overall but drops to 44.27% when source and target signs disagree. On these cases, accuracy is 11.54 percentage points (pp) lower with the signed source example than with the target input alone. Mean accuracy is below 60% under corrupted source signs. Models also struggle to predict study-reported nonsignificant effects (19.16% accuracy).
The dataset and benchmark are publicly available at this https URL.
| Comments: | 29 pages. Revised version with updated evaluation results, robustness analyses, expert validation, and references |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2510.07231 [cs.CL] |
| (or arXiv:2510.07231v5 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2510.07231 arXiv-issued DOI via DataCite |
Submission history
From: Donggyu Lee [view email]
[v1]
Wed, 8 Oct 2025 17:00:49 UTC (369 KB)
[v2]
Thu, 9 Oct 2025 16:46:30 UTC (369 KB)
[v3]
Mon, 23 Feb 2026 05:21:42 UTC (35,854 KB)
[v4]
Tue, 26 May 2026 12:27:46 UTC (1,215 KB)
[v5]
Wed, 14 Oct 2026 04:49:22 UTC (1,318 KB)
来源：arXiv：cs.CL · arxiv.org