arXiv：cs.AI· Kiwoong Park, Seyoung Song, Donghyun Woo, Alice Oh·· 22 小时前
北大还是南韩？LENS 基准诊断 LLM 对朝鲜语言、社会与知识的理解
North or South? Diagnosing LLMs' Understanding of North Korean Language, Society, and Knowledge
AI 导读
研究者提出 LENS，一个经人工校验、含 1,765 道题的基准，从语言鲁棒性、社会文化视角对齐、领域知识三个维度评估 LLM 对朝鲜的理解。评测发现模型在朝鲜语输入上性能小幅下降，社会文化视角对齐差距更大，领域知识严重不足。模型常混淆正式机构与民众现实、套用自由民主预设或依赖刻板印象，论文已被 AACL-IJCNLP 2026 主会接收。
正文
Abstract:Large Language Models (LLMs) have made substantial progress in multilingual question answering, yet their ability to understand low-resource sociolinguistic contexts remains under-explored. We study North Korea as a unique testbed where decades of division have produced distinct differences from South Korea in vocabulary, orthography, norms, and ideological discourse. We introduce LENS (Language and sociocultural Evaluation for North Korean Society), a human-validated benchmark of 1,765 questions evaluating LLMs across three dimensions: linguistic robustness, sociocultural perspective alignment, and domain-specific knowledge. Evaluating diverse recent LLMs, we find a modest performance drop on North Korean language inputs, a larger gap in sociocultural perspective alignment, and severe limitations on domain-specific knowledge. Qualitative analysis reveals that models often conflate formal institutions with lived realities, project liberal-democratic assumptions, or rely on coarse stereotypes. Our results demonstrate that linguistic similarity alone does not guarantee sociocultural or domain-specific understanding, highlighting the critical need for grounded evaluation resources for underrepresented cultural and political contexts.
| Comments: | Accepted to AACL-IJCNLP 2026 (Main Conference). 19 pages, 1 figure, 11 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15282 [cs.CL] |
| (or arXiv:2610.15282v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15282 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Kiwoong Park [view email]
[v1]
Sun, 11 Oct 2026 02:26:46 UTC (68 KB)
来源：arXiv：cs.AI · arxiv.org