arXiv：cs.CL· Ghazal Kalhor, Behnam Bahrak·· 6 小时前
MasalBench：评估 LLM 对波斯谚语语境与跨文化理解能力的基准
MasalBench: A Benchmark for Contextual and Cross-Cultural Understanding of Persian Proverbs in LLMs
AI 导读
研究者推出 MasalBench，用于评估 LLM 对低资源语言波斯谚语的语境与跨文化理解能力。八款 SOTA 模型在上下文中识别波斯谚语准确率超过 0.90，但识别对应英文谚语时最佳模型仅 0.79，暴露当前 LLM 在文化知识与类比推理上的局限。
正文
Abstract:In recent years, multilingual Large Language Models (LLMs) have become an inseparable part of daily life, making it crucial for them to master the rules of conversational language in order to communicate effectively with users. While previous work has evaluated LLMs' understanding of figurative language in high-resource languages, their performance in low-resource languages remains underexplored. In this paper, we introduce MasalBench, a comprehensive benchmark for assessing LLMs' contextual and cross-cultural understanding of Persian proverbs, which are a key component of conversation in this low-resource language. We evaluate eight state-of-the-art LLMs on MasalBench and find that they perform well in identifying Persian proverbs in context, achieving accuracies above 0.90. However, their performance drops considerably when tasked with identifying equivalent English proverbs, with the best model achieving 0.79 accuracy. Our findings highlight the limitations of current LLMs in cultural knowledge and analogical reasoning, and they provide a framework for assessing cross-cultural understanding in other low-resource languages. MasalBench is available at this https URL.
| Comments: | Accepted and forthcoming at the First Workshop on Pluralistic Value Alignment of LLMs (PlurVA-LLM 2026) at AACL-IJCNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2601.22050 [cs.CL] |
| (or arXiv:2601.22050v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2601.22050 arXiv-issued DOI via DataCite |
Submission history
From: Ghazal Kalhor [view email]
[v1]
Thu, 29 Jan 2026 17:49:44 UTC (432 KB)
[v2]
Wed, 14 Oct 2026 08:32:20 UTC (432 KB)
来源：arXiv：cs.CL · arxiv.org