arXiv：cs.CL· Khan Mahi Al Atahar, Nazifa Tasneem, Ajwad Abrar·· 16 小时前
八款大语言模型多语言摘要评测：GPT-5.6 Luna 英语领先，LLaMA 3.3 70B 孟加拉语最强
Evaluating Large Language Models for Multilingual Abstractive Summarization: A Comprehensive Study of English and Bengali
AI 导读
一项标准化零样本基准在规模匹配的英语与孟加拉语 XL-Sum 测试集上评测了八款闭源与开放权重 LLM。GPT-5.6 Luna 在多项英语词汇指标上领先，但孟加拉语优势未能延续，LLaMA 3.3 70B 取得最高 ROUGE 和 BLEU，Gemini 2.5 Flash 则录得最高孟加拉语事实性得分。不同评测方法的模型排名不一致，说明单指标难以衡量多语言摘要能力。
正文
Abstract:Abstractive summarization remains challenging across languages, where linguistic quality and factual consistency may diverge, particularly in low-resource settings. We present a standardized zero-shot benchmark of eight closed-source and open-weight LLMs on size-matched English and Bengali XL-Sum test sets. Our evaluation combines lexical and semantic metrics with reference-free factuality verification and multidimensional LLM-as-a-Judge assessment, providing a comprehensive analysis of both surface-level quality and factual reliability in multilingual summarization. Results reveal substantial cross-lingual variation: GPT-5.6 Luna leads several English lexical metrics but does not maintain the same advantage in Bengali, where LLaMA 3.3 70B achieves the highest ROUGE and BLEU scores. Gemini 2.5 Flash obtains the highest observed Bengali factuality score. Furthermore, evaluation methods produce different model rankings, highlighting the limitations of relying on a single metric for multilingual summarization assessment. These findings demonstrate that strong English summarization performance does not necessarily transfer to Bengali and emphasize the need for comprehensive evaluation frameworks for low-resource languages.
| Comments: | Accepted in MCT4SD 2026 (Manila, Philippines) |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.19003 [cs.CL] |
| (or arXiv:2610.19003v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19003 arXiv-issued DOI via DataCite |
Submission history
From: Nazifa Tasneem [view email]
[v1]
Tue, 13 Oct 2026 18:05:03 UTC (11 KB)
来源：arXiv：cs.CL · arxiv.org