arXiv：cs.CL· Nadhem Zmandar, Mo El-Haj, Paul Rayson·· 3 小时前AI 评分32
基于 Transformer 的长文档金融叙事摘要评估指标对比研究
A Comparative Study of Evaluation Metrics for Long-Document Financial Narrative Summarization with Transformers
AI 导读
研究在 FNS 2020 数据集上对比了多种预训练 Transformer 与抽取方法的长文档金融摘要效果，该数据集由伦敦证券交易所上市公司的年报及对应摘要构成。作者认为部分评估指标无法反映真实摘要能力，提出由 ROUGE-2 与 BERTscore 调和平均构成的新指标 BRUGEscore，并对结果做统计显著性检验和三种损坏方法的对抗分析。
正文
Abstract:There are more than 2,000 listed companies on the UK's London Stock Exchange, divided into 11 sectors who are required to communicate their financial results at least twice in a single financial year. UK annual reports are very lengthy documents with around 80 pages on average. In this study, we aim to benchmark a variety of summarisation methods on a set of different pre-trained transformers with different extraction techniques. In addition, we considered multiple evaluation metrics in order to investigate their differing behaviour and applicability on a dataset from the Financial Narrative Summarisation (FNS 2020) shared task, which is composed of annual reports published by firms listed on the London Stock Exchange and their corresponding summaries. We hypothesise that some evaluation metrics do not reflect true summarisation ability and propose a novel BRUGEscore metric, as the harmonic mean of ROUGE-2 and BERTscore. Finally, we perform a statistical significance test on our results to verify whether they are statistically robust, alongside an adversarial analysis task with three different corruption methods.
| Comments: | 12 pages |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.09529 [cs.CL] |
| (or arXiv:2610.09529v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09529 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Natural Language Processing and Information Systems. NLDB 2023 |
| Related DOI: | https://doi.org/10.1007/978-3-031-35320-8_28
DOI(s) linking to related resources |
Submission history
From: Mo El-Haj [view email]
[v1]
Wed, 7 Oct 2026 06:29:34 UTC (174 KB)
来源：arXiv：cs.CL · arxiv.org