arXiv：cs.CL· Gaurab Baral, Mimansak Nepal, Grishma Basnet·· 4 小时前
NEPSEE：尼泊尔语情感分析的双标注基准
NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis
AI 导读
研究者构建了尼泊尔语情感分析基准 NEPSEE，由两位标注者盲标尼泊尔公众人物推文中的 5,614 句，仅保留极性一致的 3,189 句。三分类任务上，NepBERTa 的 macro-F1 为 0.877，IndicBERTv2 为 0.867，三个商业 LLM 零样本为 0.882 至 0.896，差异不显著。
正文
Abstract:Many sentiment datasets use one annotator for each item, so the test labels show the conventions of that person. We build a Nepali benchmark from two annotators who labelled the same sentences blind. They labelled 5,614 sentences from tweets of Nepali public figures on a five-point scale. We keep only the 3,189 sentences where both give the same polarity. On the three-class benchmark, NepBERTa with a fixed tokenizer gets a macro-F1 of 0.877 and IndicBERTv2 0.867. Three commercial large language models (LLMs) get 0.882 to 0.896 zero-shot, and these differences are not significant. On the five-point scale, 766 sentences have two adjacent labels, for example Pos and SPos. We accept both labels and train on both. On this task, Claude Opus 5.5 beats the best encoders by 0.10 to 0.12 macro-F1. A tokenizer audit adds a second result. The default tokenizers of three Nepali BERT models delete the non-spacing Devanagari vowel signs and the virama. When we restore the marks, macro-F1 increases by 0.084 for NepBERTa but decreases by 0.095 for a model pretrained on stripped text.
| Comments: | 1 figure, 4 tables and 6 pages |
| Subjects: | Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2610.19190 [cs.CL] |
| (or arXiv:2610.19190v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19190 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Gaurab Baral [view email]
[v1]
Tue, 13 Oct 2026 20:54:04 UTC (40 KB)
来源：arXiv：cs.CL · arxiv.org