arXiv：cs.CL· Nevidu Jayatilleke, Nisansa de Silva, Uthpala Nimanthi, Gagani Kulathilaka, Azra Safrullah, Johan Sofalas·· 4 小时前AI 评分32
SiDiaC-v.2.0：僧伽罗语历时语料库 2.0 版发布
SiDiaC-v.2.0: Sinhala Diachronic Corpus Version 2.0
AI 导读
SiDiaC-v.2.0 是目前规模最大的僧伽罗语历时语料库，收录 185 部文学作品、共 22.9 万词，出版日期覆盖 1800 至 1955 年，书写年代跨越 5 至 20 世纪。
正文
Abstract:SiDiaC-v.2.0 is the largest comprehensive Sinhala Diachronic Corpus to date, covering a period from 1800 CE to 1955 CE in terms of publication dates, and a historical span from the 5th to the 20th century CE in terms of written dates. The corpus consists of 229k words across 185 literary works that underwent thorough filtering, preprocessing, and copyright compliance checks, followed by extensive post-processing. Additionally, a subset of 59 documents totalling 65k words was annotated based on their written dates. Texts from the National Library of Sri Lanka were selected from the SiDiaC-v.1.0 non-filtered list, which was digitised using Google Document AI OCR. This was followed by post-processing to correct formatting issues, address code-mixing, include special tokens, and fix malformed tokens. The construction of SiDiaC-v.2.0 was informed by practices from other corpora, such as FarPaHC, SiDiaC-v.1.0, and CCOHA. This was particularly relevant for syntactic annotation and text normalisation strategies, given the shared characteristics of low-resource language status between Faroese and the similar cleaning strategies utilised in CCOHA. This corpus is categorised into two layers based on genres: primary and secondary. The primary categorisation is binary, assigning each book to either Non-Fiction or Fiction. The secondary categorisation is more detailed, grouping texts under specific genres such as Religious, History, Poetry, Language, and Medical. Despite facing challenges due to limited resources, SiDiaC-v.2.0 serves as a comprehensive resource for Sinhala NLP, building upon the work previously done in SiDiaC-v.1.0.
| Comments: | 24 pages, 13 figures, 10 tables, Accepted paper at the 15th Language Resources and Evaluation Conference (LREC 2026) |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2603.10861 [cs.CL] |
| (or arXiv:2603.10861v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2603.10861 arXiv-issued DOI via DataCite |
|
| Related DOI: | https://doi.org/10.63317/2ybff4o55vrq
DOI(s) linking to related resources |
Submission history
From: Nevidu Jayatilleke Mr. [view email]
[v1]
Wed, 11 Mar 2026 15:10:32 UTC (13,205 KB)
[v2]
Tue, 29 Sep 2026 09:01:33 UTC (2,554 KB)
[v3]
Fri, 2 Oct 2026 15:21:42 UTC (2,555 KB)
来源：arXiv：cs.CL · arxiv.org