arXiv：cs.AI· Tolga \c{S}akar·· 4 小时前AI 评分36
Morpheus：面向土耳其语的形态学感知神经分词器与词嵌入模型
Morpheus: A Morphology-Aware Neural Tokenizer and Word Embedder for Turkish
AI 导读
Morpheus 是面向土耳其语的神经语素边界模型，同时充当无损、形态学感知的分词器与词嵌入生成器，通过可微 Poisson-binomial 动态规划将字符边界概率转为语素切分，保证 decode(encode(w))=w。
正文
Abstract:Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents \textbf{Morpheus}, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so $\mathrm{decode}(\mathrm{encode}(w)) = w$ holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character ($1.425$), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1 $0.61$ vs.\ ${\sim}0.32$), and uses ${\sim}19\%$ less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP $0.85$) and same-root verification (ROC-AUC $1.00$), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: this https URL model: this https URL interactive demo: this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.18717 [cs.CL] |
| (or arXiv:2606.18717v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.18717 arXiv-issued DOI via DataCite |
Submission history
From: Tolga Şakar [view email]
[v1]
Wed, 17 Jun 2026 05:55:50 UTC (5,932 KB)
[v2]
Fri, 2 Oct 2026 14:13:54 UTC (5,567 KB)
来源：arXiv：cs.AI · arxiv.org