arXiv：cs.CL· Zhongping Ji·· 3 小时前AI 评分36
HySPE：用辛对偶剪切实现位置编码
HySPE: Positional Encoding via Symplectic Dual Shears
AI 导读
研究者提出 Hyperbolic Symplectic Positional Encoding（HySPE），用辛变换的双剪切阻尼对称组合实现位置编码，替代 RoPE 的旋转（椭圆）分支，并以自适应中心执行的分块坐标重基消除表示漂移，前向延迟与缓存 RoPE 相当（RTX 4090 上 7.21ms）。
正文
Abstract:We introduce Hyperbolic Symplectic Positional Encoding (HySPE), grounding positional attention in non-compact symplectic transformations. While canonical Rotary Position Embedding (RoPE) parameterizes the compact, elliptic branch of $\Sp(2,\R)$ via rotations, HySPE operationalizes its hyperbolic branch via a damped symmetric composition of dual shears, yielding a conformally symplectic contraction with two spectral decay rates per channel pair. To eliminate the exponential representation drift inherent to naive absolute factorizations, we diagonalize the operator in its invariant eigenbasis and introduce blockwise coordinate rebasing with adaptive centered execution. This guarantees length-independent numerical bounds while matching cached RoPE forward latency (7.21\,ms on an RTX 4090). On TinyShakespeare, HySPE-UltraLong maintains an invariant perplexity of 4.810 up to $16\times$ zero-shot extrapolation ($L=4096$), whereas RoPE degrades to 131.198. Scaled to a 51M-parameter subword Transformer on WikiText-103 ($L_{\text{train}}=512$), HySPE closely matches RoPE in-domain while robustly extrapolating to length 8192, reducing tail perplexity by 83.9\% over RoPE. While these controlled experiments establish HySPE's extrapolation robustness and numerical stability, evaluating its scaling behavior on large-scale foundation models remains an important direction for future investigation.
| Comments: | 11 pages |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.10154 [cs.CL] |
| (or arXiv:2610.10154v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.10154 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zhongping Ji [view email]
[v1]
Wed, 7 Oct 2026 14:33:35 UTC (87 KB)
来源：arXiv：cs.CL · arxiv.org