FON：一种无需序列级二次注意力的起草-修订双器官语言模型

Authors:Helong Hu (1), Hongdan Pan (2), Shuiqing Hu (3) ((1) Zhanjiang Campus, Guangdong Ocean University, Zhanjiang, Guangdong, China, (2) Maoming Administrative Service Center, Maoming, Guangdong, China, (3) China Pacific Insurance (Group) Co., Ltd. Foshan Branch, Foshan, Guangdong, China)

View PDF

Abstract:Self-attention assigns two functions to one mechanism-where to read and how to mix-at the price of O(L^2) training, an O(T) decode state, and position signals bound to the training length. Existing attention-free alternatives all optimize the single function of generation. We propose FON (Fusion-Organ Network), which differentiates one weight stack into a drafting organ (multi-tap learnable delays + content-adaptive recursion; causal, O(S) training, O(1) decode state, no sequence-level quadratic attention) and a revision organ that re-processes the written prefix through multi-scale ripple kernels injected via weak gates, never generating. At 30M parameters, iso-parameter and iso-token budget, FON reaches perplexity 131.03 on WikiText-103, beating Transformer (197.73) and Mamba2 (152.53); the advantage persists under all depth-aligned baselines and at the 70M tier (-20.8%). At 8x length extrapolation FON degrades by ~8-9% without positional-encoding engineering, between ALiBi and RoPE. The revision channel provides a structural pathway for in-domain infilling that causal architectures lack in principle (+19% over on-demand coupling), at a perplexity cost indistinguishable within measurement precision. We report all costs and boundaries: impaired long-range retrieval, synthetic-task disadvantages, weaker out-of-domain code infilling than a same-recipe Transformer, and a 6.5-12.3x wall-clock gap.

Submission history

From: Helong Hu [view email]
[v1] Mon, 12 Oct 2026 07:28:24 UTC (2,118 KB)