arXiv：cs.LG· Fabien Furfaro·· 11 小时前
Kairos：面向全模态的字节级基础模型
Kairos: Toward a Byte-Level Foundation Model for Every Modality
AI 导读
Kairos 尝试用字节级基础模型统一处理文本、图像、音频、视频、激光雷达和控制流，架构结合双向门控 DeltaNet 线性注意力、滑动窗口注意力（LiZAttention2）、块级 Attention-Residual 聚合、稀疏 MoE 前馈层与多尺度字节级编解码器，并以统一噪声去噪目标训练。
正文
Abstract:Small language models trained on a single local machine leave little room for architectural trial and error. Kairos combines bidirectional gated-DeltaNet linear attention and sliding-window attention over shared projections (LiZAttention2), block-wise Attention-Residual aggregation, a sparse Mixture-of-Experts (MoE) feed-forward layer and a multi-scale byte-level codec, trained with a uniform-noise denoising objective over one byte space shared by text, image, audio, video, lidar and control streams; this report asks whether the combination optimizes at all before asking whether it is useful. We give code-derived algorithms for each component, including a cross-session recurrent-state memory gate, then run convergence sanity checks: overfittinga single 440-byte sentence at 1--15M parameters, text only, with three seeds for codec and attention choices and single-run sweeps for seven further factors. The two attention variants tested at 3\,000 steps (LiZAttention2 and window attention alone) reach a minimum loss near $10^{-3}$. After 600 steps, the mean minimum loss ranges from 0.018 (patch codec) to 0.354 (depthwise convolutional codec) with an identical ranking on three seeds. An apparent advantage of shared over separate attention projections closes with more steps or more diverse data and reverses when the projections are frozen: it is a convergence-speed effect, not a capacity effect. These checks suggest that the implementation optimizes; they say nothing about generalization, multimodal training or comparison with existing models, which a planned pretraining run is meant to address. The Discussion sizes a 200M-total/25M-active target analytically; none of its throughput figures is measured. Code is available at this https URL . Python package at this https URL .
| Comments: | 15 pages |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.12495 [cs.LG] |
| (or arXiv:2610.12495v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12495 arXiv-issued DOI via DataCite |
Submission history
From: Fabien Furfaro [view email]
[v1]
Sat, 26 Sep 2026 13:12:12 UTC (74 KB)
来源：arXiv：cs.LG · arxiv.org