Tree SAE：在稀疏自编码器中学习层级特征结构

View PDF HTML (experimental)

Abstract:Learning hierarchical features in Sparse Autoencoders (SAEs) is essential for capturing the structured nature of real-world data and mitigating issues like feature absorption or splitting. Existing works attempt to identify hierarchical relationships within independent feature sets by relying on activation coverage, the assumption that child feature should only activate when its parent feature activates. However, we demonstrate that this condition alone is insufficient; that is, it often produces false positives where parent and child concepts are semantically unrelated. To address this, we introduce a novel reconstruction condition that enforces a deeper functional link between hierarchical levels. By combining both activation and reconstruction constraints, we propose the Tree SAE, a model designed to learn hierarchical structures directly from within the feature set. Our results demonstrate that Tree SAEs significantly surpass the existing SAEs at learning hierarchical pairs while maintaining competitive performance to the state-of-the-art on several key benchmarks. Finally, we demonstrate the practical utility of our Tree SAE in mapping the geometry of child feature subspaces and uncovering the complex hierarchical concept structures encoded within large language models.

Comments:	20 pages
Subjects:	Machine Learning (cs.LG)
Cite as:	arXiv:2605.07922 [cs.LG]
	(or arXiv:2605.07922v3 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.07922 arXiv-issued DOI via DataCite

Submission history

From: Tue Minh Cao [view email]
[v1] Fri, 8 May 2026 15:57:37 UTC (5,500 KB)
[v2] Mon, 11 May 2026 02:18:14 UTC (5,500 KB)
[v3] Wed, 7 Oct 2026 03:29:03 UTC (5,479 KB)