arXiv：cs.LG· Ao Kuniya, Jun Ohkubo·· 3 小时前AI 评分28
基于逆激活回归的神经元合并：sigmoid 神经网络训练后压缩方法
Neuron merging via inverse-activation regression for post-training compression of sigmoid neural networks
AI 导读
该论文提出面向 sigmoid 神经网络的神经元合并压缩方法，通过逆激活函数将神经元响应映射回预激活空间，并用最小二乘法估计代表神经元的权重与偏置。方法同时考察了使用真实训练输入的数据辅助策略与使用随机生成输入的无数据策略，并对比了无数据贡献加权平均法。实验表明，在测试的 sigmoid 网络中，权重信息对聚类尤为有用，而激活信息则有助于合并过程中的代表神经元重建。
正文
Abstract:As neural networks continue to grow in scale, model compression is becoming increasingly important for efficient inference under limited computational resources. Structured pruning methods remove neurons or channels that are estimated to be less important, but the removed units may still contain useful information. From the viewpoint of coarse-graining a trained network, it is valuable to ask which information should be retained when multiple neuronal degrees of freedom are consolidated. In this paper, we discuss cluster-based merging methods for compression of trained neural networks. In addition to a data-free contribution-weighted averaging method, we propose neuron-merging methods in which neuron responses are mapped back to the pre-activation space via the inverse activation function, and the weights and biases of each representative neuron are estimated using the least-squares method. We also examine both a data-assisted strategy with actual training inputs and a data-free strategy using randomly generated inputs. The comparisons provide empirical evidence, in the tested sigmoid networks, that weight information is particularly useful for clustering whereas activation information is useful for representative-neuron reconstruction in the merging process.
| Comments: | 9 pages, 7 figures |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.02559 [cs.LG] |
| (or arXiv:2610.02559v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02559 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jun Ohkubo [view email]
[v1]
Thu, 1 Oct 2026 22:51:42 UTC (460 KB)
来源：arXiv：cs.LG · arxiv.org