arXiv：cs.AI· Junxian Li, Bin Shi, Erfei Cui, Hua Wei, Qinghua Zheng·· 5 小时前
面向 MLP 的图信息蒸馏：三阶段多任务知识蒸馏框架
Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework
AI 导读
针对图神经网络在大规模图数据推理时耗时耗内存的问题，研究者提出一种三阶段多任务知识蒸馏框架，将图知识蒸馏到 MLP 中以减少对图结构的依赖。该方法引入位置编码捕捉位置信息，并采用 Neural Heat Kernels 与隐藏层输出匹配来提升学生 MLP 的性能，作者称这是首个在图上对学生 MLP 做隐藏层蒸馏、并将图位置编码与 MLP 结合的工作。代码已开源。
正文
Abstract:We study the challenging problem for inference tasks on large-scale graph datasets of Graph Neural Networks: huge time and memory consumption, and try to overcome it by reducing reliance on graph structure. Even though distilling graph knowledge to student MLP is an excellent idea, it faces two major problems of positional information loss and low generalization. To solve the problems, we propose a new three-stage multitask distillation framework. In detail, we use Positional Encoding to capture positional information. Also, we introduce Neural Heat Kernels responsible for graph data processing in GNN and utilize hidden layer outputs matching for better performance of student MLP's hidden layers. To the best of our knowledge, it is the first work to include hidden layer distillation for student MLP on graphs and to combine graph Positional Encoding with MLP. We test its performance and robustness with several settings and draw the conclusion that our work can outperform well with good stability. Code implementation is in this https URL.
| Comments: | 20 pages, with Appendix |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2403.01079 [cs.LG] |
| (or arXiv:2403.01079v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2403.01079 arXiv-issued DOI via DataCite |
Submission history
From: Junxian Li [view email]
[v1]
Sat, 2 Mar 2024 03:29:11 UTC (1,780 KB)
[v2]
Sun, 11 Oct 2026 15:31:27 UTC (1,091 KB)
来源：arXiv：cs.AI · arxiv.org