arXiv：cs.LG· Kai Li, Hanlin Cai, Zihao Liu, Yichen Li, Jose Alcaraz Calero, Wei Ni, Falko Dressler, \"Ozg\"ur B. Akan·· 11 小时前
基于超图消息传递的拜占庭鲁棒联邦大语言模型 HMP-GAE
Byzantine-Resilient Federated Large Language Model via Hypergraph Message Passing
AI 导读
研究者提出 HMP-GAE 框架，用超图消息传递识别并抑制联邦聚合中被恶意操纵的模型更新，防止语义被污染的知识传播。该框架以 Qwen-2.5 和 Llama-3.2 为预训练 FedLLM 底座，在 AG News 和 Yahoo! Answers 数据集上评测，相比现有防御方案全局准确率提升 1.21%、最终模型分类语义熵降低 44.08%、困惑度降低 13.34%。
正文
Abstract:Federated Large Language Models (FedLLMs) are promising to deliver privacy-preserving, collaborative intelligence. The decentralized training process is vulnerable to Byzantine attacks that corrupt semantic consistency and factual resilience through malicious model updates. This paper presents a new resilience strategy that prevents semantically corrupted knowledge from propagating during federated aggregation. A defense problem is formulated to jointly maximize global accuracy and suppress manipulated model updates, enabling trustworthy knowledge aggregation without accessing private local data. To strengthen the resilience of FedLLMs, we develop a novel Hypergraph Message Passing Graph Autoencoder (HMP-GAE) framework, where low-rank model updates are represented as nodes in a hypergraph and high-order semantic relationships among multiple agents are exploited through iterative message passing. By leveraging historical checkpoints and cross-agent relational reasoning, HMP-GAE identifies and suppresses semantically inconsistent updates while preserving knowledge alignment among agents throughout federated training. HMP-GAE is implemented using Qwen-2.5 and Llama-3.2 as pretrained FedLLM backbones and evaluated on AG News and Yahoo! Answers datasets. Compared to existing defenses, HMP-GAE improves global accuracy by $1.21$\%, reduces final-model classification semantic entropy by $44.08$\%, and lowers perplexity by $13.34$\%. HMP-GAE outperforms the GAE-based defense by improving the accuracy $1.39$\%, reducing the semantic entropy by $54.5$\%, and lowering perplexity by $35.5$\%.
| Subjects: | Machine Learning (cs.LG); Cryptography and Security (cs.CR) |
| Cite as: | arXiv:2610.12711 [cs.LG] |
| (or arXiv:2610.12711v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12711 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hanlin Cai [view email]
[v1]
Thu, 8 Oct 2026 20:47:08 UTC (5,214 KB)
来源：arXiv：cs.LG · arxiv.org