arXiv：cs.AI· Jingnan Zheng, Yanzhen Luo, Jingjun Xu, Bingnan Liu, Yuxin Chen, Chenhang Cui, Gelei Deng, Chaochao Lu, Xiang Wang, An Zhang, Tat-Seng Chua·· 21 小时前
Risky-Bench：评估真实生活辅助场景中智能体的安全风险
Risky-Bench: Evaluating Agentic Safety Risks in Real-World Life-Assist Deployments
AI 导读
Risky-Bench 是一个面向生活辅助部署的智能体安全评测框架，从领域无关安全原则推导情境化安全准则，并在分层攻击面上通过真实长程任务执行探测风险。该框架覆盖外卖、到店协助与在线旅行预订场景，对 7 个 SOTA 智能体在 750 项任务上评测，攻击成功率为 25% 至 60%，暴露出威胁模型相关漏洞、攻击面风险集中及显式推理的模型相关效应等系统性失败模式。代码与数据已公开。
正文
Authors:Jingnan Zheng, Yanzhen Luo, Jingjun Xu, Bingnan Liu, Yuxin Chen, Chenhang Cui, Gelei Deng, Chaochao Lu, Xiang Wang, An Zhang, Tat-Seng Chua
Abstract:Large Language Models (LLMs) are increasingly deployed as agents that operate in real-world environments, introducing safety risks beyond linguistic harm through tool use, external transactions, and multi-step decision making. Existing agent safety evaluations construct risk-oriented tasks that lack a hierarchically organized taxonomy of attack surfaces, rely on predefined harm categories rather than deployment-grounded safety rubrics, and evaluate agent behavior in isolation rather than during realistic, long-horizon task execution. To address these limitations, we propose Risky-Bench, a deployment-grounded framework for systematic agent safety evaluation in life-assist settings. Risky-Bench derives context-aware safety rubrics from domain-agnostic safety principles and systematically probes safety risks across hierarchically organized attack surfaces through realistic task execution under varying threat assumptions. Applied to life-assist scenarios covering food delivery, in-store assistance, and online travel booking, Risky-Bench evaluates seven state-of-the-art agents across 750 tasks, revealing attack success rates ranging from 25% to 60% and exposing systematic failure patterns including threat-model-specific vulnerabilities, attack-surface-dependent risk concentration, and model-dependent effects of explicit reasoning. Our code and data are available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.03100 [cs.AI] |
| (or arXiv:2602.03100v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2602.03100 arXiv-issued DOI via DataCite |
Submission history
From: Jingnan Zheng [view email]
[v1]
Tue, 3 Feb 2026 04:44:11 UTC (2,570 KB)
[v2]
Sun, 11 Oct 2026 12:03:51 UTC (1,884 KB)
来源：arXiv：cs.AI · arxiv.org