arXiv：cs.AI· Kausar Moshood, Jace Bolante, Roshan Aditya Murali, Manish Motwani·· 23 小时前
LEST：面向大语言模型的规格驱动测试框架
Specification-Based Testing of Large Language Models
AI 导读
研究者提出 LEST（LLM Evaluation through Specification-driven Testing），将模型卡与系统卡中的安全声明作为黑盒测试规格，自动匹配基准提示词并变异搜索反例。
正文
Abstract:Model cards, system cards, and safety reports increasingly document LLM safety behaviors, yet safety evaluations typically start from predefined benchmarks rather than asking which documented claims those benchmarks actually test. We introduce LEST (LLM Evaluation through Specification-driven Testing), a framework that treats documented safety claims as specifications for black-box testing. In LEST, each claim determines which benchmark prompt tests it and whether a Refuse-to-Comply transition, where a model refuses a prompt but complies with its mutated version, counts as evidence against it. LEST mutates these prompts to search for counterexamples, manually validates each candidate, and traces validated counterexamples back to the claims they contradict. We evaluate LEST using the system cards of Claude Opus 4.8 and GPT-5.5 with 97 seed prompts from SORRY-Bench across Mental Health and Self-Harm, Harmful and Violent Content, and Fairness and Bias. Of 56 identified safety claims, 29 (51.8%) are prompt-testable, but only 17 (58.6%) are exercised by an applicable benchmark prompt. Both models refuse >95% of benchmark prompts. Evaluating 1,836 mutated prompts on both models reveals 312 candidate Refuse-to-Comply transitions flagged by automated response classifiers. However, human validation confirms only 19 (6.1%) as both intent-preserving and genuinely compliant, showing automated evaluation substantially overestimates behavioral failures. Tracing these 19 transitions back to the claims reveals counterexamples for 6 of 9 covered claims for which refusal is an applicable oracle. Our results show that benchmark prompts may not be sufficient to validate all documented LLM safety behavior, and automated failure detection requires manual validation. LEST provides end-to-end traceability from natural-language safety documentation to executable tests and human-validated counterexamples.
| Comments: | 21 pages, 1 figure |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14443 [cs.SE] |
| (or arXiv:2610.14443v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14443 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Manish Motwani [view email]
[v1]
Sat, 10 Oct 2026 01:17:21 UTC (67 KB)
来源：arXiv：cs.AI · arxiv.org