arXiv：cs.CL· Ke Chen, Xucheng Yu, Yufei Zhou, Haohan Wang·· 16 小时前
Prompt Stability Matters：通用系统中自动生成提示词的稳定性评估与优化
Prompt Stability Matters: Evaluating and Optimizing Auto-Generated Prompt in General-Purpose Systems
AI 导读
研究提出"语义稳定性"指标，用于衡量 LLM 在重复执行中对同一提示词响应的一致性，并据此构建了首个稳定性感知的通用提示词生成系统，通过稳定性反馈迭代提升提示词质量与系统级表现。作者通过分析系统内部结构依赖，论证了提示词稳定性是有效系统级执行的必要条件，在通用和领域特定任务上均提升了准确率与输出一致性。系统 UI 与源码已公开。
正文
Abstract:Automatic prompt generation plays a crucial role in enabling general-purpose multi-agent systems to perform diverse tasks autonomously. Existing methods typically evaluate prompts based on their immediate task performance, overlooking the intrinsic qualities that determine their reliability. This outcome-centric view not only limits interpretability but also fails to account for the inherent stochasticity of large language models (LLMs). In this work, we bring attention to prompt stability-the consistency of model responses across repeated executions-as a key factor for building robust and effective prompt generation systems. To quantify this, we propose semantic stability as a criterion for assessing the response consistency of prompts. Based on the proposed metric, we developed the first stability-aware general-purpose prompt generation system that leverages stability feedback to iteratively enhance both prompt quality and system-level performance. Furthermore, we establish a logical chain between prompt stability and task success by analyzing the structural dependencies within our system, proving stability as a necessary condition for effective system-level execution. Empirical results across general and domain-specific tasks demonstrate that our stability-aware framework improves both accuracy and output consistency. By shifting the focus from one-off results to persistent reliability, our work offers a new perspective on prompt design and contributes practical tools for building more trustworthy general-purpose systems. The UI and source code for our system are publicly available at: this https URL.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Multiagent Systems (cs.MA) |
| Cite as: | arXiv:2505.13546 [cs.AI] |
| (or arXiv:2505.13546v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2505.13546 arXiv-issued DOI via DataCite |
Submission history
From: Ke Chen [view email]
[v1]
Mon, 19 May 2025 03:28:33 UTC (527 KB)
[v2]
Tue, 13 Oct 2026 18:09:51 UTC (626 KB)
来源：arXiv：cs.CL · arxiv.org