arXiv：cs.AI· Giovanni Pio Delvecchio, Stefano Fantazzini, Giacomo Frisoni, Gianluca Moro, Lorenzo Bacchiani, Mario Bravetti, Thomas Troels Hildebrandt·· 20 小时前
SolBehave：评估 LLM 生成智能合约的行为正确性
SolBehave: Evaluating the Behavioural Correctness of LLM-Generated Smart Contracts
AI 导读
研究者推出 SolBehave 基准，用差分模糊测试、历史交易回放和 DCR 图引导的合成交易回放三种方法，在 250 个真实可复现的 Ethereum 合约上评估 LLM 生成合约的行为正确性。七款开源权重模型经一轮自动修复后编译成功率为 26.4% 至 92.7%，但通过编译的函数体中仅 31.4% 与原合约等价，六款模型在历史回放下有 13% 至 31% 的合约出现偏差。
正文
Abstract:Large language models increasingly generate Solidity smart contracts that, once deployed, can never be patched, making a single latent fault permanent and exploitable. Yet current evaluations mostly ask only whether generated code compiles. We introduce SolBehave, a benchmark for evaluating the correctness of model-generated contracts at scale, moving beyond ad hoc unit tests. Across 250 real, reproducible Ethereum contracts, we apply three complementary tests of behavioural fidelity: differential fuzzing, historical-transaction replay, and, as our core contribution, synthetic-transaction replay guided by a Dynamic Condition Response (DCR) graph that enumerates the space of legitimate function-call sequences. Differential fuzzing compares each generated function with the original on automatically generated inputs; historical replay compares the whole contract on the transactions it actually received on-chain; DCR-guided replay compares it on sequences of calls that follow its intended usage, including orderings that never occurred on-chain. We evaluate seven state-of-the-art open-weight models after one round of automated repair. Compilation success ranges from 26.4% to 92.7%, but under differential fuzzing only 31.4% of all generated function bodies that compile, pooled over models and samples, are equivalent to the original. Divergence at the contract level is the norm rather than the exception: for six of the seven models, between 13% and 31% of the generated contracts that compile diverge from the original under historical replay, and between 13% and 44% under DCR-guided replay; only Gemma-4-31B stays below 3% on both. Compilation is therefore a necessary condition for correctness and far from a sufficient one.
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.18745 [cs.SE] |
| (or arXiv:2610.18745v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18745 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Giacomo Frisoni [view email]
[v1]
Tue, 13 Oct 2026 17:12:59 UTC (3,359 KB)
来源：arXiv：cs.AI · arxiv.org