arXiv：cs.AI· Lining Hu, Ting Liu, Yuzhuo Fu·· 8 小时前
微服务 RCA 基准的汇总排行榜未说明什么：审计方法比较的范围与来源
What a Pooled Leaderboard Does Not Say: Auditing the Scope and Provenance of Method Comparisons in Microservice RCA Benchmarks
AI 导读
一项审计研究在 OpenRCA、RCAEval 和 PetShop 三个微服务 RCA 基准上运行 16 种方法的官方实现，发现汇总排行榜的领先是否在各子系统成立取决于榜单未公布的领先幅度离散度：15 个幅度超过离散度的领先中 14 个处处成立，8 个幅度低于离散度的领先中 6 个发生反转。
正文
Abstract:Offline benchmarks for microservice root cause analysis (RCA) rank methods by a score pooled over several applications, and a pooled lead leaves two things unstated: on which of the systems it holds, and how the score was produced. We audit both on three public benchmark families (OpenRCA, RCAEval, and PetShop), running the released implementations of 16 methods under recorded configurations on 11 subsystems and on a second release of three of the applications. Whether a pooled lead between published methods holds on every subsystem is decided by a quantity the leaderboard withholds, the spread of the lead across subsystems: of 15 leads whose margin exceeds this spread, 14 hold everywhere; of the 8 whose margin is below it, 6 reverse. The per-fault and per-scenario tables that the benchmark papers publish split the same way, and every error of a ranking computed without one subsystem falls on a lead of the second kind. On the second release a third of the comparisons change sign, most through one method's response to the metric kinds it records, and on several subsystems a simple probe or a shipped baseline beats every published method. How the score was produced matters as much: the input table that one released runner reads is not the one its documentation describes, and this choice alone reverses a comparison between published methods with intervals excluding zero both ways; an unpinned dependency and a duplicated column header each make a method return its input column order without an error. Of 23 RCA papers we examined, none releases the per-case outputs needed to check either premise. We recommend that a pooled comparison state the subsystems it covers and its spread across them, the input table, the handling of failed outputs, and the denominator, and be accompanied by per-case outputs, as ours are in our replication package.
| Comments: | 21 pages, 5 figures, 5 tables |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.29159 [cs.AI] |
| (or arXiv:2606.29159v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.29159 arXiv-issued DOI via DataCite |
Submission history
From: Lining Hu [view email]
[v1]
Sun, 28 Jun 2026 02:47:31 UTC (123 KB)
[v2]
Tue, 13 Oct 2026 11:12:49 UTC (221 KB)
来源：arXiv：cs.AI · arxiv.org