arXiv：cs.CL· Dmitrij \.Zatuchin·· 6 小时前
Dice Roll Method：LLM 品牌推荐重复查询审计的标准化协议
The Dice Roll Method: A Standardized Protocol for Repeated-Query Auditing of Large Language Model Brand Recommendations
AI 导读
研究者提出 Dice Roll Method，为 LLM 品牌推荐重复查询审计建立标准化协议，涵盖迭代次数、稳定性指标与可靠性阈值。方法将响应方差分解为采样、提示措辞、运行间与模型版本四部分，重析五项研究约 19 万条观测、270+ 品牌、6 种语言、迭代 5 至 40 次，发现不支持固定层级；在三个独立语料上的预注册外部验证复现 39 个单元中的 37 个。
正文
Abstract:Background: Researchers increasingly use repeated identical prompts to audit stochastic variation in large language model (LLM) brand recommendations, yet no standardized protocol exists for setting iteration counts, selecting stability metrics, or establishing reliability thresholds. Objective: We formalize the Dice Roll Method as a reusable protocol for repeated-query auditing of LLM brand recommendations, grounded in a generative model of temperature-scaled nucleus sampling. Methods: Total response variance is decomposed into sampling, prompt-phrasing, run-to-run, and model-version components. The stack: a negative-binomial mixed model with iterations as repeated measures; Cliff's delta as the distribution-free effect size; dependence-preserving bootstrap; simulation-based power; a generalizability-theory decomposition; drift diagnostics on pinned snapshots. We reanalyse five brand-recommendation auditing studies: approximately 190,000 observations, 270+ brands, 6 languages, iteration counts 5 to 40. Results: The recomputed D-study supports no fixed tiers. Across 250 distinct questions (S5), G is 0.96 at five runs and 1 to 2 runs reach 0.80, also within each model. For five versions of one question (S1 Valentine's subset, Gemini) it is 0.27 (95% CI 0.00 to 0.52), which fixes no run count. Runs follow from a pilot: n = 4 x residual / cell variance. Simulated single-cell power for a large Cliff's delta is 0.15 to 0.19 at five runs and 0.8 at 15 to 25. The four metric families (count, set, embedding, PASOR) are complementary. A pre-registered external validation on three independent corpora reproduces the D-study prediction in 37 of 39 cells with no failures; fixed tiers do not transfer. Conclusion: The protocol gives repeated-query auditing of LLM brand recommendations a statistically principled footing under the conditional, non-Gaussian structure of real autoregressive generation.
| Comments: | 32 pages. v2 corrects the reliability, tier, convergence, gender-contrast, drift and power results of Section 5 and the statements elsewhere that rest on them. Recomputed from the study data. Section 6 results are unchanged. Supersedes Research Square rs-8980233 and rs-9496731 (v1 named rs-8883056 in error) |
| Subjects: | Information Retrieval (cs.IR); Computation and Language (cs.CL) |
| ACM classes: | H.3.3; G.3 |
| Cite as: | arXiv:2609.04047 [cs.IR] |
| (or arXiv:2609.04047v2 [cs.IR] for this version) | |
| https://doi.org/10.48550/arXiv.2609.04047 arXiv-issued DOI via DataCite |
Submission history
From: Dmitrij Żatuchin [view email]
[v1]
Thu, 3 Sep 2026 16:22:06 UTC (94 KB)
[v2]
Wed, 14 Oct 2026 05:40:39 UTC (103 KB)
来源：arXiv：cs.CL · arxiv.org