arXiv：cs.AI· Ziyad Benomar, Weronika {\L}ajewska, Leonardo Perelli, Saab Mansour·· 13 小时前
基于数据驱动Persona的LLM智能体用于A/B测试模拟
Data-Driven Persona-Conditioned Agents for A/B Test Simulation
AI 导读
研究提出一种基于数据驱动persona的LLM智能体模拟框架，用真实用户行为信号预测A/B测试结果，在覆盖两类指标的40个A/B测试基准上，最佳配置取得0.75-0.80的方向准确率。该框架将A/B测试模拟建模为结构化问题任务，系统研究了问题设计格式、persona数据来源与领域对齐的影响，以及单persona行为深度与群体多样性之间的权衡和高效群体子采样。
正文
Abstract:A/B testing is the gold standard for evaluating product changes, but each experiment requires real user traffic, engineering effort, and weeks of measurement. We propose a simulation framework that predicts A/B test outcomes using LLM-powered agents conditioned on data-driven personas grounded in real user behavioral signals. Unlike prior work that relies on synthetic or rule-based personas, our agents are constructed from anonymized behavioral data - activity patterns, engagement signals, and inferred demographics - enabling more faithful population modeling. We frame A/B test simulation as a structured question task and systematically study (i) question design formats, (ii) the impact of persona data source and domain alignment, (iii) the trade-off between per-persona behavioral depth and population diversity, and (iv) efficient population subsampling. On a benchmark of 40 A/B tests spanning two metric types, our best configuration achieves 0.75-0.80 directional accuracy depending on the test metric, demonstrating that data-driven personas are a viable path toward fast, low-cost experiment pre-screening.
| Comments: | Work accepted at EMNLP 2026 Industry Track and EMNLP 2026 REALM Workshop |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.01038 [cs.AI] |
| (or arXiv:2609.01038v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.01038 arXiv-issued DOI via DataCite |
Submission history
From: Weronika Łajewska PhD [view email]
[v1]
Tue, 1 Sep 2026 10:35:49 UTC (270 KB)
[v2]
Fri, 9 Oct 2026 08:06:46 UTC (269 KB)
来源：arXiv：cs.AI · arxiv.org