arXiv：cs.CL· Natapong Nitarach, Phume Ngampornsukswadi, Pittawat Taveekitworachai, Surapon Nonesung, Warit Sirichotedumrong, Kunat Pipatanakul·· 13 小时前
AMLworld-Compact：用重要性加权降采样降低 LLM 评测成本并诊断错误
AMLworld-Compact: Importance-Weighted Downsampling for Cost-Effective LLM Evaluation and Error Diagnosis
AI 导读
AMLworld-Compact 是一套保留全部非法测试交易、仅从有监督分数分层中采样良性交易的 LLM 评测集，其 HT-Coreset 协议在 HI-Small 和 LI-Small 上分别将评测目标减少 271 倍和 611 倍。
正文
Abstract:Evaluating large language models (LLMs) on AMLworld's smallest test splits is costly because each of over a million targets requires a separate graph prompt. Sampling reduces this cost, but small balanced subsets change unweighted precision and F1. We introduce AMLworld-Compact, an evaluation set that retains every illicit test transaction and samples benign transactions from supervised-score strata. Its HT-Coreset protocol reduces evaluation targets by 271x and 611x on HI-Small and LI-Small. Horvitz-Thompson weights give unbiased full-test confusion-count estimates for fixed predictions; recall is exact, while precision and F1 are ratio estimates. In a held-out study using Graph Feature Preprocessor (GFP) features, one boosted-tree family defines strata for evaluating the other over 500 draws per configuration. At the released sizes, F1 RMSE is 0.00-2.58 percentage points versus 4.44-7.41 for uniform sampling with HT weights; conservative intervals remain broad. Under the evaluated graph prompts, seven open-weight LLMs have weighted precision near the predict-all-illicit baseline. In an outcome-stratified audit of 1,000 HI-Small few-shot outputs from one run per model, lexical checks pass for 84.5% of correct and 56.2% of incorrect binary verdicts, so these markers do not establish correctness. We release the evaluation sets, construction code, and audit tools at this https URL and this https URL.
| Comments: | 28 pages, 5 figures, 27 tables. Under review |
| Subjects: | Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2610.13627 [cs.CL] |
| (or arXiv:2610.13627v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13627 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Natapong Nitarach [view email]
[v1]
Fri, 9 Oct 2026 14:09:32 UTC (171 KB)
来源：arXiv：cs.CL · arxiv.org