arXiv：cs.AI· Yang Ba, Michelle V. Mancenido, Rong Pan·· 16 小时前
PLDiv：通过持久性景观度量数据集多样性与语义坍缩
PLDiv: Measuring Dataset Diversity and Semantic Collapse via Persistence Landscapes
AI 导读
研究者提出 PLDiv，一种基于持久性景观的多样性度量，从数据集的几何表示刻画其结构组织，输出可解释的标量多样性分数，无需任务特定监督。在文本生成与图像嵌入的内在多样性测量，以及指令数据集和推理轨迹中的语义坍缩检测上，PLDiv 表现与所评估基线相当，并在语义坍缩时与多样性标签高度一致。该指标可支持合成数据监控、数据集构建、数据增强与生成模型评估。
正文
Abstract:Modern data pipelines increasingly combine human-created and human-curated data with AI-generated data for model training and evaluation. As datasets grow, they often add semantically redundant samples rather than expanding semantic coverage, which deteriorates the quality of the whole dataset. Existing diversity metrics primarily summarize lexical variation, entropy, or pairwise similarity, but provide limited characterization of the shape of the underlying data distribution. In this paper, we propose PLDiv, a diversity metric derived from persistence landscapes that quantifies the structural organization of a dataset from its geometric representation. PLDiv summarizes how data neighborhoods evolve as nearby samples progressively connect, producing an interpretable scalar measure of structural diversity without requiring task-specific supervision. Whether measuring intrinsic diversity in text generations or image embeddings, or detecting semantic collapse in instruction datasets and reasoning trajectories, PLDiv performs competitively against evaluated baselines and aligns strongly with diversity labels during semantic collapse. Consequently, it effectively supports applications in synthetic data monitoring, dataset construction, data augmentation, and generative model evaluation.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.14031 [cs.LG] |
| (or arXiv:2610.14031v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14031 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yang Ba [view email]
[v1]
Wed, 30 Sep 2026 23:49:37 UTC (6,763 KB)
来源：arXiv：cs.AI · arxiv.org