arXiv：cs.LG· Prabhath Chellingi, Raviraja G, Viraj Bagal·· 4 小时前AI 评分41
从看似合理的层级到有用的分类体系：评估智能体框架在客户反馈上的表现
From Plausible Hierarchies to Useful Taxonomies: Evaluating Agentic Harnesses on Customer Feedback
AI 导读
研究构建了六个客户反馈分类体系（两个语料库各含 1,940 和 5,000 条记录），全部通过通用命名与结构检查，但生成树中至少 97.7% 的叶子名称只是复述祖先名称（参考树为 13.9% 和 2.9%）。作者提出结构判别器和团队可分区性两类全树指标，发现被通用检查评为同等正确的树在跨分支泄漏上相差 27 个百分点，仅一棵优于随机划分。
正文
Abstract:Taxonomies are the symbolic representations through which AI systems organize evidence, aggregate patterns, and answer questions over large document collections. Over customer feedback, the category tree decides how every record is counted and routed, which problems get seen, and which team owns them. Agentic harnesses now make it easy to generate a plausible-looking hierarchy, and such trees are checked today with generic, individually scoped checks: each name fits its description, sits under the right parent, and stays distinct from its siblings. We ask a more operational question: when is a generated hierarchy actually useful as a production taxonomy? We build six taxonomies over two proprietary feedback corpora (1,940 and 5,000 records): for each corpus, a production reference and two repeated runs of the same harness under identical inputs. All six pass every generic naming and structure check, and a deeper product-coverage check even prefers the generated trees. Yet in every generated tree at least 97.7% of leaf names merely restate an ancestor's name (13.9% and 2.9% in the references), and in one, three of every four records fall under multiple top-level categories. We introduce two families of whole-tree metrics: structural discriminators test whether a tree's shape was learned from the data or imposed by its generator; team partitionability tests whether branches split feedback into groups teams can own. Trees the generic checks rate as equally correct differ by 27 percentage points in cross-branch leakage, and only one of the two beats a random split. Surface plausibility is an insufficient measure of taxonomy quality: evaluation must measure the whole tree as well as each node.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Symbolic Computation (cs.SC) |
| Cite as: | arXiv:2610.09377 [cs.AI] |
| (or arXiv:2610.09377v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09377 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Prabhath Chellingi [view email]
[v1]
Wed, 7 Oct 2026 03:32:07 UTC (34 KB)
来源：arXiv：cs.LG · arxiv.org