arXiv：cs.CL· Chihiro Yano, Jun Hirako, Hiroaki Yamagiwa, Ryota Hirobuchi·· 7 小时前
J-tau：面向工具型智能体与用户交互的完全本地化日语基准
J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction
AI 导读
研究者发布 J-tau，首个公开的日语基准，用于评估交互式客服场景中的工具调用 LLM 智能体。该基准通过机器翻译、实体适配与人工审核对 tau-bench 进行本地化，并修复了原基准的无效任务与分数聚合错误两个缺陷。实验显示 J-tau 的评估错误少于简单机器翻译版本。
正文
Abstract:We present J-tau, the first publicly available Japanese benchmark for evaluating tool-using LLM agents in interactive customer-support scenarios. While tau-bench is widely used to evaluate LLM agents, it is English-centric and cannot evaluate agents in other languages. Existing multilingual extensions offer no public Japanese version and rely primarily on machine translation, leaving locale-specific entities such as addresses and currencies inconsistent with the target region. To enable reliable evaluation in Japanese, we carefully localize tau-bench through machine translation, entity adaptation, and human review. During this process, we also uncover and fix two flaws in the original tau-bench: invalid tasks and a score aggregation error that inflates certain models' results. Experiments show that J-tau yields fewer evaluation errors than a naively machine-translated counterpart and that our fixes resolve both flaws.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19638 [cs.CL] |
| (or arXiv:2610.19638v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19638 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Chihiro Yano [view email]
[v1]
Wed, 14 Oct 2026 06:07:40 UTC (498 KB)
来源：arXiv：cs.CL · arxiv.org