arXiv：cs.CL· Giuseppe Attanasio, Beatrice Savoldi, Daniel Chechelnitsky, Matteo Negri, Marine Carpuat, Maarten Sap, Andr\'e F. T. Martins·· 16 小时前
Ouvia：面向真实沟通场景的语音翻译可用性评估框架
Ouvia: A User-centered Framework for Measuring Usability of Speech Translation in Real-World Communication Scenarios
AI 导读
研究团队提出 Ouvia，一个衡量真实场景下语音翻译输出用户感知可用性的评估框架，聚焦英语使用者向葡萄牙语使用者传达请求的一对一沟通。通过定制 Web 应用和多阶段研究，在医疗与日常情境中采集超 1,750 次交互，涉及四个 ST 系统和三种英语方言、两种性别的说话人。结果显示现代 ST 仅约半数交互被评为可用，不同人群间可用性存在显著差距，且基于 QA 的评估比标准方法更能预测真实可用性。
正文
Abstract:Speech translation (ST) is increasingly adopted in user applications, yet its evaluation largely focuses on decontextualized testbeds and holistic quality, rather than end users' communication needs. We introduce Ouvia, an evaluation framework for measuring user-perceived usability of speech translation outputs in real-world settings. Ouvia focuses on one-to-one communication: an English speaker needs to convey a request to a Portuguese speaker, and the message is automatically translated. Through a custom web app and multi-phase study design, we collect more than 1,750 such interactions in healthcare and everyday situations, mediated by four ST systems, involving speakers from three English dialects and two genders. We find that modern ST serves people only to a limited extent---only around half of interactions are rated as usable---with significant gaps in reported usability across demographic groups. Moreover, among quality metrics, we find that QA-based evaluation is a substantially stronger predictor of real-world usability than standard approaches. Together, these findings stress the importance of situated, user-centered evaluation frameworks that go beyond holistic quality scores and attend to who the technology serves---and how well.
| Comments: | Code and data at this https URL -- Accepted at EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2606.06177 [cs.CL] |
| (or arXiv:2606.06177v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.06177 arXiv-issued DOI via DataCite |
Submission history
From: Giuseppe Attanasio [view email]
[v1]
Thu, 4 Jun 2026 13:52:21 UTC (221 KB)
[v2]
Wed, 14 Oct 2026 16:56:04 UTC (229 KB)
来源：arXiv：cs.CL · arxiv.org