arXiv：cs.CL· Tao Feng, Jiefeng Chen, Linxin Song, Yufan Zhuang, Yue Huang, Bhavana Dalvi Mishra, Chi Wang, Xiangru Tang, Jieyu Zhao, Ge Liu, Tomas Pfister, Jinsung Yoon·· 4 小时前
T2J-BENCH：在可执行验证的 PyTorch 到 JAX 仓库迁移上评测编码智能体
T2J-BENCH: Benchmarking Coding Agents on Execution-Grounded PyTorch-to-JAX Repository Migration
AI 导读
T2J-BENCH 是覆盖 18 个模型族、63 个整仓库 PyTorch 转 JAX 迁移任务的基准，以无 torch 的离线验证器按执行结果判定功能等价，torch-free 检查点转换也属交付内容。
正文
Authors:Tao Feng, Jiefeng Chen, Linxin Song, Yufan Zhuang, Yue Huang, Bhavana Dalvi Mishra, Chi Wang, Xiangru Tang, Jieyu Zhao, Ge Liu, Tomas Pfister, Jinsung Yoon
Abstract:Migrating a machine-learning codebase across frameworks--for example from PyTorch to JAX--is a routine but costly engineering task that is fundamentally cross-paradigm: eager, stateful, object-oriented modules must be rewritten as pure, jittable functions over an explicit parameter pytree. We ask whether coding agents can perform such whole-repository migrations correctly, and argue that existing code benchmarks, which grade unit-test pass rates or surface-level similarity to a reference translation, cannot answer this: they neither require the migrated system to \emph{behave} the same nor prevent a JAX reimplementation from faking equivalence by secretly importing PyTorch itself. We introduce T2J-BENCH, a benchmark of 63 whole-repository PyTorch-to-JAX migration tasks spanning 18 model families across both training and serving, where torch-free checkpoint conversion is itself part of the deliverable. T2J-BENCH grades execution-grounded functional equivalence: each candidate is run inside a separate, torch-free, offline verifier and scored against references frozen from the PyTorch original, under a hard independence gate and a continuous, threshold-free reward. Across twenty off-the-shelf agent--model configurations, the strongest recovers only 0.676 of the reward against a 0.980 oracle, with substantial numeric and behavioral divergence despite high spec pass rates. Finally, we contribute TWINHARNESS, a reference-grounded agent harness that turns the agent's executable PyTorch source into legitimate self-verification feedback; applied to an opus-5 agent it raises the reward from 0.649 to 0.723--the highest of any configuration--using only task-visible resources.
| Subjects: | Software Engineering (cs.SE); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.29054 [cs.SE] |
| (or arXiv:2605.29054v3 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2605.29054 arXiv-issued DOI via DataCite |
Submission history
From: Jiefeng Chen [view email]
[v1]
Wed, 27 May 2026 19:57:15 UTC (495 KB)
[v2]
Wed, 3 Jun 2026 18:44:55 UTC (495 KB)
[v3]
Wed, 14 Oct 2026 01:20:45 UTC (9,352 KB)
来源：arXiv：cs.CL · arxiv.org