The result hides a small physical sample
Track4Action, a preprint submitted on August 4 by researchers affiliated with Zhejiang University, Shanghai Jiao Tong University, Shanghai Innovation Institute and Noematrix, trains a vision-language-action policy with a frozen three-dimensional tracker. The tracker reads the video sequence paired with an action demonstration during training; the policy then runs from its current camera observations, instruction and robot state without the tracker or future video.[1]
The headline physical result is 67.5 percent average success across four bimanual tasks. The paper says each task used 50 demonstrations and that every compared policy received 10 trials per task. That makes the reported average 27 full completions in 40 attempts, after 200 task-specific demonstrations—not a broad field trial. The tasks were chili transfer, towel folding, putting two pens into a drawer, and placing a cabbage in a pot before closing its lid.[1]
The contribution is a training signal, not a new runtime sensor
The useful distinction is architectural. Track4Action does not ask a robot to run a three-dimensional tracker while working. It uses the realized transition in a demonstration video to teach an action-facing representation what changed in the scene, then removes that teacher at deployment. That could make already-collected, time-ordered robot demonstrations more informative without adding tracker inference to the operating robot.[1]
The authors also report 82.3 percent on the zero-shot LIBERO-Plus simulation benchmark and 80.44 percent on RoboTwin 2.0’s clean split and 81.48 percent on its randomized split. RoboTwin 2.0 is a 50-task, dual-arm simulation benchmark with three camera views and an upstream software, dataset and evaluation stack; these are author-reported comparisons, not independent reproductions of Track4Action.[1,3]
The reproducibility check is still open
The paper identifies the main technical limits: it needs temporally ordered demonstrations, inherits tracking failures under occlusion or fast motion, uses only a primary camera for the teacher, and tests physical transfer on one bimanual platform. Its project page labels buttons for checkpoints and code, but both links return to that same page. At review, the page therefore provided no linked repository, weights or evaluation artifact for an outside group to run.[1,2]
The decision-changing result is narrower than a 67.5 percent headline suggests: training-time motion supervision may be worth testing for video-rich robot datasets, but the physical result is 40 attempts on four tasks and cannot yet establish cross-embodiment reliability. The next measurable checkpoint is a public training and evaluation release, followed by independent physical tests on additional robots, camera layouts and non-visual disturbances.[1,2,3]