RoboSPA's 527,000-trajectory VLA benchmark is not yet available

A useful stress test, on paper

RoboSPA, a September 4 preprint led by Zhejiang University researchers, proposes a demanding new test for vision-language-action models. The paper reports approximately 527,000 trajectories, 997 hours of interaction video and 108 million timesteps from five simulated robot embodiments. It defines 56 base manipulation tasks, each expanded across five difficulty levels, to test fine-grained spatial reasoning and long-horizon procedures rather than a single final-success score. That is a useful direction: the task design asks where a model begins to fail as scenes get more ambiguous or a procedure gets longer.^[1]

The numbers nevertheless describe a bounded simulation study. For the main comparison, the authors use clean-scene data from one Aloha-AgileX embodiment, fine-tune a separate model for every base task and run 100 trials for each task variant. Their best reported model, pi0.5, falls from 55.2% success at the easiest level to 22.3% at the hardest; X-VLA falls from 50.4% to 19.9%. The paper reports no physical-robot trials. It therefore supports a claim about difficulty within this controlled benchmark, not a claim that an unchanged policy can operate reliably across robot jobs or workplaces.^[1]

The public links do not yet make it runnable

The more immediate limitation is access. RoboSPA's abstract says its data and code are available at a GitHub address, and the official project page repeats that claim with separate Code and Data links. But the linked GitHub repository's README says that the code and dataset are currently being prepared and will be released soon. Its visible top level carries a website directory, workflow files and that README, rather than the task definitions, collection code, evaluation scripts or trajectory package needed to rerun the paper's central result.^[1,2,3]

The linked Hugging Face dataset page points in the same direction. At access time it reported no data files, an empty README and a total file size of 2.53 kilobytes. This is not evidence that the authors' reported trajectories do not exist, nor an allegation about the research. It is a narrower and decision-changing fact: a reader currently has the paper and its claimed results, but not the public benchmark artifacts needed to inspect the task instances, reproduce the data generation or run the stated evaluation.^[2,4]

Why the access gap changes the result

That distinction matters more here than it would for a small method note. RoboSPA's value proposition is a dataset and an evaluation protocol: it says it isolates failures in target grounding, ordering, repetition and memory as task difficulty rises. The paper gives substantial implementation detail, including simulator-only data collection, manual expert-code trajectories and task-specific training. But a third party cannot yet test whether the source tasks, randomization ranges, filtering of failed trajectory collection, model configurations and result aggregation yield the same conclusions. The current evidence is an author-reported benchmark result, not a publicly reproducible release.^[1,3,4]

The contrast with RoboTwin 2.0 clarifies the practical bar. RoboSPA says it builds on RoboTwin 2.0 and reuses some of its action primitives. RoboTwin's own public repository provides installation documentation, data-collection commands, evaluation guidance and a link to more than 100,000 pre-collected trajectories. That does not validate RoboSPA or make the two benchmarks interchangeable. It does show the difference between a paper whose artefacts can be inspected and run, and a new benchmark whose release pages are still promises. RoboSPA can close that gap; it has not done so yet.^[1,5]

What the result does and does not show

The experimental signal is still worth following. The authors report that every evaluated baseline is below 25% success at the hardest level, and that all four baselines fail on their hardest memory-intensive planning settings. They also introduce a chance-adjusted target-grounding measure and a progress score for partially completed procedures. Those are sensible attempts to distinguish a wrong target choice from a failed grasp, or a nearly finished procedure from a complete one. They remain simulation-specific, task-specific measurements until the artefacts permit a wider technical check.^[1]

The decision delta is therefore not that RoboSPA has disproved current robot models, or that a 527,000-trajectory dataset is necessarily too small or too large. It is that the paper's headline scale cannot yet function as a usable shared benchmark. For a researcher, that means the published accuracy and failure curves are evidence to read, not a score to compare against. For a model team, it means a claimed weakness in memory or spatial grounding cannot yet be tested against the authors' exact implementation. For an operator, none of these simulation figures establish field reliability.^[1,3,4]

The next checkpoint is concrete

The authors' ethics statement says the task definitions, benchmark code, evaluation scripts and collected data will be released under an MIT license. That is the measurable next checkpoint. A credible public release should expose the trajectories, task and success-condition code, simulator and dependency versions, randomization settings, model checkpoints or training recipes, and the evaluation commands behind the reported tables. An independent rerun across more than the clean Aloha-AgileX setup would then test whether the reported collapse survives. Until those materials arrive, RoboSPA is a detailed and timely research claim with a clear access boundary, not an available benchmark.^[1,3,4,5]