{ "schemaVersion": 1, "slug": "robot-benchmark-scores-trial-counts-evidence", "status": "published", "articleType": "research-report", "desk": "frontier", "title": "Robot benchmark scores need a test ledger", "dek": "A new physical RoboDojo result exposes the limits of a single average. We reconcile task bundles, trial budgets and paired comparisons before treating scores as pilot evidence.", "author": { "byline": "Imani Reed, Frontier Desk" }, "dates": { "publishedAt": "2026-10-26T09:20:45.860Z", "updatedAt": "2026-10-26T09:20:45.860Z" }, "hero": { "src": "/images/articles/robot-benchmark-scores-trial-counts-evidence.webp", "alt": "Archive photograph of a white Franka Emika robot arm above a demonstration table at CeBIT 2017.", "credit": "Archive photo: Ims / Wikimedia Commons, 23 March 2017, CeBIT Hannover. CC BY-SA 4.0; cropped and resized.", "kind": "licensed-photo", "sourceUrl": "https://commons.wikimedia.org/wiki/File:Franka_Emika2.jpg", "rights": "Ims, own work via Wikimedia Commons, CC BY-SA 4.0. Crop and resize permitted; this derivative photograph retains CC BY-SA 4.0: https://creativecommons.org/licenses/by-sa/4.0/. Original and creator linked in source s 7; no endorsement implied." }, "seo": { "title": "Robot benchmark scores need a test ledger", "description": "RoboDojo, RoboArena, Macquarie and CALVIN show why task bundles, trial counts and stopping rules matter before using robot benchmark scores.", "keywords": [ "robot benchmark evidence", "RoboDojo physical evaluation", "robot trial counts", "RoboArena comparisons" ] }, "machine": { "summary": "A new RoboDojo physical result varies across three hardware/task bundles. We reconstruct its physical and simulation test plans, then reconcile RoboArena’s paired comparisons with individual rollouts. Macquarie and CALVIN add stopping-rule and sequence limits. Three original figures and a CSV show which units a pilot evaluation must preserve, without pooling incompatible scores or claiming independent reproduction.", "keyFacts": [ "The 24 October RoboDojo snapshot reports InternW0-Δ success rates of 50.0%, 31.7% and 13.3% across three different embodiment/task bundles.", "The physical protocol has 6 tasks×10 trials×3 embodiments=180 trials; the separate three-seed simulation plan has 42×50×3=6300 episodes.", "RoboArena 612 paired comparisons imply 1224 A/Brollouts plus 3060 other-policy rollouts, matching 4284 oracle evaluations.", "Task definitions, stopping rules and scene resets must match before benchmark averages guide a pilot comparison." ], "entities": [ "RoboDojo", "InternW0-Δ", "RoboArena", "Macquarie University", "CALVIN", "DROID", "Franka Emika" ] }, "signal": { "eventType": "research-result", "stage": "not-applicable", "topics": [ "data-benchmarks", "robot-intelligence", "deployment-operations" ], "industries": [ "cross-industry" ], "companies": [ "Shanghai AI Lab", "RoboDojo", "RoboArena", "Macquarie University", "University of Freiburg" ], "publicListings": [], "regions": [ "China", "Australia", "United States", "Germany" ] }, "body": [ { "type": "heading", "text": "Start with the unit behind the score" }, { "type": "paragraph", "text": "A robot-policy average is useful only when its test unit is clear. On 21 October, RoboDojo added InternW0-Δ to its physical leaderboard. The 24 October snapshot reports a 38.9 progress score and 31.7% task success rate. Those are different measures, and neither is a claim about hours of unattended work at a customer site. The new result provides a reason to examine the ledger underneath the headline.", "sourceIds": [ "s1", "s2" ], "evidence": "analysis" }, { "type": "heading", "text": "One average contains three different task bundles" }, { "type": "paragraph", "text": "The same entry reports 50.0% success for the ARX X5 bundle, 31.7% for Piper and 13.3% for Piper X. The highest-to-lowest spread is 36.7 percentage points. But the benchmark assigns different tasks to the different embodiments. This panel cannot isolate a hardware effect, and it does not show the same task transferring successfully between three arms.", "sourceIds": [ "s1", "s2" ], "evidence": "analysis" }, { "type": "figure", "src": "/images/research/robot-benchmark-scores-trial-counts-evidence-embodiment-bundles.png", "alt": "InternW0-Δ reported success rates of 50.0%, 31.7% and 13.3% for the ARX X5, Piper and Piper X task bundles.", "caption": "RoboDojo physical leaderboard, observed 24 October 2026. The 36.7 percentage-point spread mixes hardware and task differences; it is not a causal hardware comparison.", "credit": "The Robot Economy analysis from the cited evaluation records", "kind": "data-visualization", "sourceIds": [ "s1", "s2" ] }, { "type": "paragraph", "text": "For a pilot decision, the useful next request is a matched task specification: objects, initial layouts, success criteria, controller and intervention rules. Without it, selecting the highest bundle score risks choosing a different task rather than a more capable system. This is a limit on the comparison, not evidence that the reported score is wrong.", "sourceIds": [ "s1", "s2" ], "evidence": "analysis" }, { "type": "heading", "text": "Separate the physical and simulated ledgers" }, { "type": "paragraph", "text": "The RoboDojo paper specifies 18 physical tasks across three embodiments, six per embodiment, with ten trials per task. That gives 60 trials per embodiment and 180 per policy. The paper separately describes 42 simulation tasks and 50 episodes per task. Combined with the current official three-seed rule, the simulation plan implies 6300 episodes: 42×50×3. These task sets are not aligned one-to-one.", "sourceIds": [ "s2", "s3" ], "evidence": "reported" }, { "type": "figure", "src": "/images/research/robot-benchmark-scores-trial-counts-evidence-trial-ledgers.png", "alt": "Separate calculations: 6 physical tasks times 10 trials times 3 embodiments equals 180 trials; 42 simulation tasks times 50 episodes times 3 seeds equals 6300 episodes.", "caption": "Protocol arithmetic, not an audit of raw trial logs for the 21 October entry. Physical trials and simulation episodes cover different tasks. Their totals do not establish a sim-to-real transfer ratio, relative operating cost or deployment reliability.", "credit": "The Robot Economy analysis from the cited evaluation records", "kind": "data-visualization", "sourceIds": [ "s2", "s3" ] }, { "type": "paragraph", "text": "The paper also reports 202 minutes for 180 physical trials using π0.5 on a local RTX 4090 server, including resets, inference and execution. That is a measured configuration, not a completion-time guarantee for InternW0-Δ or a labour-cost estimate. A different policy, network path or task duration can change the total. Trial counts define coverage; elapsed time needs its own operating boundary.", "sourceIds": [ "s2" ], "evidence": "analysis" }, { "type": "heading", "text": "A verified-entry rule is another distinct claim" }, { "type": "paragraph", "text": "RoboDojo’s current protocol requires the official evaluation pipeline, repeated simulation seeds, all three physical embodiments, hidden-layout checks, videos and release of the evaluated artifact for verified publication. It also requires a fixed system rather than switching to a different known-best policy for each task. The protocol allows three days after listing to release the required artifacts, with removal possible after that deadline. These are the published rules. This report has not audited every checkpoint, video or compliance decision behind the current leaderboard.", "sourceIds": [ "s3" ], "evidence": "reported" }, { "type": "heading", "text": "RoboArena: reconcile comparisons with rollouts" }, { "type": "paragraph", "text": "RoboArena describes 612 pairwise comparisons involving seven policies. In its exhaustive reference evaluation, the other five policies were tested after each A/B pair on the same task. The ledger therefore contains 1224 paired-policy rollouts plus 3060 additional rollouts, totaling 4284. That matches the paper’s reported evaluation total. It does not mean there were 4284 independent head-to-head matchups.", "sourceIds": [ "s5" ], "evidence": "analysis" }, { "type": "figure", "src": "/images/research/robot-benchmark-scores-trial-counts-evidence-roboarena-denominator.png", "alt": "A stacked bar of 1224 A/B policy rollouts plus 3060 additional policy rollouts, totaling 4284 RoboArena oracle evaluations.", "caption": "Calculation: 612×2+612×5=4284. The pairwise comparison and physical rollout are different counting units. Scene and task conditions are matched within a pair but can vary between comparisons.", "credit": "The Robot Economy analysis from the cited evaluation records", "kind": "data-visualization", "sourceIds": [ "s5" ] }, { "type": "paragraph", "text": "This distinction changes sample planning. A buyer reproducing a seven-policy reference evaluation needs a different run allocation from a buyer comparing two shortlisted policies. The published totals describe those evaluation designs; they do not provide a common reliability ranking against RoboDojo. RoboArena’s DROID platform uses a Franka Panda arm, which also explains the hardware context of this report’s archive cover.", "sourceIds": [ "s5" ], "evidence": "analysis" }, { "type": "heading", "text": "Australia and Germany add two more boundaries" }, { "type": "paragraph", "text": "A Macquarie University study defaults to 50 trials per task, model and evaluation setting. Timeout, horizon and safety stop count as failures; randomized initial states are reused across models. A success percentage without those stopping rules would omit part of the experiment. We use that protocol to define a useful comparison request, not to combine its success rates with RoboDojo’s.", "sourceIds": [ "s4" ], "evidence": "analysis" }, { "type": "paragraph", "text": "CALVIN adds a different unit again: a simulated sequence of up to five language instructions. Its long-horizon result depends on completing tasks in sequence rather than merely pooling isolated attempts. A sequence, an independently reset physical trial and a matched A/B comparison answer different questions. Calling all three an episode would not make their scores interchangeable.", "sourceIds": [ "s6" ], "evidence": "analysis" }, { "type": "heading", "text": "What an operator can use next" }, { "type": "paragraph", "text": "Our synthesis uses six substantive primary records across RoboDojo, RoboArena, Macquarie and CALVIN. The projects use distinct protocols; some consortium affiliations overlap, and none independently reproduces the new RoboDojo result. We preserve the 24 October snapshot, recalculate three ledgers and publish the inputs as CSV. No confidence interval is inferred from rounded leaderboard values or unaudited trial counts. No benchmark-to-factory reliability conversion is estimated.", "sourceIds": [ "s1", "s2", "s3", "s4", "s5", "s6" ], "evidence": "analysis" }, { "type": "paragraph", "text": "Under the base scenario, current task and scoring differences persist. Treat each benchmark as evidence about its stated conditions, then ask for a pilot ledger recording complete attempts, resets, stopping reasons and human interventions. If a later release supplies matched tasks and the same fixed system across hardware, the embodiment comparison becomes stronger. If the evaluated artifact cannot be reproduced, confidence in using the published score for a pilot weakens.", "sourceIds": [ "s1", "s2", "s3", "s4", "s5", "s6" ], "evidence": "analysis" }, { "type": "paragraph", "text": "At the next physical leaderboard update, check whether the same policy artifact and task definitions accompany per-embodiment results. At the next attempted reproduction, count all planned trials and reconcile successes, stops and interventions before reporting a new average. For any RoboArena comparison, state separately the number of matched pairs and the number of individual policy rollouts. Those observations can strengthen or overturn the practical interpretation here.", "sourceIds": [ "s1", "s2", "s3", "s4", "s5" ], "evidence": "analysis" }, { "type": "paragraph", "text": "Cover context: Ims photographed a Franka Emika arm at CeBIT in Hannover on 23 March 2017. It illustrates related robot-arm hardware, not a DROID rig, a RoboArena test scene or the RoboDojo ARX/Piper experiments. Source: Wikimedia Commons, CC BY-SA 4.0; cropped and resized, with this derivative photograph under the same license.", "sourceIds": [ "s7" ], "evidence": "reported" } ], "sources": [ { "id": "s1", "title": "RoboDojo physical leaderboard and dated result news", "publisher": "RoboDojo / AI MMLab Club", "url": "https://robodojo-benchmark.com/LeaderBoard?bench=realworld", "sourceType": "dataset", "publishedAt": "2026-10-21T00:00:00Z", "accessedAt": "2026-10-24T20:25:49.394828Z", "locator": "News entry 21 October; physical InternW0-Δ row, snapshot 24 October 2026" }, { "id": "s2", "title": "RoboDojo benchmark paper, version 3", "publisher": "RoboDojo research team", "url": "https://arxiv.org/html/2607.04434", "sourceType": "paper", "publishedAt": "2026-07-08T00:00:00Z", "accessedAt": "2026-10-24T20:25:49.394828Z", "locator": "Sections 3.1–3.2, physical evaluation setup and §6.3.2 timing boundary" }, { "id": "s3", "title": "RoboDojo official evaluation and verification protocol", "publisher": "RoboDojo / AI MMLab Club", "url": "https://robodojo-benchmark.com/leaderboard/protocol", "sourceType": "official-record", "publishedAt": "2026-10-26T00:00:00Z", "accessedAt": "2026-10-26T09:18:50.175868+00:00", "locator": "Protocol linked as 2026.10.26: three seeds/embodiments, evaluated artifact, three-day artifact deadline and hidden layouts" }, { "id": "s4", "title": "Experiences from Benchmarking Vision–Language–Action Models for Robotic Manipulation", "publisher": "Macquarie University research team", "url": "https://arxiv.org/html/2511.11298", "sourceType": "paper", "publishedAt": "2025-11-14T00:00:00Z", "accessedAt": "2026-10-24T20:25:49.394828Z", "locator": "Section 4.1: 50 trials per task/model/setting, randomized initial states and failure stopping rules" }, { "id": "s5", "title": "RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies, version 2", "publisher": "RoboArena research consortium", "url": "https://arxiv.org/html/2506.18123", "sourceType": "paper", "publishedAt": "2025-11-29T00:00:00Z", "accessedAt": "2026-10-24T20:25:49.394828Z", "locator": "Introduction: 612 comparisons; §4 DROID platform; §5.1 seven-policy 4284 rollout oracle" }, { "id": "s6", "title": "CALVIN: language-conditioned long-horizon robot manipulation benchmark, version 4", "publisher": "University of Freiburg / CALVIN research team", "url": "https://arxiv.org/html/2112.03227", "sourceType": "paper", "publishedAt": "2022-07-13T00:00:00Z", "accessedAt": "2026-10-24T20:25:49.394828Z", "locator": "Benchmark definition and evaluation: five-instruction simulated sequences; four environments" }, { "id": "s7", "title": "Franka Emika arm at CeBIT 2017: original photograph and license", "publisher": "Wikimedia Commons / Ims", "url": "https://commons.wikimedia.org/wiki/File:Franka_Emika2.jpg", "sourceType": "official-record", "publishedAt": "2017-03-23T12:47:10Z", "accessedAt": "2026-10-24T20:25:49.394828Z", "locator": "Description, CeBIT Hannover context, original 6000 x 4000 photograph, creator and CC BY-SA 4.0 license" } ], "originalValue": { "type": "data-synthesis", "description": "Reconstruct a three-bundle physical score panel, separate physical/simulation protocol ledgers, and the RoboArena pair-to-rollout bridge. The original calculations expose which counting unit and operating conditions a buyer must match before choosing a pilot evaluation." }, "research": { "question": "What does a robot-policy average prove when tasks, embodiments and evaluation units differ?", "geographicScope": [ "China", "Australia", "United States", "Germany" ], "methodology": "Use a dated 21 October 2026 physical leaderboard addition as the fresh anchor, freezing the observed row on 24 October. Read six substantive primary records from four evaluation projects; conservatively merge the partially overlapping generalist-evaluation consortia when assessing publisher independence, leaving Macquarie and the Freiburg-led CALVIN project as separate groups. Recalculate the embodiment/task spread, physical and simulated protocol counts, and pair-to-rollout denominator bridge. Preserve different task sets, stopping conditions and units. Do not infer per-trial logs, independent replication, hardware causality, financial cost or factory reliability.", "assumptions": [ "Protocol arithmetic assumes every scheduled task and repetition is completed; it is a coverage plan, not independently inspected execution logs.", "Scenario discussion holds the policy artifact and task definitions fixed; it estimates no numerical deployment success rate or monetary evaluation cost." ], "limitations": [ "The leaderboard snapshot reports rounded aggregates. Raw trials and evaluated artifact compliance were not independently audited or reproduced.", "Hardware and task bundles differ, and physical and simulated tasks are not aligned one-to-one. Cross-benchmark scores are not pooled or ranked.", "Older papers provide methodological comparisons rather than fresh tests of the 21 October policy. Consortium affiliations overlap; no independent corroboration of that policy score is claimed." ], "scenarios": [ { "name": "Current protocol differences persist", "description": "Use each score within its stated scope and request a matched pilot ledger with stopping, reset and intervention counts. No shared cross-benchmark reliability estimate is supported." }, { "name": "Matched reproduction becomes available", "description": "If a fixed evaluated artifact is reproduced on matched tasks and starting conditions across hardware, compare those controlled results. Failure to reproduce or a changed task specification weakens transfer to the proposed pilot." } ], "falsifiableCheckpoints": [ "At the next physical leaderboard update, inspect exact evaluated artifact, per-embodiment task definitions and row changes. Revise the bundle interpretation if a matched task cohort is released.", "At the next independent reproduction, reconcile every planned trial into success, timeout/horizon, safety stop and intervention outcomes. A mismatch between the declared ledger and observed runs weakens the published average.", "For a new RoboArena evaluation, publish matched-pair and individual-rollout counts separately and verify the stated number of policies per pair." ], "dataAvailability": "public-download", "dataAppendixUrl": "/research/data/robot-benchmark-scores-trial-counts-evidence.csv" }, "corrections": [], "disclosures": { "sponsored": false, "sponsorName": null, "founderProductMention": false } }