{ "schemaVersion": 1, "slug": "lens-clutter-filter-fruit-tasks", "status": "published", "articleType": "analysis", "desk": "frontier", "title": "LENS filters clutter, but 22 of 40 fruit attempts still fail", "dek": "A University of Pennsylvania paper raises author-reported cluttered fruit-task success from 2 of 40 to 18 of 40, but its simulation uses ground-truth segmentation because open-vocabulary detectors performed unreliably under domain mismatch.", "author": { "byline": "Imani Reed, Frontier Desk" }, "dates": { "publishedAt": "2026-07-23T06:11:56.321Z", "updatedAt": "2026-07-23T06:11:56.321Z" }, "hero": { "src": "/images/articles/lens-clutter-filter-fruit-tasks.webp", "alt": "A conceptual robot arm focuses on a fruit in a bowl while surrounding tabletop objects fade outside an amber relevance window.", "credit": "AI-generated editorial illustration / The Robot Economy", "kind": "generated-illustration", "sourceUrl": null, "rights": "Original AI-generated editorial illustration commissioned for The Robot Economy; conceptual only and not a depiction of a reported experiment." }, "seo": { "title": "LENS clutter filtering reaches 18 of 40 attempts", "description": "LENS lifted author-reported cluttered fruit-task success from 2 of 40 to 18 of 40, but its simulation used privileged masks where detectors did not transfer.", "keywords": [ "robot clutter filtering", "vision-language-action", "robot manipulation", "LENS robotics", "robot perception" ] }, "machine": { "summary": "LENS is a University of Pennsylvania research front end that identifies task-relevant objects, filters visual clutter and passes the simplified observation to existing robot stacks. Its authors report 18 successes in 40 cluttered fruit pick-and-place attempts versus 2 in 40 for the baseline, while noting that their simulation uses ground-truth segmentation because open-vocabulary detectors did not transfer reliably under domain mismatch.", "keyFacts": [ "The paper was submitted to arXiv on July 22, 2026 by four University of Pennsylvania authors.", "Its hardware table reports 5, 3, 3 and 7 successes in ten trials for pineapple, pear, banana and apple, respectively.", "The baseline table reports 2, 0, 0 and 0 successes across the same four ten-trial groups.", "The simulation compares clean, cluttered and LENS-filtered observations over 100 episodes across ten tasks: 0.85, 0.50 and 0.69 success, respectively.", "The authors' project page marks its Code link as soon rather than linking a public LENS implementation." ], "entities": [ "LENS", "University of Pennsylvania", "Physical Intelligence", "OpenPI", "pi0.5", "GroundingDINO", "Segment Anything", "LaMa" ] }, "readerContext": [ { "term": "Vision-language-action model", "explanation": "A robot policy that combines a camera image and a language instruction to choose an action." }, { "term": "Inpainting", "explanation": "Generating pixels to replace a masked part of an image so a downstream model sees a simplified scene." }, { "term": "Privileged segmentation", "explanation": "Object masks supplied from simulator ground truth rather than produced by the perception system being tested." } ], "signal": { "eventType": "research-result", "stage": "not-applicable", "topics": [ "robot-intelligence", "computer-vision", "software-infrastructure", "research-labs" ], "industries": [ "cross-industry" ], "companies": [ "University of Pennsylvania", "LENS", "Physical Intelligence", "OpenPI" ], "publicListings": [], "regions": [ "United States" ] }, "body": [ { "type": "heading", "text": "A front end, not a new robot policy" }, { "type": "paragraph", "text": "LENS, short for LLM-guided Environment Simplification, is a University of Pennsylvania attempt to make clutter a preprocessing problem rather than a reason to retrain a robot. The July 22 paper asks a multimodal language model which objects matter to the task, then prunes irrelevant objects or combines related ones before handing the result to an existing planner, controller or vision-language-action policy. In the vision-language-action setup, excluded objects are inpainted from an overhead image and the filtered image is passed to pi0.5. The paper says this reduces distractor affordances without modifying the policy itself.", "evidence": "reported", "sourceIds": [ "s1" ] }, { "type": "paragraph", "text": "The clearest physical result is narrow but useful. The authors used pi0.5 to pick a fruit plushie into a target bowl in clutter, reporting ten attempts each for pineapple, pear, banana and apple. LENS recorded 5, 3, 3 and 7 successes; the cluttered baseline recorded 2, 0, 0 and 0. Summing the table produces 18 full successes in 40 trials for LENS and 2 in 40 for the baseline. That is a meaningful decision delta for a team already using a vision-language-action policy: visual relevance filtering may be a lower-friction experiment than replacing the policy. It is not evidence that cluttered manipulation is solved.", "evidence": "reported", "sourceIds": [ "s1" ] }, { "type": "heading", "text": "The simulation result is not an end-to-end vision result" }, { "type": "paragraph", "text": "The paper's simulation offers a second signal, but it defines the boundary. Across 100 evaluation episodes over ten Spatial LIBERO tasks, the authors report pi0.5 success of 0.85 in the clean condition, 0.50 after adding clutter and 0.69 after LENS filtering. Yet the simulated masking did not use the perception pipeline intended for hardware. The appendix says it used ground-truth segmentation because open-vocabulary detectors such as GroundingDINO performed unreliably in simulated scenes under domain mismatch. That means the 0.69 result tests the abstraction idea with ideal masks, not the complete visual system a deployer would run.", "evidence": "reported", "sourceIds": [ "s1" ] }, { "type": "paragraph", "text": "The hardware loop is more representative, but still leaves a wide failure band. There, the authors pass LENS selections through GroundingDINO, Segment Anything and LaMa to detect, mask and inpaint distractors before the policy runs. The resulting 18 of 40 total means 22 attempts still did not complete the reported pick-and-place task. The physical result therefore supports testing a relevance-filtering layer on a real table; it does not establish a general reliability level, an independent replication, or success across different cameras, robots, clutter types and task horizons.", "evidence": "reported", "sourceIds": [ "s1" ] }, { "type": "heading", "text": "The implementation boundary matters" }, { "type": "paragraph", "text": "The authors' project page currently marks its Code link as soon rather than linking an implementation. That does not weaken the reported table, but it does set the present operator decision: the paper can be reviewed, while the exact LENS perception, prompting and inpainting path cannot yet be independently rerun from a public project link. Treat this as an evidence packet for a focused internal experiment, not as a ready-made retrofit for a production robot stack.", "evidence": "reported", "sourceIds": [ "s1", "s2" ] }, { "type": "paragraph", "text": "It is also important to separate the upstream policy from the proposed front end. Physical Intelligence maintains the OpenPI repository associated with pi0.5, while LENS is the authors' extra scene-simplification layer placed before that policy. A reader evaluating the result should not infer that the existing pi0.5 code base contains LENS or that a pi0.5 user can reproduce these trials without the missing layer. The claimed gain belongs to the combined system and its author-run configuration.", "evidence": "context", "sourceIds": [ "s1", "s3" ] }, { "type": "heading", "text": "What would change the decision" }, { "type": "paragraph", "text": "The next decisive artifact is a public, versioned LENS release with prompts, object-selection rules, GroundingDINO, Segment Anything and LaMa settings, plus per-trial logs for the four physical groups. The stronger follow-up would be an independent hardware comparison that holds the policy, camera, fruit set and clutter constant while comparing the original and filtered images across more repetitions and longer tasks. If that work preserves the gain without privileged masks or author-only tuning, relevance filtering could become a practical reliability lever for existing robot policies. Until then, the result is promising but bounded: 18 completions, 22 failures, and no public end-to-end implementation.", "evidence": "analysis", "sourceIds": [ "s1", "s2", "s3" ] } ], "sources": [ { "id": "s1", "title": "LENS: LLM-guided Environment Simplification for Planning and Control in Clutter", "publisher": "LENS research team", "url": "https://arxiv.org/abs/2607.19633", "sourceType": "paper", "publishedAt": "2026-07-22T00:05:00.000Z", "accessedAt": "2026-07-23T06:03:30.000Z" }, { "id": "s2", "title": "LENS project page", "publisher": "LENS research team", "url": "https://lens-2026.github.io/", "sourceType": "technical-document", "publishedAt": "2026-07-17T07:25:36.000Z", "accessedAt": "2026-07-23T06:02:44.000Z" }, { "id": "s3", "title": "OpenPI repository", "publisher": "Physical Intelligence", "url": "https://github.com/Physical-Intelligence/openpi", "sourceType": "code-release", "publishedAt": "2024-10-21T15:23:28.000Z", "accessedAt": "2026-07-23T06:03:30.000Z" } ], "originalValue": { "type": "calculation", "description": "This analysis sums the four per-fruit hardware rates in the primary paper's Table 1 to make the full task count legible: 18 LENS successes and 2 baseline successes across 40 attempts. It then separates that author-reported physical result from the ideal-mask simulation and the unavailable end-to-end LENS implementation." }, "corrections": [ { "at": "2026-07-23T06:11:13.063Z", "note": "Editorial Safety Editor changed the headline and SEO title from tabletop tasks to fruit attempts, and aligned the dek and Code-link wording with the reopened paper and project page. The evidence, denominator, and conclusion are unchanged." } ], "disclosures": { "sponsored": false, "sponsorName": null, "founderProductMention": false } }