LENS filters clutter, but 22 of 40 fruit attempts still fail

A front end, not a new robot policy

LENS, short for LLM-guided Environment Simplification, is a University of Pennsylvania attempt to make clutter a preprocessing problem rather than a reason to retrain a robot. The July 22 paper asks a multimodal language model which objects matter to the task, then prunes irrelevant objects or combines related ones before handing the result to an existing planner, controller or vision-language-action policy. In the vision-language-action setup, excluded objects are inpainted from an overhead image and the filtered image is passed to pi0.5. The paper says this reduces distractor affordances without modifying the policy itself.^[1]

The clearest physical result is narrow but useful. The authors used pi0.5 to pick a fruit plushie into a target bowl in clutter, reporting ten attempts each for pineapple, pear, banana and apple. LENS recorded 5, 3, 3 and 7 successes; the cluttered baseline recorded 2, 0, 0 and 0. Summing the table produces 18 full successes in 40 trials for LENS and 2 in 40 for the baseline. That is a meaningful decision delta for a team already using a vision-language-action policy: visual relevance filtering may be a lower-friction experiment than replacing the policy. It is not evidence that cluttered manipulation is solved.^[1]

The simulation result is not an end-to-end vision result

The paper's simulation offers a second signal, but it defines the boundary. Across 100 evaluation episodes over ten Spatial LIBERO tasks, the authors report pi0.5 success of 0.85 in the clean condition, 0.50 after adding clutter and 0.69 after LENS filtering. Yet the simulated masking did not use the perception pipeline intended for hardware. The appendix says it used ground-truth segmentation because open-vocabulary detectors such as GroundingDINO performed unreliably in simulated scenes under domain mismatch. That means the 0.69 result tests the abstraction idea with ideal masks, not the complete visual system a deployer would run.^[1]

The hardware loop is more representative, but still leaves a wide failure band. There, the authors pass LENS selections through GroundingDINO, Segment Anything and LaMa to detect, mask and inpaint distractors before the policy runs. The resulting 18 of 40 total means 22 attempts still did not complete the reported pick-and-place task. The physical result therefore supports testing a relevance-filtering layer on a real table; it does not establish a general reliability level, an independent replication, or success across different cameras, robots, clutter types and task horizons.^[1]

The implementation boundary matters

The authors' project page currently marks its Code link as soon rather than linking an implementation. That does not weaken the reported table, but it does set the present operator decision: the paper can be reviewed, while the exact LENS perception, prompting and inpainting path cannot yet be independently rerun from a public project link. Treat this as an evidence packet for a focused internal experiment, not as a ready-made retrofit for a production robot stack.^[1,2]

It is also important to separate the upstream policy from the proposed front end. Physical Intelligence maintains the OpenPI repository associated with pi0.5, while LENS is the authors' extra scene-simplification layer placed before that policy. A reader evaluating the result should not infer that the existing pi0.5 code base contains LENS or that a pi0.5 user can reproduce these trials without the missing layer. The claimed gain belongs to the combined system and its author-run configuration.^[1,3]

What would change the decision

The next decisive artifact is a public, versioned LENS release with prompts, object-selection rules, GroundingDINO, Segment Anything and LaMa settings, plus per-trial logs for the four physical groups. The stronger follow-up would be an independent hardware comparison that holds the policy, camera, fruit set and clutter constant while comparing the original and filtered images across more repetitions and longer tasks. If that work preserves the gain without privileged masks or author-only tuning, relevance filtering could become a practical reliability lever for existing robot policies. Until then, the result is promising but bounded: 18 completions, 22 failures, and no public end-to-end implementation.^[1,2,3]