Robot learning is tightening the link between visual simulation and contact execution

GS-Playground connects a parallel physics engine with batched 3D Gaussian Splatting (3DGS), a rendering method for photorealistic scene reconstruction. The goal is to keep visual inputs close to real camera data without losing the simulation scale needed for reinforcement learning.

The reported numbers are the main signal. The paper claims about 10,000 FPS 3DGS rendering at 640×480 on an RTX 4090-class setup, supports up to 2048 rendered scenes at that resolution, and lists up to 4096 3DGS environments. Its pruning step removes more than 90% of Gaussians while keeping PSNR loss below 0.05. The simulator also binds Gaussian assets to rigid bodies, so rendered objects move with the physics state during contacts.

This matters for robot learning because contact-rich manipulation, navigation, and locomotion need both many trials and useful images. The Real2Sim pipeline adds another practical piece: it turns RGB captures into assets, meshes, poses, scales, and collision-ready scene elements, reducing the manual work normally needed to build simulation scenes.

HANDFUL studies a common failure mode in robot hands: a stable first grasp can consume the fingers or contact regions needed for the next action. The setup asks a LEAP Hand to hold one object, then push, press, twist, pull, or pick another object while preserving the initial grasp.

The method assigns fingers to the first grasp and keeps other fingers available. Its reward encourages contact for active fingers and penalizes contact force on inactive fingers. It then trains second-stage policies from terminal grasp states and uses a curriculum to keep the grasp candidates that work best for each follow-up task.

The gains are clearest in the ablations. HANDFUL reports 69.90% success on Push Object, 77.75% on Press Button, 61.52% on Twist Knob, 78.94% on Pull Drawer, and 76.54% on Pick Second in simulation. Removing finger constraints drops Pick Second to 0.00% and lowers several other tasks. The curriculum cuts second-stage training from 90 million to 54 million steps while keeping similar final success.