Recoleta
GitHub
Source note

SkyJEPA: Learning Long-Horizon World Models for Zero-Shot Sim-to-Real Control of Quadrotors

Authors
Pratyaksh Rao, Wancong Zhang, Randall Balestriero, Yann LeCun, Giuseppe Loianno
Source
arXiv
Published
Collection
Embodied AI
Quadrotor ControlWorld ModelsSim2realLatent DynamicsModel Based Control
Open arXivMarkdown source

Summary

SkyJEPA trains a JEPA-style latent world model for quadrotor control, then uses it inside a sampling-based controller for zero-shot sim-to-real flight. The paper targets long-horizon prediction without state reconstruction at every step.

Problem

  • Learned quadrotor dynamics models often predict the next state autoregressively, so small one-step errors compound over long control horizons.
  • A controller needs predictions in physical state variables such as position, velocity, attitude, and angular velocity, not only abstract embeddings.
  • Real quadrotor data collection is costly and can damage hardware, especially for aggressive maneuvers and platform changes.

Approach

  • The model encodes a history window of states and actions into latent vectors, then predicts future latent states over a horizon instead of reconstructing full future states.
  • Training uses a multi-step latent prediction loss plus SIGReg, an anti-collapse regularizer that pushes latent embeddings toward an isotropic Gaussian through random 1D projections.
  • A second-stage physics-inspired prober freezes the latent model and maps predicted latents to metric states through residual-corrected kinematics.
  • The prober predicts residual translational acceleration in R^3 and a 3x4 residual angular-acceleration map tied to the 4 rotor forces.
  • The learned model is placed inside a sampling-based optimal controller and trained on domain-randomized simulation data for zero-shot outdoor deployment.

Results

  • The provided excerpt gives no exact metric values, dataset sizes, baseline scores, or closed-loop success rates.
  • Claimed evaluations include open-loop prediction tests and outdoor closed-loop flight experiments.
  • Claimed comparisons include current dynamics learning baselines and ablations over key design choices, but the excerpt does not give numeric deltas.
  • Claimed effects are improved long-horizon prediction, real-time execution on embedded hardware, and zero-shot sim-to-real transfer across trajectories and platform variations.
  • Concrete numeric details in the method include a quadrotor action vector with 4 motor forces, state terms for 3D position, 3D velocity, SO(3) attitude, and 3D angular velocity, plus a 3x4 residual angular map in the prober.

Link