Recoleta
GitHub
Source note

Dream-MPC: Gradient-Based Model Predictive Control with Latent Imagination

Authors
Jonathan Spieler, Sven Behnke
Source
arXiv
Published
Collection
Embodied AI
World ModelsModel Predictive ControlGradient Based PlanningModel Based RLContinuous Control
Open arXivMarkdown source

Summary

Dream-MPC is a gradient-based MPC method that plans through a learned latent world model using a policy prior, an uncertainty penalty, and action reuse. It targets faster planning for high-dimensional continuous control while improving over the base policy and, in BMPC experiments, over MPPI.

Problem

  • Sampling-based MPC methods such as CEM and MPPI often evaluate hundreds or thousands of action sequences per step, which is costly for high-dimensional control and harder to run on limited hardware.
  • Pure policy networks are cheaper at inference time, but they can generalize poorly because they do no online planning at test time.
  • Prior gradient-based planners with learned models often underperformed gradient-free planners, partly because gradients can drive actions into model-error regions.

Approach

  • Dream-MPC samples a small number of candidate action sequences from a stochastic policy prior, then rolls them out in a learned latent world model.
  • It optimizes each action sequence by gradient ascent on predicted return, using predicted rewards and a terminal Q-value.
  • It penalizes epistemic uncertainty estimated from an ensemble of Q-functions, so plans with high model uncertainty get lower objective values.
  • It reuses previously optimized actions across MPC steps with a reuse coefficient, which carries optimization work forward in the receding-horizon loop.
  • The reported default planner uses 5 candidates, 1 gradient step, horizon 3, step size 0.1, reuse coefficient 0.1, and uncertainty coefficient 0.01.

Results

  • On 24 continuous-control tasks from DeepMind Control Suite, HumanoidBench, and Meta-World, Dream-MPC with BMPC improves IQM normalized score by 26.7% and mean normalized score by 20.5% over BMPC.
  • With TD-MPC2 as the base model, Dream-MPC improves over the TD-MPC2 policy-only baseline by 144.7% in IQM and 43.4% in mean score, but it does not consistently match TD-MPC2 with MPPI.
  • The planner uses 15 world-model evaluations per time step in the given setup, compared with 9216 for MPPI under the cited TD-MPC2 configuration.
  • Mean domain scores for Dream-MPC on TD-MPC2 are 433 ± 259 on DMControl, 379 ± 897 on HumanoidBench, and 0.62 ± 0.31 on Meta-World; TD-MPC2 with MPPI scores 657 ± 225, 761 ± 1617, and 0.67 ± 0.33 on the same domains.
  • On 6 image-observation DMControl tasks with BMPC, Dream-MPC scores 725 ± 141 on Cartpole Swingup Sparse, 643 ± 9 on Cheetah Run, 275 ± 3 on Hopper Hop, 435 ± 76 on Quadruped Walk, and 762 ± 6 on Walker Run; it is roughly tied on Acrobot Swingup at 288 ± 31 versus BMPC policy-only 292 ± 18.
  • The paper claims Dream-MPC can outperform gradient-free MPC when paired with a strong policy prior from BMPC, while using far fewer model evaluations than MPPI.

Link