Source note
Event-Centric World Modeling with Memory-Augmented Retrieval for Embodied Decision-Making
Summary
This paper proposes an event-centric world model for embodied decision-making that stores past situations and retrieves similar ones to choose actions. The system is aimed at dynamic, safety-critical control, with a UAV navigation study used as the main validation.
Problem
- Autonomous agents in dynamic environments need decisions that are fast, physically consistent, and easy to inspect.
- End-to-end policies can map observations to actions, but the paper argues that they hide why an action was chosen and make it hard to check physical feasibility.
- This matters in safety-critical settings such as UAV flight, where inconsistent or opaque actions can lead to collisions or unstable behavior.
Approach
- The environment is converted into a set of semantic events: detected objects, their positions and motion, global context, the agent state, and goal direction.
- A permutation-invariant encoder maps that event set into a latent code, so the agent can compare the current situation with stored past situations.
- A knowledge bank stores triples of latent event code, maneuver, and reward or feasibility score; action selection uses nearest-neighbor retrieval and similarity weights over the top-k matches.
- To avoid averaging incompatible actions, retrieved maneuvers are clustered by direction, and the final action is averaged only inside the highest-weight cluster.
- The latent dynamics are constrained to be contractive and physics-aware, using a stable linear transition model, Lyapunov-style bounds, a physics-consistency regularizer, and ANN retrieval with FAISS for real-time use.
Results
- In NVIDIA Isaac Sim UAV experiments, the expert dataset used for pretraining contains 27,075 collision-free trajectories generated by a Virtual Potential Field supervisor.
- The control loop is designed for 20-50 ms update intervals, and the paper claims sub-millisecond retrieval latency on embedded hardware using FAISS-based ANN search.
- In an adversarial curriculum with 5 episodes, the training loss reportedly spikes to about 77.9 in Episode 2 and then drops to about 2.6 by Episode 5.
- In the reported navigation evaluation, the agent achieves 100% success rate with 0 collisions across all five curriculum episodes.
- The average trajectory length is about 680 steps per 100 m task.
- The excerpt does not provide baseline comparisons against other methods, ablations, or standard benchmark metrics beyond these internal results.