Source note
Echo-Memory: A Controlled Study of Memory in Action World Models
Action World ModelsMemory MechanismsVideo DiffusionState Space ModelsRevisit Consistency
Summary
Echo-Memory is a controlled study of memory mechanisms for action-conditioned video world models. It finds that replay quality and revisit memory can rank methods differently, with block-wise state-space recurrence scoring highest on open-domain return.
Problem
- Action world models can change a scene or replace a salient object when the camera leaves and later returns, even when local video looks plausible.
- Prior memory comparisons mix changes in backbone, training, retrieval, sampling, and metrics, so the memory mechanism is hard to isolate.
- This matters because camera-following video is not enough for a world model; the model must preserve object identity and scene state across generated segments.
Approach
- The study fixes the video diffusion-transformer backbone, optimizer, camera-action representation, sampler, training recipe, and evaluation pipeline.
- It varies only the memory profile: raw context, compression-based memory, spatial summaries with different read-out paths, and state-space recurrence.
- The shared input interface uses a first frame, text prompt, historical context, and a per-frame 12D relative-RT camera-action sequence.
- Training uses 81-frame segments at 352×640 resolution, AdamW, 8 A100-80G GPUs, 5k steps, target-frame-only supervision, and a 10% overlap-drop policy.
- Evaluation has three branches: replay PSNR/SSIM/LPIPS, in-domain loop return PSNR/SSIM/LPIPS, and open-domain return scored by Qwen3-VL-30B-A3B on a 0-100 VLM scale.
Results
- Raw context is a strong baseline: open-domain VLM rises from 12.25 for anchor-only I2V to 50.75 with K=5 and 58.63 with K=20.
- Block-wise State-Space gets the best open-domain return score in the main table: 69.00 O-V, compared with 58.63 for Context K=20 and 34.75 for legacy-hybrid State-Space.
- Replay metrics do not predict revisit memory: Spatial Memory has high replay PSNR at 13.60 but low open-domain VLM at 6.00.
- Context K=20 has the best main-table replay SSIM and LPIPS: 0.449 SSIM and 0.496 LPIPS, while its open-domain VLM is 58.63.
- In-domain return also ranks methods differently: State-Space legacy hybrid has 12.23 ID-PSNR, while Context K=20 has 11.07 ID-PSNR but better open-domain VLM.
- The VLM-judge sanity check reports correlations above 0.90 with Claude Opus 4.6, GPT-5.5, and a human anchor; Pearson correlations are 0.93, 0.94, and 0.96 versus Qwen3-VL.