Recoleta
GitHub
Source note

Awaking Spatial Intelligence in Unified Multimodal Understanding and Generation

Authors
Lin Song, Wenbo Li, Guoqing Ma, Wei Tang, Bo Wang, Yuan Zhang, et al.
Source
arXiv
Published
Collection
Embodied AI
Spatial ReasoningMultimodal Foundation ModelImage GenerationImage EditingWorld Models
Open arXivMarkdown source

Summary

JoyAI-Image is a unified image understanding, generation, and editing model that adds stronger spatial reasoning to a Qwen3-VL-based MLLM and a 16B-parameter diffusion transformer.

Problem

  • Current unified visual models often connect understanding, generation, and editing weakly, so image edits and generated views do not reliably use scene geometry.
  • The paper targets layout, depth, object relations, viewpoint change, and cross-view consistency, which matter for controllable image editing and may help later vision-language-action systems and world models.

Approach

  • The model uses Qwen3-VL-8B-Instruct as the MLLM for image/text understanding and instruction parsing.
  • For generation and editing, the MLLM outputs hidden-state features that condition a 16B MMDiT diffusion model; a Wan-2.1 VAE compresses images into latent tokens.
  • The authors build OpenSpatial, a 3D box-centered data engine that turns scans and web videos into spatial QA pairs using 3D oriented boxes, projected masks, visibility checks, and multi-view consistency checks.
  • Training uses about 11.3M samples, including 6.1M general-understanding samples, 3.4M spatial-understanding samples, 1.4M instruction-rewriting samples, and 137.4K spatial-editing samples.
  • The MLLM is fine-tuned with supervised learning plus KL distillation on general data only, so it keeps general skills while learning new spatial skills.

Results

  • On 9 spatial benchmarks, JoyAI-Image-Und reports a 64.4 spatial average, up +5.3 points over Qwen3-VL-8B-Instruct at 59.1 and equal to Gemini-2.5-Pro at 64.4.
  • The largest gains over the base model are AllAngles +11.5, 3DSR_C +7.7, MMSI +7.7, ERQA +4.9, and VSI +4.5.
  • On general benchmarks, it keeps similar performance: MMBench_CN 83.7 versus base 83.3, MathVista 74.4 versus 75.0, MMStar 71.3 versus 70.1, and OCRB 87.9 versus 90.3.
  • OpenSpatial contains about 3M entries across 5 spatial capability groups and 19 sub-tasks; the full spatial training subset is reported as 3.4M samples.
  • The excerpt claims state-of-the-art or competitive results for generation, long-text rendering, and editing, but it does not provide the detailed numeric generation/editing tables in the supplied text.

Link