Recoleta
GitHub
Source note

CodeMMR: Bridging Natural Language, Code, and Image for Unified Retrieval

Authors
Jiahui Geng, Qing Li, Fengyu Cai, Fakhri Karray
Source
arXiv
Published
Collection
Software Intelligence
Multimodal Code RetrievalCode SearchRetrieval Augmented GenerationVision Language CodeBenchmark Datasets
Open arXivMarkdown source

Summary

CodeMMR is a multimodal code retriever that puts natural language, code, and images into one embedding space so the same model can retrieve across all three. The paper also introduces MMCoIR, a benchmark for this setting, and shows that standard multimodal retrievers miss much of the visual meaning in code artifacts.

Problem

  • Existing code retrieval models mostly treat code as text and miss visual outputs tied to code, such as web pages, charts, SVGs, diagrams, and UML.
  • This matters for code search and code RAG because developers often need to retrieve code from an image, retrieve images from code, or combine text and visuals when editing or generating code.
  • Before this work, there was no broad benchmark for multimodal multilingual code retrieval across domains, languages, and retrieval directions.

Approach

  • The paper builds MMCoIR, a benchmark covering 5 visual domains, 8 programming languages, and 11 libraries, with tasks such as text-to-code, image-to-code, code-to-image, and composed queries like text+image-to-code.
  • It trains CodeMMR, a single retriever initialized from a pretrained vision-language model and adapted to encode text, code, and images into a shared semantic space.
  • Each query includes an instruction that states the retrieval intent, such as retrieving code that matches an image. The model embeds the query plus instruction and matches it against candidate targets.
  • Training uses a contrastive InfoNCE objective: matched query-target pairs are pulled together, while in-batch and hard negative examples are pushed apart.
  • The design aims to generalize across modalities, programming languages, and unseen retrieval settings used in editing and repair tasks.

Results

  • On MMCoIR, CodeMMR (2B) gets 68.0 nDCG@10 average, beating VLM2Vec-v2 (2B) at 58.0, GME (7B) at 53.8, and GME (2B) at 51.5. The abstract states an average gain of about 10 absolute nDCG@10 points over strong baselines such as UniIR, GME, and VLM2Vec.
  • On Hit@1 over MMCoIR, CodeMMR (2B) reaches 65.4 average, ahead of VLM2Vec-v2 (2B) at 53.3 and GME (7B) at 49.5.
  • Strong per-dataset numbers include 99.8 nDCG@10 on Chart2Code image-to-code, 98.6 on Web2Code image-to-code, 97.8/97.0 on DATIKZv3 code-to-image / image-to-code, and 100.0/100.0 on PlantUML code-to-image / image-to-code.
  • The task remains hard on SVG-heavy settings: on MMSVG, CodeMMR reports 11.2, 12.3, 9.4, and 58.2 nDCG@10 across the listed retrieval directions, which is still much better than several baselines but far below its scores on WebUI, charts, and UML.
  • For downstream generation with retrieval, the paper claims gains on unseen image-to-code tasks: +10.0 points in Execution Rate on ChartMimic Direct and +9.4 points in Visual Accuracy on WebCode2M-Mid over a non-retrieval baseline, while also beating RAG systems that use baseline retrievers.

Link