VLURes: Benchmarking Long-Text Grounding and Cross-Lingual Robustness in Vision Language Models

🎉 Accepted to ACL2026 Findings. See you in San Diego, CA🌴

Jesse Atuhurra1, Iqra Ali2, Tomoya Iwakura3, Hidetaka Kamigaito1, and Tatsuya Hiraoka1,4,5
1 NAIST 2 QMUL 3 Meiji University 4 MBZUAI 5 RIKEN

This project started when Jesse Atuhurra and Iqra Ali were interns at the AI Lab of Fujitsu Ltd., Japan, supervised by Tomoya Iwakura and Tatsuya Hiraoka.

Code Data on Hugging Face Paper
🌍 Motivation: A Multilingual, Multimodal World Needs Multilingual, Multimodal AI

Despite recent advances in Vision-Language Models (VLMs), most benchmarks evaluate models in English, with limited regard for non-English languages or rich, real-world contexts. This monolingual bias severely limits how we assess AI’s true generalization capabilities, especially for low-resource languages.

VLURes is designed to change that. It rigorously evaluates visual and linguistic understanding across English, Japanese, Swahili, and Urdu, using diverse tasks, rich prose, and grounded cultural contexts.

VLURes Task Overview Figure 1: VLURes Task Overview

We envision a world comprising generalist intelligent agents, such as robots, that accomplish several Vision-Language tasks.
🌍 What We Built: The VLURes Benchmark

VLURes is more than just a dataset; it’s a comprehensive testbed for the next generation of intelligent agents.

🧠 What Is VLURes?

VLURes is a multilingual vision-language benchmark designed to test intelligent agents under realistic conditions. Each input contains an image and an article-level text (not just captions), and the benchmark tests a model’s ability to perform both image-only and image+text reasoning.

VLURes covers 8 tasks:

🏗️ Dataset Construction

We collected articles and images from multiple web sources, including Wikipedia, Wikinews, blogs, and forums. The collection covers diverse topics such as animals, locations, food, buildings, and events.

We used CLIP similarity scores to align the most relevant image to each article. All data was cleaned manually, filtered for quality, and checked for NSFW or offensive content.

🎯 New Task: The "Unrelatedness" Challenge

The proposed Unrelatedness task. Left: The VLM inputs consist of two modalities, a pair of images and texts. The image undergoes a series of transformations in the vision encoder and connector, generating visual tokens that are ready for alignment in a shared embedding space. Similarly, a tokenizer tokenizes text, generating textual tokens. Textual and visual tokens are aligned in a shared embedding space and fed as input to the LLM. Right. The LLM uses its multimodal understanding to decide what textual information is relevant to different parts of the image. We see that the text painted green (marked with a cross sign) is directly related to the region of the image shown inside a green square box. That is, the text matches the image part shown in green. But in this task, we are interested in text unrelated to the image. Hence, yellow text (marked with a check sign) answers our Unrelatedness task.

VLURes Task Overview Figure 2: Our proposed Unrelatedness Task

Unlike traditional matching tasks, Unrelatedness tests whether a model can identify irrelevant information. This is vital in noisy, multimodal environments like news feeds or social media.

Can the model ignore text that does not describe or relate to the image?
This is the inverse of standard grounding tasks and pushes models to reason beyond associations.

📊 Summary of the Benchmark Pipeline
  1. Task Definition: 8 vision-language tasks
  2. Data Collection: From native-language web sources
  3. Alignment: Image selection via CLIP similarity
  4. Evaluation: Via human and automatic judges
  5. Results: Quantitative accuracy + qualitative rationale analysis
🔬 Evaluation Protocols

Models were tested under:

We used both:

VLURes Task Performance

🧪 Experiment Results: Key Findings
📉 Challenges Highlighted
🔓 Open Access

We believe in open science. The dataset is publicly available:

🧑‍💻 Authors, BibTeX, Usage and License Notices

🧑‍💻 Authors

For questions about this research, please get in touch with the corresponding authors:

📚 BibTeX

@inproceedings{atuhurra-etal-2026-vlures,
    title = "{VLUR}es: Benchmarking Long-Text Grounding and Cross-Lingual Robustness in Vision Language Models",
    author = "Atuhurra, Jesse  and
      Ali, Iqra  and
      Iwakura, Tomoya  and
      Kamigaito, Hidetaka  and
      Hiraoka, Tatsuya",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-acl.1367/",
    pages = "27426--27481",
    ISBN = "979-8-89176-395-1",
    abstract = "We introduce ***VLURes***, a multilingual benchmark for evaluating Vision-Language Models (VLMs) under *long-text grounding*: selecting and reasoning over the image-relevant subset of article-length text that contains distractors and ungrounded claims. *VLURes* contains **4,000** web-curated *image + long-text* pairs across **English (En), Japanese (Ja), Swahili (Sw), and Urdu (Ur)** and **10** topical categories, and defines **eight** tasks spanning image-only perception (OR, SU, RU, SS, IC) and image+text grounding (ITM, *Unrelatedness*, VQA). To construct web-realistic pairs, we apply language-adapted CLIP alignment to select representative images and filter weakly grounded pages. Across **10** proprietary and open VLMs evaluated under zero-shot and one-shot prompting, with and without rationales, the best model (GPT-4o) reaches **90.8{\%}** overall accuracy but remains **6.7** points below human performance (**97.5{\%}**) on Object Recognition, and cross-lingual sensitivity persists, while open models are substantially weaker and often lack reliable multilingual VL support. *VLURes* provides a practical testbed for long-text grounding and multilingual robustness in web-realistic agent settings."
}

and

@misc{atuhurra2025vluresbenchmarkingvlmvisual,
      title={VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages}, 
      author={Jesse Atuhurra and Iqra Ali and Tomoya Iwakura and Hidetaka Kamigaito and Tatsuya Hiraoka},
      year={2025},
      eprint={2510.12845},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.12845}, 
}

Usage and License Notices

The code, annotations, and other original materials in this repository are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0).