Visually Grounded Language Learning: a review of language games, datasets, tasks, and models

Abstract: In recent years, several machine learning models have been proposed. They are trained with a language modelling objective on large-scale text-only data. With such pretraining, they can achieve impressive results on many Natural Language Understanding and Generation tasks. However, many facets of meaning cannot be learned by `listening to the radio" only. In the literature, many Vision+Language (V+L) tasks have been defined with the aim of creating models that can ground symbols in the visual modality. In this work, we provide a systematic literature review of several tasks and models proposed in the V+L field. We rely on Wittgenstein's idea oflanguage games' to categorise such tasks into 3 different families: 1) discriminative games, 2) generative games, and 3) interactive games. Our analysis of the literature provides evidence that future work should be focusing on interactive games where communication in Natural Language is important to resolve ambiguities about object referents and action plans and that physical embodiment is essential to understand the semantics of situations and events. Overall, these represent key requirements for developing grounded meanings in neural models.

• The paper provides a comprehensive review of language games that connect visual data with linguistic processing in AI.
• The paper categorizes tasks and datasets into discriminative, generative, and interactive approaches, highlighting their roles in multimodal learning.
• The paper discusses challenges in compositionality and transferability, prompting further research towards robust, human-like language comprehension.

In the field of machine learning, particularly within the domain of Artificial Intelligence, there's a burgeoning subfield focused on Visually Grounded Language Learning (VGLL). VGLL is intriguing because it seeks to train models not just to understand and generate language, but to do so in a context where meaning is tied to visual data—the same way humans often associate words with images or experiences.

So, what does this mean concretely? Picture a scenario where an AI is given a photo of a busy street scene and asked, "Which vehicle looks like it's moving the fastest?" A VGLL model would not only parse the language but also assess the visual cues in the photo to provide a sensible answer.

A significant breakthrough in this area involves using "language games," essentially tasks that require the AI to interact with and interpret visual data, often while in communication with humans or other AIs. These games are categorized based on how the AI generates its outputs: discriminative (choosing from given options), generative (creating descriptions), and interactive (engaging in dialogue to solve tasks).

Interactive games are especially notable as they closely replicate real-world learning scenarios—think of how children learn language while interacting with their environment and people around them. This is seen as a promising path towards models that grasp language in a more human-like, contextually-rich way.

However, developing AIs that excel in these language games is no simple feat. Researchers have been actively creating varied datasets and models aiming to thrive on these tasks. Models are designed to encode multimodal contexts, such as joint text-and-image input, while maintaining the capacity to hold multi-turn dialogues that can resolve ambiguities and facilitate learning. An interesting aspect is the focus on embodiment—the idea that the AI has a presence in a simulated 3D environment, enabling it to perform actions or manipulate objects, making the learning process even more akin to human experience.

One of the challenges faced by current models is the aspect of compositionality—the ability to create new, meaningful expressions from known components. Just how humans can intuitively understand a “zebra” as a “striped horse,” AI models aspire to understand and generate language that reflects a similar level of intuitive recombination and generalization.

Yet, open questions linger, for instance, how can these visually grounded representations be transferable across multiple tasks, and how robust are they against missing or noisy data? It’s also essential to closely examine whether these AIs can still perform well when dealing with just text or images, not necessarily both.

Looking ahead, the VGLL field ponders how to best represent and learn language from rich multimodal interactions, move beyond specific task-driven models to more generic, world-savvy systems, and how usage in diverse, realistic environments can drive better language understanding.

In summary, VGLL represents a facet of AI research committed to a deeper understanding of how artificial agents interpret and use language, grounding linguistic elements in the rich tapestry of visual perception, much like we do. As this field progresses, we're inching closer to machines capable not just of understanding our words, but also the world they hint at.