Chasing Ghosts: Instruction Following as Bayesian State Tracking

Abstract: A visually-grounded navigation instruction can be interpreted as a sequence of expected observations and actions an agent following the correct trajectory would encounter and perform. Based on this intuition, we formulate the problem of finding the goal location in Vision-and-Language Navigation (VLN) within the framework of Bayesian state tracking - learning observation and motion models conditioned on these expectable events. Together with a mapper that constructs a semantic spatial map on-the-fly during navigation, we formulate an end-to-end differentiable Bayes filter and train it to identify the goal by predicting the most likely trajectory through the map according to the instructions. The resulting navigation policy constitutes a new approach to instruction following that explicitly models a probability distribution over states, encoding strong geometric and algorithmic priors while enabling greater explainability. Our experiments show that our approach outperforms a strong LingUNet baseline when predicting the goal location on the map. On the full VLN task, i.e. navigating to the goal location, our approach achieves promising results with less reliance on navigation constraints.

Bayesian State Tracking for Vision-and-Language Navigation

This essay provides an expert analysis of a paper that introduces a novel approach to Vision-and-Language Navigation (VLN) through Bayesian state tracking. Traditional VLN involves an agent interpreting natural language instructions to navigate 3D environments. The presented research redefines instruction following by applying a Bayesian framework to model navigation as state estimation from partial observations.

Methodology and Implementation

The core innovation in the paper lies in representing navigation instructions as latent descriptions of expected actions and observations, thereby formulating instruction-following as Bayesian state tracking. This is accomplished by leveraging a differentiable Bayes filter to predict the likely trajectory of a hypothetical human demonstrator within a semantic spatial map constructed on-the-fly. The model integrates three main components: a mapper, a filter, and a policy.

• Mapper: Constructs a semantic spatial map from first-person views during navigation. This map consists of grid-based representations derived from projected CNN features, using depth images and a pinhole camera model.
• Filter: Implements Bayesian state tracking using a histogram-based belief representation that tracks a distribution over potential states in the map. This process integrates motion and observation models, both conditioned on latent representations crafted from the language instructions via a sequence-to-sequence model with attention.
• Policy: Executes a series of actions determined by predicting the most probable trajectories within the map to reach the goal.

Empirical Results

The approach significantly outperformed a strong LingUNet baseline in predicting the goal location given partially-observed maps and limited training data, achieving a higher success rate across both seen and unseen environments. On the comprehensive VLN task, while the model trained via imitation learning (sans data augmentation or RL) was less competitive with state-of-the-art methods that leverage complex strategies, it nonetheless demonstrated credible performance. Notably, the system boasted reduced overfitting and maintained success across diverse test environments.

Implications and Future Directions

The implications of this work are multifold. The Bayesian framework, with its explicit probabilistic reasoning over trajectories, enriches interpretability and aligns closely with sim-to-real transfer scenarios. The integration of depth sensing into the Matterport simulator further fortifies the approach's adaptability to real-world robotics applications where environmental navigation graphs are unavailable.

Future research could focus on optimizing the interplay between the intrinsic and extrinsic state representations for improved task generalization. There exists potential synergy with reinforcement learning paradigms and advanced data augmentation techniques to enhance performance metrics sustainably. Additionally, extending the Bayesian methodology could enrich multi-agent collaborative systems through shared belief networks.

In summary, this Bayesian state tracking model underscores a novel approach in VLN, emphasizing an explainable, probabilistic architecture that provides meaningful insights into agent-level navigation strategies while maintaining a robust framework suited to complex, real-world operational domains.