Towards Automated Semantic Interpretability in Reinforcement Learning via Vision-Language Models

Abstract: Semantic interpretability in Reinforcement Learning (RL) enables transparency and verifiability by making the agent's decisions understandable and verifiable. Achieving this, however, requires a feature space composed of human-understandable concepts, which traditionally rely on human specification and may fail to generalize to unseen environments. We introduce interpretable Tree-based Reinforcement learning via Automated Concept Extraction (iTRACE), an automated framework that leverages pre-trained vision-LLMs (VLM) for semantic feature extraction and interpretable tree-based models for policy optimization. iTRACE first extracts semantically meaningful features, then maps them to policies via interpretable trees. To address the impracticality of running VLMs in RL loops, we distill their outputs into a lightweight model. By leveraging Vision-LLMs (VLMs) to automate tree-based reinforcement learning, iTRACE eliminates the need for human annotation traditionally required by interpretable models, while also addressing the limitations of VLMs alone, such as their lack of grounding in action spaces and inability to directly optimize policies. iTRACE outperforms MLP baselines that use the same interpretable features and matches the performance of CNN-based policies, producing verifiable, semantically interpretable, and human-aligned behaviors without requiring human annotation.