CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models

Abstract: Users often ask dialogue systems ambiguous questions that require clarification. We show that current LLMs rarely ask users to clarify ambiguous questions and instead provide incorrect answers. To address this, we introduce CLAM: a framework for getting LLMs to selectively ask for clarification about ambiguous user questions. In particular, we show that we can prompt LLMs to detect whether a given question is ambiguous, generate an appropriate clarifying question to ask the user, and give a final answer after receiving clarification. We also show that we can simulate users by providing LLMs with privileged information. This lets us automatically evaluate multi-turn clarification dialogues. Finally, CLAM significantly improves LLMs' accuracy on mixed ambiguous and unambiguous questions relative to SotA.