Causal Discovery with Language Models as Imperfect Experts

Abstract: Understanding the causal relationships that underlie a system is a fundamental prerequisite to accurate decision-making. In this work, we explore how expert knowledge can be used to improve the data-driven identification of causal graphs, beyond Markov equivalence classes. In doing so, we consider a setting where we can query an expert about the orientation of causal relationships between variables, but where the expert may provide erroneous information. We propose strategies for amending such expert knowledge based on consistency properties, e.g., acyclicity and conditional independencies in the equivalence class. We then report a case study, on real data, where a LLM is used as an imperfect expert.

• The paper formalizes an optimization framework using imperfect LLM inputs to minimize the Markov equivalence class size while ensuring the true causal graph remains included.
• The authors propose a greedy method with Bayesian updates that incrementally orients graph edges, balancing uncertainty reduction with the risk of excluding valid causal links.
• Empirical studies on real datasets demonstrate that cautious integration of LLMs, including GPT-3.5, effectively refines causal discovery even with overconfident expert outputs.

Causal Discovery with LLMs as Imperfect Experts: An Analysis

The paper "Causal Discovery with LLMs as Imperfect Experts" explores the integration of expert knowledge, specifically that of LLMs, into the process of causal discovery. The primary objective is to enhance the identification of causal graphs beyond the constraints of Markov equivalence classes (MECs), utilizing experts who may not always provide accurate information. The authors address the inherent challenge of enhancing causal graph identification through imperfect experts by proposing a strategy that ensures fundamental consistency properties such as acyclicity and conditional independencies remain intact.

Core Contributions

1. Problem Formalization: The paper formalizes the use of imperfect experts in causal discovery as an optimization problem. The goal is to minimize the size of the MEC while ensuring that the true causal graph remains a member, with a high probability, of the reduced set of solutions. The authors characterize this as a trade-off between reducing uncertainty in possible causal structures and the risk associated with erroneous expert inputs.
2. Methodological Innovation: A greedy approach is proposed that incrementally incorporates expert knowledge via Bayesian inference, optimizing the objective with Bayesian updates on the expert's decisions. This strategy involves iterative selection of edges to orient based on either minimizing the resultant MEC size or minimizing the risk of excluding the true causal graph from the equivalence class.
3. Empirical Evaluation: The strategy is evaluated on several case studies using real datasets, with experiments involving both simulated "imperfect" experts and LLM-based experts, specifically GPT-3.5 models. Results show that the proposed approach effectively reduces MEC size while maintaining a high likelihood of retaining the true graph under the guidance of imperfect knowledge. The robust performance highlights its potential utility in real-world applications where expert knowledge is partial or uncertain.

Theoretical and Practical Implications

The theoretical contribution lies in extending causal discovery methodologies by incorporating Bayesian strategies to ameliorate uncertainties stemming from imperfect expert inputs. The practical implications are significant, particularly in fields where acquiring full interventional data sets is impractical or ethically questionable, such as in certain medical and ecological studies.

Model Performance Insights: The research identifies that the performance of LLM-based experts may vary significantly based on their training and calibration. For instance, the newer GPT-3.5 models were observed to be potentially overconfident, leading to inaccuracies in some cases, indicating the necessity for cautious integration of LLM insights into causal inference frameworks.

Future Directions: The paper suggests further investigations into alternative noise models better aligned with the capabilities and limitations of LLMs, alongside improved strategies for querying and extracting causally relevant information from LLMs. Moreover, the study opens avenues for coupled methodologies integrating this approach with Bayesian causal discovery algorithms, enhancing the probabilistic graphical modeling frameworks for complex system analyses.

In sum, this work advances the domain of causal inference by addressing uncertainties in expert knowledge and integrating modern machine learning models as potential aids in the causal discovery process. Researchers in this field are thereby encouraged to further refine the methodologies and explore broader applications of these insights within complex data environments.