Church: a language for generative models

Abstract: We introduce Church, a universal language for describing stochastic generative processes. Church is based on the Lisp model of lambda calculus, containing a pure Lisp as its deterministic subset. The semantics of Church is defined in terms of evaluation histories and conditional distributions on such histories. Church also includes a novel language construct, the stochastic memoizer, which enables simple description of many complex non-parametric models. We illustrate language features through several examples, including: a generalized Bayes net in which parameters cluster over trials, infinite PCFGs, planning by inference, and various non-parametric clustering models. Finally, we show how to implement query on any Church program, exactly and approximately, using Monte Carlo techniques.

• The paper introduces Church, a universal language based on lambda calculus that recasts evaluation as sampling for generative processes.
• It employs a Lisp-like syntax and a stochastic memoizer to efficiently capture and share computations across non-parametric models.
• Church supports both exact and approximate inference methods, including rejection sampling and Metropolis-Hastings, enabling robust probabilistic queries.

Church: A Language for Generative Models

The paper introduces "Church," a universal language grounded in the lambda calculus to describe stochastic generative processes. This language builds upon the Lisp model, specifically the functional language Scheme, and employs lambda expressions to construct generative models that can be arbitrarily composed and abstracted.

Key Features and Contributions

The paper outlines several unique features and main contributions of Church:

1. Lisp-Based Language Specification: Church is expressed in a Lisp-like syntax where evaluation is equated to sampling, and queries are conditional samplings.
2. Stochastic Memoizer: This construct allows separate evaluations to share generative history efficiently, facilitating the straightforward description of non-parametric models.
3. Inference Techniques: Church implements both exact and approximate generic inference schemes that can execute any Church program, circumventing the necessity for special-purpose inference code.

Language Semantics and Construction

Church expressions define generative processes through the recursive evaluation procedure, eval. This procedure maps expressions to concrete values. Randomness is introduced via stochastic primitive functions, with memoization ensuring reuse of computed values for efficiency. The language also supports constructing procedures dynamically, leveraging the power of Lisp environments.

Evaluative Procedure and Stochastic Memoization

Evaluation in Church involves repeated application of eval, recursively interpreting expressions within an environment, yielding spaces of nested lambda expressions, conditional constructs (like if), and variable bindings. The stochastic memoizer, particularly the Dirichlet Process (DP) based DPmem, handles cases where random objects have persistent properties. This memoization is critical for implementing complex models involving infinite structures and ensures that the values generated remain consistent across evaluations.

Probabilistic Inference

Inference in Church is achieved using the query primitive, which samples values based on specified conditions (predicates). An essential aspect of this is ensuring admissibility—expressions must converge (halt with probability one) and generate valid distributions. The paper details a basic rejection sampling approach and outlines a Monte Carlo method with Metropolis-Hastings to manage complex queries that involve conditioning on random variables.

Example Applications

Several examples illustrate the versatility of Church:

• Causal Reasoning in Diagnostic Models: The paper demonstrates representing simple causal models where dependencies between events (e.g., rain and wet grass) are described using memoized functions to capture persistent states and their probabilistic interactions.
• Non-Parametric Clustering Models: By defining classes and their persistent characteristics using memoizers, Church succinctly implements structured models like the infinite Gaussian mixture model.
• Stochastic Transition Models: Expressing PCFGs (Probabilistic Context-Free Grammars) and extensions like HDP-HMMs (Hierarchical Dirichlet Process HMMs) showcases how Church can generalize traditional models to handle infinite and adaptive structures effectively.
• Planning by Inference: Leveraging Church's capability to embed queries, the paper outlines planning problems where actions are chosen to maximize expected rewards, translating goals into probabilistic conditions to be met.

Practical Implications and Future Directions

Church's capability to elegantly represent and manipulate complex generative processes offers significant implications for AI development. By integrating stochastic processes directly into the language semantics, Church supports the modularity and reusability of models, which is crucial for scalable AI applications. Future work can focus on optimizing inference algorithms for practical use, such as more efficient implementations of MCMC methods or leveraging deterministic program analyses for exact inference.

This dual use of Church as a data representation and procedural tool bridges declarative probabilistic modeling with functional programming processes. The language's extensibility, such as through nested queries or self-reflective reasoning, shows potential for advanced AI tasks like hierarchical modeling, program synthesis, and adaptive learning algorithms. As methodologies for probabilistic programming advance, Church provides a robust foundation to exploit these innovations comprehensively.