Brownian motion, bridges and Bayesian inference in phylogenetic tree space

Abstract: Billera-Holmes-Vogtmann (BHV) tree space is a geodesic metric space of edge-weighted phylogenetic trees with a fixed leaf set. Constructing parametric distributions on this space is challenging due to its non-Euclidean geometry and the intractability of normalizing constants. We address this by fitting Brownian motion transition kernels to tree-valued data via a non-Euclidean bridge construction. Each kernel is determined by a source tree $x_0$ (the Brownian motion's starting point) and a dispersion parameter $t_0$ (its duration). Observed trees are modelled as independent draws from the transition kernel defined by $(x_0, t_0)$, analogous to a Gaussian model in Euclidean space. Brownian motion is approximated by an $m$-step random walk, with the parameter space augmented to include full sample paths. We develop a bridge algorithm to sample paths conditional on their endpoints, and introduce methods for sampling a Bayesian posterior for $(x_0, t_0)$ and for marginal likelihood evaluation. This enables hypothesis testing for alternative source trees. The approach is validated on simulated data and applied to an experimental data set of yeast gene trees. These methods provide a foundation for future development of a wider class of probabilistic models of tree-valued data.