The Yule-$Λ$ Nested Coalescent: Distribution of the Number of Lineages

Abstract: We study a model of a population with individuals sampled from different species. The Yule-$\Lambda$ nested coalescent describes the genealogy of the sample when each species merges with another randomly chosen species with a constant rate $c$ and the mergers of individuals in each species follow the $\Lambda$-coalescent. For the Yule-$\Lambda$ nested coalescent with $c<\int_0^1x^{-1}\Lambda(dx)<\infty$, where $\Lambda$ is the measure that characterizes the $\Lambda$-coalescent, we show that under some initial conditions, the distribution of the number of individual lineages belonging to one species converges weakly to the distribution $\nu_c^*$, which is the solution to some recursive distributional equation (RDE) with finite mean. In addition, we show that for some values of $c$, the RDE has another solution with infinite mean.