Comparing multilevel and fixed effect approaches in the generalized linear model setting

Abstract: We extend prior work comparing linear multilevel models (MLM) and fixed effect (FE) models to the generalized linear model (GLM) setting, where the coefficient on a treatment variable is of primary interest. This leads to three key insights. (i) First, as in the linear setting, MLM can be thought of as a regularized form of FE. This explains why MLM can show large biases in its treatment coefficient estimates when group-level confounding is present. However, unlike the linear setting, there is not an exact equivalence between MLM and regularized FE coefficient estimates in GLMs. (ii) Second, we study a generalization of "bias-corrected MLM" (bcMLM) to the GLM setting. Neither FE nor bcMLM entirely solves MLM's bias problem in GLMs, but bcMLM tends to show less bias than does FE. (iii) Third, and finally, just like in the linear setting, MLM's default standard errors can misspecify the true intragroup dependence structure in the GLM setting, which can lead to downwardly biased standard errors. A cluster bootstrap is a more agnostic alternative. Ultimately, for non-linear GLMs, we recommend bcMLM for estimating the treatment coefficient, and a cluster bootstrap for standard errors and confidence intervals. If a bootstrap is not computationally feasible, then we recommend FE with cluster-robust standard errors.