Post-treatment problems: What can we say about the effect of a treatment among sub-groups who (would) respond in some way?

Abstract: Investigators are often interested in how a treatment affects an outcome for units responding to treatment in a certain way. We may wish to know the effect among units that, for example, meaningfully implemented the intervention or passed an attention check. Simply conditioning on the observed value of the post-treatment variable introduces problematic biases. Further, the identification assumptions required of several existing strategies are often indefensible. We propose the Treatment Reactive Average Causal Effect (TRACE), which we define as the total effect of treatment in the group that, if treated, would realize a particular value of the relevant post-treatment variable. By reasoning about the effect among the "non-reactive" group, we can identify and estimate the range of plausible values for the TRACE. We demonstrate the use of this approach with three examples: (i) using hypothetical data to illustrate how we can point identify the effect of police-perceived race on police violence during traffic stops, (ii) estimating effects of a community-policing intervention in Liberia, in locations where the project was meaningfully implemented, and (iii) studying how in-person canvassing affects support for transgender rights in the United States, among participants whose feelings towards transgender people become more positive.