Sequential Counterfactual Evaluation

• Sequential counterfactual evaluation is the principled quantification of what-if scenarios in sequential decision-making environments using causal models and structured interventions.
• Advanced methodologies including dynamic programming, genetic algorithms, and importance weighting are employed to efficiently estimate counterfactual outcomes in complex sequential systems.
• Decomposition frameworks and model-agnostic recourse methods have facilitated practical applications in healthcare, recommender systems, and process mining while highlighting ongoing challenges in scalability and identifiability.

Sequential counterfactual evaluation is the principled quantification of "what-if" scenarios in systems where actions, interventions, or decisions are taken sequentially over time, possibly by multiple agents and under confounded or adaptive conditions. This paradigm lies at the intersection of causal inference, reinforcement learning, off-policy evaluation, and explainability, and has deep connections to sequential decision-making under uncertainty, potential-outcome models, and algorithmic recourse. The following sections present a comprehensive account of its foundations, methodologies, current advances, and open challenges as established in the latest research.

1. Formalisms and Core Counterfactual Policy Spaces

Sequential counterfactual analysis formalizes an agent's (or agents') interactions with an environment as a (structural) causal model augmented with time-indexed actions and outcomes. Fundamental constructs include:

• Causal Graphical Model and Structural Causal Model (SCM): Variables $V$ are related via a directed acyclic graph and structural functions; exogenous noise $U$ introduces stochasticity (Skalnes, 2022).
• Policy/Intervention Classes:
  • Soft-Intervention ($\pi$): Replaces a node's update rule by sampling from a policy map $\pi_N$ conditioned on ancestors.
  • Counterfactual Policy ($\rho$): The agent's natural action is first computed, exposed as an input to a stochastic remapping $\rho_N$ which can randomize or conditionally modify the action in light of the counterfactual intent.
  • Conditional-Twin Construction: Each decision node is duplicated to allow simultaneous reasoning over both factual and counterfactual trajectories, central for formal equivalence between policy classes (Skalnes, 2022).

In Markov decision process terms, these generalize to finding alternative action sequences or policies $\pi_{\mathrm{cf}}$ that optimize counterfactual outcomes under explicit sequential, partial-budget, or recursion constraints (Tsirtsis et al., 2021, Kobialka et al., 14 May 2025).

2. Algorithmic Approaches to Sequential Counterfactual Evaluation

2.1. Dynamic Programming and Constrained Optimization

Dynamic-programming–based formulations are used to yield optimal counterfactual explanations in finite-horizon MDPs, subject to sparsity constraints (e.g., at most $k$ deviations from the observed action sequence). Value functions are augmented to account for the change budget; recurrences dictate the maximal expected reward under any allowable alternative policy and can be solved in polynomial time in the trajectory length and action space (Tsirtsis et al., 2021).

2.2. Genetic and Heuristic Search in Discrete Sequences

In domains where sequences are categorical (e.g., item recommendations), the minimal-edit counterfactual generation task is NP-Complete (Scarcelli et al., 5 Aug 2025). Tailored metaheuristics such as the GECE genetic algorithm are employed: chromosomes encode entire action/item histories, and fitness blends Hamming distance to the observed sequence with the magnitude of outcome change. Realistic settings require specialized mutation, crossover, and selection parameters.

2.3. Off-Policy Evaluation and Importance Weighting

Off-policy counterfactual evaluation for sequential (slate or contextual bandit) decision making leverages reweighted estimators—most notably, the Recursive Importance-Weighted Probability (RIPS) estimator, which normalizes nested importance weights at each sequence position to simultaneously retain asymptotic unbiasedness and greatly reduce variance relative to plain IPS or decoupled estimators. This approach directly builds on an assumed causal graph of action–reward dependencies and enables data-efficient estimation of the expected cumulative reward under target sequential policies (McInerney et al., 2020).

2.4. Nonparametric and Generative Modeling for Time-Varying Treatments

For general treatments with arbitrary temporal structure, flexible generative models—conditional VAEs, guided diffusion models—are combined with longitudinal g-formula–based inverse-propensity weighting losses to enable sampling from full counterfactual trajectories. These methods admit evaluation and estimation in extremely high-dimensional or continuous state domains, under standard identification assumptions (consistency, positivity, sequential ignorability) (Wu et al., 2023).

2.5. Nearest Neighbor and Latent Factor Methods

In large-scale adaptive experiments, latent-factor models and nearest-neighbor estimators are invoked for inference of the counterfactual mean outcome at each unit-time-action triple. The approach leverages empirical distances over partially-observed realization histories and enables the construction of confidence intervals via non-asymptotic, entrywise error bounds (Dwivedi et al., 2022).

3. Decomposition and Attribution of Sequential Counterfactual Effects

Recent frameworks for sequential counterfactual effect explanation in multi-agent settings decompose total counterfactual effects (TCFE) into:

• Agent-Propagated Effects: Quantifying how much of an outcome difference is due to cascading behavioral changes among all agents (e.g., via interventions on all future actions). Attribution via the Shapley value yields a game-theoretic split among agents (Triantafyllou et al., 2024).
• State-Propagated Effects: Quantifying the loss attributable to the environment transitions shaped by early actions. Intrinsic Causal Contribution (ICC) assigns credit to particular state variables, normalized so that contributions sum to the state-propagated share of the TCFE. Computation relies on successive conditioning and Monte Carlo estimation.

Such decomposition supports granular interpretability and fairness analyses in multi-agent and dynamically evolving environments.

4. Model-Agnostic and Consequence-Aware Recourse Methods

Sequential counterfactual recourse incorporates the realities of actionable recommendation:

• LocalFACE (Small et al., 2023): Constructs step-wise recourse as locally feasible traversals in the data manifold, explicitly verifying high-density support and minimizing discrete path cost via graph search. Algorithmic recourse proceeds via sequential, privacy-preserving, and model-agnostic steps.
• Consequence-Aware Sequencing (Naumann et al., 2021): Encodes recourse as a multi-objective optimization over action sequences, incorporating not only direct feature costs but also context-dependent "consequential discounts", reflecting how earlier actions may facilitate or constrain subsequent edits. Optimization over sequence orderings is performed by non-dominated sorting genetic algorithms.

Both paradigms address feasibility (existence of realistic paths), interpretability (minimum steps or edits), and user-centric utility.

5. Applications, Empirical Evaluations, and Limitations

Empirical evaluation spans synthetic and real-world deployments:

• Healthcare Decision Support: Cognitive behavioral therapy and mobile health studies leverage sequential counterfactuals to suggest minimal action sequences (e.g., treatment plan changes) that would improve outcomes or recommend discharge (Tsirtsis et al., 2021, Dwivedi et al., 2022, Small et al., 2023).
• Recommender Systems: Both off-policy sequential evaluation (McInerney et al., 2020, Zenati et al., 2023) and explainability via counterfactual edits of user interaction histories (Scarcelli et al., 5 Aug 2025, Ren et al., 2023) are demonstrated, with model fidelity and edit-distance metrics.
• Process Mining and Markov Decision Processes: Counterfactual synthesis in learned MDPs (e.g., in customer process logs or streaming behaviors) empirically produces sparse and effective strategy perturbations that reduce undesirable outcomes as measured by reachability and probabilistic targets (Kobialka et al., 14 May 2025, Kinjo, 16 May 2025).
• Generative Modeling of Time-Varying Interventions: Full trajectory distribution estimation and sampling elucidate the impact of complex, temporally extended intervention regimens (Wu et al., 2023).

Major limitations identified include computational complexity (e.g., combinatorial action spaces or MIQCQP complexity for MDPs), incomplete theoretical understanding of identifiability in the presence of multiple confounded actions, and absence of regret, sample complexity, or robustness guarantees in many approaches. Empirical findings often rely on toy or moderately sized datasets; deployment at scale is constrained by either optimization or data estimation costs.

6. Open Challenges and Directions for Future Research

• Identifiability and Full Estimation in Multi-Action Settings: Consistent estimation of the multi-action counterfactual distribution remains open (Skalnes, 2022); bridging the gap between twin-network–style constructions and identifiability under general observational distributions is ongoing.
• Sample Complexity, Regret, and Confidence Guarantees: Finite-sample error bounds, regret analyses, and robust confidence intervals for sequential counterfactual estimators (especially in adaptive, non-i.i.d. settings) are areas of active research (Skalnes, 2022, Dwivedi et al., 2022, Zenati et al., 2023).
• Computational and Algorithmic Scalability: Handling large action/state spaces—especially in practical domains such as recommendation systems or real-world process logs—requires more efficient optimization and search methodologies without sacrificing interpretability or proximity guarantees (Kobialka et al., 14 May 2025, Scarcelli et al., 5 Aug 2025).
• Generalization to Partial Observability and Continuous Domains: Many existing algorithms assume finite, fully observable MDPs; extension to POMDPs and continuous state-action domains via function approximation is a natural next step (Tsirtsis et al., 2021, Wu et al., 2023).
• Causal Attribution and Fairness: Attributing and explaining counterfactual influences in multi-agent or multi-stage settings with fairness or transparency constraints, such as via Shapley or ICC decomposition, remains underdeveloped but critical (Triantafyllou et al., 2024).

References:

• (Skalnes, 2022) Sequential Counterfactual Decision-Making Under Confounded Reward
• (Triantafyllou et al., 2024) Counterfactual Effect Decomposition in Multi-Agent Sequential Decision Making
• (Tsirtsis et al., 2021) Counterfactual Explanations in Sequential Decision Making Under Uncertainty
• (Small et al., 2023) Counterfactual Explanations via Locally-guided Sequential Algorithmic Recourse
• (Wu et al., 2023) Counterfactual Generative Models for Time-Varying Treatments
• (Scarcelli et al., 5 Aug 2025) Demystifying Sequential Recommendations: Counterfactual Explanations via Genetic Algorithms
• (McInerney et al., 2020) Counterfactual Evaluation of Slate Recommendations with Sequential Reward Interactions
• (Dwivedi et al., 2022) Counterfactual inference in sequential experiments
• (Zenati et al., 2023) Sequential Counterfactual Risk Minimization
• (Kinjo, 16 May 2025) Analysis of Customer Journeys Using Prototype Detection and Counterfactual Explanations for Sequential Data
• (Kobialka et al., 14 May 2025) Counterfactual Strategies for Markov Decision Processes
• (Naumann et al., 2021) Consequence-aware Sequential Counterfactual Generation
• (Ren et al., 2023) Disentangled Counterfactual Reasoning for Unbiased Sequential Recommendation