WhatIf: Interactive Exploration of LLM-Powered Social Simulations for Policy Reasoning

Abstract: Policymakers in domains such as emergency management, public health, and urban planning must make decisions under deep uncertainty, where outcomes depend on how large populations interpret information, coordinate, and adopt over time. Existing tools only partially support this process: tabletop exercises enable collaborative discussion but lack dynamic feedback, while computational simulations capture population dynamics but are designed for offline analysis. We present WhatIf, an interactive system that enables policymakers to steer, inspect, and compare LLM-powered social simulations in real time. Informed by a formative study in emergency preparedness planning, we derive four design requirements for interactive policy simulations: fluid steering, real-time scale, collaborative exploration, and multi-level interpretability. We developed WhatIf guided by these requirements and evaluated it with five preparedness professionals across three disaster evacuation scenarios. Our findings show that participants used the system as a space for iterative branching and comparison rather than evaluating fixed plans; reflected on tacit planning assumptions when agent behavior violated expectations; surfaced previously unrecognized planning vulnerabilities; and grounded their reasoning in inspectable agent-level cases rather than aggregate outputs alone. These findings suggest broader design implications for LLM-powered social simulation systems: designing such systems as interactive, shared reasoning environments -- rather than offline predictive tools -- can better support expert decision-making under deep uncertainty.

• The paper presents an innovative LLM-empowered simulation platform that enables interactive, collaborative policy reasoning under deep uncertainty.
• It decouples LLM-based agent decision-making from deterministic spatial dynamics, achieving scalability with real-time feedback for over 12,000 simulated agents.
• Empirical evaluations reveal that iterative branching and in-depth agent inspection help experts refine policies in complex emergency management scenarios.

WhatIf: Interactive Exploration of LLM-Powered Social Simulations for Policy Reasoning

Introduction and Motivation

The "WhatIf" system represents a significant advance in LLM-driven agent-based simulation, explicitly targeting the needs of policy reasoning under deep uncertainty. Conventional policy simulation platforms have notable deficiencies: tabletop exercises foster collaboration but lack large-scale dynamic feedback, while computational simulations, including agent-based models (ABMs), generally remain offline and monolithic, offering only post hoc aggregate insights. WhatIf is engineered to fill this gap, supporting interactive, real-time, multi-user exploration of social simulations powered by LLM agents. Its guiding focus is the operational context of emergency management (e.g., evacuations), but the architecture is domain-agnostic.

Central to the paper's thesis is the claim that simulation for policy support is as much an interactive and collaborative process as a modeling one. WhatIf is not merely an agent society simulator but a fully-fledged shared reasoning environment designed to support fluid steering, real-time scale, collaborative exploration, and multi-level interpretability. The implication is that policy experts require tools for iterative, evidence-grounded reasoning—enabling branching, inspection, and comparison rather than one-shot plan validation.

System Architecture and Interaction Model

WhatIf structurally decouples two simulation layers: an LLM-based agent decision module and a deterministic spatial engine. The LLM layer provides open-ended deliberation and rationalization capabilities for agents, producing heterogeneous, context-dependent behaviors, while the spatial module ensures feasible execution of movement, hazards, and density effects among thousands of agents. This separation achieves scalability without sacrificing expressivity in agent reasoning.

The simulation cycle involves repeated rounds, where user interventions (e.g., altering announcements, placing security staff, introducing threats) update shared state and agent context. Agents enter LLM deliberation when their context changes, yielding new destinations, messages, and justifications. When context remains static, state propagation is purely deterministic. This enables selective batching of LLM calls, ensuring computational efficiency for large populations (over 12,000 agents in evaluation scenarios) while maintaining interactivity (Figure 1).

Figure 1: The WhatIf system overview illustrates the interplay between user interventions, batched LLM-driven agent decision-making, and deterministic world dynamics, all supporting real-time outcome inspection at individual and collective scales.

Fluid steering is supported through a multimodal interface: direct manipulation on the canvas, structured tool selection, and a natural-language command interface (the "What-If Agent") that maps user queries to atomic interventions (Figure 2). Intervention semantics are unified across modalities, ensuring rapid, low-friction scenario modifications with immediate visualization of propagation effects.

Figure 2: The system supports rapid intervention through direct manipulation, structured tools, and natural language, all yielding real-time feedback on agent behavior and macro-level dynamics.

Collaborative and Interpretability Features

A distinguishing aspect of WhatIf's design is its emphasis on synchronous, multi-user collaborative exploration. Users operate within a shared simulation workspace where state, interventions, and cursor presence are synchronized. Live annotations and shared canvas operations support situated sensemaking and discussion, closely approximating the informal, iterative style of in-person tabletop planning but at realistic, scalable population sizes (Figure 3).

Figure 3: Multiple users can collaborate in real time within the same simulation workspace, jointly steering, annotating, and inspecting scenario evolution.

Interpretability is realized at multiple abstraction levels. The system allows inspection of individual agent rationales, persona details, local group communications, and in-character interviews. At the macro-level, users can examine evacuation progress, congestion patterns, and outcome metrics, all linked to local actions through synchronized visual views. Temporal navigation enables rewind, comparison, and forking for direct counterfactual analysis across scenario branches (Figure 4).

Figure 4: Multi-level interpretability integrates agent inspections, rationale interrogation, system-level metrics, temporal navigation, and side-by-side scenario comparison.

Empirical Evaluation and User Study

The evaluation employed five emergency preparedness professionals, all embedded in the real-world context of university-scale disaster planning. Across guided, open-ended, and collaborative tasks, the users exercised the system on three main scenarios (e.g., tornado, active shooter). Data collection combined detailed interaction logs, structured questionnaires (NASA-TLX, SUS), free-form interviews, and scenario forking records.

A salient behavioral pattern observed was the cyclical engage-simulate-inspect-fork strategy. Rather than converging on a single "best" policy, participants habitually branched from expectation-violating or otherwise critical states, testing and iteratively refining plans through rapid intervention and round-trip inspection (Figure 5).

Figure 5: Time-aligned visualizations highlight recurring cycles of intervention, simulation, inspection, and forking across participants, revealing a fundamentally iterative exploratory pattern.

The system demonstrated robust usability: mean SUS scores were generally above standard benchmarks, and the system maintained real-time interactivity (∼1–2 seconds per round for the bulk of simulation steps, Figure 6).

Figure 6: After initial startup overhead, per-round simulation time remains stable at around 1–2 seconds, supporting iterative, real-time exploration with large agent populations.

Usability and cognitive workload ratings suggest that users perceived moderate to high mental and temporal demands, but low physical burden, aligning with the complexity of the task rather than system friction (Figure 7).

Figure 7: Workload ratings across NASA-TLX dimensions indicate cognitively engaging but not physically demanding interaction.

Usability was broadly positive (Figure 8), and system features targeting collaborative awareness and interpretability were rated especially high (Figure 9).

Figure 8: Most participants reported above-benchmark SUS scores for the WhatIf system.

Figure 9: System capabilities in collaboration, interpretability, and iterative exploration received uniformly strong Likert ratings from participants.

Notable Outcomes and Claims

Strong claims and observations include:

• Domain experts do not treat simulation as a static evaluation tool but as a space for interactive, evidence-driven reasoning, structuring their exploration around iterative cycles of branching, inspection, and collaborative negotiation.
• Inspectability at the agent level enables experts to surface and revise tacit planning assumptions, to diagnose model/pathology (e.g., agent homogeneity or implausible behaviors), and to construct stakeholder-relevant evidence from rich, concrete behavioral cases rather than aggregate outputs alone.
• High interpretability and collaboration features—such as explainable agent rationale inspection, forking, and synchronous state sharing—are essential for supporting expert reasoning and organizational knowledge transfer.

Identified tensions and limitations:

• Behavioral fidelity of LLM agents under extreme stress is still a challenge; agent decision homogeneity and lack of sufficient behavioral diversity pose risks for realistic policy modeling.
• System effectiveness for collaborative reasoning may depend on established group norms; simultaneous control is less crucial than shared deliberation and the visibility of collective sensemaking artifacts.
• Unmodeled environmental heterogeneity remains a limiting factor for scenario generalization.

Implications and Theoretical Perspective

By reframing social simulation as an interactive, shared reasoning environment, WhatIf advances a new design paradigm. The core implication is that real utility arises not solely from predictive accuracy, but from the degree to which the simulation enables experts to iteratively externalize, test, and contest their planning assumptions in light of inspectable evidence.

This has significant consequences for the design of future policy support systems: simulation should allow for robust branching, rapid counterfactual analysis, and deep inspection at both micro- and macro-levels. Trust in simulation becomes a dynamic, context-dependent property, shaped less by static model fidelity than by the transparency and critique-ability of assumptions and behaviors.

Directions for Future Research

Further research should address several open challenges highlighted by the study:

• Improving agent heterogeneity and plausible variability in LLM-driven behavior, particularly under high-stress, non-normative scenarios.
• Extending environmental and scenario modeling to better capture setting-specific complexities (e.g., infrastructures, geography, and emergent social structures).
• Scaling evaluation to multi-institutional and cross-domain contexts to validate generality of observed sensemaking patterns.
• Deepening the integration of interpretability features to proactively surface model failures and unresolved assumptions during expert workflow.

Conclusion

WhatIf demonstrates that LLM-powered, real-time, interactive simulation platforms can effectively support policy reasoning under uncertainty—not as predictive oracles, but as collaborative environments for exploratory, evidence-based deliberation. The system's architecture, interaction model, and interpretability instrumentation collectively enable iterative branching, deep inspection, and shared understanding, suggesting a new standard for decision support tools in complex sociotechnical domains.