JamBayes: User-Expectation Surprise

• The paper introduces a computational framework that quantifies surprise as a deviation from user expectations using Bayesian and marginal probabilistic models.
• It constructs context-sensitive models from large-scale, historical data to detect and forecast unexpected events in real-time.
• Applications span urban traffic management and indoor spatial analysis, providing actionable alerts for improved decision-making.

User-Expectation Centered Surprise (JamBayes) is a principled computational framework for quantifying, detecting, and forecasting surprising events relative to an explicit model of user expectation. “Surprise” is not treated as an intrinsic anomaly in data, but is rigorously formulated as a function of how observed or predicted states deviate from what a typical user expects in a given context. The JamBayes approach has been deployed in large-scale traffic forecasting systems for urban environments and extended to domains such as spatial perception in indoor environments. The methodology centers on deriving expectation from historical, contextually-stratified data, using marginal and Bayesian probabilistic models, and operationalizing surprise as expectation violations that are salient and alert-worthy for the end user (Horvitz et al., 2012, Feld et al., 2020).

1. Defining Surprise as Expectation Violation

At the core of JamBayes is the mathematical reification of user expectation. Rather than flagging all statistical outliers, JamBayes builds, for each observable (e.g., a traffic bottleneck or spatial feature), a marginal, time-indexed, and context-sensitive probabilistic model of typical behavior. In the deployed traffic setting, the expected state $P(S_i(t) = s \mid C(t))$ at each bottleneck $i$ is conditioned on context $C(t)$, including time of day (15-minute intervals), day of week, holiday status, and weather variables. Surprise is then operationalized as a binary indicator:

$$\text{Surprise}_i(t) = \begin{cases} 1 & \text{if } P(S_i(t)=s\mid C(t)) \leq \epsilon \ 0 & \text{otherwise} \end{cases}$$

where $\epsilon$ (default $\approx 0.10$) is a tunable rarity threshold. This formulation captures the intuition that commuters are only surprised by states markedly inconsistent with their empirically learned contextual expectations—for instance, a traffic jam in a location and at a time with low historical jam probability (Horvitz et al., 2012).

2. Marginal Model Construction and Feature Sources

Expectation models are computed via empirical case-counting over large, multi-year datasets. For each bottleneck and stratified context (defined by 15-minute time slots, weather, and holiday information), the system records the empirical frequency distribution over discrete congestion states. This process produces a multinomial representation of normalcy from the perspective of regular users.

The predictive and surprise models exploit a rich feature library, with continuous updates at minute-scale granularity. Features include:

• Sensor-derived traffic statistics: fraction of congested cells ($\text{pct\_black}_i$), time since jam onset ($\text{since\_black}_i$), recent velocity change rates, and their flowing analogs.
• Incident reports: binary accident flags from transportation authorities.
• Exogenous factors: precipitation, visibility, temperature, major event indicators (e.g., local sports games), and holiday/school vacation status.
• Temporal attributes: time of day, day of week, and specific time windows.

This extensive feature set enables both base-level forecasting and real-time surprise detection (Horvitz et al., 2012).

3. Probabilistic Learning, Inference, and Surprise Forecasting

Long-term predictions such as time to clearing or jamming are learned using large-scale Bayesian networks over all monitored hotspots and contextual variables. Structure learning employs a greedy hill-climbing algorithm with the Bayesian Dirichlet score; targets are handled via binary-Gaussian leaves for censored observations. The system infers posterior predictive distributions for future clear or jam times.

Forecasting future surprise—a distinguishing JamBayes advance—involves assembling training cases from time points with detected surprise under the marginal model. For each flagged event, the system uses features available at $t-\Delta$ (typically $\Delta=30$ minutes) to train a secondary Bayesian network whose output variable is $[\text{Surprise}_i(t+\Delta)]$. This enables the computation of $$P(\text{FutureSurprise}_i=1\mid \text{evidence at }t)$$ and proactive alerting for upcoming deviations from expectation (Horvitz et al., 2012).

The pipeline for surprise detection and forecasting is summarized as follows:

Step	Description	Main Output
1	Marginal user model computes $P(S_i(t)=s|C(t))$ for all $i,t$	Current surprise flag per bottleneck
2	For each surprise event, gather feature vector at $t-\Delta$	Training case for future surprise model
3	Train Bayesian net with binary target (future surprise)	Forecast model $P(\text{FutureSurprise}_i=1)$
4	At runtime, flag if this probability exceeds threshold	User alert or visualization update

4. Interface, User Alerts, and Visualization

JamBayes presents surprise information to users through both smartphone and desktop interfaces. Current bottleneck states are encoded as color segments (green/yellow/red/black) with superimposed “clock” icons:

• Red wedge: ML time until congestion clears (if jammed)
• Green wedge: time until next predicted jam (if open, and within 60 minutes)
• Tick marks: $\pm1\sigma$ confidence intervals around predictions
• Exclamation point: overlays the clock for a currently “surprising” state
• Question mark: indicates low-confidence model regions as flagged by an auxiliary reliability Bayesian net

The desktop interface (“Deskflow”) enables users to select routes, monitoring windows, and personalize alert criteria. Alerts for either predicted jams/clears or real-time/future surprises can be delivered via various channels, including on-screen, SMS, ringtones, or vibrations, reflecting flexible integration into commuter workflows (Horvitz et al., 2012).

5. Empirical Validation and Performance Characteristics

Empirical evaluations, conducted on the Greater Seattle area dataset with 22 monitored hotspots and over 2,500 daily users, establish the predictive and alerting efficacy of the JamBayes methodology:

• Base-Level Traffic Forecasting: Per-standard accuracy (prediction within $\pm15$ minutes) for “time until clear” ranges from 0.65 to 0.87, and for “time until jam,” from 0.84 to 0.98 across the network.
• Surprise Forecasting: Since surprise events are rare, overall prediction accuracy is uninformative. ROC curves flexibly display performance by plotting false-negative versus false-positive rates as the alert threshold varies. Accepting a 50% miss rate yields false positive rates as low as 5%, capturing about half of surprise events 30 minutes in advance with minimal over-alerting.
• User-Centric Design: The approach demonstrates practical feasibility at scale, with real users utilizing surprise alerts for actionable commuting decisions (Horvitz et al., 2012).

6. Extensions to Bayesian Surprise and Isovist Analysis in Spatial Domains

The general framework of expectation-centered surprise in JamBayes extends beyond traffic to spatial cognition and robotics. In “Bayesian Surprise in Indoor Environments,” an agent traverses a 2D floorplan and collects isovist measurements—quantitative descriptors of visible space (area, perimeter, occlusion, etc.)—at each step. For each feature, a Dirichlet-multinomial model maintains the agent’s expectation across K discrete bins.

Observing a new datum, the system uses Bayesian inference to compute the posterior $P(M|D_t)$ and quantifies surprise as the Kullback–Leibler divergence $D_{KL}(P(M|D_t)\|P(M))$, with analytic forms for Dirichlet-Dirichlet KL. Surprise maps generated from isovist features robustly highlight human-relevant salience in floorplans, support “fingerprinting” for summarizing trajectories, and guide exploration and alerting behaviors in mobile robots and spatial LBS agents (Feld et al., 2020).

Framework limitations include quantization artifacts from binning, fixed spatial step sizes that may not adapt to heterogeneous layouts, and independence assumptions across features. Proposed improvements involve moving to continuous distributions, temporal smoothing, fusion with multi-modal context, and addressing high-dimensional dependencies (Feld et al., 2020).

7. Significance and Implications

User-Expectation Centered Surprise (JamBayes) exemplifies a rigorous, operationalized approach to anomaly detection grounded in user-centric priors rather than data-intrinsic outlierness. By leveraging contextual marginal probability models and Bayesian inference, it supports scalable, interpretable, and actionable alerting across domains from traffic management to indoor spatial cognition. Empirical validation highlights both forecasting accuracy and end-user utility, and extensions demonstrate generality to robotics and intelligent LBS. This paradigm enables systems not merely to detect statistical anomalies, but to identify, forecast, and summarize those events that are meaningfully “surprising” relative to the expectations and needs of individual users or agents (Horvitz et al., 2012, Feld et al., 2020).