Dual-Pathway Framework DSTED

• The paper introduces a dual-pathway neural architecture that decouples temporal stabilization (RMP) and discriminative enhancement (UPR) to mitigate prediction jitter and class ambiguity.
• Empirical results on the AutoLaparo–hysterectomy benchmark demonstrate significant improvements in accuracy and reduction of over 50% frame-to-frame classification flips.
• The framework employs a confidence-driven gating mechanism to dynamically fuse features, offering enhanced reliability and potential applications in other sequential decision-making tasks.

The Dual-Pathway Framework DSTED is a neural architecture developed to address the challenges of prediction jitter and class ambiguity in surgical workflow recognition. By explicitly decoupling temporal stabilization and discriminative enhancement into two cooperative processing streams, DSTED achieves greater prediction smoothness and improved class separation compared to conventional single-pathway models, culminating in state-of-the-art performance on the AutoLaparo-hysterectomy benchmark (Chen et al., 22 Dec 2025).

1. Motivation and Architectural Overview

Surgical workflow recognition targets the assignment of surgical phase labels to video frames, enabling downstream context-aware assistance and automatic skill assessment. Prevailing models are limited by (a) substantial temporal instability—manifested as frame-to-frame label jitter—and (b) poor handling of visually ambiguous or under-represented phase transitions. These limitations stem from the intrinsic conflict between enforcing temporal smoothness and maximizing inter-class discrimination within a unified representation space.

DSTED addresses this by factoring the problem into two distinct, specialist pathways, each addressing a complementary objective:

• The Temporal Stabilization Pathway (@@@@1@@@@, RMP) filters and propagates features from temporally adjacent, reliable frames to promote coherent labeling.
• The Discriminative Enhancement Pathway (Uncertainty-Aware Prototype Retrieval, UPR) dynamically injects prototypical features from previously observed hard samples, enhancing separability in ambiguous cases.

Fusion is accomplished via a confidence-driven gating mechanism, ensuring that auxiliary pathway contributions are modulated in accordance with the baseline model's certainty estimates.

2. Reliable Memory Propagation (RMP): Temporal Stabilization Pathway

RMP is designed to suppress prediction jitter by selectively incorporating stable temporal context from past frames. For each timestep $t$, a sliding window memory bank $M_t = \{f_{t-K}, ..., f_{t-1}\}$ holds features from the $K$ most recent frames. Each memory entry $f_i$ is assessed for reliability using three criteria:

1. Feature similarity: $$s_\mathrm{sim}(f_t, f_i) = \frac{f_t \cdot f_i}{\|f_t\| \cdot \|f_i\|}$$
2. Class consistency: $$s_\mathrm{cls}(f_t, f_i) = (p_t^{\mathrm{base}})^{\top} \cdot p_i^{\mathrm{base}}$$, where $p^{\mathrm{base}}$ denotes the softmaxed baseline logits.
3. Temporal proximity: $$s_\mathrm{temp}(i, t) = \exp\left(-\frac{|t-i|}{\tau}\right)$$ with $\tau > 0$.

These are aggregated into a composite reliability score: $$r_i = s_\mathrm{sim}(f_t, f_i) + s_\mathrm{cls}(f_t, f_i) + s_\mathrm{temp}(i, t)$$. Only memory entries with $r_i > \theta$ (typically $\theta=0.75$) are retained for fusion, weighted as $$w_i = \frac{\exp(r_i)}{\sum_{j:\,r_j > \theta} \exp(r_j)}$$. The stabilized memory feature $f_t^m$ is extracted via:

$$f_t^m = \mathrm{Conv}\left(f_t, \left\{w_i f_i : r_i > \theta\right\}\right).$$

This mechanism ensures that sudden, low-confidence phase transitions do not introduce noise from unreliable temporal context, thereby substantially reducing high-frequency label flips.

3. Uncertainty-Aware Prototype Retrieval (UPR): Discriminative Enhancement Pathway

UPR targets improved discrimination, particularly at phase boundaries and in ambiguous frames. During training, for each class $c$, DSTED maintains a fixed-size prototype bank $P_c$ of the $N$ most uncertain samples, evaluated via $u_t = 1 - \max(p_t^{\mathrm{base}})$. Feature update decisions are driven by a lightweight policy network $\pi_\theta(s_t)$, where state $s_t$ contains $u_t$, entropy, margin, and bank size. Selected features are added to $P_c$ by ejecting the prototype with the lowest uncertainty if the bank is full.

At inference, for input feature $f_t$, cosine similarity is computed against all $p_j \in P_c$. Each similarity is weighted by the model's baseline class probability $p_t^{\mathrm{base}}[c_j]$ for the prototype's class $c_j$, giving $$s_j = p_t^{\mathrm{base}}[c_j] \cdot \mathrm{sim}(f_t, p_j)$$. The top-$k$ matches by $s_j$ are selected, weighted via a softmax over similarities, and aggregated:

$f_t^u = f_t + \sum_{j \in \text{top-}k} w_j p_j,$

with $$w_j = \frac{\exp(\mathrm{sim}(f_t, p_j))}{\sum_{\ell \in \text{top-}k} \exp(\mathrm{sim}(f_t, p_\ell))}$$. This injects "hard" feature variation, strengthening separation near ambiguous transitions.

4. Gated Fusion Mechanism

Combination of the baseline, stabilized, and enhanced features is governed by confidence-dependent gates. Defining $c_t = \max(p_t^{\mathrm{base}})$, the gate weights are:

$$g_m = \sigma(a_m (\tau_m - c_t)), \qquad g_u = \sigma(a_u (\tau_u - c_t))$$

where $\sigma$ is the sigmoid, $\tau_m, \tau_u$ are learnable thresholds, and $a_m, a_u$ are fixed scaling factors. The final representation is fused as

$f_{\mathrm{final}} = f_t + g_m f_t^m + g_u f_t^u.$

When model confidence is low, both auxiliary pathways contribute strongly; at high confidence, only the baseline is largely retained, minimizing spurious corrections.

5. Objective Function and Optimization

Training employs:

• Class-balanced cross-entropy loss $L_\mathrm{CE}$ for phase classification.
• Temporal smoothness regularization $$L_\mathrm{KL} = \sum_{t=2}^T \mathrm{KL}(p_t \parallel p_{t-1})$$ to penalize abrupt prediction changes.

The total objective is $L = L_\mathrm{CE} + L_\mathrm{KL}$. The UPR policy network is trained jointly to maximize the primary loss reduction, without requiring an explicit prototype regularization loss.

6. Experimental Configuration and Quantitative Results

DSTED utilizes a VideoMAE-V2 backbone with 16-frame clip inputs, optimized via AdamW (initial learning rate $10^{-3}$, 50 epochs, batch size 12, single NVIDIA A100). RMP maintains $K=60$ features per memory (about 1 minute of video), with threshold $\theta=0.75$. UPR banks store $N=256$ prototypes per class, retrieving top-$k=8$ at inference.

Evaluation is conducted on the AutoLaparo–hysterectomy dataset (21 cases, 83,243 frames, 7 phases) via three-fold cross-validation. Full DSTED achieves:

Method	Accuracy	Jaccard	Precision	Recall	F1
VideoMAE-V2	79.61 %	48.85 %	63.33 %	58.42 %	57.41 %
+ RMP only	81.80 %	51.83 %	65.04 %	61.79 %	60.21 %
+ UPR only	81.54 %	52.51 %	65.66 %	62.73 %	61.31 %
RMP + UPR (no gate)	83.27 %	55.13 %	65.72 %	65.86 %	63.80 %
DSTED (full)	84.36 %	57.60 %	67.05 %	67.59 %	65.51 %

RMP yields +2.19% accuracy and +2.80% F1, UPR adds +1.93% accuracy and +3.90% F1. Synergistic effects are observed when the pathways are combined, with the gating mechanism conferring a further +1.09% accuracy gain.

7. Impact and Analytical Findings

DSTED suppresses over 50% of frame-to-frame classification flips in jitter-prone video segments using RMP, and UPR corrects up to 15% of boundary-phase errors. These improvements in stability and discrimination outperform prior approaches on temporal consistency and ambiguous phase transitions, establishing DSTED as a robust standard for workflow recognition in minimally invasive surgery (Chen et al., 22 Dec 2025).

A plausible implication is that the dual-pathway decomposition—explicitly separating temporal stabilization and discriminative enhancement—may offer broader benefits in other sequential decision-making and time-series classification domains where analogous conflicts between history filtering and class sharpening arise.