Phase-Aware Cross-Attention (PACA)

• Phase-Aware Cross-Attention (PACA) is a mechanism that decomposes inputs into phase-specific components using explicit phase embeddings to align temporal features accurately.
• It has been empirically validated in medical imaging and video generation, showing improvements in classification AUC and synchronization of time-dependent outputs.
• PACA’s design supports diverse applications by enabling precise phase-aware conditioning, though its success depends on accurately segmented phase data.

Phase-Aware Cross-Attention (PACA) is a cross-modal and cross-temporal attention mechanism designed to selectively attend to phase-relevant features or tokens in structured, multi-phase data or temporally-resolved prompts. PACA has emerged as a key architectural primitive in domains requiring precise temporal or phase-aware conditioning, including fine-grained medical imaging analysis and temporally-aligned video generation with rich, structured conditioning cues. The mechanism has been formalized and empirically validated in recent works such as the Lesion-Aware Cross-Phase Attention Network for renal tumor classification (Uhm et al., 2024) and ActAvatar for temporally-controllable talking avatar synthesis (Peng et al., 22 Dec 2025).

1. Conceptual Foundation

PACA is fundamentally motivated by the need to model explicit relationships between discrete temporal or acquisition phases present in multi-phase datasets. Standard attention mechanisms treat all conditioning inputs identically over time or across structural phases; PACA decomposes the context into temporally-anchored or phase-specific components and introduces explicit phase embeddings, allowing features or tokens pertinent to a given temporal segment to be prioritized at the appropriate computational step.

In medical imaging, this addresses the variability in lesion enhancement patterns across CT phases, improving discriminative feature aggregation. In generative video models, such as ActAvatar, PACA enables the model to "fire" action-related tokens only during specified temporal intervals, sharply aligning semantics and generation time.

2. Mathematical Formulation and Workflow

The technical realization of PACA generally follows a two-stage formulation: (i) prompt or feature decomposition with explicit phase labeling and (ii) phase-aware cross-attention with customized query-key-value (QKV) computations and learnable phase embeddings.

Lesion-Aware PACA for Multi-Phase CT (Uhm et al., 2024)

1. Multi-phase CT scan $$\mathcal I = \{I_i \in \mathbb R^{H \times W \times D}\}_{i=1}^N$$ is first segmented via a 3D U-Net to yield binary lesion masks $\hat S_i$.
2. For each phase $i$, $Q_i$, $K_i$, $V_i \in \mathbb R^C$ vectors are extracted via three parallel 3D convolutional streams and masked average pooling (MAP) over $\hat S_i$.
3. Phase embeddings $P_i \in \mathbb R^C$ are added to $Q_i$, $K_i$, $V_i$, yielding $\tilde Q_i = Q_i + P_i$ etc.
4. The attention matrix $A \in \mathbb R^{N \times N}$ is calculated by the scaled dot-product softmax:

$$A_{ij} = \frac{\exp(\tilde Q_i \cdot \tilde K_j/\sqrt C)}{\sum_{j'}\exp(\tilde Q_i \cdot \tilde K_{j'}/\sqrt C)}$$

1. The attended value is $F_{\mathrm{out}} = V + \lambda AV$, flattened and passed to a multi-layer perceptron for classification.

Hierarchical PACA for Video Prompt Alignment (Peng et al., 22 Dec 2025)

1. The prompt $P$ is decomposed into a base block $B$ (global, static semantics) and $K$ phase blocks ${P_k}$, each associated with temporal windows $T_k = [\tau_k^{start}, \tau_k^{end}]$.
2. After tokenization, context tokens $C = \{c_i\}_{i=1}^M$ are partitioned via binary masks $M_\text{base}$ and $M_k$ to select relevant tokens.
3. Each $c_i$ is augmented with a phase embedding $e_k$ if $c_i \in I_k$, so $c_i' = c_i + e_k$.
4. At each transformer block and video frame $f$ with normalized time $\tau$, cross-attention is performed:

$$\text{Attention}_\text{PACA}(X_f, C') = \text{softmax}(Q_f K'^{T}/\sqrt D) V'$$

where $Q_f = X_f W_Q$, and $K', V'$ constructed from the phase-embedded $C'$.

1. The softmax is empirically observed to concentrate on phase blocks $P_k$ for which $\tau \in T_k$ plus $B$, achieving temporal selectivity.

3. Architectural Instantiations

The specific implementation of PACA varies by application but exhibits shared elements:

Component	Multi-Phase CT (LACPANet) (Uhm et al., 2024)	Video Generation (ActAvatar) (Peng et al., 22 Dec 2025)
Input context	$\{I_i\},\{\hat S_i\}$ (CT images, masks)	Tokenized prompt $C$ with base/phase blocks, time intervals $T_k$
Phase embedding	$P_i\in \mathbb R^C$ (per-phase lesion vectors)	$e_k\in \mathbb R^{D_c}$ (per-phase text embeddings)
Attention backbone	Single-head dot-product across $N$ CT phases	Multi-block diffusion transformer with parallel audio/text attention
Output integration	Flattened per-phase attended vectors	Residual addition with progressive audio weighting

In LACPANet, PACA modules operate at both low- and high-level feature scales, jointly fusing multi-scale lesion patterns. In ActAvatar, PACA is paralleled with audio cross-attention, with a depth-aware scaling function governing the contribution of modalities at different network blocks.

4. Training, Hyperparameters, and Implementation

LACPANet (Uhm et al., 2024)

• Lesion segmentation: 3D U-Net backbone, trained with Dice loss.
• PACA Q/K/V channels: $C=8$ (base), $2C$ (high-level), single-head attention.
• Phase embeddings are raw vectors added to Q/K/V.
• Classification: three-layer MLP (FFN) with softmax.
• Loss: weighted cross-entropy over fused multi-scale outputs, with loss balance factor $\beta=0.1$.
• Input preprocessing: image resampling, intensity clipping, ROI cropping.

ActAvatar (Peng et al., 22 Dec 2025)

• Text tokens ($M\approx64$), embedding dim $D_c=1024$.
• Phase embeddings ${e_k}_{k=0}^K$ initialized to zero, learned during Stage 2.
• 30-layer diffusion transformer backbone, parallel PACA and audio cross-attention per block.
• Stage 1: freeze backbone, train only audio-attention on diverse clips.
• Stage 2: full fine-tuning on structured prompts with phase blocks.
• Optimization: AdamW, two-stage curriculum, classifier-free guidance.

5. Empirical Results and Evaluation

PACA consistently improves both discriminative and generative performance in structured, temporally-complex tasks.

• In lesion subtype classification (Uhm et al., 2024), adding PACA to the baseline yields a gain of 2–4 AUC points across both semi- and fully-automated settings, with further ≈1 pp improvement from multi-scale aggregation. Precision, recall, and F1-score also increased by 5–10 percentage points compared to baseline attention-free models.
• In talking avatar synthesis (Peng et al., 22 Dec 2025), PACA delivers phase-level temporal-semantic synchronization, ensuring that model outputs (gestures, lip movements) are precisely aligned to prompt-anchored time intervals. This obviates the need for low-level control signals such as pose skeletons and improves both action control and visual quality compared to non-phase-aware or flat-prompt architectures.

6. Domain Significance and Extensions

The introduction of PACA demonstrates that explicit phase or temporal anchoring within attention mechanisms confers substantial benefits in both interpretability and task performance for structured, time-indexed data. In medical contexts, this supports the formulation of lesion signatures that better capture phase-dependent enhancement, directly reflecting clinical diagnostic practice. In generative video, PACA enables nuanced, hierarchically-structured prompts that can control complex, multi-modal outputs without auxiliary labels.

A plausible implication is that PACA or related phase-aware mechanisms may generalize to any domain characterized by distinct, interpretable temporal or acquisition segments, including physiological time series, staged procedures, or dialogue systems requiring tight temporal-semantic coupling.

7. Limitations and Considerations

While PACA produces robust gains when temporal or phase structure is explicit, its efficacy may diminish if phase labels are noisy or if prompt decomposition is under-specified. Both cited works assume the availability of phase-segmented data or structured prompts. The mechanism introduces additional learnable embeddings (per phase or block), potentially increasing training and inference cost. Neither work reported catastrophic forgetting in multi-modal contexts, but this remains a potential risk in continual learning environments.

The precise mathematical formalism and empirical effects described in (Uhm et al., 2024) and (Peng et al., 22 Dec 2025) establish PACA as an effective and generalizable method for phase- or time-resolved cross-attention in both discriminative and generative deep learning architectures.