Attend-and-Excite Semantic Guidance

• Attend-and-Excite is an inference-time semantic guidance technique for text-to-image diffusion models that addresses catastrophic neglect and attribute-binding errors.
• It intervenes in the reverse diffusion process by nudging latent codes using gradient-based excitation of weak cross-attention regions, enhancing semantic accuracy.
• Quantitative evaluations reveal improvements of up to 8% in similarity metrics and high user preference, confirming its effectiveness in correcting modality-specific failures.

Attend-and-Excite is an inference-time semantic guidance technique for text-to-image diffusion models, designed to address the failure modes of catastrophic neglect and attribute-binding errors. Such failures frequently arise when models neglect subjects referenced in a prompt or incorrectly bind attributes to subjects, correlating with cross-attention distributions that assign negligible probability mass to neglected tokens. Attend-and-Excite, developed by Chefer, Alaluf et al., embodies a distinct instance of @@@@1@@@@ (GSN) methods and operates by intervening in the latent space during the reverse diffusion process to refine cross-attention activations. The method enhances the semantic faithfulness of generated imagery without additional model training or fine-tuning, preserving the original model’s learned distribution (Chefer et al., 2023).

1. Failure Modes in Text-to-Image Diffusion

Text-to-image generative systems such as Stable Diffusion exhibit impressive diversity and visual fidelity. Nonetheless, qualitative and quantitative analysis reveals persistent problems:

• Catastrophic Neglect: The model may omit subjects entirely. For instance, given a prompt such as “a horse and a dog,” output may depict only a horse, with the dog missing due to its token receiving negligible attention.
• Attribute-Binding Failures: Discrepancies in color or accessory bindings arise. For example, the prompt “a yellow bowl and a blue cat” sometimes results in swapped colors (a blue bowl and yellow cat).
• Correlation with Cross-Attention Maps: Neglect coincides with the near-absence of attention mass on the relevant subject token(s). If a spatial feature does not attend to token $n$, its concept is often omitted.

This diagnostic is foundational for subsequent intervention strategies.

2. Generative Semantic Nursing: Framework and Attend-and-Excite Instantiation

Generative Semantic Nursing (GSN) encompasses inference-time interventions aimed at “nursing” the latent code $z_t$ during the denoising trajectory, such that the resulting image more faithfully reflects the input prompt’s semantics. Attend-and-Excite concretely instantiates GSN by:

• Targeting the UNet cross-attention layers at each reverse diffusion timestep $t$.
• Identifying subject token indices $S$ in the prompt (typically nouns, selected via a part-of-speech tagger).
• Computing attention maps per subject and spatial patch.
• Defining a loss to selectively amplify the weakest maximal attention, i.e., encourage each neglected subject to receive sufficient mass.
• Nudging $z_t$ along the gradient of the loss prior to denoising to $z_{t-1}$.

Attend-and-Excite operates exclusively during inference, requiring neither retraining nor fine-tuning. It leverages the pretrained model's knowledge base and manipulates only the sample generation process (Chefer et al., 2023).

3. Mathematical Formalism: Cross-Attention and Excitation

For a prompt tokenization of length $N$ and spatial feature grid of $P^2$ patches, the cross-attention mechanism at each layer and head is defined as follows.

• Let $H\in\mathbb{R}^{P^2 \times d}$ denote image feature vectors, and $E\in\mathbb{R}^{N \times d}$ encode text tokens. Learned projections yield: $$Q = H W_Q,\quad K = E W_K,\quad V = E W_V,\quad W_Q, W_K, W_V \in \mathbb{R}^{d \times d}$$.
• Attention from patch $i$ to token $n$ at timestep $t$ is:

$$\alpha_t(i,n) = \frac{\exp(Q_t(i) K(n)^\top/\sqrt{d})}{\sum_{n'=1}^{N} \exp(Q_t(i) K(n')^\top/\sqrt{d})}$$

• Aggregated attended feature for patch $i$:

$H'_t(i) = \sum_{n=1}^{N} \alpha_t(i, n) V(n)$

• After updating the latent $z_t$ via gradient descent on the semantic excitation loss,

$$z_t \leftarrow z_t - \alpha_t \cdot \nabla_{z_t} \mathcal{L}$$

the new features $Q'_t$ induce recalibrated attention weights,

$$\alpha'_t(i, n) = \operatorname{Softmax}_n(Q'_t(i) K(n)^\top/\sqrt{d})$$

which is equivalent to adding excitation biases to the logits for neglected tokens:

$$\alpha^{\text{exc}}_t(i, n) = \operatorname{Softmax}_n(Q_t(i) K(n)^\top/\sqrt{d} + \delta_{t,n}(i))$$

where $\delta_{t,n}(i) > 0$ for the most neglected subject and its maximal patch.

4. Attend-and-Excite Algorithmic Workflow

The Attend-and-Excite refinement procedure is interleaved with the denoising steps of the pretrained UNet. At designated refinement timesteps $T_{\text{refine}} = \{ t_1, ..., t_k \}$ with pre-set thresholds $\{ \tau_1, ..., \tau_k \}$ and step schedule $\alpha_t$, the steps are:

1. Compute raw cross-attention maps $A^-$ for latent $z_t$ and prompt $P$.
2. Exclude start-of-text tokens, apply softmax normalization: $A = \operatorname{Softmax}(A^-_{:,1:N})$.
3. For each subject $s \in S$:
   • Extract attention map $A_s$ (reshaped as $P\times P$ grid).
   • Apply Gaussian smoothing (kernel $3\times 3$, $\sigma=0.5$) to prevent patch collapse.
   • Compute loss $L_s = 1 - \max_{i,j} G_s(i,j)$ (with $G_s$ the smoothed map).
4. Aggregate semantic loss $\mathcal{L} = \max_{s \in S} L_s$.
5. Update latent: $$z_t \leftarrow z_t - \alpha_t \cdot \partial \mathcal{L}/\partial z_t$$.
6. If $t \in T_{\text{refine}}$ and $\mathcal{L} > 1 - \tau_t$, repeat steps 3–5.
7. Denoise via UNet: $z_{t-1} \leftarrow \text{UNetDenoise}(z_t, P, t)$.

Timesteps for intervention typically span $t=T$ down to $t_\text{end}=25$ (for $T=50$ total), focusing on early structural formation. Minimal max-attention thresholds are enforced at $t=\{0,10,20\}$ with respective $\tau=\{0.05,0.5,0.8\}$, stopping early upon attainment.

5. Hyperparameters and Semantic Control Mechanisms

Key design choices include:

• Subject Token Selection ($S$): Noun tokens are automatically extracted via part-of-speech tagging as default. User override is supported for fine control.
• Gaussian Smoothing: Kernel size $3\times 3$, standard deviation $\sigma=0.5$, prevents the degenerate localization of attention (‘patch collapse’).
• Refinement Timesteps: Gradient intervention occurs primarily in the initial half of the denoising schedule, reflecting the early establishment of spatial structure.
• Step-Size ($\alpha_t$): Linearly decreased from 20 to 10, scaled to latent norm.
• Classifier-Free Guidance: Scale parameter $s=7.5$, consistent with Stable Diffusion defaults.

These mechanisms ensure balanced control over the excitation applied to neglected subjects.

6. Empirical Evaluation and Observed Outcomes

Attend-and-Excite was evaluated on three prompt subsets, each constituted by conjoined subjects:

Subset	#Prompt Pairs	Typical Prompt Construction
Animal–Animal	66	"a horse and a dog"
Animal–Object	144	"a blue cat and a yellow bowl"
Object–Object	66	"a chair and a lamp"

64 seeds per prompt yielded approximately 15,000 images in total. Four primary metrics were used:

• Full-Prompt CLIP Similarity: Cosine between CLIP-encoded prompt text and generated image.
• Minimum-Object Similarity: For each prompt-image pair, split prompt into constituent subjects, compute individual CLIP similarities, take minimum, then average across dataset.
• Text-Text Similarity: Caption each image with BLIP, measure CLIP cosine between prompt and caption.
• User Study: 65 raters select best result among four methods for 10 prompts per subset.

Baselines include Stable Diffusion (uninfluenced), Composable Diffusion (Liu et al. ’22), and StructureDiffusion (Feng et al. ’22).

Quantitative improvements:

• Minimum-Object Similarity: +7–8% absolute gain over Stable Diffusion and StructureDiffusion on all subsets.
• Full-Prompt CLIP Similarity: Typically +4–5% improvement.
• Text-Text Similarity: +4.7% to +16.5% absolute gain.
• User Preference: Attend-and-Excite chosen for 90.7% of Animal–Animal, 77.6% Animal–Object, and 77.2% Object–Object prompts.

Qualitatively, Attend-and-Excite exhibits robust mitigation of forgotten subjects and correct attribute bindings, with cross-attention maps serving as reliable local explanations once neglect is eliminated.

7. Limitations and Scope of Guidance

Attend-and-Excite is restricted to the semantic scope learned by the underlying base model; it cannot synthesize combinations outside of the pretrained distribution—subject pairs with unnatural relations (e.g., "an elephant with a sombrero") may induce artifacts. The approach leverages and preserves the pretrained model’s generalization capabilities without augmenting training data or network parameters.

A plausible implication is that inference-time GSN-type interventions offer an efficient pathway for improving semantic faithfulness in generative models without retraining, provided that guidance remains within the learned distribution (Chefer et al., 2023).