Decoupled Guidance: Disentangling Subject and Context Pathways in Text-to-Image Personalization

Abstract: Text-to-image personalization aims to generate a user-provided subject in novel scenes described by text. However, most existing methods encode subject identity (fidelity) and context (editability) through the same conditioning pathway, forcing the two to compete for attention-map resources. We refer to this phenomenon as conditioning entanglement and show that it induces a fidelity-editability trade-off. We further provide causal evidence by replacing the target subject token with a generic subject token, which produces shifts in attention allocation and corresponding changes in context adherence. To this end, we propose Decoupled Guidance (DeGu), a plug-and-play framework that routes subject identity and scene context through two independent guidance streams. We further introduce a spatial mixing mechanism that dynamically fuses these streams, ensuring each operates within its semantically relevant region without interference. Furthermore, DeGu can be readily applied to existing personalization methods without modifying the underlying backbone models, consistently improving the overall personalization performance while enabling inference-time control over the fidelity-editability balance, across diverse methods and backbones, including flow-matching Diffusion Transformers (DiTs).

• The paper introduces DeGu, a framework that decouples subject identity and scene context into separate guidance streams to resolve conditioning entanglement.
• It employs novel modules—CAE for context-agnostic embeddings, DGM for dual guidance, and TCAM for spatial fusion—to mitigate attention collapse and improve performance.
• Empirical evaluations demonstrate significant improvements in subject fidelity and compositional controllability across diverse personalization backbones with minimal storage and computational overhead.

Decoupled Guidance for Text-to-Image Personalization: Eliminating Conditioning Entanglement

Introduction

Text-to-image personalization aims to synthesize user-provided subjects within novel textual contexts while preserving subject-specific visual identity. Existing paradigms—including embedding optimization (e.g., Textual Inversion [TI]), finetuning (e.g., DreamBooth), and encoder-based methods—route both subject fidelity and contextual editability through a single conditioning pathway. This design induces what the authors define as conditioning entanglement, manifesting as a structural bottleneck characterized by attention collapse: subject tokens dominate attention maps, suppressing context tokens and yielding a monotonically increasing trade-off between fidelity and editability.

This work, "Decoupled Guidance: Disentangling Subject and Context Pathways in Text-to-Image Personalization" (2607.00766), presents Decoupled Guidance (DeGu), a modular framework that addresses this bottleneck. DeGu routes subject identity and context information through disjoint guidance streams, dynamically fuses them via spatial mixing, and offers architectural and paradigm-agnostic, plug-and-play integration for SOTA personalization backbones.

Analysis of Conditioning Entanglement

The paper presents a thorough causal analysis of conditioning entanglement in the unified-prompt setting. By quantifying the distribution of per-token cross-attention (designated as Attention Collapse Score, ACS), it shows that subject tokens in personalized models absorb a disproportionate share of attention compared to the pretrained model.

This effect is illustrated quantitatively: after TI personalization, the subject token captures on average 86% of the total cross-attention, suppressing context tokens to below 10%, with ACS > 80% in 94% of prompts (Figure 1).

Figure 1: Unified prompts lead to attention collapse—TI-trained subject token $S^\ast$ absorbs ~86% of cross-attention, context tokens approach zero.

Notably, increased subject-token capacity (n-token TI) intensifies this collapse; although fidelity improves with additional tokens, editability degrades monotonically (see Table S.3 and Figure 2/16), falsifying the hypothesis that the trade-off is a simple consequence of embedding dimensionality. Instead, these findings demonstrate that the primary impediment is architectural: conditioning pathways enforce resource competition that cannot be mitigated by token expansion.

DeGu: Decoupled Personalization Architecture

Context-Agnostic Embeddings (CAE)

DeGu introduces CAE to structurally eliminate context co-occurrence. The learnable subject token $[S^\ast]$ is trained exclusively on a padding-only sequence, ensuring no textual context is present during optimization. This isolates subject identity from prompt semantics, enabling CAE to encode subject appearance independently.

Decoupled Guidance Mixer (DGM)

DGM consists of two parallel guidance streams for subject fidelity and scene context, each assigned an independently tunable guidance scale ($\gamma_f$ and $\gamma_e$). Each guidance stream passes its own forward pass through the diffusion backbone, yielding disjoint key-value sets for the cross-attention mechanism. The overall noise prediction is then composed at the score level, essentially implementing a product-of-experts (PoE) score with negligible interference between objectives.

Figure 3: DeGu framework—CAE for context-independent subject embeddings; DGM for dual guidance; TCAM for spatial fusion.

Test-Time Cross-Attention Masking (TCAM)

Although pathway-level competition is eliminated by DGM, spatial leakage remains—fidelity streams may contribute extraneous subject-background information to the context region. TCAM addresses this by constructing a binary subject mask $\mathbf{M}_t$ directly from non-subject token attention in the latent space (e.g., $[\text{SOT}]$ or $[\text{EOT}]$), inverting and binarizing to partition pixelwise influence between guidance streams.

Qualitative and quantitative ablation (Figure 4) shows that TCAM outperforms segmentation-based approaches (e.g., Grounded SAM 2) both in mask reliability under noise and in subject fidelity at denoising time.

Figure 4: TCAM robustly localizes subject regions via latent cross-attention—external segmentation fails under high noise.

Plug-and-Play Integration

The modularity of DeGu enables seamless integration with any SOTA personalization method, including TI, P+, NeTI, DreamBooth, and MS-Diffusion, without modifying backbone parameters or retraining per-method. It injects CAE tokens and dual guidance streams as an additive score-level composition along with the host’s native conditioning.

Empirical Results

Quantitative Performance

Across compositional benchmarks (39 subjects; 25 scene prompts each), DeGu sets new Pareto frontiers for the fidelity-editability trade-off on SD v2.1, SDXL, and SD v3.5 (DiT). Masked DINO and CLIP-I (subject-only) and context-following CLIP-T are reported. Notably, significant improvements are seen in subject fidelity (MDINO gain up to +0.0786 over TI, +0.0449 over DreamBooth) without degrading editability. When integrated plug-and-play, DeGu consistently lifts the overall metric for all tested baselines, as summarized below:

Backbone	Paradigm	Baseline + DeGu (Overall)
SD v2.1	Embedding (TI)	0.5236 (+0.0283)
SD v2.1	Finetuning (DB)	0.5254 (+0.0127)
SDXL	Embedding (DTI)	0.5082 (+0.0477)
SDXL	Encoder (MS)	0.5046 (+0.0052)

Plug-and-play integration incurs negligible per-subject storage (320KB for CAE tokens compared to 3.4GB for DreamBooth checkpoint).

Qualitative Analysis

Figure 5 (SDXL) and Figure 6 (SD v2.1) delineate contrasting outputs: embedding optimization methods preserve scene composition but lose fine-grained subject identity, while finetuning methods preserve identity but overfit context; DeGu achieves both via decoupling.

Figure 5: DeGu harmonizes identity preservation and compositional scene fidelity—outperforming baselines and in plug-and-play mode.

TCAM is further validated to enable precise region control; Figure 7 demonstrates continuous and independent editability/fidelity manipulation at inference time by varying $\gamma_f$, $\gamma_e$.

Figure 7: Inference-time control—raising $\gamma_f$ or $[S^\ast]$0 influences subject fidelity and context adherence independently.

Ablation, Analysis, and Architectural Generality

Systematic ablations (Table S.4; Figure 8) reveal that each component (CAE, DGM, TCAM) is essential: removing CAE re-introduces context entanglement, DGM eliminates pathway interference but causes spatial leakage, and only TCAM confines each stream to its region for optimal performance.

DeGu's architecture-agnostic design is validated with SD v3.5 (DiT/Transformer backbone), achieving parity with the SOTA regardless of underlying cross-attention mechanics (Figure 9).

Figure 9: DeGu generalizes to DiT-based SD v3.5—subject identity is maintained across diverse compositional prompts.

Theoretical Significance and Broader Implications

By dissecting and eliminating conditioning entanglement, this work reveals the structural limitations in current personalization pipelines. Attention collapse is shown not merely as a symptom but as a direct causative failure mode inherent to shared-pathway designs. The DeGu architecture demonstrates that compositional controllability and high-fidelity subject preservation are not fundamentally at odds, but require explicit architectural factorization.

The product-of-experts perspective of DGM, together with CAE and TCAM, enables inference-time control—transforming fidelity/editability into a continuous user-adjustable trade-off rather than a fixed training-time decision. This has substantial implications for interactive AI media systems, multi-concept generation, and general personalized generative modeling.

Moreover, the framework is backward compatible, imposing minimal computational or storage overhead, and is extendable to multi-subject and advanced architectures. Future work can explore vector quantized representations for CAE, mask derivation in large-scale scene layouts, and extending decoupling principles to video and 3D personalized diffusion.

Conclusion

DeGu addresses and resolves the essential structural bottleneck of conditioning entanglement in text-to-image personalization, providing disjoint subject/context guidance via CAE, DGM, and TCAM modules. This disentanglement eliminates attention competition, achieves marked improvements in both subject fidelity and compositional controllability, and offers paradigm-agnostic, inference-time flexible personalization. The work establishes a new methodological foundation for future personalized generative systems.