Directional Textual Inversion (DTI)

• Directional Textual Inversion (DTI) is a technique that addresses embedding norm inflation by constraining updates to the unit hypersphere.
• It utilizes Riemannian Stochastic Gradient Descent and a von Mises–Fisher prior to optimize embedding direction while preserving in-distribution magnitude.
• The method enables semantically coherent interpolation and robust text alignment, offering enhanced prompt conditioning for personalized text-to-image generation.

Directional Textual Inversion (DTI) is a technique for personalized text-to-image generation that addresses fundamental limitations of standard Textual Inversion (TI) methods, particularly the phenomenon of embedding norm inflation that undermines prompt conditioning in pre-norm Transformer architectures. DTI constrains the update of learned token embeddings to directional movements on the unit hypersphere, employing Riemannian Stochastic Gradient Descent (RSGD) and a von Mises–Fisher (vMF) prior for robust and scalable semantic representation. This leads to improved text fidelity and supports semantically-coherent interpolation between personalized concepts, a capability unattainable with conventional unconstrained embedding optimization (Kim et al., 15 Dec 2025).

1. Motivation: Embedding Norm Inflation and Pre-Norm Transformer Degradation

Standard Textual Inversion (TI) introduces a new token embedding $\mathbf{e} \in \mathbb{R}^d$, optimized through backpropagation across a frozen text encoder and diffusion model. Empirical evidence indicates that the $\ell_2$-norm of learned embeddings undergoes extreme inflation, frequently exceeding $20$ compared to in-distribution vocabulary tokens (typically $\approx 0.4$). This out-of-distribution (OOD) magnitude causes the learned embedding to dominate LayerNorm or RMSNorm operations in pre-norm Transformer blocks. The result is an attenuated influence of contextual and positional signals, impairing prompt conditioning, especially for complex prompts.

Theoretical analysis of a generic pre-norm Transformer block substantiates these empirical failures. Consider the update equation:

$$x^{(\ell+1)} = x^{(\ell)} + F_\ell\bigl(\mathrm{Norm}(x^{(\ell)})\bigr),\quad \ell=0,\ldots,L-1,$$

where $x^{(0)} = m\,\mathbf{v} + \mathbf{p}$, $\|\mathbf{v}\|=1$, $m>0$, and $\mathbf{p}$ is the positional embedding. For large $m$:

• Positional attenuation: $$\bigl\|\mathrm{Norm}(m\mathbf{v}+\mathbf{p}) - \mathrm{Norm}(m\mathbf{v})\bigr\|_2 = O(\|\mathbf{p}\|/m)$$ as $m\to\infty$.
• Residual-update stagnation: For $\|F_\ell(\cdot)\|\le B_\ell$, the hidden state’s directional update is bounded:

$$\angle(x^{(\ell)},x^{(\ell+1)}) \leq \arcsin(B_\ell/\|x^{(\ell)}\|) \approx \frac{\pi}{2} \frac{B_\ell}{\|x^{(\ell)}\|}.$$

• Accumulated drift: Across $L$ layers,

$$\angle(x^{(0)},x^{(L)}) \leq \frac{\pi}{2} \frac{S_L}{\|x^{(0)}\|-S_L}, \quad S_L = \sum_{\ell=0}^{L-1} B_\ell.$$

As $\|x^{(0)}\|\to\infty$, the embedding direction “freezes,” and model dynamics are crippled.

2. Semantic Role of Direction Versus Magnitude

Nearest-neighbor retrieval in CLIP token space reveals that cosine similarity—capturing direction alone—recovers semantically similar words, whereas Euclidean distance (which conflates direction and magnitude) fails at semantic clustering. This demonstrates that the majority of semantic content is encoded in the unit-norm direction of the embedding vector. Therefore, effective text-to-image personalization should fix the embedding norm to its in-distribution value and optimize the embedding’s direction exclusively.

3. Methodological Framework: Hyperspherical Parameterization and Riemannian Optimization

Directional Textual Inversion (DTI) decouples the personalized embedding into magnitude and direction:

$$\mathbf{e} = m^\star\,\mathbf{v}, \quad \mathbf{v}\in\mathbb{S}^{d-1},\, \|\mathbf{v}\|_2=1, \quad m^\star\text{ fixed}.$$

All learning occurs over the unit hypersphere $\mathbb{S}^{d-1}$, preserving the embedding’s norm at $m^\star$, which is set to the average norm of vocabulary tokens to ensure in-distribution scaling.

Optimization proceeds via Riemannian Stochastic Gradient Descent (RSGD) on the sphere. For the direction $\mathbf{v}$, given the Euclidean loss gradient $\nabla_{\rm euc}$, the update projects $\nabla_{\rm euc}$ onto the tangent space at $\mathbf{v}$:

$$g = \nabla_{\rm euc} - (\mathbf{v}^\top \nabla_{\rm euc})\,\mathbf{v},$$

followed by retraction to the sphere:

$$\mathbf{v}_{\rm new} = \frac{\mathbf{v} - \eta\,g}{\|\mathbf{v} - \eta\,g\|_2}.$$

This update guarantees unit-norm, direction-only optimization at every step.

The prior regularization employs a von Mises–Fisher (vMF) distribution,

$$p(\mathbf{v}\mid\mu,\kappa) = C_d(\kappa)\, \exp(\kappa\,\mu^\top \mathbf{v}),$$

resulting in a Maximum A Posteriori loss of

$$\mathcal{L}(\mathbf{v}) = \mathcal{L}_{\rm data}(\mathbf{v}) - \kappa\,\mu^\top \mathbf{v},$$

with the prior contributing a constant Euclidean gradient $-\kappa\mu$, thus exerting a stable attraction toward the mean direction $\mu$.

The DTI procedure is detailed in the following pseudocode:

\begin{algorithm}[H] \caption{Directional Textual Inversion (DTI)} \begin{algorithmic}[1] \State \textbf{Input:} Diffusion model %%%%30%%%%, text encoder %%%%31%%%%, initial token %%%%32%%%%, fixed magnitude %%%%33%%%%, prior direction %%%%34%%%%, prior strength %%%%35%%%%, learning rate %%%%36%%%%, steps %%%%37%%%%. \State Initialize direction %%%%38%%%%. \For{%%%%39%%%%} \State Sample a minibatch of images and noises %%%%40%%%%. \State Compute text embedding %%%%41%%%%. \State Compute diffusion loss gradient %%%%42%%%% via backprop. \State Add prior gradient: %%%%43%%%%. \State Project to tangent space: %%%%44%%%%. \State Optionally normalize %%%%45%%%%. \State Retract on sphere: %%%%46%%%%. \EndFor \State \Return final embedding %%%%47%%%%. \end{algorithmic} \end{algorithm}

4. Experimental Evaluation and Ablation Studies

Performance of DTI versus standard and modified TI methods is established using image- and text-based metrics drawn from DINOv2 and SigLIP, respectively. In SDXL-based experiments, DTI achieves superior text alignment while retaining competitive subject similarity:

Method	Image (↑)	Text (↑)
TI	0.561	0.292
TI-rescaled	0.243	0.466
CrossInit	0.545	0.464
DTI (ours)	0.450	0.522

On SANA 1.5 and larger architectures, DTI's text alignment advantage further increases (up to $0.757$ compared to $0.646$). Ablation studies confirm:

• RSGD on the hypersphere is markedly more effective than Euclidean AdamW with subsequent re-projection.
• Setting $m^\star$ to the mean vocabulary norm is optimal compared to minimum or OOD values.
• A nonzero prior strength $\kappa$ (e.g., $10^{-4}$) best balances subject fidelity and text alignment.

Human evaluation (Amazon MTurk) preferentially endorses DTI for both image fidelity (43.45%) and text alignment (66.77%) relative to TI and CrossInit (Kim et al., 15 Dec 2025).

5. Interpolation and Creative Applications: Hyperspherical SLERP

A salient feature of DTI's hyperspherical parameterization is its support for spherical linear interpolation (SLERP) between personalized concepts. Given two unit directions $\mathbf{v}_A, \mathbf{v}_B$ and angle $\Omega = \arccos(\mathbf{v}_A^\top\mathbf{v}_B)$, interpolation for $\alpha\in[0,1]$ is given by:

$$\mathrm{slerp}(\mathbf{v}_A,\mathbf{v}_B;\alpha) = \frac{\sin((1-\alpha)\Omega)}{\sin\Omega}\,\mathbf{v}_A + \frac{\sin(\alpha\Omega)}{\sin\Omega}\,\mathbf{v}_B.$$

Multiplying by $m^\star$ and using the resulting embedding in the diffusion pipeline yields smooth, semantically valid blends of learned concepts. Linear interpolation of unconstrained embeddings, by contrast, fails to preserve semantic coherence. This practical capability expands the personalization and creative manipulation toolkit available in text-conditional generative pipelines.

6. Theoretical and Practical Significance in Text-to-Image Personalization

DTI provides a principled solution to degeneracies introduced by unconstrained embedding optimization in TI. By constraining optimization to the sphere and incorporating an explicit direction prior, DTI mitigates norm inflation, preserves prompt conditioning, and enhances semantic control. The approach unifies theoretical insights—such as norm-induced positional signal attenuation and residual update stagnation—with a robust, efficient practical algorithm. The resulting improvements in text fidelity and creative manipulation position DTI as a canonical solution to prompt-faithful text-to-image personalization in pre-norm Transformer text encoders (Kim et al., 15 Dec 2025).

A plausible implication is that future research in text-conditioned generative models may increasingly exploit hyperspherical optimization to address interpretability, control, and compositionality challenges.