OmniAlpha: Unified RGBA Image Generation

• OmniAlpha is an end-to-end unified sequence-to-sequence framework that synthesizes multi-layer RGBA images across 21 diverse tasks.
• It introduces key innovations such as an alpha-aware VAE with opaque initialization and a novel 3D rotary positional encoding for concurrent layer processing.
• The framework leverages a high-fidelity AlphaLayers dataset and multi-task training to achieve state-of-the-art results in image synthesis, matting, and editing.

OmniAlpha is an end-to-end, unified sequence-to-sequence framework for multi-task RGBA image generation and editing, introducing a methodology for concurrently modeling and synthesizing multi-layer images (including alpha channels) across 21 tasks. It directly addresses the previous fragmentation between specialized alpha-aware models and unified RGB-only frameworks by enabling powerful, layer-aware generative modeling. The framework is enabled by two technical innovations: an alpha-aware VAE initialized via “opaque initialization,” and a novel 3D rotary positional encoding, MSRoPE-BiL, within a multi-image Diffusion Transformer architecture. OmniAlpha leverages a curated, high-fidelity dataset, AlphaLayers, to train a shared representation that achieves or surpasses state-of-the-art results on a variety of image synthesis, matting, composition, and decomposition tasks (Yu et al., 25 Nov 2025).

1. Architectural Innovations

1.1 Alpha-Aware VAE with Opaque Initialization

OmniAlpha employs an encoder–decoder VAE, initialized from a pretrained 3-channel (RGB) VAE. The transition to RGBA employs “opaque initialization,” whereby only the first convolutional layer of the encoder and the final convolution of the decoder are replaced. For the encoder, the weights for the RGB channels are copied and the alpha channel is zero-initialized. For the decoder, the RGB weights are copied, the alpha channel is zero-initialized, and the bias for the alpha output is set to one. The VAE is then fine-tuned on RGBA data using a composite objective:

$$\mathcal{L}_{\text{VAE}}(E, D) = \lambda_{\text{rec}}\|x - D(E(x))\|_1 + \lambda_{\text{perc}} \mathcal{L}_{\text{perc}} + \lambda_{\text{KL}}\text{KL}[q(z|x)\|\mathcal{N}(0,I)] + \lambda_{\text{ref}} \text{KL}[q(z)\|\mathcal{N}(0,I)] + \lambda_{\text{GAN}}\mathcal{L}_{\text{GAN}}(D(E(x)))$$

This enables rapid adaptation from large-scale RGB pretraining to effective RGBA representation learning.

1.2 Multi-Image Diffusion Transformer with MSRoPE-BiL

The denoising backbone is a sequence-to-sequence latent diffusion transformer that accepts variable-length sequences of input and target RGBA layers. The core tokenization includes text tokens from a vision-LLM (VLM), latent codes for multiple input/output images (flattened as patch tokens), and a task-identifying token. The MSRoPE-BiL embedding introduces a third “layer index” axis to standard 2D RoPE, enabling the model to distinguish and process all input/output layers concurrently. The token positional tuple is $(x, y, z)$, with $z$ (layer index) bi-directionally extendable to accommodate both inputs and outputs. For a token at $(x, y, z)$, the embedding is:

$$R(q; x, y, z) = \text{RoPE}_{2D}(q; x, y) \oplus \text{RoPE}_{1D}(q; z)$$

This facilitates stable multi-image, multi-layer modeling and robust sequence-to-sequence generation.

2. Construction of the AlphaLayers Dataset

AlphaLayers is a 1,000-sample, high-fidelity RGBA triplet dataset specifically synthesized and filtered to ensure multi-layer structural diversity and fine-grained alpha variation. The pipeline begins with roughly 10,000 RGBA foregrounds from public matting datasets, which are processed as follows:

1. Each foreground is captioned using a VLM (Qwen3-VL).
2. A composite scenario is imagined and described by the VLM.
3. A background-replacement instruction is synthesized and executed by Qwen-Image-Edit, yielding a composite.
4. The composite is masked to extract a background and again captioned.
5. Consistency metrics—foreground-to-composite MSE and recomposite-to-composite MSE—are computed; a score combines these and the top 1,000 triplets are retained.
6. Each triplet is augmented with four mask variants: continuous alpha, precise mask, trimap, and rough mask via morphological operations.

The dataset is split into 900 training and 100 test images (AlphaLayersTest), and is supplemented with out-of-distribution evaluations on AIM-500, RORD, and RefMatte-RW100.

3. Multi-Task Sequence-to-Sequence Formulation

OmniAlpha formulates 21 RGBA tasks as unified sequence-to-sequence prediction problems. Conditioning is achieved by prepending a task token and providing relevant embeddings (e.g., for masks, images, or text) to the input token sequence; text is always processed by the VLM stream, while spatial data uses a parallel stream into the DiT. The task suite spans text-to-image generation, various layer-conditioned completion tasks, mask-free and mask-conditioned matting, object removal, and layer decomposition.

Category	Task Example	Input(s)	Output(s)
Text-to-Image	Text-to-Image Generation	T_fg	I_fg (RGBA)
Layer-Cond. Comp.	FG→BG Generation	{I_fg}_i, T_bg	I_bg
Image Matting	Mask-Free Matting	I_comp, “Extract subject” prompt	I_fg
Object Removal	α-Cond. Removal	I_comp, α_fg	I_bg
Layer Decomp.	Mask-Free Decomposition	I_comp	{I_fg}_i, I_bg

All tasks share a latent diffusion loss:

$$\mathcal{L}_{\text{diff}} = \mathbb{E}_{D, t, \epsilon}\left[(1/m_D) \sum_{k=1}^{m_D} \|\epsilon_k - \epsilon_\theta(Z_t, t, c)_k\|_2^2\right]$$

For mask-conditioned cases with ground-truth alpha, a pixel-space $L_1$ loss on the alpha channel is added:

$$\mathcal{L}_\alpha = \lambda_\alpha \mathbb{E}_x|\alpha_{\text{true}}(x) - \alpha_{\text{pred}}(x)|$$

Total loss is $$\mathcal{L} = \mathcal{L}_{\text{diff}} + \mathcal{L}_\alpha$$ (if applicable).

4. Joint Training Protocol and Implementation

Training is conducted in two stages:

• Stage 1: The RGBA VAE is fine-tuned for 32k steps (batch size 16) using AdamW with learning rate 1.5×10⁻⁵ and a warmup/cosine decay schedule.
• Stage 2: The VAE is frozen; the DiT backbone is trained for 100k steps (batch size 8) with LoRA (rank 256) on all attention and MLP modules, and AdamW (constant LR 5×10⁻⁵).
• Mixed precision training is employed on 8× NVIDIA H20 GPUs.

This joint, multi-task training leverages all 21 tasks simultaneously, enabling rich cross-task representation sharing.

5. Evaluation Methodology and Results

Quantitative evaluation uses a combination of standard generative and matting metrics:

• Sum of Absolute Differences (SAD)
• Mean Squared Error (MSE) on alpha
• Mean Absolute Deviation (MAD), Gradient error (GRAD), Connectivity error (CONN)
• FID, CLIP-Score, LPIPS, PSNR, CLIP-FID

Notable results include:

• OmniAlpha reduces SAD for mask-free matting on AIM-500 by 83–85% vs. SmartMatting and AIM.
• In text-to-image generation, OmniAlpha matches or outperforms LayerDiffuse and AlphaVAE in FID and CLIP-Score.
• For layer-conditioned image completion, OmniAlpha is preferred in 85–95% of pairwise comparisons, both by human annotators and automated VLMs.
• In decomposition and object removal on RORD, OmniAlpha achieves superior LPIPS, FID, CLIP-FID, with up to 25.14 dB PSNR for direct removal.

6. Ablation Studies and Insights

Ablation experiments reveal:

• MSRoPE-BiL is essential: removing the layer axis increases SAD on AIM from 7.8 to 10.1 and reduces human preference in compositional completion tasks by ~15 points.
• Multi-task training is superior to single-task: single-task matting yields SAD of 12.6 versus OmniAlpha’s 7.8; single-task text-to-image models show 22% worse FID.
• For data scale, 500 triplets yield SAD of 12.3 on AIM, while extending to 2,000 (with lower-quality samples) gives only marginal improvement (SAD = 8.1); a carefully filtered 1,000-triplet set is optimal.

The dominance of unified, multi-task modeling is substantiated, indicating that transformer architectures, extended via alpha-aware VAE and layer-wise positional encoding, are highly effective for complex RGBA image generation and editing.

7. Significance and Implications

OmniAlpha demonstrates that a single transformer backbone, equipped with suitable architectural and representational advances, can simultaneously solve a broad range of RGBA synthesis and editing problems. The substantial error reductions in matting, compositional preference rates, and generalization over both in-distribution and out-of-distribution benchmarks establish a new baseline for unified RGBA modeling. The sequence-to-sequence formulation, along with the MSRoPE-BiL embedding, facilitates multi-layer, multi-condition modeling previously unattainable in both single-task and RGB-confined frameworks. A plausible implication is that future generative image architectures could universally adopt multi-layer representations and multi-task protocols to achieve both flexibility and performance (Yu et al., 25 Nov 2025).