E2E-Transformer-DiT Architecture

• The paper presents the E2E-Transformer-DiT architecture, unifying pure-transformer principles to handle complex visual, temporal, and multimodal tasks efficiently.
• It integrates the Diffusion Transformer (DiT) with advanced temporal dynamics and streaming mechanisms, showcasing improved performance in video generation and autonomous driving.
• It employs blockwise attention and spatiotemporal compression techniques to reduce computation and enhance inference robustness across diverse applications.

The E2E-Transformer-DiT architecture refers to a class of unified, end-to-end frameworks that integrate the Diffusion Transformer (DiT) backbone with principled transformer methodologies, enabling highly scalable and performant processing across complex visual, temporal, or multimodal domains. These systems generalize the DiT’s pure-Transformer design, extending it from static 2D images to long videos and task-parallel autonomous control, and they support advanced training schedules, parameter efficiency, and efficient inference via architectural innovations. Key examples include EasyAnimate for video generation and DriveTransformer for autonomous driving.

1. Architectural Principles and Foundations

The E2E-Transformer-DiT family leverages the pure-Transformer stack as its computational backbone, characterized by a sequence of pre-normalized Transformer blocks utilizing multi-head self-attention (MHSA), residual connections, and a feed-forward (MLP) layer. Unlike prior U-Net or CNN+Transformer hybrids, these architectures operate entirely within the transformer formalism—flattening spatiotemporal data into sequences of tokens, embedding positional and condition information, and enabling blockwise composition.

The standard DiT block processes token sequences $x^{(\ell)} \in \mathbb{R}^{N \times C}$ with layer pre-normalization, computing

$$\begin{aligned} &x_0 = \text{flatten}(z_t) + P,\ &Q = x^{(\ell-1)}W_Q,\ K = x^{(\ell-1)}W_K,\ V = x^{(\ell-1)}W_V,\ &A = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right),\ h_{\text{att}} = AV,\ &h_{\text{ffn}} = W_2\,\text{GELU}(W_1 x^{(\ell-1)}),\ &x^{(\ell)} = x^{(\ell-1)} + h_{\text{att}} + h_{\text{ffn}}. \end{aligned}$$

All implementations incorporate learned (or sinusoidal/rotary) positional encodings and, where appropriate, timestep and conditioning (e.g., text or class) embeddings.

2. Key Innovations: Task-Parallelism, Temporal Dynamics, and Streaming

E2E-Transformer-DiT extensions address modalities beyond static images by introducing:

• Task Parallelism: Every task (e.g., perception, prediction, planning in driving) is encoded as a set of learnable queries, processed jointly through the shared stack. Each block supports direct mutual attention between all tasks, abolishing the conventional pipeline order and enabling synergistic feature sharing. This design stabilizes training and mitigates cascading errors (Jia et al., 7 Mar 2025).
• Temporal Dynamics Handling: For applications such as video synthesis, temporal dependencies are modeled explicitly. EasyAnimate, for example, introduces the Hybrid Motion Module: a dual-pathway system performing both temporal attention (framewise) and global spatiotemporal attention, gated via a learned $\alpha$,

$$h_{\text{out}} = \alpha \cdot h_{\text{temp}} + (1-\alpha) \cdot h_{\text{global}},$$

ensuring frame-consistent motion and high-fidelity transitions across long clips (Xu et al., 2024).

• Streaming Processing: To scale to long horizons (temporal or sequential), queries from previous timesteps are stored and brought into current processing blocks via temporal cross-attention—with geometric and temporal embeddings aligned and compensated for ego-motion if necessary. This enables both online fusion of long-range context (as in DriveTransformer) and high-FPS recurrent processing (Jia et al., 7 Mar 2025).

3. Spatiotemporal Compression and Efficient Latents

For video and long sequence processing, the computational burden is alleviated by condensing the temporal axis and spatial resolution:

• Slice VAE: EasyAnimate utilizes a two-stage VAE scheme. Each video is divided into temporal slices, and each slice independently mapped via the VAE encoder into a latent tensor $$z_0^{(i)} \in \mathbb{R}^{S \times h \times w \times c}$$. For decoding, neighbor-consistent concatenation ensures continuity: $$\tilde z_0^{(i)} = [z_0^{(i-1)}; z_0^{(i)}; z_0^{(i+1)}],$$ with upsampling returning to the original frame count. This enables the system to handle video lengths up to $T = 144$ (Xu et al., 2024).
• Latent Structuring for Sparse Queries: In autonomous driving settings, raw image features from multiview sensors are flattened and indexed into queries for agents, ego-vehicle, and map entities; dense BEV grids are abandoned in favor of direct task-to-sensor attention (Jia et al., 7 Mar 2025).

4. Blockwise Attention Mechanisms and Core Operations

The core DiT block structure is augmented with application-specific attention mechanisms:

• Sensor Cross-Attention (SCA): Each task's query attends directly to raw sensor features (e.g., image backbone tokens with position encodings): $$\begin{aligned} Q_T &= (H_T + \text{PE}_T) W_Q,\ K_S &= (H_\text{sensor} + \text{PE}_\text{sensor}) W_K,\ A &= \text{softmax}(Q_T K_S^\top/\sqrt{d}),\ H'_T &= AV_S W_O. \end{aligned}$$
• Task Self-Attention (TSA): Concatenated task queries self-attend: $$[H'_{\text{ego}}, H'_{\text{agent}}, H'_{\text{map}}] = A V_{\text{all}} W_O.$$
• Temporal Cross-Attention (TCA): Each task query fuses information from its past states, motion-corrected, and time-embedded for streaming fusion.

In video generation, the Hybrid Motion Module is injected after specific DiT blocks and controls spatiotemporal coherence via dual attention pathways (Xu et al., 2024).

5. Training Protocols, Fine-Tuning, and Scalability

All E2E-Transformer-DiT systems rely on multi-stage training with VAE alignment, staged or joint domain adaptation, and optionally low-rank adaptation (LoRA)-style parameter efficiency:

Stage	Example (EasyAnimate)	Batch/Steps / LR
VAE Train (Stage 1/2)	MagViT init, slice VAE, decoder-tune	350k+200k+100k / 1e-4
DiT Image Alignment	Images only, VAE-align	20k / 1024 / 2e-5
Motion Module Pretrain	Images + video, freeze other layers	11k / 1024 / 2e-5
Full Video Pretrain/Fine-tune	All layers, high-res upscaling	60k+scale-up+HQ / 1024+
LoRA Fine-Tune (optional)	Low-rank adapters, domain adaptation	5k / 1e-4
DriveTransformer One-Stage (driving)	End-to-end (all tasks)	30 epochs / AdamW

Key optimizations include temporal slicing, mixed-precision batch bucketing by resolution/length, zero-initialized long skip connections for training stability of deep stacks, and fused CUDA kernels for high-throughput.

6. Performance and Quantitative Characteristics

E2E-Transformer-DiT architectures demonstrate superior throughput, memory efficiency, and task-level accuracy across domains:

• Video Generation: EasyAnimate can generate 144-frame high-resolution videos ($512^2$) at $\sim10$ fps with effective temporal consistency and spatiotemporal diffusion modeling (Xu et al., 2024).
• Autonomous Driving: DriveTransformer achieves state-of-the-art closed-loop driving scores (68.2 on Bench2Drive) and open-loop planning accuracy (average L2 0.40 m on nuScenes), with significant latency reduction versus BEV-centric approaches (e.g., $211.7$ ms vs $663.4$ ms for UniAD). Ablation shows that task-parallel and temporal attention pathways are critical to high robustness and low error rates (Jia et al., 7 Mar 2025).
• Scalability: Temporal slicing and feature sharing enable efficient processing of long video sequences on hardware of moderate capacity (e.g., A100 80GB), without the overhead of dense grids or repeated VAE-encoding (Xu et al., 2024).

7. Significance and Research Directions

The emergence of E2E-Transformer-DiT signifies a unification of vision, sequence, and control tasks under pure-transformer formalisms with powerful attention mechanisms. Their ability to replace explicit multi-stage pipelines with joint, attention-mediated inference reduces system complexity, minimizes error compounding, and facilitates scaling. The efficiency gains from sparse representation, auto-sliced temporal encoding, and streamed cross-frame fusion enable deployments in domains where legacy architectures are infeasible (e.g., long-form video, real-time autonomous driving).

Research avenues include further reduction of computational overhead via parameter-sharing or block distillation, enhancing cross-modal conditioning (e.g., richer prompt or multimodal context), and extension to reinforcement learning or interactive environments. The E2E-Transformer-DiT paradigm provides architectural and methodological templates that generalize across standard diffusion/image tasks and complex, streaming, or hierarchical visuomotor scenarios (Xu et al., 2024, Jia et al., 7 Mar 2025).