Arachne: Orchestrating Cascades for Efficient Text-to-Video Model Training

Abstract: The rising demand for AI-generated videos is fueled by advances in large-scale Text-to-Video (T2V) models, trained on extensive datasets of video clips spanning diverse resolutions and durations. To address this data heterogeneity, current training methods often use a bucketing strategy that groups samples into discrete buckets for efficiency. However, this approach struggles to scale with compute and data volumes under static parallelism schemes, such as data and sequence parallelism, leading to significant workload imbalances and hardware under-utilization. In this paper, we present Arachne, a novel training framework for efficient T2V model training at scale. Arachne decomposes the training process into fine-grained computational units, called \textit{cascades}, orchestrating their distributed execution and synchronization across the cluster through coordinated spatial and temporal optimization. Our comprehensive evaluation demonstrates that Arachne reduces iteration time by up to 65\% over leading frameworks, exhibiting a positive scaling trend where its performance advantages amplify as training scale grows.

• The paper introduces Arachne, a system that decomposes training workloads into fine-grained cascades to mitigate GPU idle times in text-to-video model training.
• It employs a cascade-level planner, topology-aware resource mapper, and runtime executor to dynamically optimize both spatial and temporal scheduling across GPU clusters.
• Empirical results demonstrate up to 65% faster iteration times compared to existing methods, highlighting its scalability and efficiency in heterogeneous training environments.

Arachne: Fine-Grained Spatio-Temporal Orchestration for Efficient Large-Scale Text-to-Video Model Training

Motivation and Background

Text-to-Video (T2V) generative models, primarily based on the latent diffusion paradigm, have pushed the boundaries of media generation, simulation, and creative applications. State-of-the-art T2V models leverage massive, heterogeneous video-text datasets for training, but unique challenges arise due to inherent variability in video duration and resolution. Unlike LLMs, whose sequence-length distributions are long-tailed and skewed toward smaller contexts, T2V datasets are right-shifted, concentrated, and highly discrete in sequence length, requiring the processing of large, computationally intensive tensors even for short video clips (see Figure 1). Traditional parallelization strategies—including data parallelism (DP), sequence parallelism (SP), and bucketing—fall short in this scenario by inducing severe workload imbalance, resource under-utilization, and exacerbated GPU idle times, especially as data and model scales increase.

Figure 1: Sequence-length distributions for two T2V datasets (Koala and Lynx) highlighting right-shift, concentration, and discreteness compared to LLM corpora.

Bucketing has been widely adopted to mitigate these issues by grouping videos of similar lengths and resolutions. However, bucketing alone cannot fully eliminate imbalance due to strict static scheduling assumptions: each device group processes its own bucket in lock step, making the fastest devices frequently idle as they wait for stragglers with longer, more complex samples (see Figure 2).

Figure 2: Training step showing significant GPU idle time due to load imbalance under bucketing with static DP/SP settings.

Existing work on dynamic scheduling, particularly from the LLM domain, addresses communication overhead. Yet, for T2V, computation (not communication) is the primary bottleneck due to the quadratic complexity of self-attention over long video sequences. LLM-targeted optimizations hence provide limited speedups. A paradigm shift is required to enable both spatial and temporal flexibility in distributed training.

Arachne System Design

Arachne tackles these core inefficiencies by decomposing the training workload into fine-grained, schedulable computational units termed “cascades.” At its core, Arachne employs a unified spatio-temporal orchestration strategy. The training process is formulated as a Directed Acyclic Graph (DAG) of cascades, which are dynamically scheduled across GPU clusters to maximize hardware utilization and minimize idle times. The system comprises three components:

Figure 3: Overview of Arachne.

• Cascade-Level Parallelism Planner: This planner performs temporal optimization by decomposing computation into cascades, then solving a global makespan minimization problem. Optimization variables include per-cascade parallelism, schedule times, and resource allocation, subject to dependency, memory, and hardware constraints. The optimization is solved efficiently via a genetic algorithm-based heuristic, supported by module-specific cost models for latency and memory for both 3D VAEs and DiT backbones.
• Topology-Aware Resource Mapper: After logical planning, this component adapts the cascade schedule to the physical hardware topology. It applies a hierarchical placement algorithm prioritizing intra-node placements, balanced cross-node assignments, and communication-minimizing GPU selections. Inter-cascade placement aims to maximize GPU group overlap for efficient in-place data handoff, reducing explicit data transfers between pipeline stages.
• Runtime Executor: The executor enacts the optimized plan, orchestrating asynchronous cascade launches, managing inter-cascade data handoffs (with in-place or locality-driven communication), and handling heterogeneous gradient accumulation using a representative-based, two-phase synchronization protocol to ensure mathematically correct updates across a dynamically changing landscape of partial gradients.

Empirical Results

Arachne was evaluated using three premier T2V architectures—Wan2.1 (1.3B), CogVideoX (5B), and HunyuanVideo (13B)—over realistic curriculum-based, high-resolution datasets. Baseline comparisons include Megatron-LM (with ZeRO and CP), DeepSpeed, and FlexSP, covering both static and communication-oriented dynamic scheduling systems.

End-to-End Performance

Arachne consistently yields substantial speedups, reducing average training iteration time by up to 65% over Megatron-LM, 59% over DeepSpeed, and 35% over FlexSP in high-resolution regimes. Notably, this advantage scales positively with growing data heterogeneity and computational demands (see Figure 4). GPU idle ratio under Arachne drops to 8% compared to over 33% for communication-centric schedulers, confirming that fine-grained orchestration converts heterogeneity from a liability into a source of scheduling flexibility.

Figure 4: Average iteration time (in seconds) across three training stages for HunyuanVideo-13B, with Arachne’s speedup annotations.

Figure 5: Average GPU idle ratio across training stages; Arachne achieves an 8.1% ratio versus 33.6% for the best communication-based baseline.

Scalability

Thorough scalability studies assess Arachne’s efficacy over increasing model sizes, workloads with expanded heterogeneity (maximum frame windows), and cluster size scales to 64 GPUs. Results demonstrate Arachne’s performance advantage intensifies in all dimensions: throughput speedup over Megatron-LM rises from 35% at 1.3B to 43% at 13B parameter scale, from 21% to 35% as workload heterogeneity increases, and remains consistently 30–40% across expanded clusters (see Figure 6).

Figure 6: Throughput scalability of Arachne on HunyuanVideo-13B across dimensions of model size, data heterogeneity, and hardware scale.

Ablation and Case Studies

Ablation studies attribute the majority of iteration time reductions to intra-cascade and inter-cascade topology-aware resource mapping, with slowdowns of up to 19% in their absence—highlighting the criticality of adapting execution plans to hardware structure.

Execution timeline visualizations in case studies confirm Arachne’s ability to stagger independent cascades temporally and reallocate GPU resources spatially within a single iteration, thereby collapsing GPU idle windows and improving iteration makespan by over 30% relative to dynamic communication-oriented baselines.

Figure 7: Execution timeline visualization for a case study iteration, showing dense, staggered scheduling with minimal idle time.

Spatial workload balancing is quantitatively substantiated by per-rank TFLOPS analyses, with Arachne showing lower Coefficient of Variation (CV) and tighter distribution, and consistent temporal stability over continuous iterations (see Figures 10 and 11).

Figure 8: Per-rank TFLOPS distribution within an iteration; Arachne achieves tightly balanced computation compared to static baselines.

Figure 9: Temporal stability (CV) of per-rank TFLOPS across 50 iterations; Arachne maintains low temporal variance.

Theoretical and Practical Implications

Arachne shifts the design paradigm in distributed generative model training from coarse-grained, static or communication-bottlenecked strategies toward fine-grained, compute-centric spatio-temporal optimization. Critical implications include:

1. Enabling Efficient Utilization at Scale: The demonstrated positive scaling trend ensures Arachne remains effective as both models and datasets grow—an essential capability as generative models approach hundreds of billions of parameters and leverage even larger, more heterogeneous multi-modal corpora.
2. Flexibility Toward Newer Hardware and Model Classes: The system’s topology-awareness and generic cascade abstraction allow straightforward adaptation to evolving hardware configurations and potentially to non-diffusion architectures with analogous computational characteristics.
3. Implications for Foundation Models: For large multi-modal foundation models, where data heterogeneity and long-context window training are norm rather than exception, Arachne’s approach provides a blueprint for overcoming under-utilization barriers in foundational AI infrastructure.

Conclusion

Arachne introduces a principled, scalable framework for orchestration of large-scale T2V model training. By embracing workload decomposition to cascades and enabling dynamic, simultaneous spatio-temporal optimization, it achieves up to 65% faster iteration times in practical scenarios and robust scalability across key dimensions. Experimental evidence validates the central claim that fine-grained orchestration is a necessary and sufficient condition to harness the ever-increasing heterogeneity and scale of modern generative models, positioning Arachne as a template for the next generation of efficient, production-scale AI training systems.

(2607.01701)