OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens

Abstract: OmniLottie is a versatile framework that generates high quality vector animations from multi-modal instructions. For flexible motion and visual content control, we focus on Lottie, a light weight JSON formatting for both shapes and animation behaviors representation. However, the raw Lottie JSON files contain extensive invariant structural metadata and formatting tokens, posing significant challenges for learning vector animation generation. Therefore, we introduce a well designed Lottie tokenizer that transforms JSON files into structured sequences of commands and parameters representing shapes, animation functions and control parameters. Such tokenizer enables us to build OmniLottie upon pretrained vision LLMs to follow multi-modal interleaved instructions and generate high quality vector animations. To further advance research in vector animation generation, we curate MMLottie-2M, a large scale dataset of professionally designed vector animations paired with textual and visual annotations. With extensive experiments, we validate that OmniLottie can produce vivid and semantically aligned vector animations that adhere closely to multi modal human instructions.

• The paper introduces a tokenization strategy for Lottie JSON that efficiently linearizes nested animation data for high-fidelity auto-regressive generation.
• It presents MMLottie-2M, a large-scale, multi-modal vector animation dataset with rigorous benchmarks for semantic alignment and motion fidelity.
• Experimental results show improved rendering reliability and controllability across text, image, and video modalities compared to existing methods.

OmniLottie: A Unified Framework for Multi-Modal Vector Animation Generation

Introduction

The synthesis of vector animations from multi-modal prompts constitutes a key challenge in content creation, enabling resolution-independent graphics with explicit temporal control. Traditional approaches address either static vector graphic synthesis or post-hoc animation via keyframe manipulation, rarely integrating appearance, effects, and motion in an end-to-end manner. "OmniLottie: Generating Vector Animations via Parameterized Lottie Tokens" (2603.02138) proposes a paradigm shift by introducing a tokenization framework for Lottie's hierarchical JSON format, establishing an auto-regressive generative model conditioned on arbitrary input modalities (text, images, videos) and yielding expressive, editable vector animations. The paper also contributes MMLottie-2M, the first large-scale, multi-modal vector animation dataset, and MMLottie-Bench, a rigorous evaluation protocol for multi-modal fidelity and motion correctness.

Vector Animation Modeling and Lottie Tokenization

Direct generation of standard Lottie JSON is both inefficient and unreliable due to the verbosity and redundancy of its structure—models trained on the raw representation waste token budget on irrelevant formatting and frequently produce invalid or unrenderable outputs. The central innovation is a Lottie tokenizer formalizing a transformation from nested JSON structures into a linearized token sequence that encodes metadata, hierarchical layering, commands, and continuous parameters via discrete tokens with optimized offset mapping. This process abstracts away schema-induced noise, focusing capacity on the underlying parametric representation of animation primitives and effects.

Figure 1: Overview of the Vector Animation Data Construction Pipeline, detailing data collection, conversion, processing, video rendering, and annotation to yield a large-scale, well-annotated dataset.

This tokenization preserves all generative flexibility, supports lossless round-tripping (detokenization), and admits type-specific discretization, offset allocation informed by dataset-based parameter value distributions (see the parameter distribution figure in the appendix), and explicit end/command markers to define sequence boundaries. Notably, by reserving dedicated token subspaces for categories (binary flags, categorical variables, continuous scales, text identifiers) and maintaining structural markers, the model enables autoregressive generation with fine-grained semantic control.

Figure 2: Parameter Distribution Analysis for Vocabulary Offset Design, highlighting dense concentration areas and informing the tokenizer’s range allocation for efficient code space usage.

Figure 3: Token Efficiency Through Progressive Abstraction; token sequence length is reduced $\sim$5.3$\times$ from raw JSON, with semantic content preserved for high-fidelity generation.

MMLottie-2M Dataset and Standardized Benchmarking

To facilitate large-scale, instruction-conditioned training, the authors curate MMLottie-2M—a composition of two million Lottie animation samples, accompanied by rich text captions (coarse-to-fine, object- and motion-grounded), reference images, and rendered videos. The data sources span both web-crawled professional Lottie files (after strict cleaning/normalization) and animated SVG-conversions with diversified synthesized motions, ensuring architectural, temporal, and compositional diversity for robust learning.

A modular annotation pipeline, leveraging VLMs for semantic grounding and conformance to strict labeling guidelines, provides multi-granularity annotation tailored to appearance, color, geometry, and motion. The dataset includes detailed statistics—layer counts, duration, category distributions, and hierarchical complexity—that expose the non-trivial structure and diversity, positioning it as a foundation for vector animation research.

Figure 4: Visual summary of dataset composition, temporal and spatial statistics pre- and post-normalization, emphasizing diversity and cross-domain coverage.

The authors propose MMLottie-Bench, a dual-track benchmark partitioned into real (hold-out, professionally authored) and synthetic (prompt-based, model-generated) splits, ensuring train–test separation and robustness to future data leakage. The protocol scores both visual fidelity (FVD, PSNR, SSIM, DINO) and semantic alignment (using LLM-based object/motion alignment metrics), supporting automation, reproducibility, and human consistency validation.

The OmniLottie Model: Architecture and Training

OmniLottie builds on a pretrained vision-LLM (VLM)—specifically, Qwen2.5-VL—augmented with embedding tables for the new Lottie token vocabulary. The model consumes interleaved multi-modal instructions and generates Lottie token sequences autoregressively. The discrete token stream is subsequently detokenized into valid Lottie JSON, ensuring renderable, highly-editable vector animation outputs.

The training objective is standard next-token prediction under multi-modal conditioning, optimized with cross-entropy. Ablation studies confirm that fine-tuning directly on raw JSON is significantly inferior to training with the proposed tokenization, both in deterministically improving rendering reliability and boosting all fidelity/alignment metrics.

Experimental Results

Quantitative Benchmarks

Across all modalities (Text-to-Lottie, Text-Image-to-Lottie, Video-to-Lottie), OmniLottie exhibits superior performance versus LLMs, VLMs, and optimization-based/decomposition baselines:

• Text-to-Lottie: Success rate 88.3% (Real), FVD 202.14, best or second-best semantic and visual alignment scores—significantly ahead of Qwen2.5-VL, DeepSeekV3, and even the commercial Recraft system.
• Text-Image-to-Lottie: Success rate 93.3%, FVD 180.27, object and motion alignment at or slightly below AniClipart/Livesketch, but with an order-of-magnitude lower runtime and higher validity.
• Video-to-Lottie: Success rate 88.1%, best FVD (227.11), PSNR, SSIM, DINO, and far higher consistency than GPT-5/Gemini3.1.

These numerical results decisively establish the advantage of structured tokenization and unified training for generative controllability, compactness, and success.

Qualitative Analysis

Rendering samples demonstrate (1) precise adherence to input prompts, (2) plausible, varied, and visually coherent motions, and (3) robust handling of compositionally complex or multimodal conditioning. Comparison with VLM/LLM generators reveals chronic failures in the latter: high rates of schema violations, empty/malformed renderings, and inability to reliably link content structure with temporal dynamics.

Figure 5: Qualitative comparison of Text-to-Lottie outputs, revealing precise motion, object integrity, and clear semantic adherence from OmniLottie while other systems fail in prompt alignment or structural validity.

Figure 6: Text-Image-to-Lottie results; only OmniLottie consistently achieves both visual and semantic alignment with input images and captions.

Figure 7: Video-to-Lottie examples highlight the efficacy of parameterized tokenization and multi-modal inference—baseline LLM/VLMs fail to reconstruct valid outputs in most cases.

Analyses of ablations on (1) data mixing (SVG proportion) reveal that moderate inclusion of synthetic SVG-animated sequences enhances geometric coverage without sacrificing realistic motion, and (2) the tokenization strategy verifies it as indispensable to reliability and performance.

Figure 8: Qualitative impact of data composition on animation fidelity and prompt alignment.

Figure 9: Lottie Tokenizer ablation study with visualizations and qualitative metrics, illustrating failure rates without structured tokenization.

Failure Analysis

Failure mode taxonomy identifies the dominant error types for each class of method:

• LLM/VLMs: Specification/structural errors dominate (schema hallucination, attribute misplacement), reflecting lack of format-induced priors.
• Optimization-based: Input-dependency and convergence pathologies, due to multi-stage pipelines and keypoint/feature extraction failures.
• OmniLottie: Residual errors are confined to rendering-level (missing styles, timing errors, opacity/scale mispredictions), a significantly more tractable domain for improvement.

Implications, Limitations, and Future Prospects

From a practical standpoint, OmniLottie enables one-shot multi-modal animation synthesis, significantly reducing latency and token cost associated with template-based design or iterative optimization pipelines. Its outputs preserve editability, scalability, and cross-platform compatibility, in contrast to raster or non-vector approaches. Theoretically, the work illustrates that format-aware tokenization is essential for generative modeling of structured, parameter-rich domains, especially where schema conformance and renderability are non-negotiable.

Remaining limitations reside primarily in residual invalid sequences from autoregressive decoding, limited context in extreme compositional depth scenarios, and the brittleness of direct parameter regression under adversarial or ambiguous input conditions. Directions for future work include constrained decoding schemes, reinforcement/active learning-based reward shaping (renderability and perceptual feedback), and agentic integrations with professional DCC pipelines to close the designer-in-the-loop gap.

Conclusion

OmniLottie establishes a new foundation for multi-modal vector animation generation by introducing a robust tokenization and representation framework tailored to Lottie. The model’s auto-regressive design, enabled by large-scale, richly-annotated data and formalized benchmarks, yields marked improvements in reliability, semantic control, and generative quality across all major modalities of vector animation synthesis. This work sets a methodological precedent for future research on high-level scene parameterization and editable, instruction-driven creative toolchains in vision and graphics.