Anime Image Synthesis Benchmarking

• Anime image synthesis benchmarking is a systematic evaluation of generative models for anime art, integrating diverse datasets and advanced architectures.
• It leverages benchmark datasets like Anime-Face and MagicAnime to drive tasks such as image-to-video, audio-driven animation, and detailed facial synthesis.
• Quantitative metrics such as FID, IS, and MOS, along with ablation studies, guide improvements in model design and performance.

Anime image synthesis benchmarking encompasses the systematic evaluation of generative models tasked with producing high-fidelity anime content. This multidisciplinary effort brings together innovations in generative adversarial networks (GANs), neural ordinary differential equations (NeuralODEs), contrastive semi-supervised learning, and large-scale multimodal datasets. The principal goal is to provide reproducible, quantitative, and qualitative measures of model performance for anime image and animation synthesis across a variety of tasks, datasets, and benchmarks.

1. Benchmark Datasets and Annotation Protocols

Benchmarking in anime image synthesis critically depends on annotated, high-quality datasets. The Anime-Face-Dataset provides a focused setting for character generation, comprising 27,588 facial images (256×256, resized for experiments to 64×64, tensors normalized in $[-1,1]$), with splits of 80% for training, 10% each for validation and testing (Lu, 2024). In contrast, MagicAnime introduces a hierarchically annotated, multimodal, and multitask resource scaled for animation: 400,000 clips for image-to-video synthesis, 50,000 video-keypoint pairs, 12,080 for video-driven face animation, and 2,900 for audio-driven tasks (Xu et al., 27 Jul 2025).

The annotation hierarchy involves scene segmentation (via PySceneDetect), multimodal pairings (e.g., video–keypoints, image–video, text–video, and audio–video), and keypoint or landmark detection (68 facial, 133 full-body), with rigorous filtering for single-character scenes, valid pose/face detections, and clean, lip-synced speech for audio-driven animation.

2. Model Architectures and Innovations

Current benchmarks highlight divergent architectural advances:

• USE-CMHSA-GAN extends DCGAN by inserting a channel-attentive Upsampling Squeeze-and-Excitation (USE) module and a Convolution-based Multi-Head Self-Attention (CMHSA) module into the generator, preserving the original discriminator design (Lu, 2024). The USE module recalibrates channel-wise feature importances, while the CMHSA module introduces long-range spatial self-attention between deconvolutional layers, augmenting spatial coherence and feature integration.
• NijiGAN employs NeuralODEs to replace discrete ResNet bottlenecks within a CUT-style generator. The generator evolves hidden states $h(t)$ continuously by solving ODEs ${d h(t)}/{d t} = f_\theta(h(t), t)$ using Dormand–Prince solvers, advocating for continuous dynamic feature transformation (Santoso et al., 2024). This is paired with a contrastive semi-supervised learning framework exploiting both pseudo-paired (Scenimefy-generated) and unpaired anime samples.
• MagicAnime-based benchmarks facilitate benchmarking across multimodal generation tasks (audio-driven, video-driven, pose-/text-driven, and image-to-video) and encourage model development for animation scenarios far beyond static image synthesis (Xu et al., 27 Jul 2025).

3. Benchmarking Metrics and Evaluation Protocols

Anime image synthesis benchmarking utilizes both conventional and domain-specific quantitative metrics:

Metric	Definition/Formula	Role
Fréchet Inception Distance (FID)	$$FID = \|\mu_r-\mu_g\|^2 + Tr(\Sigma_r + \Sigma_g - 2(\Sigma_r\Sigma_g)^{1/2})$$	Distributional closeness (lower is better)
Inception Score (IS)	$$IS = \exp(\mathbb{E}_{x\sim p_g}[D_{KL}(p(y|x)\|p(y))])$$	Quality/diversity (higher is better)
Mean Opinion Score (MOS)	Human rater mean on style/content/quality (1–3 scale)	Subjective fidelity
PSNR, SSIM, LPIPS, L1	Standard perceptual and image quality measures in video/image-to-video tasks	Fidelity, structure, perceptual similarity
Valid Sample Ratio (VSR)	Fraction with valid output in test set	Task completion

All FID/IS comparisons are conducted on standardized samples (e.g., 10k images on anime-face-dataset), and FID is routinely computed to the same anime reference pool. MOS is assessed by 20–30 human raters on style clarity, content consistency, and overall quality (Santoso et al., 2024). MagicAnime expands to LPIPS, PSNR, SSIM, L1, and VSR, especially for video and audio-driven tasks (Xu et al., 27 Jul 2025).

4. Quantitative and Qualitative Benchmark Results

Empirical comparisons on standardized datasets enable rigorous evaluation of both baseline and advanced models.

Static Anime Face Synthesis

Model	FID	IS
VAE-GAN	64.45	2.60
WGAN	79.34	2.35
DCGAN	63.92	2.52
USE-GAN	58.99	2.69
CMHSA-GAN	55.82	2.68
USE-CMHSA-GAN	53.74	2.85

USE-CMHSA-GAN achieves lowest FID and highest IS by combining channel attention (USE) and spatial attention (CMHSA), confirming complementary effects: USE boosts channel discrimination, CMHSA enhances spatial/global consistency. Visual inspection reveals sharper eyes and higher-fidelity hair textures versus convolution-only architectures, while certain facial details (eyebrows, nose, mouth) remain underrepresented due to dataset biases (Lu, 2024).

Scene-Level Anime Translation

Model	FID	MOS	Params (M)
CartoonGAN	45.79	2.708	--
AnimeGAN	56.88	2.160	--
Scenimefy	60.32	2.232	11.38
NijiGAN	58.71	2.192	5.48

NijiGAN outperforms Scenimefy in FID with a reduced parameter count, at comparable or slightly lower MOS. CartoonGAN records the highest MOS, potentially due to explicit edge emphasis but often at the expense of content faithfulness. NijiGAN offers artifact-free outputs and consistent coloration, with ablation studies demonstrating NeuralODE's superiority over residual blocks in reducing checkerboard artifacts (Santoso et al., 2024).

Multimodal Anime Animation (MagicAnime-Bench)

Task	Top Baseline(s)	VSR (%)	PSNR↑	SSIM↑	LPIPS↓	L1↓
Audio-Driven Face Animation	Hallo	55	18.01	0.586	0.245	0.081
Face Reenactment	AniPortrait (FT)	54	24.65	0.920	0.067	0.023
Image-to-Video Animation	SVD, DynamiCrafter	97–99.5	30–31	0.73	0.19	--

Domain-specific finetuning (e.g., AniPortrait retrained on MagicAnime) yields marked improvements in anime-specific facial alignment and stylization. For I2V, SVD and DynamiCrafter attain high VSR, PSNR, and SSIM, demonstrating strong structural and perceptual correspondence (Xu et al., 27 Jul 2025).

5. Ablation Studies, Failure Modes, and Analysis

Ablation experiments underscore the importance of each architectural component. For USE-CMHSA-GAN, the USE module alone reduces FID by ≈4.9, and CMHSA by ≈8.1 over DCGAN, with their combination yielding optimal performance (Lu, 2024). In NijiGAN, NeuralODE bottlenecks lead to smoother generation than ResNet blocks, and the addition of a VGG perceptual loss increases global structuring at the expense of fine anime strokes (Santoso et al., 2024). In MagicAnime, curated keypoints improve face reenactment tasks marginally, and modeling limitations persist for head-pose variety and complex expressions.

Failure cases frequently arise from dataset biases, such as missing facial contours in training data, which impede the synthesis of nuanced features (e.g., eyebrows or mouths). Anime lip synchronization remains challenging, with audio tone/rhythm affecting output minimally, consistent with the stylized “open–close” mouth conventions in anime productions (Xu et al., 27 Jul 2025).

6. Benchmarking Best Practices and Future Directions

Best practices for reproducible benchmarking include:

• Publishing code, pretrained models, and data preprocessing scripts.
• Fixing random seeds for sampling, initialization, and ODE solver tolerances.
• Reporting optimizer parameters, batch sizes, and evaluation settings in detail.
• Utilizing standardized test sets (such as “AnimeScene1000” or MagicAnime splits), and consistent sample sizes (e.g., 10k images for FID/IS).
• Including extensive user studies (≥20 raters) for subjective measures like MOS.

Recommended future work involves expanding datasets to reduce style bias, enhancing discriminators (e.g., with self-attention or spectral normalization), integrating hybrid perceptual/objective losses, and evaluating on additional metrics such as precision/recall, LPIPS, and human preference scores. Extensions to longer sequences, multi-character scenes, and the addition of richer emotional/3D priors in audio-driven settings are encouraged (Lu, 2024, Santoso et al., 2024, Xu et al., 27 Jul 2025).

A plausible implication is that standardized, openly available multimodal datasets and benchmarks—such as MagicAnime-Bench—will accelerate the development of models capable of high-fidelity, controllable anime generation across domains and modalities.