1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On

Abstract: Virtual Try-On (VTON) has become a crucial tool in ecommerce, enabling the realistic simulation of garments on individuals while preserving their original appearance and pose. Early VTON methods relied on single generative networks, but challenges remain in preserving fine-grained garment details due to limitations in feature extraction and fusion. To address these issues, recent approaches have adopted a dual-network paradigm, incorporating a complementary "ReferenceNet" to enhance garment feature extraction and fusion. While effective, this dual-network approach introduces significant computational overhead, limiting its scalability for high-resolution and long-duration image/video VTON applications. In this paper, we challenge the dual-network paradigm by proposing a novel single-network VTON method that overcomes the limitations of existing techniques. Our method, namely MNVTON, introduces a Modality-specific Normalization strategy that separately processes text, image and video inputs, enabling them to share the same attention layers in a VTON network. Extensive experimental results demonstrate the effectiveness of our approach, showing that it consistently achieves higher-quality, more detailed results for both image and video VTON tasks. Our results suggest that the single-network paradigm can rival the performance of dualnetwork approaches, offering a more efficient alternative for high-quality, scalable VTON applications.

• The paper introduces a single-network approach with tailored normalization, significantly enhancing virtual try-on segmentation accuracy and temporal consistency.
• It leverages state-of-the-art tools like GroundingDINO-V1.5 with SAM-2 and SAPIENS to address garment and arm segmentation challenges in videos.
• Comprehensive evaluations on multiple datasets demonstrate improved SSIM, LPIPS, KID, FID, and VFID metrics, confirming the method’s superiority.

An Overview of "1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On"

The paper, titled "1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On," addresses critical issues in the domain of Virtual Try-On (VTON), focusing particularly on video segmentation and modality-specific normalization structures to improve garment visualization accuracy and realism. The paper proposes advancements to overcome the limitations present in existing Video Virtual Try-On (VITON) methodologies, aiming to enhance the fidelity and applicability of VTON systems.

Challenges and Proposed Improvements

The authors of this paper identify several deficiencies in the current approaches to video segmentation for Virtual Try-On applications. Primarily, they highlight the inaccuracy in garment segmentation, particularly with lower-body clothing. Temporal continuity is compromised when segmentation occurs at the image level rather than accounting for video dynamics. Additionally, there are challenges in segmenting arms when subjects face away from the camera. To counter these, the paper introduces improvements leveraging GroundingDINO-V1.5 with SAM-2 for better clothing segmentation and SAPIENS for refining arm segmentation. These enhancements are shown to significantly improve segmentation quality across video frames, indicating enhanced temporal continuity and segmentation precision.

Dataset Utilization and Performance Metrics

To test and validate their approaches, the authors utilize several datasets, notably the VITONHD, DressCode, VVT, and VIVID datasets, with extensive pairing and categorizations that facilitate a comprehensive evaluation framework. Each dataset provides ample paired samples, serving as a reliable testbed for measuring performance improvements introduced by the proposed methods.

The authors evaluate image quality using Structural Similarity Index Measure (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS). They also consider Kernel Inception Distance (KID) and Fréchet Inception Distance (FID) to assess realism and fidelity. For video-specific results, Video Fréchet Inception Distance (VFID) is used to capture both visual quality and temporal consistency, thereby ensuring that the evaluation accounts for dynamic elements of video try-ons.

Quantitative and Qualitative Analysis

A significant portion of the quantitative results focuses on a novel Modality-specific Normalization strategy, explored through three distinct variants (V1, V2, and V3). The analysis reveals V3 as the superior strategy, effectively handling inputs from various modalities and improving performance metrics significantly over V1. The paper provides empirical evidence to justify their preference for V3, demonstrating clear advancements in both training and inference phases.

Qualitatively, the presented results show impressive garment detail preservation and length estimation across diverse clothing categories. Comparative analysis on the DressCode dataset illustrates the superior performance of the proposed approach relative to state-of-the-art methods in maintaining garment features and textures.

Demonstration and Future Implications

The paper includes a demo highlighting the capabilities of the proposed approach, presenting sequences that underscore the method's effectiveness in long-duration try-ons and high-resolution outputs. The results showcase the system's potential for practical applications, particularly in fashion retail where realistic and seamless virtual fittings are crucial.

Theoretical implications of the research suggest a promising future for VTON applications, especially with the rise of more integrated and precise segmentation strategies. The improved network paradigm advocated by this study might influence future works to focus on refining segmentation technology and modality-specific normalization techniques, potentially leading to widespread adoption in commercial applications.

Conclusion

This paper contributes noteworthy enhancements in the segmentation methodologies and normalization strategies within the VTON field. By addressing existing shortcomings and proposing a more streamlined, single-network paradigm, this research sets a foundation for future advancements in virtual try-on technologies, aiming for a more realistic and efficient user experience in digital fashion environments. Further exploration of integration techniques and dataset diversity could extend the applicability and robustness of such systems in increasingly complex virtual settings.