Stitched Embeddings: A Unified Latent Space for 3D Garments and 2D Patterns

Abstract: While garments are essential for realistic digital humans, their topological variety makes them much harder to model than parametric bodies. Traditional tailoring relies on 2D sewing patterns, yet bridging these patterns to 3D geometry currently requires physical simulations. We present Stitched Embeddings, the first simulation-free framework to unify 3D garment reconstruction and sewing pattern inference within a single bidirectional latent space. By leveraging the geometric priors of a pretrained 3D foundation model, our approach overcomes the data scarcity typically associated with high-quality garment modeling. We propose to use the BoxMesh as a critical intermediate representation to align 2D panels into 3D configurations without the computational overhead of a simulator. This architecture achieves state-of-the-art accuracy in pattern reconstruction while significantly improving efficiency. Furthermore, our differentiable pipeline enables novel applications, including pattern recovery from meshes and 3D editing from 2D patterns. Finally, this work provides a scalable link between neural 3D vision and the physical garment manufacturing pipeline. Project Page: https://andreus00.github.io/stitchedembeddings

• The paper presents StEm-Net, which unifies 3D garment reconstruction and 2D sewing pattern inference in a single differentiable latent space.
• It employs a VAE-based architecture with BoxMesh representation to ensure precise topological correspondences and test-time latent optimization.
• Experiments show up to 15% improvements in accuracy metrics, demonstrating robust bidirectional translation and real-time editing capabilities.

Stitched Embeddings: A Unified Latent Space for 3D Garments and 2D Patterns

Introduction

The synthesis and reconstruction of digitally represented garments are increasingly central in digital human modeling. A persistent bottleneck has been the translation between 2D sewing patterns—integral to the physical garment pipeline—and 3D garment surfaces, particularly in the presence of high topology complexity and non-rigid deformations. "Stitched Embeddings: A Unified Latent Space for 3D Garments and 2D Patterns" (2607.00829) presents StEm-Net, the first framework to jointly model and infer both 3D garments and their manufacturing patterns within a shared, simulation-free, fully differentiable latent space. This work directly addresses the fundamental disconnect between neural 3D representations (e.g., unsigned distance fields) and the discrete, semantic structure required for manufacturing and animation, facilitating reciprocal prediction, efficient refinement via test-time latent optimization, and edit propagation across modalities.

Architecture and Methodology

Sewing Patterns and BoxMesh Representation

The foundational modeling assumption is that parametric sewing patterns, though semantically rich and physically grounded, are nontrivial for regression as they map to variable-length, high-dimensional sets of 2D curves. StEm-Net leverages GarmentCode [GarmentCode2023], parameterizing patterns through a fixed-dimensional code that abstracts away complex curve details into 122 interpretable parameters covering design intent, panel shape, topology, and connectivity.

Crucially, the framework introduces BoxMesh—a 3D spatial proxy generated from GarmentCode parameters—which layouts sewing panels in space before physics-based simulation. BoxMesh serves as an intermediate modality, encoding both geometric priors and sewing semantics, and provides precise topological correspondences between pre-simulated structures and final draped meshes.

StEm-Net Architecture

StEm-Net adapts the VAE backbone from Hunyuan3D 2.1 [hunyuan3d2025hunyuan3d], replacing SDFs with unsigned distance fields to handle the open manifolds of real garments. The architecture consists of two encoders mapping (a) 3D garment point clouds with normals ($\mathcal{E}_g$) and (b) BoxMesh surfaces ($\mathcal{E}_{bm}$) into a unified, high-capacity latent set ($\mathbb{R}^{256 \times 64}$). Correspondence between BoxMesh and garment mesh is enforced at the sampling stage, guaranteeing that corresponding physical locations map to similar latent codes. Decoding proceeds via separate heads: $\mathcal{D}_{3D}$ reconstructs the UDF, while $\mathcal{D}_{SP}$ transforms the latent set to sewing pattern parameters using transformer-based cross-attention with panel-specific queries.

Losses and Optimization

A composite objective governs training: mean squared error on the UDF, cross-entropy and MSE/BCE on the parametric pattern outputs, and KL-divergence for both regularizing the latent distribution and enforcing tight coupling between BoxMesh and garment mesh encodings. This design ensures that the method learns a latent space supporting direct and differentiable mapping between the geometric and parametric domains.

At inference, StEm-Net enables bidirectional translation between 3D surfaces and sewing patterns. Its differentiable structure supports test-time latent optimization: by minimizing UDF reconstruction loss with respect to the latent vector, the predicted sewing patterns adapt to better explain the given 3D observation, handling ambiguities and out-of-distribution geometries.

Experimental Evaluation

3D-Based Pattern Recovery

On established benchmarks (NeuralTailor and GarmentCodeData), StEm-Net achieves substantial improvements over NeuralTailor [NeuralTailor2022] and other baselines, most notably in Panel L2 error, stitch precision, and edge accuracy. The approach remains robust as garment complexity increases, succeeding where sequential decoders fail to converge. Test-time adaptation further reduces errors, yielding up to a 15% gain in L2 accuracy and stitch precision. These strong numerical results highlight the efficacy of shared latent supervision and the power of test-time optimization.

Comparison with Image-Based Baselines

Despite the strong priors of image-based methods such as ChatGarment [bian2024chatgarment], StEm-Net consistently outperforms 2D-to-pattern pipelines, highlighting the significance of 3D geometric information—especially panel connectivity and draping cues—over visual cues alone, even when provided with highly favorable image segmentation.

Editing and Generalization

A defining feature of StEm-Net is seamless propagation of edits from 2D pattern space to the 3D garment, obviating expensive physics-based simulation in CAD workflows. Edits to GarmentCode parameters yield updated 3D geometries in real time; empirical results show strong correspondence with physically simulated ground truth for modifications including global length adjustment and topological changes (e.g., sleeve removal/addition).

Generalization studies on real scans (4DDress, CloSe) demonstrate that despite exclusive training on synthetic data, StEm-Net robustly reconstructs realistic garments, vastly outperforming NeuralTailor on bidirectional Chamfer Distance and validating the transfer strength of the BoxMesh-based embedding.

Robustness and Ablations

StEm-Net exhibits strong resilience to both vertex and normal perturbation on the BoxMesh, suggesting that the latent space does not overfit to synthetic artifacts and encodes meaningful, noise-resistant features.

Implications and Future Directions

The introduction of a bidirectional, unified latent space fundamentally transforms the interface between neural graphics, computer vision, and garment manufacturing. Practically, StEm-Net provides an efficient, non-simulation-based approach for digital garment reconstruction, manufacturing, and interactive editing. Theoretically, this work establishes that semantic garment attributes, geometric closure, and parametric editability can all be harmonized within a learned, differentiable representation.

Scalability to multilayered garments, complex material simulations, and body-driven deformation remains contingent on dataset enrichment and further architectural innovations. The modularity and efficiency of the StEm latent space, however, position it as an attractive foundation for forthcoming work in large-scale neural garment modeling, semantics-conditioned generation, and advanced manufacturing feedback cycles.

Conclusion

StEm-Net inaugurates a simulation-free, bidirectional link between 3D garment geometry and 2D manufacturing patterns. By leveraging a BoxMesh-aligned, differentiable latent space, the system dramatically improves efficiency and accuracy in both garment reconstruction and editability tasks, as evidenced by strong quantitative and qualitative results across synthetic and real datasets. The positioning of garment modeling within a unified, robust, and editable space marks a significant advance in the scalability and application of neural methods for practical garment engineering and digital human modeling (2607.00829).