Mesh-Agnostic Latent Embedding

• Mesh-Agnostic Latent Embedding is a framework that abstracts mesh connectivity by encoding geometric and physical properties into continuous latent spaces, supporting various 3D representations.
• The approach employs domain functions, latent basis fields, spectral encodings, and diffusion pipelines to achieve robust shape manipulation, simulation, and cross-format generalization.
• Its design eliminates the need for remeshing, scales with topology variations, and facilitates efficient deformation transfer along with generative modeling across different 3D data formats.

Mesh-agnostic latent embedding refers to a class of representations and neural architectures enabling 3D geometric learning, analysis, and simulation independently of a particular mesh connectivity or discretization. These frameworks encode geometric, topological, or physical properties into continuous (often low-dimensional) latent spaces such that the resulting learned functions, codes, or field representations are applicable across point clouds, surface meshes, volumetric grids, implicit neural fields, and other 3D formats. This property enables cross-domain generalization, robust shape manipulation, and efficient simulation, avoiding remeshing and manual retopology.

1. Fundamental Principles and Mathematical Definitions

Mesh-agnostic latent embedding approaches unify diverse input representations by abstracting away explicit mesh connectivity during encoding and downstream processing. Central methods achieve this through:

• Domain functions: All geometric data are converted to a queryable function, typically an occupancy or signed-distance function, $o(x)$, which identifies whether $x\in\mathbb{R}^3$ lies inside the solid, providing a topology-neutral basis for downstream algorithms (Modi et al., 2024).
• Latent basis fields: Instead of fixed mesh nodes, models use spatially varying continuous embeddings $z(x)\in\mathbb{R}^H$ queried by arbitrary $x$, serving as input to reduced bases or directly defining deformations and properties.
• Spectral/sparse encodings: Approaches such as SVD-based eigenfeatures (Fan et al., 9 Mar 2025) or functional map-based spectral pooling (Hahner et al., 2023) project geometric data into mesh-reparameterization-invariant latent codes, which act as the basis for generation and analysis.
• Diffusion and generative pipelines: Latent diffusion in these continuous or spectral spaces enables both deterministic encoding/decoding and stochastic sampling, mesh-agnostic interpolation, and generation (Lyu et al., 2023, Shi et al., 9 Jun 2025).

Mathematically, a typical mesh-agnostic field-basis representation takes the form:

$$u(x) = z(x)^\top W, \quad \text{with} \quad z(x) = f_\theta(x)$$

where $f_\theta$ is an MLP producing per-point latent vectors, $W$ is a global learnable or problem-specific basis, and $u(x)$ is the deformation or property at $x$ (Modi et al., 2024).

2. Architectural and Algorithmic Paradigms

Several architectures have been proposed, optimized for geometry, simulation, or generative tasks:

• Implicit Neural Embeddings: Fully-connected MLPs (with sinusoidal or coordinate-based activations) encode both occupancy/signature values and local latent embedding vectors. These networks are independent of mesh structure and can process SDFs, point cloud fields, radiance fields, or mesh vertex queries uniformly (Modi et al., 2024).
• Transformer-based Latent Sets: Architectural patterns using transformer or cross-attention modules over subsampled "latent tokens" abstracted from the input mesh, enabling mesh-agnostic framewise diffusion for sequence or animation generation (Shi et al., 9 Jun 2025).
• Sparse Latent Skeletons: Hierarchical set-abstraction, farthest point sampling, and feature transfer modules create sparse semantically-meaningful latent points, reducing point clouds to compact latent sets suitable for conditional and controllable mesh generation via latent diffusion (Lyu et al., 2023).
• Spectral Pooling: Functional maps and Laplace–Beltrami eigenbasis define a canonical spectral domain across a collection of shapes, with per-shape latent codes pooled into the shared basis, thus achieving invariance to connectivity and mesh refinement (Hahner et al., 2023, Fan et al., 9 Mar 2025).
• Per-vertex Connectivity Embeddings: Each mesh vertex receives adjacency and permutation vectors, providing a continuous latent description from which manifold mesh edges and face cycles are reconstructed, with manifoldness enforced by Sinkhorn normalized assignments (Shen et al., 2024).
• Intrinsic Spectral Descriptors and Attention: Intrinsic geometry descriptors (HKS/WKS) are combined with mesh-agnostic DiffusionNets and cross-attention to produce decoupled identity and expression fields applicable to any surface with or without annotation (Wang et al., 10 Jan 2026).

3. Training Objectives and Optimization Procedures

Optimization schemes for mesh-agnostic embeddings are adapted to the field representation and target application:

• Monte Carlo Volumetric Integration: For physical simulation, total energy or deformation cost is computed as a Monte Carlo average over points sampled in ambient space, weighted by learned occupancy and latent codes. An $\ell_2$ regularization term is added on basis weights (Modi et al., 2024).
• Reconstruction and Structure Losses: Point-wise $\ell_2$ loss, normal and Jacobian regularization, and spectral distance preservation are common. For autoencoders, latent-to-geometry pipelines are trained to minimize both direct reconstruction error and structural/shape distance discrepancies (Hahner et al., 2023, Cha et al., 28 May 2025, Wang et al., 10 Jan 2026).
• Generative/objective Losses: In pipeline frameworks with diffusion or variational decoding, composite losses combine ELBO (variational reconstruction + KL), EDM denoising losses in latent space, and, if applicable, conditional structure supervision from images or segmentation data (Shi et al., 9 Jun 2025, Lyu et al., 2023).
• Manifoldness and Connectivity Regularizers: For mesh manifoldness, adjacency and permutation losses are enforced using cross-entropy or Sinkhorn-based cycle assignments, further ensuring geometric and topological regularity (Shen et al., 2024).
• Domain Adaptivity: Some pipelines support domain adaptation by segmenting loss objectives according to input structure (e.g., FACS-based supervision versus regularization for out-of-domain data in facial retargeting) (Cha et al., 28 May 2025).

4. Mesh-Agnosticism and Generalization Properties

Mesh-agnostic latent framework properties include:

• Geometry- and format-invariance: All frameworks process point clouds, meshes (of any connectivity), implicit fields, and even volumetric CT or NeRF data through common interfaces, requiring no remeshing or category-specific pre-processing (Modi et al., 2024, Shi et al., 9 Jun 2025, Hahner et al., 2023).
• Scalability with topology and resolution: Representations support arbitrary vertex counts, valences, and face structures, with performance and fidelity unaffected by input mesh granularity (Shen et al., 2024). Embeddings rely on field queries or spectral pooling rather than fixed-vertex lookup or edge-walk computations.
• Consistent deformation and transfer: Embedding-based pipelines enable zero-shot transfer of deformations (e.g. human-to-animal expression transfer) or animation (e.g. arbitrary mesh deformation from monocular video) without per-mesh tuning, reference template correspondence, or skeleton extraction (Wang et al., 10 Jan 2026, Shi et al., 9 Jun 2025).
• Interoperable latent spaces: Embedding spaces (spectral, diffusion-based, or global code) allow smooth interpolation, latent editing, and pose transfer, supporting cross-category shape generation and manipulation (Lyu et al., 2023, Hahner et al., 2023, Lei et al., 2023).

5. Representative Applications and Quantitative Results

Mesh-agnostic latent embeddings have enabled advances in several domains:

Domain	Paper (arXiv)	Key Results
Physics simulation	(Modi et al., 2024)	$<$1% displacement error vs. FEM; 77–1400ms/step
Mesh animation	(Shi et al., 9 Jun 2025)	Chamfer 0.018; PSNR 24.39dB; cross-category generaliz.
Mesh generation	(Lyu et al., 2023, Fan et al., 9 Mar 2025)	SLIDE 0.2s/sample; SpoDify 512D code, 5.82e–7 error
Expression transfer	(Wang et al., 10 Jan 2026)	Zero-shot human$\to$animal; sub-mm human error
Manifold retrieval	(Shen et al., 2024)	Edge-F1 0.42; CD 1.39e–3 (ABC CAD dataset)

Simulation approaches enable mesh- and grid-free reduced-order modeling for nonlinear elastic objects of arbitrary representation (Modi et al., 2024). Cross-modal pipelines for deforming or animating assets from raw video outperform skeleton or mesh-registered pipelines on structural and perceptual metrics (Shi et al., 9 Jun 2025). Sparse latent diffusion models and spectral-domain schemes provide efficient sampling/generation and outperform point-cloud-based methods in MMD, coverage, and normal consistency (Lyu et al., 2023, Fan et al., 9 Mar 2025). Expression transfer frameworks achieve robust decoupling of identity and deformation, producing semantically plausible transfers across taxonomy with no animal training data (Wang et al., 10 Jan 2026).

6. Limitations and Open Challenges

While mesh-agnostic embeddings exhibit wide applicability and robust generalization, specific limitations are reported:

• Self-intersections and geometric artifacts: Manifoldness and regularity of produced geometry is guaranteed up to combinatorial consistency, but practical artifacts such as non-planar faces or self-intersections may occur depending on training data (Shen et al., 2024).
• Bounds on size and connectivity: Some approaches, especially transformer-based or spectral pooling pipelines, require fixing a maximal vertex count or spectral basis size. Memory scaling and computational costs still rise quadratically with maximal size (Shen et al., 2024, Hahner et al., 2023).
• Domain-specificity and structural outliers: Out-of-distribution inputs or rare topological anomalies may lead to failures, highlighting ongoing challenges in mesh-agnostic generalization and in handling open-boundary or non-manifold cases (Shen et al., 2024).
• Decoding quality for fine-scale or high-valence features: While spectral and diffusion compressions achieve high fidelity, loss of fine-scale or topologically exceptional details remains an open research problem in latent encoding (Fan et al., 9 Mar 2025, Lei et al., 2023).
• Absence of explicit correspondence: Bypassing pre-established templates forfeits certain interpretability or alignment properties, although embedding analytic techniques (e.g., functional maps, spectral visualization) partially reconstitute semantic meaning (Hahner et al., 2023, Wang et al., 10 Jan 2026).

7. Comparative Overview of Recent Mesh-Agnostic Latent Embedding Strategies

Method	Input Format	Latent Representation	Decoder / Generation	Mesh-Agnostic Generalization
Simplicits (Modi et al., 2024)	Any (SDF, PC, mesh)	MLP per-point latent $z(x)$ + global basis $W$	Deformation field via basis	Yes, no remeshing or domain conversion
DriveAnyMesh (Shi et al., 9 Jun 2025)	Mesh (PC), Video	Transformer VAE, $M$ tokens ($M\ll N$)	Spatiotemporal diffusion, cross-attn.	Yes, all mesh types and motions
SLIDE (Lyu et al., 2023)	Point cloud	$M$-point sparse skeleton + feature	Sparse → dense upsampling via MLP	Yes, explicit structural control
SpaceMesh (Shen et al., 2024)	Mesh/PC	Per-vertex connectivity embeddings	Diffusion on vertex+connectivity space	Yes, arbitrary genus, valence, face types
SpoDify (Fan et al., 9 Mar 2025)	Mesh (SDF grid)	SVD spectral code ($d$-vector)	Inverse SVD + DWT, marching cubes	Yes, single 512D code encodes 15k vertices
Domain-agnostic face expr. (Wang et al., 10 Jan 2026)	Mesh	Intrinsic spectral (HKS/WKS) + DiffNet	Cross-attn, local Jacobian MLP	Yes, identity/expression disentangled, any mesh

Each pipeline demonstrates a unique tradeoff between latent compactness, mesh-agnosticism, generative fidelity, and structural interpretability. Contemporary research continues to refine such frameworks for wider generalization, reduced compute, and enhanced semantic editing and analysis capabilities.