FMPose3D: monocular 3D pose estimation via flow matching

Abstract: Monocular 3D pose estimation is fundamentally ill-posed due to depth ambiguity and occlusions, thereby motivating probabilistic methods that generate multiple plausible 3D pose hypotheses. In particular, diffusion-based models have recently demonstrated strong performance, but their iterative denoising process typically requires many timesteps for each prediction, making inference computationally expensive. In contrast, we leverage Flow Matching (FM) to learn a velocity field defined by an Ordinary Differential Equation (ODE), enabling efficient generation of 3D pose samples with only a few integration steps. We propose a novel generative pose estimation framework, FMPose3D, that formulates 3D pose estimation as a conditional distribution transport problem. It continuously transports samples from a standard Gaussian prior to the distribution of plausible 3D poses conditioned only on 2D inputs. Although ODE trajectories are deterministic, FMPose3D naturally generates various pose hypotheses by sampling different noise seeds. To obtain a single accurate prediction from those hypotheses, we further introduce a Reprojection-based Posterior Expectation Aggregation (RPEA) module, which approximates the Bayesian posterior expectation over 3D hypotheses. FMPose3D surpasses existing methods on the widely used human pose estimation benchmarks Human3.6M and MPI-INF-3DHP, and further achieves state-of-the-art performance on the 3D animal pose datasets Animal3D and CtrlAni3D, demonstrating strong performance across both 3D pose domains. The code is available at https://github.com/AdaptiveMotorControlLab/FMPose3D.

• The paper introduces FMPose3D, which uses flow matching as a novel and efficient alternative to iterative diffusion models for 3D pose estimation.
• It employs a conditional ODE-based velocity field and explicit Euler integration to map Gaussian noise to plausible 3D poses from 2D inputs.
• The proposed RPEA module robustly aggregates multiple hypotheses, achieving state-of-the-art accuracy and computational efficiency on standard benchmarks.

FMPose3D: Monocular 3D Pose Estimation via Flow Matching

Introduction and Context

Monocular 3D pose estimation from a single image presents an ill-posed inverse problem, exhibiting inherent depth ambiguities and frequent self-occlusions. These challenges substantially limit the reliability of deterministic regression models that map 2D keypoints directly to a unique 3D pose. Existing literature addresses these ambiguities by advancing probabilistic frameworks—MDNs, CVAEs, normalizing flows, and recently, diffusion models—to generate multi-modal and diverse pose hypotheses. However, diffusion-based methods are hampered by high computational complexity due to their iterative denoising schemes.

The paper "FMPose3D: monocular 3D pose estimation via flow matching" (2602.05755) introduces FMPose3D, a conditional generative model leveraging flow matching (FM) to efficiently sample diverse and plausible 3D pose hypotheses, while sidestepping the sampling inefficiency of diffusion models. The framework learns a deterministic ODE velocity field conditioned on observed 2D poses, enabling rapid inference with a small number of integration steps. Furthermore, the work proposes the Reprojection-based Posterior Expectation Aggregation (RPEA) module, a Bayesian aggregation mechanism for hypothesis integration, yielding theoretically justified and empirically superior single-prediction outputs.

Methodology

Conditional Flow Matching Formulation

FMPose3D recasts 3D pose estimation as a conditional distribution transport problem. For every input 2D pose $x^{2D} \in \mathbb{R}^{J \times 2}$, the model learns a parameterized, time-continuous velocity field $v_\theta(x_t, t, c)$, governing the ODE transporting noise samples $x_0 \sim \mathcal{N}(0, I)$ onto the manifold of plausible 3D poses $x_1 \in \mathbb{R}^{J \times 3}$, conditioned on $c = x^{2D}$. The model is trained using a Conditional Flow Matching (CFM) objective, minimizing the MSE between predicted and target velocities along linearly interpolated paths:

$$x_t = (1-t)x_0 + t x_1, \quad t \in [0,1], \qquad v_t = x_1 - x_0$$

This loss encourages the network to learn smooth, conditional flows that efficiently morph Gaussian noise into data samples consistent with the observed 2D projection.

Figure 1: Overview of the training process: the flow network learns to transport noise samples to 3D pose targets conditioned on 2D projections.

Efficient Inference and Multi-Hypothesis Generation

At test time, inference consists of solving the learned ODE using explicit Euler integration (typically $S=3$ steps), starting from distinct noise seeds, yielding diverse plausible hypotheses consistent with the 2D input. By varying the initial noise vectors, the method supports native multi-hypothesis prediction useful for ambiguity quantification and downstream uncertainty estimation.

Figure 2: Illustration of the parallel multi-hypothesis sampling and subsequent aggregation during inference.

Reprojection-based Posterior Expectation Aggregation (RPEA)

To consolidate multiple generated hypotheses into a single output, RPEA is introduced, inspired by MMSE (minimum mean squared error) Bayesian estimation. As the true posterior over 3D poses is intractable, RPEA uses a pseudo-likelihood based on 2D projection error: hypotheses whose 3D projection aligns with the observed 2D input are weighted more heavily. Both joint-wise and pose-wise aggregation are possible, with joint-wise variant ensuring structural plausibility and robust error reduction even with large $N$.

Figure 3: Comparison of different hypothesis aggregation schemes: RPEA maintains monotonic improvements as hypothesis count grows, superior to mean or greedy selection schemes.

Architecture

The velocity field is parameterized by a deep network combining local GCN and global self-attention (Transformer) branches, jointly capturing articulated body topology and long-range dependencies. Input embeddings for the current 3D estimate, the 2D conditions, and the temporal context are fused before being processed by the dual-branch backbone. This design outperforms GCN-only, attention-only, or serial configurations.

Experimental Results

Human Pose Estimation

On Human3.6M, FMPose3D sets a new state-of-the-art under probabilistic formulations, with an MPJPE of 47.3 mm (using $N=40$ hypotheses and RPEA), surpassing the best prior diffusion-based method DiffPose (49.7 mm) and also outperforming deterministic methods. Superior P-MPJPE and systematic advantage over alternative aggregation strategies are consistently observed.

Generalization and Cross-Domain Evaluation

Without any retraining, FMPose3D generalizes robustly from Human3.6M to unseen subjects in the MPI-INF-3DHP benchmark, achieving state-of-the-art AUC and PCK scores in both indoor and outdoor scenes. On animal datasets (Animal3D, CtrlAni3D), it delivers the lowest P-MPJPE compared to mesh-based and regression-based baselines, with improved performance sustained even without hypothesis aggregation.

Computational Efficiency

A notable outcome is the drastic speedup over diffusion counterparts—FMPose3D attains over $5 \times$ faster inference (145.59 FPS vs. 27.15 FPS with DDIM, 3.36 FPS with vanilla DiffPose), thus facilitating the deployment of multi-hypothesis 3D pose estimation in real-time and resource-constrained scenarios.

Qualitative Assessment

The qualitative results reveal superior anatomical plausibility, structural consistency, and improved handling of challenging articulations and occlusions compared to diffusion models and mesh-fitting approaches.

Figure 4: Qualitative comparison of FMPose3D and DiffPose on Human3.6M; blue: prediction, red: ground truth.

Practical and Theoretical Implications

By leveraging flow matching rather than diffusive or normalizing flow processes, the method combines the expressivity of generative models with orders-of-magnitude improvements in inference efficiency. The conditional ODE-based construction naturally supports multi-hypothesis modeling, meaningful uncertainty quantification, and principled hypothesis aggregation. The proposed RPEA mechanism advances probabilistic pose lifting by more closely approximating the Bayesian posterior, showing monotonic improvements as the hypothesis set grows.

The fusion of local (GCN) and global (self-attention) processing establishes an effective architectural paradigm for articulated structure modeling, and the algorithm generalizes robustly beyond human domains, validating its wider applicability in bio-behavioral phenotyping, robotics, and animal motion analysis.

Future Directions

Potential research trajectories include:

• Extending the conditional FM paradigm for explicit joint distribution modeling over full meshes, or to multi-modal sensor input.
• Investigating learned likelihoods or explicit uncertainty prediction for more expressive posterior estimation.
• Exploring applications in markerless behavioral neuroscience, interactive systems, and out-of-distribution species, where annotational scarcity and non-human variation are dominant.

Conclusion

FMPose3D establishes a new standard for efficient, uncertainty-aware monocular 3D pose estimation, introducing flow matching as a compelling alternative to diffusion and flow-based generative models in conditional vision tasks. The framework combines theoretical soundness, strong empirical performance, architectural modularity, and computational tractability (2602.05755).