Left–Right Disparity Consistency in Stereo Vision

• Left–right disparity consistency is a constraint that enforces agreement between the disparities predicted from left and right views to maintain geometric coherence.
• It is integrated into loss functions through pixelwise discrepancy and warping strategies, thereby enhancing depth estimation and reducing artifacts.
• Adaptive methods like bilateral cyclic and census-based consistency further refine disparity predictions in varied setups including unsupervised and event-based systems.

Left–right disparity consistency refers to the constraint or loss term that enforces or encourages agreement between the predicted disparities of the left and right views in a stereo or multi-view system. This principle is used widely in both supervised and unsupervised frameworks for stereo matching, monocular depth estimation, event-based vision, and semi-supervised learning. Its central purpose is to impose geometric coherence across views, suppressing artifacts and improving the reliability of dense correspondence or depth prediction.

1. Mathematical Formulations of Left–Right Consistency

The canonical left–right consistency loss computes the pixelwise discrepancy between predicted left and right disparities, sampled at their respective epipolar correspondences. For rectified stereo, let $d^L(x,y)$ and $d^R(x,y)$ denote disparities at pixel $(x,y)$ predicted from the left and right images, respectively. One typical left→right consistency loss is

$$L_{LR}^L = \frac{1}{N} \sum_{x,y} | d^L(x,y) - d^R(x-d^L(x,y), y) |$$

and symmetrically for right→left. This form appears in Godard et al. (Godard et al., 2016), Amiri et al. (Amiri et al., 2019), and other works.

Alternative formulations exist:

• The bilateral cyclic constraint (Wong et al. (Wong et al., 2019)) applies a full cycle: $d^L \to d^R \to \hat{d}^L$, enforcing $\hat{d}^L(x) \approx d^L(x)$ for co-visible pixels, with adaptive weighting to suppress penalties in occluded or dis-occluded regions.
• In event-based stereo (EV-MGDispNet (Jiang et al., 2024)), left–right consistency is imposed not directly on disparity fields, but via a transitive census-transform loss between left event frames and their right-view warps.

The defining property in all cases is penalizing geometric disagreement between left-view prediction and the corresponding right-view prediction, mapped by the known or predicted epipolar shift.

2. Integration into Loss Functions and Training Objectives

Left–right consistency is incorporated into multi-component loss frameworks, balancing data terms (photometric, supervised depth) and regularization:

Method/paper	Loss term for LR consistency	Other loss terms	Loss weighting strategy
Godard et al. (Godard et al., 2016), Amiri et al. (Amiri et al., 2019)	$L_{LR}^L + L_{LR}^R$ (L1)	photometric (stereo), smoothness, (LiDAR)	All terms weighted per scale; typical LR weight 1.0
Wong et al. (Wong et al., 2019)	Bilateral cyclic consistency $\ell_{bc}$ ($\ell_1$ with adaptive $\alpha(x)$)	photometric, SSIM, smoothness (adaptive)	Bilateral cyclic and adaptive smoothness use per-pixel weights
EV-MGDispNet (Jiang et al., 2024)	Charbonnier census-based LR loss	Smooth L1 disparity regression	Census term weight $\lambda_c$ set to 10–20% of overall loss

All systems apply the left–right term at multiple pyramid scales. In semi-supervised frameworks, the term is combined with explicit supervision (e.g., LiDAR) to further constrain disparity consistency beyond sparse ground truth.

3. Implementation Details and Architectural Integration

Key implementation aspects are:

• Warping: All methods use differentiable warping (e.g., bilinear samplers as in Spatial Transformer Networks) to sample right-view predictions at left-view correspondence and vice versa. This allows gradients to flow through the reprojection step during backpropagation (Godard et al., 2016, Amiri et al., 2019, Jiang et al., 2024).
• Disparity and Inverse-Depth: Some approaches (e.g., (Amiri et al., 2019)) formulate consistency in the inverse-depth space, computing $d = bf\rho$ for camera baseline $b$ and focal length $f$.
• Occlusion Handling: Some models, notably (Wong et al., 2019), modulate the strength of the consistency loss per pixel using adaptive weights derived from photometric residuals ($\alpha(x) = \exp(-c\rho(x)/\sigma)$), reducing the penalty in regions likely to be occluded.
• Event-Based Vision Specifics: In EV-MGDispNet (Jiang et al., 2024), left–right consistency is enforced on event representations via a census transform after warping, with the consistency loss placed between EAA (edge-aware aggregation) and feature extraction.

Some architectures exploit attention mechanisms or recurrent inference to iteratively refine and enforce left–right consistency, as in the LRCR model (Jie et al., 2018), where error maps guide further refinement in a ConvLSTM-based network.

4. Variants: Bilateral Cyclic and Census-Based Consistency

Beyond the basic left$\leftrightarrow$right terms, notable variants are:

• Bilateral Cyclic Constraint (Wong et al., 2019): Rather than enforcing simple mutual agreement, the model composes left→right and right→left projections (e.g., $d^L \to d^R \to \hat{d}^L$), enforcing cycle-consistency. An adaptive regularizer weights the loss, discounting ambiguous or unexplainable regions (occlusions, dis-occlusions), which improves robustness in real data.
• Census-Based Consistency (Jiang et al., 2024): Instead of comparing raw intensities or disparities, the difference is measured in census-encoded "texture-rank" space, which is robust to illumination changes and sparse events. The census loss term is efficiently implemented as a fixed, parameter-free convolutional layer, with gradients propagated through the warping and census stages.

5. Empirical Impact and Quantitative Evaluation

Quantitative ablations across multiple studies confirm the utility of left–right consistency:

Paper	Dataset	Baseline (w/o LR)	With LR consistency	Observed improvements
EV-MGDispNet (Jiang et al., 2024)	DSEC (stereo, events)	MAE 0.622, RMSE 1.463	MAE 0.612, RMSE 1.432	+LR improves disparity and event-frame alignment
Godard et al. (Godard et al., 2016)	KITTI (monocular)	AbsRel 0.152, RMSE 6.098 (Eigen)	AbsRel 0.148, RMSE 5.927	Sharper depth edges; fewer “texture-copy” artifacts
Amiri et al. (Amiri et al., 2019)	KITTI Eigen	AbsRel 0.082 (unsup), 0.108 (semi-sup no LR), 0.078 (full semi-sup)	+LR: gains of 1–3.5% AbsRel	Clear improvement in both unsupervised and semi-supervised settings
Wong et al. (Wong et al., 2019)	KITTI 2015/Eigen	D1-all 30.27%, AbsRel 0.148	D1-all 27.15%, AbsRel 0.133	Bilateral cyclic + adaptive regularization yields best results
LRCR (Jie et al., 2018)	KITTI, Middlebury	–	SOTA disparity via iterative attention-based LR matching	Mismatch-guided refinement reduces error

Qualitative improvements are consistently reported in terms of boundary sharpness, thin structure recovery, and reduced “ramp” or copy artifacts.

6. Applications and Extensions

Left–right consistency is now foundational in stereo matching, unsupervised monocular depth estimation, semi-supervised depth prediction, and event-based vision. Its flexibility enables integration in systems using only stereo pairs (Godard et al., 2016, Amiri et al., 2019), monocular videos augmented with stereo (Zhou et al., 2019), or event frames (Jiang et al., 2024).

The constraint is also critical for generalization in semi-supervised settings where sparse supervision (e.g., LiDAR) alone is insufficient. It provides mid-level geometric supervision that does not depend on ground-truth depth, making it scalable for domains where such labels are unavailable or costly to collect.

Extensions have addressed occlusions (via adaptive weighting), ambiguity (bilateral cyclic, attention), and robustness to photometric noise (census transform, structural losses).

7. Open Challenges and Future Directions

While current forms of left–right consistency are effective given accurate calibration and rectification, outstanding challenges remain:

• Occlusion sensitivity still limits the precision of direct left–right penalties; bilaterally-aware and adaptively-weighted variants partially remedy this.
• For extreme lighting, sparse events, or motion blurs (as in event-based vision), direct photometric or structural comparison is non-trivial; census-based methods are a step forward but may not capture all fine-grained structure.
• Expanding to more general multi-view setups (beyond simple stereo) introduces additional geometric complexity for consistency enforcement.
• Handling errors in predicted disparities that propagate across the cycle (left→right→left) remains sensitive to accumulative drift, particularly in regions with weak texture or parallax.

Recent architectural advances suggest further improvements are possible through end-to-end differentiable geometric modules, context-aware attention, and integration with learned occlusion/mask predictors.

---

References:

• EV-MGDispNet (Jiang et al., 2024)
• Godard et al., "Unsupervised Monocular Depth Estimation with Left–Right Consistency" (Godard et al., 2016)
• LRCR (Jie et al., 2018)
• Wong et al., "Bilateral Cyclic Constraint and Adaptive Regularization for Unsupervised Monocular Depth Prediction" (Wong et al., 2019)
• Amiri et al., "Semi-Supervised Monocular Depth Estimation..." (Amiri et al., 2019)
• "Unsupervised Video Depth Estimation Based on Ego-motion and Disparity Consensus" (Zhou et al., 2019)