mmWave Radar-Based Perception System

• mmWave radar-based perception systems are defined by FMCW radar operating in the 24–81 GHz band to enable robust 3D object detection in challenging environments.
• They integrate distributed multi-sensor pipelines and advanced neural architectures to fuse radar, vision, and other modalities for precise tracking and semantic scene analysis.
• Key results include up to 9× mAP improvements and reliable detection in rain, fog, and dust with sub-centimeter accuracy under adverse conditions.

Millimeter-Wave (mmWave) Radar-Based Perception System

Millimeter-wave (mmWave) radar-based perception systems employ frequency-modulated continuous wave (FMCW) radar operating in the ~24–81 GHz band for environmental sensing, with a focus on robust 3D object detection, tracking, and semantic scene understanding. These systems are widely adopted in domains such as automated driving, intelligent transportation systems (ITS), autonomous robotics, industrial monitoring, and safety-critical applications in adverse or visibility-degraded environments. The key advantage is their resilience to rain, fog, dust, glare, and low illumination, enabling perception in conditions where cameras or LiDAR degrade or fail.

1. System Architectures and Sensor Integration

Modern mmWave perception systems are architected as distributed multi-sensor pipelines, with a mmWave radar front-end and additional modalities—cameras and GPS-RTK (vehicular), LiDAR (for ground-truth or multimodal fusion), or IMU/odometry (for motion compensation). Radar front-ends, such as the Texas Instruments AWR1843/AWR2243/AWR6843 (3–12 TX × 4–16 RX), provide raw ADC data representing reflections at each range/angle sample.

System architectures fall into three major categories:

• Single-vehicle or static deployments: Utilize a single radar or a composite (multi-radar) platform for on-board perception or static surveillance.
• Cooperative multi-agent systems: Enable real-time sharing of spatial/radar features and/or predictions via V2V links to expand field-of-view, resolve blind spots, and improve detection reliability (Song et al., 22 Aug 2025).
• Omnidirectional and multi-surface coverage systems: Employ multiple radars for 360-degree (UAV) or hemispherical (roadside/robotic) perception and redundancy (Malle et al., 3 Feb 2026).

Crucial systems implementing these architectures include:

• CoVeRaP: Multi-vehicle perception through mmWave FMCW radars, with cooperative feature/prediction fusion and a public cooperative dataset (Song et al., 22 Aug 2025).
• RadarNeXt: Real-time, memory-optimized 3D object detection pipeline for 4D radar, leveraging depthwise/deformable convolutions (Jia et al., 4 Jan 2025).
• Achelous, ASY-VRNet, WRCFormer: Multimodal frameworks fusing 4D radar and monocular RGB vision for panoptic perception or robust detection under all-weather conditions (Guan et al., 2023, Guan et al., 2023, Guan et al., 28 Dec 2025).
• DREAM-PCD: Signal processing and deep learning hybrid for point-cloud densification, angular super-resolution, and denoising (Geng et al., 2023).

2. mmWave Radar Signal Model, Preprocessing, and Representation

All systems rely on a core FMCW chirp model:

• Transmit: $$s_{\text{tx}}(t) = \exp\left(j\,2\pi\left(f_c t + \frac{B}{2T} t^2\right)\right)$$
• Echo and beat frequency: $f_b = \frac{2B R}{c T}$, with range recovery $R = \frac{c f_b T}{2B}$

Signal processing pipeline typically involves:

1. Range FFT (per chirp) for range estimation.
2. Doppler FFT (across chirps) for radial velocity.
3. Angle-of-Arrival estimation (across antennas), via FFT/MUSIC/compressive sensing.
4. CFAR detection and clustering, extracting significant peaks from the (range, Doppler, angle) tensor.
5. Multipath/clutter suppression and ghost point filtering—using Doppler/velocity, RCS, intensity, and physical range constraints (Liu et al., 19 Jan 2026, Liu et al., 19 Jan 2026).

Resulting point clouds are stored as sets $\{ (x, y, z, v_{\mathrm{rad}}, I) \}$ and may be pillarized (scattered to BEV pseudo-images) for downstream learning (Jia et al., 4 Jan 2025, Han et al., 2024).

3. Learning Architectures and Feature Fusion Strategies

Radar-based perception systems leverage highly specialized neural architectures:

Radar-only perception:

• PointNet-style encoders: Multi-branch networks separately process position, velocity/Dynamics, and intensity (return power), fusing with attention to capture spatial and Doppler structure (Song et al., 22 Aug 2025).
• Backbone networks: Re-parameterizable depthwise convolutions (in RadarNeXt) and deformable convolutions for memory-efficient, multi-scale foreground enhancement (Jia et al., 4 Jan 2025).
• Nonlinear denoising: U-Net-style pre-processing (removal of device- and scene-dependent noise) prior to classification tasks (gesture recognition, safety) (Baek et al., 2022).

Multimodal fusion:

• Early/middle/late fusion: Features or predictions from multiple vehicles or sensors are spatially aligned (using GPS-RTK or extrinsics) and fused at different DNN layers, with middle fusion generally vastly outperforming late fusion for high IoU accuracy (Song et al., 22 Aug 2025, Luo et al., 1 Jun 2025).
• Wavelet attention and geometry-guided fusion: To efficiently integrate raw radar, vision, and Doppler information, WRCFormer applies wavelet-based FPN modules and two-stage geometry-driven cross-attention, achieving high adverse-weather robustness (Guan et al., 28 Dec 2025).
• Asymmetric fair fusion: Separate per-task fusion paths for detection and segmentation accommodate irregular radar/camera features, exploiting spatial and channel attention (Guan et al., 2023).
• Radar-vision “pseudo-images”: Projecting radar points into the image plane and constructing multi-channel radar maps allows for joint pixel-wise representation and synchronous backbone processing (Guan et al., 2023, Zong et al., 2024).

4. Postprocessing, Training Objectives, and Performance Metrics

Decoders predict 3D bounding boxes (w, h, l, x, y, z, θ), per-point or per-frame depth-confidence, and pixel-wise or pointwise semantic segmentation.

Loss functions typically involve:

• Bounding-box regression: Smooth L1 or L1 on box coordinates.
• IoU/dIoU loss: Directly optimizing volumetric/combinatorial overlap.
• Depth/centerness/objectness loss: Binary cross-entropy or focal loss on detection heatmaps.
• Segmentation: Dice loss (for drivable/water areas), negative log-likelihood (pointwise labeling), and cross-entropy (semantic masks).
• Super-resolution: Novel SDE-driven diffusion models, specifically optimized for background (ghost point) suppression and target fidelity, with explicit foreground/background split in the residual loss (Luan et al., 2024).

Systems are benchmarked using mean Average Precision (mAP at various IoUs), recall/precision across modalities and adverse settings, FID/Chamfer/MHD for point cloud reconstructions, and real-time latency on embedded NVIDIA platforms.

Notable results include:

• Up to 9× mAP increase at IoU 0.9 using middle-fusion radar sharing (Song et al., 22 Aug 2025).
• Detection of power lines as thin as 1.2 mm at >90% reliability, with ≲6 cm RMSE at 1 m (Malle et al., 3 Feb 2026).
• State-of-the-art super-resolution of radar clouds with large FID/MHD reduction and improved registration accuracy (RR@5°/0.5 m = 93.1%) (Luan et al., 2024).
• Robust panoptic waterway and urban traffic monitoring (mAP₃ᴅ=58.7% in fog/sleet, mIoU_area=99%) (Guan et al., 2023, Guan et al., 28 Dec 2025, Han et al., 2024).

5. Robustness in Adverse and Challenging Environments

mmWave radar’s independence from visible light and strong weather penetration afford significant robustness:

• Rain, Snow, Fog, and Dust: Systems maintain consistent detection and segmentation accuracy (>90% recall for radar; vision recall drops by >80% in heavy dust) (Liu et al., 19 Jan 2026, Liu et al., 19 Jan 2026, Gao et al., 2021).
• Multipath and Clutter: Through threshold-based filtering on RCS, velocity, and angular bounds, and by Doppler-aware cluster refinement, false-positive (ghost) rates are reduced by ∼50% (Liu et al., 19 Jan 2026).
• 4D imaging radars: Large virtual arrays (12×16 = 192 elements) support sub-degree DoA accuracy even in highly cluttered or metal-heavy indoor and industrial environments, where LiDAR and IR fail (Liu et al., 19 Jan 2026, Gao et al., 2021).

A dedicated causal denoising pipeline (RDMNet) and cluster-level rule-based classification further suppress spurious returns without ML retraining, ensuring interpretability and computational efficiency.

6. Application Domains, Challenges, and Open Directions

mmWave radar-based perception underpins a broad range of critical applications:

Application Domain	Key System Features	Critical Metrics
Autonomous vehicles (single/multi)	Cooperative feature fusion, robust 3D detection/tracking	mAP@IoU, latency, recall
UAV collision avoidance	Omnidirectional radar array, high refresh rate, fast wire detection	Min detectable target, <6 cm RMSE
Adverse industrial/safety	Model-driven, thresholded, and motion-compensated clustering	Real-time recall, false alarms
Waterway navigation/USV	Panoptic detection, radar-vision fusion, lightweight inference	mAP, mIoU_area, FPS
Robotics and mapping	Super-resolved BEVs, sensor fusion, SLAM integration	FID, CD, RR@θ/d

Emerging challenges include:

• Low-latency, high-bandwidth V2V communication for cooperative fusion (Song et al., 22 Aug 2025).
• Calibration drift and real-time synchronization across distributed sensors.
• Handling domain gaps, ghost points, and severe radar sparsity in complex or cluttered scenes.
• Fusion with camera/LiDAR under calibration uncertainty and missing modalities.
• Scalability to large-scale, multi-agent, or dense urban environments, including non-line-of-sight object detection and multi-class/attribute generalization.

Planned advances emphasize learning end-to-end from raw or minimally preprocessed FMCW tensors, exploiting phase-coherent temporal stacks, compressive sensing approaches for non-uniform virtual arrays, and integrating advanced uncertainty modeling for multi-task learning.

7. Outlook and Research Directions

The mmWave radar-based perception landscape is rapidly evolving, with public datasets such as CoVeRaP (Song et al., 22 Aug 2025), RadarEyes (Geng et al., 2023), CRUW3D (Wang et al., 2023), and WaterScenes (Guan et al., 2023) enabling reproducible and comparative research. Prominent directions include model-driven vs. learning-based fusion, radar-vision BEV generation and multitask reasoning, radar-specific augmentation and super-resolution, and universal frameworks for cross-domain generalization.

A plausible implication is that continued reduction in radar hardware cost, tighter networked sensor integration, and robust front-end denoising/filtering will establish mmWave radar as a mainstay of real-time environmental perception not only in safety-critical ground and air mobility but also in harsh industrial, subterranean, and low-visibility scenarios (Liu et al., 19 Jan 2026, Malle et al., 3 Feb 2026, Guan et al., 2023).