StreetSurf: Urban Surface Reconstruction

• StreetSurf is a comprehensive research program for high-fidelity urban street surface understanding, integrating multi-view implicit reconstruction and semantic mapping pipelines.
• It employs scalable techniques such as Gaussian splatting and active learning to achieve state-of-the-art geometry accuracy and rendering robustness without relying on LiDAR.
• The system fuses multimodal sensing—including camera, audio, and IMU data—with real-time simulation to support autonomous mobility and urban analytics.

StreetSurf is a multicomponent research program and system family targeting high-fidelity urban street surface understanding, real-time road condition detection, city-scale semantic mapping, and photorealistic simulation for autonomous mobility and urban analytics. The term encompasses pipelines for implicit surface reconstruction from street-view data (Guo et al., 2023), scalable mesh/raster field generation using Gaussian splatting (Cui et al., 2024), multimodal real-time surface-type detection for safe driving (“SmartRSD”) (Tayeb et al., 2024), and foundational territory mapping and dataset curation methods for supervised learning (Kapp et al., 2024, Hosseini et al., 2022). Comparative studies against instant-NGP, 3DGS, and other NeRF-type frameworks position StreetSurf as state-of-the-art for geometry accuracy and rendering robustness on long, unbounded street trajectories without LiDAR.

1. Multi-View Implicit Surface Reconstruction and StreetSurf Architecture

StreetSurf, as originally introduced, is a multi-view implicit surface reconstruction technique facilitating high-accuracy geometry and appearance recovery from street-view sequences, exemplified by Waymo-perception datasets. Unlike object-centric pipelines, StreetSurf partitions unbounded outdoor space into:

• Close-range SDF cuboid: Occupied by the ego-vehicle trajectory and modeled as a signed-distance function S_cr(x), enabling sphere-tracing and physically consistent surface extraction.
• Distant-view neural field (N_dv): Occupied by landscape-scale or block-scale context, implemented as a hyper-cuboid NeRF variant for efficient far-field color/density modeling.
• Sky/background MLP (N_sky): Optionally overlays infinite background radiance conditioned on direction, using sky/light masks.

The volume rendering equation follows

$$\widehat{C}(r) = \sum_{i=1}^K T_i \alpha_i c_i, \;\; T_i = \prod_{j<i} (1-\alpha_j)$$

with sky compositing as

$$\widehat{C}^{(\mathrm{all})}(r) = \widehat{C}^{(\mathrm{cr,dv})}(r) + \left[1-\widehat{O}^{(\mathrm{cr,dv})}(r)\right]\, \widehat{c}^{(\mathrm{sky})}(v).$$

Hash-grid encoding for close-range SDF aligns multi-level 3D grids with the axis-aligned bounding box of the street trajectory. Geometry initialization leverages a prior that zero-level sets the SDF to the recovered ground-plane via monocular or sensor-based estimation. This initialization is essential: initializing as a sphere/capsule (object-centric) yields collapsed or muddled reconstructions; modeling the ground/road surface explicitly enables tractable separation of road, curb, and adjacent environments (Guo et al., 2023).

Supervision combines RGB photometric error, eikonal loss, entropy/sparsity regularizers, and geometric consistency from either dense/sparse LiDAR or monocular depth/normal priors. Multi-stage ray marching first prunes empty space using occupancy grid strides, then applies dense upsampling with NeRF-style importance sampling to exploit fine surface detail. Empirically, training on standard Waymo segments completes in 1–2 hours on a single RTX 3090, producing watertight meshes at ∼5 cm voxel resolution and per-ray (LiDAR-style) depth errors <0.5 m.

2. Scalable Street Surface Reconstruction: Gaussian Splatting and StreetSurfGS

The “StreetSurfGS” extension employs planar-based octree Gaussian splatting tailored to street scene structure. The pipeline consists of:

• Planar-based octree partitioning of initial COLMAP Structure-from-Motion (SfM) points, allocating anchor Gaussians at octree leaf centroids with cell-scale-dependent covariance.
• 3D→2D splatting, rendering each Gaussian to an elliptical footprint on the image plane via the projection Jacobian.
• Segmented and progressive training: Long trajectories are divided into overlapped temporal segments; each fuses rendered depth and color via Truncated Signed Distance Fields (TSDF) and Marching Cubes for mesh extraction.
• Guided smoothing regularization: Normal consistency is enforced within planar regions only, using a depth-derived normal mask and edge-aware filtering from SAM segment boundaries.
• Dual-step multi-view matching: Local overlapping cameras are used for geometric and photometric consistency (neighbor matching). Distant viewpoint pairs provide far-field depth regularization to stabilize geometry in underconstrained, textureless, or sparse-view regions.

StreetSurfGS achieves PSNR/SSIM of 28.7/0.935 on Waymo Open and depth RMSE of 1.51 cm on MatrixCity, substantially outperforming instant-NGP, 3DGS, and NeuS in both memory efficiency (11–12 GB for 1 km) and surface reconstruction accuracy (Cui et al., 2024). Limitations include reliance on adequate SfM seed coverage and sufficient intersegment overlap for depth transfer.

3. Multimodal Road Surface Detection for Safe Driving

The “SmartRSD” system embodies a StreetSurf-like architecture for real-time road surface condition classification by fusing camera and audio sensors in-vehicle:

• Sensor suite: Forward-facing RGB camera at ≈30 FPS; unidirectional microphone sampling wheel–road contact at 16 kHz.
• Image and audio processing: Images are normalized and resized; audio is resampled and transformed to 64-bin Mel-spectrograms.
• Lightweight feature extractors: Improved MobileNet for vision ($d_v ≈ 128$), YAMNet variant for audio ($d_a ≈ 1024$), both regularized via batch-norm/dropout.
• Score-level weighted fusion: Final class probabilities $s = w_v s_v + w_a s_a$, with $w_v + w_a = 1, w_v, w_a \ge 0$, trained and fine-tuned jointly.
• Latency and deployment: Sub-50 ms per update on Intel i7 or NVIDIA Jetson TX2 (≤15 W). End-to-end accuracy reaches 94.91% across three classes (dry, wet, snow), about 2–4 percentage points above best unimodal designs, with notable robustness in poor visibility (Tayeb et al., 2024).

4. Dataset Construction and Crowdsourced Labeling for Street Surface Classification

High-quality, heterogeneous annotated datasets are foundational for supervised and self-supervised StreetSurf systems:

• StreetSurfaceVis (Kapp et al., 2024): 9,122 Mapillary images annotated by both OpenStreetMap (OSM) 'surface' tags and five-level 'smoothness' quality attributes, with per-tile and per-sequence spatial control to ensure class diversity.
• Semi-automated labeling: OSM-based pre-labeling reduces search by up to 10×; custom EfficientNetV2-S classifiers and iterative pseudo-label sampling further increase precision (89–95% on common classes). GPT-4o prompt-based labeling attains 39–65% precision and recall 52–83% on rare classes, outperforming embedding similarity search.
• Sampling strategies: Embedding-based similarity search (CLIP, DINOv2, fine-tuned EfficientNetV2-S) is used to select candidates for minor classes; thresholds are tuned per class to achieve maximum F₁ on held-out sets.
• Annotation rigor: Central road segment focus, quality-over-quantity selection, and removal of ambiguous or blurry images are enforced by annotator protocol.

Dataset assembly strategies and recommendations include retraining classifiers for new geographies and updating OSM mappings to reflect regional label conventions.

5. Semantic Segmentation and City-Scale Surface Inventory

“CitySurfaces” demonstrates scalable city-level semantic segmentation using an active-learning powered hierarchical multi-scale attention (HMSA) model with HRNet-OCR backbone (Hosseini et al., 2022):

• Seed label transfer from municipal GIS inventories (e.g., Boston Sidewalk Inventory), fused with Cityscapes-trained HRNet semantic masks.
• Active learning loop: Alternates uncertainty sampling and failure-case hard-mining for iterative label enrichment. New materials (e.g., granite, hex pavers) are integrated by geofiltered image acquisition and targeted relabeling.
• Training regime: 20 active-learning stages (final pool: 3,226 images, 8 pavement materials + road + background; mIoU 90.5% overall, 88.4% on pavements).
• Cross-city robustness: Zero-shot transfer yields mIoU 82–87% across four major US cities, demonstrating spatial generalizability.

The methodology scales to road, bike-lane, defect, and accessibility mapping, supporting citywide dashboards, block-level urban heat island analysis, and GIS export compatible with StreetSurf’s downstream simulation and planning applications.

6. Sensor Modalities Beyond Vision: IMU-Based Surface Recognition

Surface classification via inertial measurement units (IMUs) has been explored as a lightweight alternative or complement.

• Pipeline: Smartphone accelerometer and gyroscope signals (100 Hz native, decimated to 20 Hz), preprocessed with low-pass filtering and body-to-navigation rotation, are segmented into 2–3 s windows.
• Models: LSTM–CNN fusion network ($≈0.4\,\mathrm{MB}$, 100 k params) achieves 93% accuracy binary distinction of road vs. sidewalk, outperforming classical SVM and Random Forest baselines (Eweida et al., 2023).
• Real-time suitability: Inference latency is 5–10 ms per window on consumer ARM CPUs, permitting deployment in battery-constrained micro-mobility scenarios.
• Limitations: Restricted to binary classes; sensitive to vibration and motion artifacts; potential for extension to multiclass (e.g., cobblestone, gravel, grass) and sensor fusion with GPS or magnetometer.

7. Photorealistic Street Simulation and Environmental Style Editing

The “StyledStreets” simulator (Chen et al., 27 Mar 2025) provides StreetSurf-compatible, instruction-driven urban environment rendering through temporally and spatially consistent 3D Gaussian Splatting:

• Scene representation: Static Gaussian field for persistent geometry and dynamic object Gaussians for vehicles, each parameterized by mean ($\mu$), covariance ($\Sigma$), opacity, and view-dependent color.
• Hybrid embeddings: Disentangle camera and style attributes via concatenated embeddings; style parameters are CLIP-encoded and can be linearly interpolated for continuous transitions (e.g., sunny↔rainy).
• Training: Multi-view, joint pose optimization enforces SE(3) consistency for up to 7 cameras and all dynamic objects.
• Style editing: Appearance MLPs modulate Gaussian parameters under uncertainty-masked supervision. Loss terms include photometric, style, and regularization. Uncertainty masks (from DINOv2 features) filter unreliable 2D guidance, e.g., from diffusion priors, confining style change to infrastructure pixels.
• Performance: Full-image PSNR/SSIM 36.40/0.967 (train), 34.28/0.958 (novel view); depth error CD/RMSE 0.201 m/1.67 m (novel view). Real-time rasterization at ≈30 FPS is achievable with tile-based GPU pipelines.

This structured separation between geometry and style enables StreetSurf to simulate street environments under arbitrary seasonal, weather, or temporal conditions with preservation of underlying urban spatial relationships.

---

Major references:

• Multi-view implicit reconstruction methods, hash-grid SDF, and ray marching: (Guo et al., 2023)
• Gaussian splatting and octree/planar organization: (Cui et al., 2024)
• Multimodal real-time surface type classification: (Tayeb et al., 2024)
• Crowdsourced annotated datasets and sampling: (Kapp et al., 2024)
• Active-learning segmentation at city scale: (Hosseini et al., 2022)
• IMU-driven surface recognition: (Eweida et al., 2023)
• Realistic multi-style, pose-stable simulation systems: (Chen et al., 27 Mar 2025)