Semantic Proxies & 3D Box Representations

• Semantic proxies are formal abstractions that encode geometric, semantic, and relational features using parameterized 3D box representations.
• They bridge low-level sensor data and high-level reasoning tasks, enabling efficient shape completion, scene synthesis, and annotation.
• Advanced methods integrate learnable proxies, transformer decoders, and weak supervision to achieve state-of-the-art performance across benchmarks.

Semantic proxies are formal abstractions—often instantiated as 3D box representations—that encode geometric, semantic, or relational structure in three-dimensional data and systems. These proxies bridge low-level sensory or point cloud data and higher-level reasoning tasks, facilitating efficient annotation, completion, inference, and scene understanding. The 3D box representation, typically a parameterized oriented or axis-aligned cuboid, is prevalent for its mathematical tractability, annotation efficiency, and scalability across applications, ranging from shape completion and instance segmentation to spatial reasoning and scene control.

1. Formal Definitions of Semantic Proxies and 3D Box Representations

Semantic proxies serve as parameterized intermediaries, encapsulating both geometric extent and semantic traits of an object or region. The standard 3D box is described by its center $c \in \mathbb{R}^3$, extents $E = (w, h, d)$ for width, height, and depth, and orientation $R \in SO(3)$ (or scalar yaw $\theta$ for up-axis rotations). Formally, an oriented box:

$$B = (c, E, R), \quad B = \{ p \in \mathbb{R}^3 \mid p = c + R u,\, u \in [-\tfrac{w}{2}, \tfrac{w}{2}] \times [0, h] \times [-\tfrac{d}{2}, \tfrac{d}{2}] \}$$

Semantic proxies may extend this form by associating type, instance ID, or semantic label, as in $$b_i = (c_i, E_i, R_i, c_{\text{class}}, i_{\text{ID}})$$ (Schult et al., 2023, Chen et al., 2 Jan 2026, Häsler et al., 25 Apr 2025, Liu et al., 14 Nov 2025).

In advanced structured shape completion—e.g., UniCo (Chen et al., 2 Jan 2026)—proxies are learnable query embeddings $\mathbf{r}_k$ passing through transformer decoders that output primitive parameters:

• For mixed-type primitives: quadric matrices $\mathbf{A}_k$ such that $\mathbf{x}^\top \mathbf{A}_k \mathbf{x} = 0$ for point $\mathbf{x}$ in homogeneous coordinates.
• For standard boxes: $(c_k, q_k, s_k)$ from MLP($\mathbf{r}_k$), where $q_k$ is a quaternion, $s_k$ scale.

2. Methodologies and Pipeline Integration

Proxies are central to multiple technical pipelines:

a. Structured Shape Completion (UniCo):

• Initialize $K$ learnable proxies $\mathbf{r}_k$, contextualize via cross/self-attention to input shape features.
• Decode each proxy to primitive parameters, semantic class probabilities, and inlier membership scores.
• Training leverages online target updates, permutation-invariant matching (Hungarian), and composite losses comprising parameter, semantic, membership, and Chamfer terms (Chen et al., 2 Jan 2026).

b. Semantic Scene Generation (ControlRoom3D):

• User specifies proxy room $M = \{b_i\}$, with axis-aligned boxes and semantic labels.
• Render boxes into 2D maps ($S$, $I$, $D_n$, $D_f$), feed these into a multi-view latent diffusion U-net via adapters.
• Post-process with geometry alignment and normal preservation losses to enforce box-constrained 3D recovery and plausible mesh completion (Schult et al., 2023).

c. Weakly-supervised 3D Detection (ALPI):

• Construct proxy objects as synthetic 3D boxes using 2D annotations and size priors.
• Inject proxies into point clouds during training; only 2D losses supervise real objects, full 3D losses for proxies.
• Offline pseudo-labeling refines proxy pool, enabling closed-loop improvement over successive iterations (Lahlali et al., 2024).

d. Graph Contact Representations:

• Represent graph vertices as axis-aligned boxes; edges become box contacts (shared facet of nonzero area).
• Linear-time algorithms use Schnyder woods; L-shaped polyhedra are used for certain 1-planar graphs where boxes alone are insufficient (Alam et al., 2015).

e. Spatial Reasoning:

• Proxies instantiate nodes in a spatial knowledge graph; directed edges encode predicates (directionality, adjacency, topology, connectivity).
• Pipeline processes boxes into triples, supporting dynamic rule evaluation for qualitative and geometric queries (Häsler et al., 25 Apr 2025).

3. Semantic Proxy–Based Reasoning and Inference

By abstracting objects as proxies, systems perform symbol-level reasoning on spatial predicates and relations. The knowledge graph formalism, as deployed in XR applications (Häsler et al., 25 Apr 2025), relies on the following:

Predicate Catalog

• Directionality: left/right/ahead/behind ($\Delta = c_i - c_j$ projected on $e_x^j, e_y^j$).
• Adjacency: leftside, beside, ontop, beneath—tested via minimum box-to-box distances $d_{\min}(B_i, B_j)$.
• Topology: disjoint, inside, containing, overlapping, crossing, touching, meeting.
• Connectivity: on, at, by, in.
• Sectoriality, proximity, visibility (egocentric), comparability, similarity, geographical predicates.

Dynamic rule pipelines parse and deduce these relations, enabling scene queries and symbolic production (e.g., assigning type "cup" to objects on a table).

4. Annotation, Supervision, and Efficiency

Box-based proxies systematically reduce annotation cost and enable weakly supervised learning regimes where dense point-level labels are infeasible:

Box2Mask (Chibane et al., 2022):

• Only axis-aligned bounding boxes are required for training.
• Per-point deep features vote for box parameters; clustering (non-maximum clustering by IoU) yields instance masks.
• Points-to-box assignments are heuristically resolved for ambiguous cases; empirical performance reaches 97% of fully-supervised mAP.

ALPI (Lahlali et al., 2024):

• Synthetic proxies created from 2D labels and priors inject precise 3D supervision during pre-training.
• Depth-invariant 2D losses stabilize across object scales/distances.
• Pseudo-label pool eventually replaces proxies as the model matures.

These approaches are validated on benchmarks such as ScanNet, ARKitScenes, KITTI, and nuScenes, achieving near-supervised accuracy with substantial annotation overhead reduction.

5. Applications Across Domains

Semantic proxies and 3D box representations are foundational in diverse computational settings:

• Shape Completion: Learnable proxies (UniCo) bridge incomplete input point clouds to primitive-based reconstructions with joint geometry, semantics, and inlier membership predictions, outperforming cascade methods in Chamfer distance and normal consistency (Chen et al., 2 Jan 2026).
• Room and Scene Synthesis: Proxy rooms guide latent diffusion models to generate plausible scene layouts with strong box-to-mesh alignment and perceptual quality (Schult et al., 2023).
• Instance Segmentation: Box2Mask demonstrates box-driven voting and clustering for instance masks, matching fully supervised precision (Chibane et al., 2022).
• Graph Theory: Contact representations of graphs in 3D via box proxies encode adjacency, duality, and optimal 1-planarity, offering constructive algorithms for complex planar/1-planar topologies (Alam et al., 2015).
• Spatial Augmentation for VLMs: Abstract bounding box proxies (SandboxVLM) equip vision-LLMs with symbolic 3D structure, unlocking gains in spatial question answering without additional pre-training (Liu et al., 14 Nov 2025).
• XR/Cognitive Reasoning: Oriented box proxies power spatial pipelines for knowledge graph inference, enabling spatial ontology construction, logical queries, and downstream entity production (Häsler et al., 25 Apr 2025).

6. Quantitative Performance and Robustness Analysis

Proxy-based systems have demonstrated superior metrics and robustness:

Shape Completion: UniCo reduces Chamfer distance by up to 50% (CD = 2.18 vs. 4.33 for ODGNet+HPNet) and raises normal consistency by over 6 points (NC = 0.935 vs. 0.873). Robust to input incompleteness (CD only rises from 1.8 to 2.7 under 25–75% missing data), far surpassing baselines (Chen et al., 2 Jan 2026).

Weakly Supervised Segmentation: Box2Mask achieves mAP@50 of 67.7 vs. 69.9 for fully supervised methods, with an 18 point lead over prior weakly supervised variants (Chibane et al., 2022).

Object Detection with 2D Labels: ALPI on KITTI attains 78.3% (val) and 85.98% (test) mAP for cars, approaching supervised SOTA, and is robust to class/instance variability in nuScenes (Lahlali et al., 2024).

Scene Synthesis: ControlRoom3D improves CLIP score (+2.8), Layout-Plausibility (+1.5), and Perceptual Quality (+1.2) over text-only baselines (Schult et al., 2023).

VLM 3D Intelligence: SandboxVLM yields up to +8.3% boost in SAT-Real and notable gains in physical reasoning scores (Liu et al., 14 Nov 2025).

7. Limitations, Extensions, and Future Directions

While 3D boxes are efficient proxies, limitations arise in certain theoretical and practical contexts:

• Not all 1-planar graphs can be represented solely by boxes; L-shaped polyhedra are required for specific structures, incurring higher computational complexity ($O(n^2)$) (Alam et al., 2015).
• Proxy abstractions trade fine metric accuracy for symbolic tractability in spatial reasoning pipelines (Häsler et al., 25 Apr 2025).
• Extensions towards richer proxy designs (dynamic attributes, physical properties) are plausible, as suggested by future work in embodied AI (Liu et al., 14 Nov 2025).

Further research may target scalable symbolic reasoning, proxy-based neural architectures for embodied agents, and algorithmic advances for more complex topological representations.

---

In summary, semantic proxies—anchored in rigorous 3D box parameterizations—provide a unifying geometric and symbolic abstraction for structured shape completion, weak and self-supervised learning, spatial inference, scene synthesis, and semantic reasoning. Their prevalence across recent research reflects their mathematical expressivity, annotation efficiency, and adaptability for both low-level reconstruction and high-level cognitive tasks.