Anchor DETR: Interpretable Object Detection

• The paper introduces an explicit anchor-point query mechanism that replaces learned queries in DETR, enhancing optimization and interpretability.
• The method uses Row-Column Decoupled Attention to reduce memory complexity while maintaining competitive accuracy and fast inference speeds.
• Experimental results on the COCO dataset demonstrate that combining anchor-based queries with RCDA yields significant improvements in average precision and training efficiency.

Anchor DETR is a transformer-based object detection framework that introduces an interpretable, anchor-point-driven query formulation for the DETR family of models. By replacing the opaque, learned object queries of the original DETR with explicit anchor point queries—paralleling strategies from CNN-based detectors—Anchor DETR achieves enhanced optimization, faster convergence, and competitive detection accuracy with reduced computational overhead. This approach establishes a bridge between anchor-based detection paradigms and end-to-end transformer detectors, and has influenced subsequent generations of DETR-like models.

1. Anchor-Based Query Design in DETR

Anchor DETR reformulates the object query design in the transformer decoder to embed explicit spatial priors, which allows each query to focus on a physically-defined region in the image. Each anchor point is a normalized two-dimensional coordinate: $\mathrm{Pos}_q \in [0,1]^{N_A \times 2},$ where $N_A$ denotes the number of anchor points. Each anchor point is encoded to a query-position embedding via a two-layer MLP applied to the standard sine–cosine encoding: $$Q_p = \mathsf{Encode}(\mathrm{Pos}_q) \in \mathbb{R}^{N_A \times C},$$ where $C$ is the model dimension. To support detection of multiple objects at a single location ("one region, multiple objects"), $N_p$ learnable pattern embeddings are introduced and tiled across all anchors: $$N_q = N_A \times N_p,\qquad Q_f^{\mathrm{init}}=[P;\ldots;P]\in \mathbb{R}^{N_q \times C}.$$ Initial decoder queries are the sum $$Q_{\mathrm{init}} = Q_f^{\mathrm{init}} + Q_p \in \mathbb{R}^{N_q \times C}$$ (Wang et al., 2021).

This formulation ensures that each object query has an explicit spatial focus, improving optimization by imparting a clearer division of labor to the prediction slots and more direct interpretability.

2. Row-Column Decoupled Attention (RCDA)

Anchor DETR introduces Row-Column Decoupled Attention (RCDA) as an efficient variant of standard multi-head attention to address the memory bottleneck in high-resolution feature processing. RCDA factorizes attention computation into separate row and column interactions:

• Given feature $K_f\in\mathbb{R}^{H \times W \times C}$, average along rows and columns to get $K_{f,x}$ and $K_{f,y}$.
• Row attention: $$A_x = \mathsf{softmax}((Q_x K_x^T)/\sqrt{d_k})\in \mathbb{R}^{N_q \times W}$$.
• Intermediate result: $Z_{n,i,c} = \sum_{w=1}^W A_x(n,w)V_f(n,i,w,c)$.
• Column attention: $$A_y = \mathsf{softmax}((Q_y K_y^T)/\sqrt{d_k})\in \mathbb{R}^{N_q \times H}$$, final output is summed accordingly.

RCDA reduces the memory complexity from $O(N_q\,H\,W\,M)$ (standard) to $O(N_q\,(H\,C + H\,M + W\,M))$, substantially saving memory especially at high feature map resolutions (Wang et al., 2021).

3. Loss Functions and Training Regime

Anchor DETR adopts the same set-based Hungarian matching paradigm as the original DETR for bipartite assignment between predictions and ground-truth targets. The cost and total loss combine focal classification loss, $\ell_1$ box regression loss, and GIoU loss: $$\mathcal{C}(i,j) = -\log p_i(c_j) + \lambda_1 \|b_i - b_j\|_1 + \lambda_2 (1-\mathrm{GIoU}(b_i,b_j)),$$

$$\mathcal{L} = \sum_{i=1}^{N_q}\Bigl[\lambda_{\mathrm{cls}}\,\mathrm{Focal}(p_i,\,c_i^*) + \lambda_{\mathrm{box}}\|b_i - b_i^*\|_1 + \lambda_{\mathrm{giou}}(1-\mathrm{GIoU}(b_i,b_i^*))\Bigr].$$

Typical weights are $\lambda_{\mathrm{cls}}=2.0$, $\lambda_{\mathrm{box}}=5.0$, $\lambda_{\mathrm{giou}}=2.0$ (Wang et al., 2021).

Training is performed for 50 epochs on COCO with AdamW and ResNet-50-DC5 as backbone, using standard data augmentations and learning rate scheduling.

4. Experimental Performance and Ablation Studies

On the COCO dataset, Anchor DETR attains strong metrics while training with 10× fewer epochs compared to DETR. For single-scale setups with ResNet-50-DC5:

Method	AP	AP$_{50}$	AP$_{75}$	FPS
DETR (500 ep)	43.3	63.1	45.9	12
Deformable DETR (50 ep)	43.8	62.6	47.7	15
SMCA (50 ep)	43.7	63.6	47.2	10
Anchor DETR (50 ep)	44.2	64.7	47.5	19

Ablation reveals:

• Anchor-based queries alone contribute $+4.9$ AP;
• RCDA adds $+1.0$ AP;
• Using both yields the highest AP and best speed. Grid vs. learned anchor initialization yields nearly identical AP (44.1 vs. 44.2) (Wang et al., 2021).

5. Comparative Analysis and Influence on Successors

Anchor DETR marks a pivotal shift by rendering the queries interpretable and physically grounded. Its anchor point formulation and RCDA were adopted and extended in subsequent work:

• DAB-DETR generalizes to dynamic box queries, updating anchor box parameters layer by layer while interpreting their cross-attention as soft ROI pooling (Liu et al., 2022).
• Conditional DETR V2 formalizes "box queries" as embeddings of a reference point and learned box transform, explicitly drawing a parallel to anchor box refinement in Faster R-CNN (Chen et al., 2022).
• Box-DETR further replaces box centers in conditional queries with head-specific agent points encompassing full box information for each cross-attention head, accelerating convergence and improving AP (Liu et al., 2023).

These descendant models empirically show further improvements in both training efficiency and detection AP, substantiating the value of the anchor-inspired query mechanism introduced by Anchor DETR.

6. Practical Impact and Implementation Considerations

Anchor DETR achieves end-to-end, anchor-free inference; no non-maximum suppression or region assignment modules are required at test time. The inference pipeline remains as in DETR: all predictions are directly interpretable as detection candidates, assigned via bipartite matching, and scored by a unified loss. Implementation requires only minimal deviation from standard DETR, with public code and trained models available at https://github.com/megvii-research/AnchorDETR (Wang et al., 2021).

Anchor DETR demonstrates that explicit anchor-point queries can reconcile the optimization advantages of second-stage CNN detectors with the global, permutation-invariant processing of transformers, resulting in faster training, interpretable queries, and state-of-the-art transformer detection performance with lower computational and memory costs.