A Graph Neural Network Approach for Solving the Ranked Assignment Problem in Multi-Object Tracking

Abstract: Associating measurements with tracks is a crucial step in Multi-Object Tracking (MOT) to guarantee the safety of autonomous vehicles. To manage the exponentially growing number of track hypotheses, truncation becomes necessary. In the $δ$-Generalized Labeled Multi-Bernoulli ($δ$-GLMB) filter application, this truncation typically involves the ranked assignment problem, solved by Murty's algorithm or the Gibbs sampling approach, both with limitations in terms of complexity or accuracy, respectively. With the motivation to improve these limitations, this paper addresses the ranked assignment problem arising from data association tasks with an approach that employs Graph Neural Networks (GNNs). The proposed Ranked Assignment Prediction Graph Neural Network (RAPNet) uses bipartite graphs to model the problem, harnessing the computational capabilities of deep learning. The conclusive evaluation compares the RAPNet with Murty's algorithm and the Gibbs sampler, showing accuracy improvements compared to the Gibbs sampler.

• The paper introduces RAPNet, a novel GNN-based method that addresses the NP-hard ranked assignment problem in multi-object tracking for improved hypothesis management.
• It encodes assignment costs in bipartite graphs and employs alternating GCN, GAT, and LSTM layers to predict sequential k-best assignment solutions.
• Experimental evaluations show RAPNet outperforms the Gibbs sampler in assignment accuracy, wp-score, and cost, enabling more efficient real-time tracking.

A Graph Neural Network Approach for Solving the Ranked Assignment Problem in Multi-Object Tracking

Introduction and Motivation

The ranked assignment problem plays a pivotal role in the update step of the $\delta$-Generalized Labeled Multi-Bernoulli ($\delta$-GLMB) filter within Multi-Object Tracking (MOT), particularly in scenarios characterized by the necessity to associate noisy, uncertain measurements with a dynamic set of object tracks. Traditional approaches, such as Murty’s algorithm and the Gibbs sampler, either incur substantial computational cost or provide only suboptimal solutions, especially in NP-hard extensions like multi-sensor MOT (MS-MOT). This paper introduces a deep learning-based method—Ranked Assignment Prediction Graph Neural Network (RAPNet)—which leverages the expressive power of Graph Neural Networks (GNNs) to approximate the ranked assignment solution efficiently and with high fidelity, offering important progress toward scalable and tractable inference in combinatorial MOT assignment problems.

Formalization of Assignment in the MOT Context

Under the RFS and FISST frameworks, the MOT problem—central to environmental awareness for autonomous driving systems—is optimally addressed by the GLMB filter and its $\delta$-GLMB extension. In these Bayesian multi-object tracking pipelines, the data association task is succinctly cast as a ranked assignment problem. Specifically, a cost matrix $C_Z$ is constructed, encoding the negative log-probabilities (or other suitable costs) of assigning each measurement to each track, including the possibility of missed detections. Assignment matrices enforcing appropriate assignment constraints (unique track-to-measurement association, possibility for missed detections, gate-enforced $\infty$ penalties) are employed to model candidate hypotheses. Subsequent tractable implementation, owing to exponential hypothesis proliferation, requires k-best assignments—posing an inherently NP-hard problem.

Related Work and the Case for GNNs

Murty's algorithm provides an exact k-best solution with prohibitive cost for non-trivial problem sizes, while the Gibbs sampler sacrifices solution optimality for runtime but exhibits pronounced inaccuracy for assignments beyond the first. Deep learning methods, particularly those employing GNNs such as GLAN [GLAN: A Graph-based Linear Assignment Network, (Liu et al., 2022)], have previously been tailored to the linear assignment problem but have been limited to best-assignment outputs and often rigid cost matrix dimensions. This work addresses these limitations by proposing a graph-based formulation capable of resolving the general ranked assignment problem, effectively decoupling accuracy from the traditional bottlenecks of classic combinatorial optimization.

RAPNet: Problem Encoding and Model Architecture

RAPNet constructs bipartite graphs encoding the assignment problem, where nodes correspond to tracks and measurements, and edges encapsulate assignment costs derived from $C_Z$. To ensure compatibility with neural architectures, node features are statically summarized via five statistics—ratio of finite entries, min, max, mean, and L2-norm—generated per track or measurement, bypassing shape inconsistency issues for variable-sized inputs. Edge attributes are directly mapped from cost values, and absence of assignment possibilities ($\infty$ in $C_Z$) is enforced via edge exclusion.

The GNN is realized via a multi-stage encoder-decoder sequence: node and edge features are respectively propagated and refined using alternating GCN and GAT layers, culminating in concatenated, edge-wise representations. The decoder leverages LSTM modules to sequentially predict up to $k_\text{max}$ assignments, treating each potential assignment as a separate binary classification sub-task. The model is trained using a weighted binary cross-entropy loss, with per-assignment supervision derived from Murty's algorithm outputs.

Post-Processing and Extraction of Assignments

Owing to soft assignment outputs and possible assignment ambiguities, a greedy post-processing heuristic interprets the network’s output tensor into valid assignment sets. The procedure first selects the argmax prediction per track (row), then incrementally adds high-confidence alternative assignments above a threshold while ensuring assignment validity by eliminating column-wise (measurement) collisions. The result is a set of candidate assignments, which are then ranked and pruned to the desired k-best predictions according to their costs.

Experimental Evaluation

Dataset and Training Paradigm

RAPNet was trained on a mixture of synthetic and simulated MOT datasets, leveraging parameterized cost distributions to emulate assignment patterns reflective of real-world tracking scenarios. Cost matrices ranged up to $15 \times (|Z| + |I|)$ dimensions, with assignment counts $\delta$0 following a Poisson prior to mirror track split dynamics. Training was conducted with standard AdamW optimizer and learning rate scheduling, and the evaluation protocol encompassed both synthetic sweeps and simulation data representative of automotive scenarios.

Metrics

Three evaluation metrics were prioritized:

• Assignment Accuracy: Fraction of matching assignments versus Murty’s optimal outputs, evaluated per assignment position.
• Weighted Position (wp) Score: A custom metric valuing assignment position accuracy (higher weight for lower ranks) and tolerant to moderate misalignment. Maximal for perfect rank-order preservation.
• Cost: Aggregate cost of selected assignments, penalized for incomplete assignment sets.

Results

RAPNet (with post-processing) consistently outperformed the Gibbs sampler in assignment accuracy (notably beyond $\delta$1), wp-score, and cost for small- and medium-size matrices (up to $\delta$2), which dominate real-world MOT workloads. For $\delta$3 or larger problem instances, performance gently degraded relative to the Gibbs sampler due to the hard network output size cap; however, this rarely affected practically relevant use cases.

Key quantitative findings (from Table 1 in the paper):

Framework	Acc ($\delta$4=1)	Acc ($\delta$5=2)	Acc ($\delta$6=3)	Acc ($\delta$7=4)	wp	Cost
RAPNet-a	0.99	0.91	0.73	0.54	2.74	5.68
RAPNet-PP	0.99	0.95	0.82	0.66	2.80	3.23
Gibbs	1.00	0.18	0.06	0.04	2.11	14.10

For completeness, Murty's algorithm yields $\delta$8, $\delta$9, $\delta$0 for assignment accuracy, wp-score, and cost respectively, reflecting its theoretical optimality.

Notably, RAPNet’s bottleneck is computational overhead in the single-instance (unbatched) regime; however, when batch inference is used, RAPNet achieves competitive performance (e.g., for $\delta$1, RAPNet's per-instance inference time is comparable to the Gibbs sampler and vastly outpaces Murty's algorithm for growing $\delta$2).

Theoretical and Practical Implications

The main theoretical contribution is the demonstration that learned GNN representations can recover multiple high-quality assignment solutions for combinatorial optimization problems, capturing the structured dependencies inherent in ranked assignments—well beyond typical one-best association learning. In practice, this substantially accelerates hypothesis truncation in $\delta$3-GLMB-based filtering, making real-time deployment on multi-object multi-sensor systems more feasible without conceding accuracy as in Gibbs-based approximations.

Moreover, the flexibility of the network design admits natural integration of richer, domain-specific features (e.g., explicit state or class representations for tracks/measurements), something not trivially achievable in combinatorial sampling-based solvers. The approach is thus amenable to further improvement in contexts where assignment “cost” may itself be partially learned or described by deep feature spaces.

Future Directions

Several future research axes emerge from this work:

• Scalability for NP-hard MS-MOT: Extending RAPNet’s architecture and post-processing to efficiently support truly large-scale, multi-sensor association (tens to hundreds of objects and sensors).
• Feature Augmentation: Utilizing richer input attributes (spatial, semantic, contextual) perhaps jointly learned with cost estimation, to further improve assignment accuracy.
• End-to-End Integration: Embedding RAPNet (or its variants) directly within differentiable tracking pipelines, enabling fully end-to-end training.

Conclusion

This paper demonstrates that GNN-based architectures, specifically RAPNet, can substantially improve both the tractability and performance of the ranked assignment problem fundamental to hypothesis management in multi-object tracking. RAPNet bridges the gap between classical combinatorial optimization and learned assignment policies, outperforming popular samplers in critical regime while attaining near-optimal accuracy and cost—laying a foundation for further neural combinatorial optimization advances within autonomous systems and real-time tracking filters.

Citation: "A Graph Neural Network Approach for Solving the Ranked Assignment Problem in Multi-Object Tracking" (2604.01696)