Dense Point-to-Mask Optimization with Reinforced Point Selection for Crowd Instance Segmentation

Abstract: Crowd instance segmentation is a crucial task with a wide range of applications, including surveillance and transportation. Currently, point labels are common in crowd datasets, while region labels (e.g., boxes) are rare and inaccurate. The masks obtained through segmentation help to improve the accuracy of region labels and resolve the correspondence between individual location coordinates and crowd density maps. However, directly applying currently popular large foundation models such as SAM does not yield ideal results in dense crowds. To this end, we first propose Dense Point-to-Mask Optimization (DPMO), which integrates SAM with the Nearest Neighbor Exclusive Circle (NNEC) constraint to generate dense instance segmentation from point annotations. With DPMO and manual correction, we obtain mask annotations from the existing point annotations for traditional crowd datasets. Then, to predict instance segmentation in dense crowds, we propose a Reinforced Point Selection (RPS) framework trained with Group Relative Policy Optimization (GRPO), which selects the best predicted point from a sampling of the initial point prediction. Through extensive experiments, we achieve state-of-the-art crowd instance segmentation performance on ShanghaiTech, UCF-QNRF, JHU-CROWD++, and NWPU-Crowd datasets. Furthermore, we design new loss functions supervised by masks that boost counting performance across different models, demonstrating the significant role of mask annotations in enhancing counting accuracy.

• The paper proposes DPMO-RPS, integrating point-to-mask optimization with NNEC constraints and reinforcement-based point selection to tackle dense crowd segmentation.
• It leverages a custom mask-supervised loss to enhance both segmentation quality and crowd counting metrics, outperforming current methods under extreme density.
• Experimental results on datasets like ShanghaiTech demonstrate significant gains in IoU, F1, MAE, and RMSE, validating the framework’s effectiveness.

Dense Point-to-Mask Optimization with Reinforced Point Selection for Crowd Instance Segmentation

Introduction and Motivation

Crowd instance segmentation remains a central challenge in high-density crowd analysis due to severe occlusions, minute object scale, and annotation scarcity. Traditional approaches utilize point annotations and density maps, which are decoupled from the genuine morphology and distributions of individuals. This work introduces the crowd instance segmentation problem, defining it as predicting accurate segmentation masks for each individual in dense crowds. This paradigm enables more precise region-level supervision, bridges the gap between point annotations and density maps, and allows for robust instance-level analysis—a pivotal requirement for safety-critical applications in surveillance and public management.

The paper proposes the DPMO-RPS framework, an end-to-end method integrating Dense Point-to-Mask Optimization (DPMO) and Reinforced Point Selection (RPS). DPMO leverages the Segment Anything Model (SAM) and the Nearest Neighbor Exclusive Circle (NNEC) constraint to convert point annotations into reliable mask pseudo-labels, while RPS—trained with Group Relative Policy Optimization (GRPO)—selects optimal segmentation prompts for fine-grained instance mask generation.

Figure 1: The proposed segmentation mask provides a determined mapping between origin coordinates and the density map—addressing a fundamental limitation of prior frameworks.

DPMO-RPS Framework

The DPMO-RPS framework fundamentally advances the state-of-the-art by integrating segmentation, prompt selection, and region-level supervision, crucial for improved performance under crowding, occlusion, and low-resolution conditions.

Figure 2: DPMO-RPS pipeline: DPMO utilizes point annotations and NNEC constraints to derive separation-assured masks, then RPS (trained with GRPO) refines point prompts for SAM to improve instance segmentation.

Dense Point-to-Mask Optimization (DPMO)

DPMO augments SAM with NNEC constraints such that each mask uniquely corresponds to a single annotated point. Masks are constructed with variable-radius circles centered at points, dynamically adjusted to be either non-overlapping or overlapping, accounting for the variable scale and density of the heads. The NNEC constraint plays a critical role in ensuring that each resulting mask encompasses exactly one point while maximizing discriminative information.

Figure 3: Examples of NNEC-constrained mask generation: left, non-overlapping circles; right, overlapping and unbounded circles—illustrating their effect on the resulting instance masks.

Reinforced Point Selection (RPS) with GRPO

The RPS component leverages a reinforcement learning paradigm, using GRPO to train a Point Selection Network (PSNet) that adaptively selects the best candidate point from a Gaussian-sampled group around initial predictions. Rewards signal mask quality relative to ground truth pseudo-labels. This selection is supervised by a cross-entropy objective that aligns high reward with the assigned score, narrowing the gap between predicted points and optimal mask cues.

Mask-Supervised Loss Functions

The authors develop novel loss functions with mask supervision for both density map prediction and point prediction models. In density-based counting, a custom mask-based loss directly enforces region-level consistency by penalizing deviations between predicted density and mask-derived labels. For point-based models, a matching cost under mask constraints refines localization, ensuring improved correspondence with true head positions. This dual strategy systematically addresses ambiguity in point-dense region correspondence and leverages richer mask-level supervision.

Figure 4: Mask-supervised loss function cases—single, multiple, and absent predicted points per mask—underpinning effective point-region correspondence.

Experimental Results

DPMO-RPS achieves SOTA performance across major benchmarks: ShanghaiTech, UCF-QNRF, JHU-CROWD++, and NWPU-Crowd. The improvements are manifest in both mask quality (IoU, F1) and downstream counting (MAE, RMSE).

Figure 5: DPMO-RPS instance segmentation results on representative datasets, demonstrating robustness to density, occlusion, and annotation noise.

(Tabled results omitted as per directions; numerical values referenced.)

On ShanghaiTech A, DPMO-RPS attains IoU 0.5431 and F1 0.5533—substantially above competitive methods, including base SAM, FastSAM, and CrowdSAM. Similar relative margins are observed on other datasets. Mask-supervision also consistently enhances crowd counting accuracy across point and density-based models (e.g., PET, ZIP, CLIP-EBC).

Comparisons with recent tiny object detectors (YOLOv7, LTDnet) further underscore the superiority of DPMO-RPS for handling extreme crowd density where bounding-box-based detectors fail.

Figure 6: YOLOv7, LTDnet, and DPMO-RPS: only DPMO-RPS produces accurate individual segmentations in extreme-density crowds.

Ablation studies demonstrate:

• NNEC is essential; unconstrained SAM produces overlapping/ambiguous masks.
• Point prompts yield better results than box prompts, which are imprecise in dense regions.
• RPS is critical for maximizing instance-level F1; omitting it degrades performance.
• Mask-supervised loss outperforms box supervision, especially for NWPU-Crowd.

Hyperparameter analysis validates the selected Gaussian and NNEC radii, with segmentation quality degrading outside chosen ranges.

Figure 7: Ablation on kernel radius (RPS sampling) and NNEC constraint—showing sensitivity and informing best-practice parameterization.

Implications and Future Directions

The explicit mapping from points to masks via DPMO introduces a new pseudo-labeling strategy for dense instance segmentation in limited-annotation regimes. This approach enables explicit topological and geometric supervision, facilitating the training of scalable segmentation, localization, and counting networks with minimal manual labeling.

The reinforcement-driven RPS framework demonstrates the utility of RL techniques in prompt selection and offers a template for similar RL-based enhancements in interactive segmentation pipelines.

Future research possibilities include:

• Extending mask-level supervision to settings with massive unannotated data.
• Joint end-to-end optimization of both the SAM backbone and prompt-selection module.
• Adaptation to multi-modal or multi-attribute scenario (beyond heads, to limbs or pose).
• Integrating temporal information for video-based dense crowd analysis.

Conclusion

DPMO-RPS advances the field of crowd analysis by fusing promptable segmentation, NNEC-based region assignment, and RL-driven prompt selection. Empirically, it substantially improves performance across segmentation and counting metrics in dense crowd settings, far surpassing SAM, FastSAM, and current detection models. The mask-supervision mechanism demonstrates transferability, enhancing network performance in both density and localization tasks.

The architectural innovations, loss designs, and demonstrated empirical strengths position DPMO-RPS as a viable blueprint for future crowd analysis systems requiring both precision and scalability.

Figure 8: Segmentations by DPMO-RPS, SAM, FastSAM, and CrowdSAM; rightmost column—robust DPMO-RPS masks confirm qualitative improvements over all baselines.