Data-centric HITL in Machine Learning

• Data-centric HITL is an interactive approach that integrates automated pre-labeling with expert verification to iteratively build high-quality datasets.
• It employs iterative retraining, dynamic seed set expansion, and precise metric tracking to enhance model generalization across diverse domains.
• The approach delivers significant efficiency gains, as evidenced by a 51% reduction in manual annotation time in dental image applications.

Data-centric Human-in-the-Loop (HITL) refers to supervised machine learning and interactive algorithmic workflows that prioritize human involvement at critical points in the data pipeline—labeling, correction, feedback propagation, and iterative enrichment—rather than solely in model selection or tuning. The paradigm aims to maximize annotation and curation efficiency, accelerate iterative dataset enlargement, and improve downstream model generalization by continuously leveraging domain expertise within structured, quantifiable annotation loops. Data-centric HITL systems have been instrumental in applications requiring precise, high-quality labels and adaptive learning, such as medical image analysis, financial fraud detection, document layout segmentation, robotics, and autonomous navigation.

1. Defining Data-centric HITL: Principles and Schematics

A data-centric HITL pipeline situates the human expert as an iterative collaborator in annotation and verification, using model-driven provisional labels to boost throughput and minimize redundant manual labeling. The pipeline fundamentally consists of:

• Seed annotation: Start with a small, diverse set of meticulously labeled examples by experts.
• Model-based pre-labeling: Deploy a state-of-the-art network trained on this seed to provisionally label the next batch.
• Human verification/correction: Experts review, edit, and approve model predictions via graphical interfaces.
• Iterative retraining: Verified labels expand the training pool, the model is retrained, and the cycle repeats with exponential pool growth.

This workflow is codified, for example, in the OdontoAI dental radiograph pipeline (Silva et al., 2022):

Repeat: 1. Train network on N labeled images 2. Predict masks/labels on N new images 3. Human expert verifies/corrects 4. Merge to training pool (size doubles) 5. Retrain model Until dataset size or performance saturates

The focus is on data quality and annotation leverage rather than solely on model architecture search or hyperparameter tuning.

2. Model Architectures and Training Strategies in HITL Systems

HITL platforms typically benchmark multiple architectures but ultimately converge on those with maximal annotation-leverage and performance gains. For OdontoAI, seven instance segmentation models (ResNeSt Mask-R-CNN, Cascade R-CNN, DetectoRS, HTC+deformable conv, etc.) were evaluated; Hybrid Task Cascade (HTC, ResNeXt-101 ⊗ 4d backbone) yielded optimal results (Silva et al., 2022).

Key elements:

• Multi-task loss function:

$$L_\text{total} = L_\text{cls} + L_\text{box} + L_\text{mask}$$

where $L_\text{cls}$ is cross-entropy, $L_\text{box}$ is smooth L1, $L_\text{mask}$ is pixel-wise binary cross-entropy.

• Data augmentation: Horizontal flips with label remapping, cropping.
• Early stopping criteria: Maximization of validation mAP or other generalization metric.
• Batch size/scaling: E.g., 8 images per iteration on 8×V100 GPUs.

Generalization beyond domain is achieved by stratified seed sets, explicit correction guidelines, GUI customization for annotation speed, and by retaining a hold-out test partition untouched by the HITL loop.

3. Annotation Efficiency, Metrics, and Quantitative Gains

Data-centric HITL schemes systematically quantify annotation savings and performance improvement at each iteration. In OdontoAI, annotation time per image fell from 14 min 43 s (purely manual) to 7 min 12 s using HITL, corresponding to a 51% reduction (over 390 hours across 3,150 images) (Silva et al., 2022).

Segmentation and labeling metrics tracked include:

Iteration	Train Size	Test mAP (seg)	ΔmAP	Numbering Accuracy (%)
1	450	0.720	-	64.5
2	900	0.746	0.026	66.1
3	1,800	0.756	0.010	67.0
4	3,600	0.774	0.018	67.9

Time-saving is calculated as:

$$\text{TimeSaving} = 1 - \frac{T_{\text{HITL}}}{T_{\text{manual}}} \approx 0.51$$

Instance segmentation mean average precision (mAP) and label matching rate both increased monotonically with pool expansion.

4. HITL Platform Design and Benchmarking

OdontoAI embodies the data-centric HITL paradigm as a full-featured online platform (Silva et al., 2022):

• Dataset: 4,000 panoramic dental images, split evenly between public (2,000 labeled for training/validation) and private (2,000 reserved for evaluation).
• Leaderboards and tasks: Instance segmentation (COCO-style mAP), semantic segmentation (IoU, Accuracy, F₁), and tooth numbering (exact match, micro-precision/recall, Hamming loss).
• Submission protocol: Standardized JSON formats (COCO masks, FDI label lists).
• Baseline visibility: Performance floor for all benchmarked architectures.
• Test set secrecy: Hidden labels in the evaluation pool prevent overfitting to iteratively-corrected training data.

Versioning, submission tracking, and task centralization enable standardized, reproducible research and fair external benchmarking.

5. Best Practices, Domain Generalization, and Pitfalls

Critical best practices for data-centric HITL, explicitly drawn from the OdontoAI implementation (Silva et al., 2022), include:

• Seed set strategy: Initial stratified, diverse sample to cover all relevant subcategories avoids early model bias.
• Batch doubling: Exponential annotation pool growth via doubling mitigates batch bias and minimizes drift.
• Annotation tooling: GUI customization (control point simplification, opacity toggles) and explicit correction guidelines (mask serrations, root coverage) ensure rapid, high-fidelity edits.
• Metric tracking: Simultaneous monitoring of model-side (mAP, AP50) and annotation-side (time per image, correction count) metrics flags diminishing returns.
• Bias avoidance: Retaining a strictly manual hold-out test set curtails overfitting to iterative corrections.
• Systematic error mining: Re-sampling and hand-labeling of rare edge cases not surfaced by the provisional model prevent error propagation.

Common pitfalls are over-trust in the model's provisional labels, leading to overlooked systematic errors, and annotation drift if packetwise labeling is not randomized.

6. Cross-domain Applicability and Impact

The OdontoAI paradigm—the iterative, data-centric HITL pipeline—extends to any domain with high annotation cost and heterogeneous sample difficulty, including but not limited to medical imaging (segmentation, diagnostic labeling), computer vision (fine-grained detection, rare event annotation), document analysis (semantic layout), fraud detection (sparse node labeling and feedback propagation) and preference-alignment/fine-tuning for RL agents.

This approach enables construction of large-scale, high-quality labeled datasets at a fraction of time and annotation effort, delivers robust gains in task-specific accuracy and interpretability, and provides a scalable, reusable template for interactive data curation in both research and deployment settings. By instrumenting each workflow stage, integrating annotation as part of iterative learning, and codifying platform-level best practices, the data-centric HITL model demonstrably accelerates dataset creation and model adaptation in domains where expert labeling is expensive and ground truth is non-uniformly distributed (Silva et al., 2022).