Expert-in-the-Loop Models

• Expert-in-the-loop models are AI systems that incorporate human and artificial experts to handle uncertainty in classification tasks.
• They utilize dynamic routing with OOD detection (e.g., ODIN, Mahalanobis) to segregate in-distribution data from ambiguous cases.
• Empirical evaluations show high accuracy and reduced human intervention, though challenges remain in scalability and integrating non-ideal expert feedback.

Expert-in-the-loop models are a class of AI and ML systems that systematically integrate domain experts into the model development and decision-making pipeline. These models generalize traditional human-in-the-loop (HITL) frameworks by allowing AI systems, often equipped with out-of-distribution (OOD) detection and dynamic routing, to collaborate with human experts and artificial experts (learnt specialist models) for optimal efficiency and accuracy. While human expertise remains critical for resolving instances that cannot be reliably classified or processed by automated systems, expert-in-the-loop approaches aim to transfer the handling of certain unknowns to trainable AI components, thereby reducing repetitive human workload and enhancing overall system performance (Jakubik et al., 2023).

1. System Architectures: Hybridization of Human and Artificial Experts

Contemporary expert-in-the-loop systems, such as the AI-in-the-Loop (AIITL) paradigm, extend classic HITL designs with a modular bank of artificial experts. Each incoming instance $x$ is processed as follows:

• General Model $f_0$: Classifies $x$ among the known classes $C_0$, equipped with an OOD detector $s_0(x)$.
• Expert Consultancy Decision: If $s_0(x)$ indicates in-distribution, $f_0(x)$ is accepted. Otherwise, $x$ is routed to Expert Selection.
• Expert Selection: A suite of $n$ artificial experts $f_1,\dots, f_n$, each managing a disjoint non-overlapping class set $C_1, \dots, C_n$, with distinct OOD detectors $s_i(\cdot)$.
  • If exactly one artificial expert claims $x$ (in-distribution), its output is used.
  • If no artificial expert claims $x$, or multiple claimants exist, the instance is escalated to the human expert.
• Human Expert: Assigns the correct label or, for novel classes, instantiates a new artificial expert trained on these previously unseen categories.

This architecture ensures the gradual absorption of unknown data by specialist models, thus systematically offloading routine classification from humans as the knowledge base of the system evolves (Jakubik et al., 2023).

2. Algorithms and Expert Engagement Strategies

Expert-in-the-loop models rely on a set of algorithmic primitives for effective partitioning of decision space between machinery and human specialists:

• OOD Detection and Deferral: Out-of-distribution scores ($s_j(x)$) determine model self-confidence. Popular algorithms include:
  • ODIN: $s_j(x) = \max_k \text{softmax}(z_j(x; T)/T)[k]$, augmented by temperature scaling and adversarial noise.
  • Mahalanobis-based: Computes the minimal Mahalanobis distance between $z_j(x)$ and per-class embedding means.
  • Thresholds $\tau_j$ and a mixture-of-experts gating network orchestrate assignment and escalation decisions.
• Artificial Expert Training: Each artificial expert $f_i$ is incrementally trained on class-specific datasets $D_i$ via cross-entropy minimization:

$$\text{Loss}_i(\theta_i) = -\frac{1}{|D_i|} \sum_{(x, y) \in D_i} \sum_{c \in C_i} 1_{[y=c]} \log p_{\theta_i}(c|x) + \lambda \|\theta_i\|^2,$$

with $\theta_i$ parameters updated by stochastic optimization and an activation threshold (e.g., 95% validation accuracy) gating expert participation.

• Gating and Allocation: The deferral mechanism can be formalized as:

$$A(x) = \begin{cases} f_0(x) & \text{if } s_0(x) > \tau_0, \ f_i(x) & \text{if } s_0(x) \le \tau_0 \text{ and } \exists! i: s_i(x) > \tau_i, \ \text{Human}(x) & \text{otherwise}. \end{cases}$$

This stratification of expertise and training enables low-latency, high-confidence decisions for in-distribution data, iterative expansion of machine-specialized domains, and dynamic expert engagement for ambiguous or novel instances (Jakubik et al., 2023).

3. Performance Metrics and Experimental Protocols

The assessment of expert-in-the-loop models relies on multiple quantitative measures:

• Accuracy $\varphi(X,\hat{Y})$: Fraction of correct predictions on the test set.
• Human Effort $\rho(X, \hat{Y})$: Proportion of instances routed to human review.
• Combined Utility $U(X, \hat{Y})$: Weighted accuracy minus human effort, $$U(X, \hat{Y}) = \alpha \varphi(X, \hat{Y}) - \beta \rho(X, \hat{Y})$$ with sensitivity to the human cost parameter $\beta$.

Empirical evaluations use benchmark setups such as CIFAR-10 (known domain), with SVHN, MNIST, Fashion-MNIST as incrementally revealed unknowns. The general model (e.g., Wide-ResNet-28-10) is trained on known classes and extended as new artificial experts are activated upon reaching accuracy thresholds. Iterative testing tracks both accuracy and reduction in human interventions (Jakubik et al., 2023).

4. Empirical Findings and Comparative Analysis

Notable findings from experimental studies include:

• AIITL with Mixture-of-Experts Gating: Achieves $\varphi=0.92$, $\rho=0.00$, $U=0.92$—retiring the human expert once sufficient artificial expert coverage is achieved.
• Mahalanobis and ODIN OOD Detectors: Both yield significant reductions in human effort and increased utility ($U=0.73$ and $U=0.61$ respectively) over traditional HITL ($U=0.39$).
• Baseline and Upper Bound: Even a perfect HITL setting (with optimal human allocation, $U=0.51$) is outperformed by AIITL variants.

Statistical significance is confirmed via paired t-tests ($p<0.01$) across repeated runs. The modular artificial expert bank also circumvents catastrophic forgetting—a key issue in monolithic model updates—by task decomposition (Jakubik et al., 2023).

5. Theoretical Insights, Practical Benefits, and Limitations

Key insights and practical consequences for expert-in-the-loop systems:

• Effort–Accuracy Tradeoff: AIITL dominates for any nontrivial value of $\beta$ (human review cost), except in regimes where human effort is nearly free.
• Learning Dynamics: Artificial experts require enough labeled support to meet activation thresholds; ODIN and Mahalanobis scoring can be deployed earlier, while learned gating ultimately offers higher utility at the price of increased initial annotation.
• Scalability Challenges: Linear growth in the number of experts per novel class introduces potential scalability issues if the domain contains hundreds of emerging classes—a limitation yet to be resolved.
• Domain and Feedback Limitations: Evaluations are currently restricted to the vision domain with idealized oracle humans. Real-world studies involving noisy or late expert feedback, or application to text and structured data, remain open areas for development.
• Activation Heuristic Flexibility: Reliance on fixed accuracy thresholds could be replaced or complemented by uncertainty-driven or PAC-style criteria to further optimize engagement (Jakubik et al., 2023).

6. Broader Context and Future Directions

The expert-in-the-loop paradigm generalizes to a variety of settings beyond image classification, as a formalization of mixed-initiative, modular, and incrementally adaptive machine intelligence architectures. Core design patterns—dynamic routing via OOD estimation, modular expert instantiation, incremental specialization, and human escalation—are applicable to structured data, continual learning, and lifelong adaptation. Open questions remain in managing expert bank complexity, efficiently handling non-ideal expert behaviors, and formalizing optimal engagement schedules under cost constraints. The architecture outlined sets the foundation for next-generation hybrid AI systems with provable benefits in utility and expert resource efficiency (Jakubik et al., 2023).