Bayesian Class Adaptation Plus (BCA+)

• Bayesian Class Adaptation Plus (BCA+) is a training-free unified framework for vision-language models that uses cache-based Bayesian inference.
• It employs a dual adaptation mechanism to dynamically adjust both likelihoods and class priors, boosting robustness under distribution shifts.
• Its dynamic cache efficiently fuses evolving likelihoods and adaptive priors, achieving state-of-the-art accuracy in real-time object recognition and detection.

Bayesian Class Adaptation Plus (BCA+) is a training-free, unified test-time adaptation framework for vision-LLMs (VLMs), extending the Bayesian Class Adaptation (BCA) paradigm to address both object recognition and detection under significant real-world distribution shifts. BCA+ dynamically adapts the model’s outputs using a cache-based Bayesian inference mechanism that fuses evolving likelihoods and adaptive priors derived from historical predictions, efficiently correcting semantic predictions and contextual confidence without backpropagation. This approach is designed to achieve state-of-the-art performance in accuracy and robustness, maintaining practical efficiency suited for real-time deployment scenarios.

1. Motivation and Background

Test-time adaptation (TTA) techniques seek to counteract performance degradation of VLMs such as CLIP and Grounding DINO when faced with out-of-distribution (OOD) data. Previous methods either perform computationally expensive gradient-based adaptation or restrict adaptation to the likelihood term, ignoring the critical influence of the prior over class predictions. BCA+ directly addresses these limitations by generalizing BCA to handle both object recognition and detection tasks, and by introducing a dynamic cache and a dual-adaptation mechanism for likelihood and prior (Zhou et al., 3 Oct 2025).

2. Bayesian Inference Formulation

BCA+ operationalizes adaptation as a Bayesian inference problem at test time. For a test input $x_{ij}$, the posterior probability of class label $Y$ is determined by integrating over all cache entries $\mu_m$: $$P(Y | x_{ij}) = \sum_m P(\mu_m | x_{ij}) \cdot P(Y | \mu_m),$$ where $P(\mu_m | x_{ij})$ is determined via Bayes’ theorem as: $$P(\mu_m | x_{ij}) = \frac{P(x_{ij} | \mu_m) P(\mu_m)}{\sum_j P(x_{ij} | \mu_j) P(\mu_j)}.$$ Here, $P(x_{ij} | \mu_m)$ captures the feature (and scale, for detection) similarity between the query and cache entry, and $P(Y | \mu_m)$ is the dynamically updated class prior.

For object recognition, similarity is computed as: $$P(x_{ij} | \mu_m) \propto \exp\left(\cos(f_{ij}^v, f_m^{\mathrm{cache}})\right),$$ where $f_{ij}^v$ is the visual feature and $f_m^{\mathrm{cache}}$ is the cached feature entry. For object detection, the likelihood additionally incorporates the normalized scale difference of bounding boxes.

3. Dynamic Cache Mechanism

BCA+ maintains a cache storing class embeddings, spatial scales (detection), and adaptive class priors. The cache is updated online:

• Cache matching uses cosine similarity; if no entry exceeds a threshold $\tau_2$, a new cache entry is created for the sample.
• Statistical updates: For the matched cache entry, class embeddings and spatial scales are updated by weighted running averages, and class priors are updated by aggregating the pseudo-label distributions of high-confidence samples.
• Adaptive priors: Rather than fixed one-hot priors, the class prior is updated to reflect empirical class probabilities in the evolving test stream, enabling contextual adaptation.

This mechanism ensures rapid, memory-efficient adaptation without retraining or additional backpropagation.

4. Dual Adaptation: Likelihood and Prior

The BCA+ framework adapts both the likelihood and the prior components in Bayesian inference:

• Likelihood adaptation acts on class embeddings and spatial scales, incorporating accumulating evidence about the appearance and context of each class/object.
• Prior adaptation learns the evolving class distribution from historic predictions, allowing the model to correct contextual misclassifications that would persist with static priors.

This dual mechanism enables more robust correction of both semantic misunderstandings and shifts in class prevalence.

5. Uncertainty-Guided Fusion

For final prediction, BCA+ fuses the initial VLM output ($p_{\mathrm{init}}$) and the cache-based prediction ($p_{\mathrm{cache}}$) using entropy-based weights:

$$p_{\mathrm{final}} = \frac{\exp(-E(p_{\mathrm{init}})) p_{\mathrm{init}} + \exp(-E(p_{\mathrm{cache}})) p_{\mathrm{cache}}}{\exp(-E(p_{\mathrm{init}})) + \exp(-E(p_{\mathrm{cache}}))}$$

where $E(p)$ is the Shannon entropy. This uncertainty-guided fusion prioritizes more confident (lower entropy) predictions, yielding calibrated outputs under severe distribution shift.

6. Empirical Performance and Efficiency

BCA+ demonstrates state-of-the-art metrics across diverse OOD benchmarks: for object recognition (e.g., CLIP ResNet-50, ViT-B/16) and detection (e.g., Grounding DINO on FoggyCityscapes, PASCAL-C, COCO-C), consistent accuracy improvements (0.6–0.9% over BCA for recognition; multi-point mAP₅₀ gains for detection) have been observed (Zhou et al., 3 Oct 2025). The cache-based mechanism is low-overhead, avoiding backpropagation and keeping inference latency and memory footprint competitive for real-time systems.

Performance Table

Task	Backbone	BCA+ Accuracy / mAP₅₀	Baseline Accuracy / mAP₅₀
Recognition OOD	CLIP RN50	61.81%	61.35% (TDA), lower (Zero)
Detection OOD	Swin-T DINO	26.65%	23.88% (Baseline DINO)

7. Significance and Implications

BCA+ generalizes Bayesian adaptation principles to a broad set of vision-language tasks. Its training-free, dual adaptation mechanism offers a rigorous way to correct VLM predictions under distribution shift, with both semantic and contextual error-correction. By shifting from fixed priors and static caches to adaptive, uncertainty-aware fusion of prior and likelihood, BCA+ advances test-time adaptation to a robust, efficient paradigm anchored in Bayesian inference.

A notable feature is the applicability to both recognition and detection within a single framework and its suitability for real-time deployment due to computational simplicity and minimal memory expansion.

8. Connections to Broader Bayesian Adaptation and Future Directions

The cache-based, dual-adaptation strategies in BCA+ are consistent with PAC-Bayesian and empirical Bayesian approaches in adaptation literature (Germain et al., 2012, Sicilia et al., 2022). The use of dynamic priors aligns with recent calls to model class-frequency shift and contextual uncertainty in real-world applications. A plausible implication is that BCA+ or similar frameworks could be extended to multimodal or sequential scenarios, incorporating metadata, temporal dependencies, or additional hierarchical priors, provided these augment the cache-based statistics in principled Bayesian ways.