Linguistic Relative Policy Optimization for Video Anomaly Reasoning

Abstract: Video anomaly detection (VAD) with multimodal LLMs has shown strong potential, yet most existing methods still depend on large-scale annotations or expert-designed priors, limiting their ability to acquire anomaly knowledge with as little human intervention as possible. To address this, we propose Linguistic Relative Policy Optimization (LRPO), which distills group-relative semantic advantages from multiple reasoning trajectories into a linguistically expressed anomaly experience prior, and adapts the model by injecting this prior into the context to steer its output distribution without any parameter updates. LRPO builds two complementary experience representations: general experience captures transferable anomaly preferences across scenarios, while scenario experience models context-dependent anomaly rules for targeted refinement. To further improve the learned experience, we introduce an anomaly alignment reward that guides trajectory optimization to match human risk preferences and reinforce temporally grounded reasoning. Extensive experiments on XD-Violence, UCF-Crime, and UBnormal demonstrate that LRPO significantly outperforms existing state-of-the-art methods under tuning-free settings.

• The paper introduces a tuning-free adaptation framework (LRPO) that leverages linguistic anomaly experience to steer frozen vision-language model outputs.
• It employs dual experience repositories and a group-relative reward structure, achieving state-of-the-art performance on benchmarks like XD-Violence and UCF-Crime.
• The method reduces annotation costs while improving interpretability and scalability for robust, context-sensitive video anomaly reasoning.

Linguistic Relative Policy Optimization for Tuning-Free Video Anomaly Reasoning

Introduction and Problem Statement

The paper "Linguistic Relative Policy Optimization for Video Anomaly Reasoning" (2607.00654) addresses the limitations of existing multimodal LLM (MLLM)-based video anomaly detection (VAD) paradigms. Traditional instruction-tuned VAD approaches require costly large-scale, fine-grained annotations to update model weights, while training-free methods, despite preserving model generalization, depend heavily on expert-designed priors and handcrafted pipelines. Both hinder the automated and scalable acquisition of anomaly knowledge, especially under distribution shifts and limited supervision.

The central proposal, Linguistic Relative Policy Optimization (LRPO), reframes adaptation for VAD by eschewing parameter updates: instead, it distills semantically linguistic anomaly experience into an editable contextual repository, injected at inference to steer the output distribution of a frozen vision-LLM (VLM). This strategy significantly reduces annotation costs and engineering overhead while preserving foundation model generality.

Figure 1: Summary of recent VAD paradigms, positioning LRPO as context-driven, low-annotation, and tuning-free.

Methodology: LRPO Framework

The LRPO pipeline maintains a dual-representation of domain knowledge: a general experience repository, capturing transferable anomalous patterns, and a scenario experience repository, encoding context-dependent event rules. Experience is distilled through group-relative reinforcement signals without model fine-tuning, and an LLM optimizer iteratively reflects on sampled reasoning outputs to edit and improve the experience pool over time.

Figure 2: The LRPO pipeline – interaction between a learner VLM and an optimizer LLM drives iterative optimization of language-based anomaly experience under reward constraints.

Adaptation occurs exclusively within a linguistic experience space. At each iteration, multiple reasoning trajectories are sampled per video using the learner VLM under the current experience context. The optimizer LLM summarizes and contrasts these outputs based on specific reward signals (accuracy, risk-preference alignment, and temporal-dependency consistency), identifying and preserving semantic advantages while correcting or removing suboptimal rules. The scenario experience is efficiently constructed via weak-label prompting, and efficient retrieval is realized through a dual visual-text retrieval scheme.

Reward Structure and Optimization

LRPO’s reward framework is multifaceted:

• Accuracy reward provides basic supervision, pushing anomaly scores towards weak video labels.
• Anomaly preference reward aligns reasoning to human-defined risk biases, leveraging scenario experience as positive preference and LLM-generated negative perturbations for contrast, enhancing robustness against shortcut learning.
• Temporal dependency reward ensures anomaly assessment is sensitive to causal/temporal evidence, penalizing static semantics and encouraging dynamic reasoning across frame orderings.

Optimization is fundamentally group-relative, minimizing overfit to singular labels or spurious details and facilitating stable policy improvements.

Empirical Results

State-of-the-Art Benchmarking

LRPO achieves strong sample efficiency and generalization:

• XD-Violence: 73.17% AP using only 2.5% of the training set, outperforming prior tuning-free and tuning-based methods.
• UCF-Crime: 85.36% AUC with 6% labeled videos, closely matching full-data performance.
• UBnormal: 76.24% AUC by direct experience transfer from XD-Violence, indicating substantial cross-domain robustness.

Notably, as shown in the ablation, marginal improvement arises from increasing annotated data beyond a small, balanced subset. LRPO’s context-editable experience mechanism drives efficient use of human annotation.

Reward and Experience Component Analysis

Ablations demonstrate that:

• Combining general and scenario experience increases AP by over 10% compared to model-only or single-experience baselines.
• The addition of preference and temporal rewards yields steady accuracy improvements, with full reward integration achieving state-of-the-art frame-level results.
• Visual-semantic retrieval for scenario experience contributes an additional 1.3% performance boost over visual-only or random selection.

Model and Optimizer Scalability

Performance improvements from LRPO are consistent across both LLaVA-NeXT-7B and InternVL3_5-8B, showing applicability to different VLM backbones and optimizer LLMs. Optimizer scale moderately affects the ceiling, but is not the determining factor for success.

Figure 3: Scaling and stability of LRPO – increasing scenario experience at inference improves performance; training-time scaling leads to overfitting. Results are robust to different random subset samplings.

Figure 4: Hyperparameter ablation – group size, general experience memory size, and Top-K retrieval parameter exhibit robust and interpretable effect on performance.

Qualitative Analyses

LRPO outputs interpretable, human-readable reasoning trajectories that transparently cite underlying experience rules—both general (e.g., multi-class conflict-resolution priorities, evidence composition) and scenario-specific (e.g., “ice-hockey fighting is normal absent hostile intent”). Injecting both experience types yields context-sensitive and risk-aligned conclusions, with anomaly score temporal curves exhibiting improved localization and reduced false positives/negatives compared to baselines.

Figure 5: Qualitative example – general and scenario experience enable robust, preference-aligned anomaly reasoning.

Figure 6: Frame-level anomaly score visualization – LRPO achieves superior event localization and temporal consistency.

Figure 7: Normal-case reasoning – experience injection leads to correct rejection of false anomalies in context-specific highway scenarios.

Figure 8: Additional frame-level qualitative results on XD-Violence – LRPO provides more discriminative and temporally stable anomaly confidence.

Theoretical and Practical Implications

This work decouples adaptation from parameter modification, opening a new axis for foundation model specialization via context-editable, natural-language experience repositories. The approach:

• Provides a formal framework where linguistic experience editing induces analogous distribution shifts to parameter updates, but remains stable via group-relative RL objectives and KL clipping.
• Reduces reliance on costly annotation and fragile pipeline engineering, promoting scalable generalization to new anomaly domains.
• Enhances traceability, interpretability, and controllability of foundation VLM-based reasoning, which is crucial for high-stakes applications (e.g., safety-critical surveillance, forensic review).

Future Directions

Extensions include automated and scalable experience expansion to increase repository coverage, validation for open-world and rare anomaly scenarios, and integration with online or continual feedback mechanisms for real-time adaptation. Formalizing generalization bounds and potentially coupling with light-weight parameter-efficient tuning could further strengthen robustness.

Conclusion

Linguistic Relative Policy Optimization establishes an effective, sample-efficient, and tuning-free adaptation paradigm for video anomaly detection via context-driven, editable experience optimization. Empirical benchmarks confirm state-of-the-art sample-efficiency, transferability, and reasoning consistency. The LRPO framework offers a robust pathway for human-in-the-loop and risk-aligned deployment of video reasoning systems in complex, rapidly evolving real-world settings.

---