A Multi-Turn Framework for Evaluating AI Misuse in Fraud and Cybercrime Scenarios

Abstract: AI is increasingly being used to assist fraud and cybercrime. However, it is unclear whether current LLMs can assist complex criminal activity. Working with law enforcement and policy experts, we developed multi-turn evaluations for three fraud and cybercrime scenarios (romance scams, CEO impersonation, and identity theft). Our evaluations focused on text-to-text model capabilities. In each scenario, we measured model capabilities in ways designed to resemble real-world misuse, such as breaking down requests for fraud into a sequence of seemingly benign queries, and measuring whether models provide actionable information, relative to a standard web search baseline. We found that (1) current LLMs provide minimal practical assistance with complex criminal activity, (2) open-weight LLMs fine-tuned to remove safety guardrails provided substantially more help, and (3) decomposing requests into benign-seeming queries elicited more assistance than explicitly malicious framing or system-level jailbreaks. Overall, the results suggest that current risks from text-generation models are relatively minimal. However, this work contributes a reproducible, expert-grounded framework for tracking how these risks may evolve with time as models grow more capable and adversaries adapt.

• The paper presents a multi-turn evaluation framework that uses expert-validated pipelines to simulate real-world fraud and cybercrime scenarios.
• It shows that well-aligned LLMs typically yield generic outputs, while uncensored models deliver more actionable, though still limited, assistance.
• The study underscores how safety alignment and prompt decomposition critically impact responses, calling for dynamic red-teaming and robust cross-turn monitoring.

Multi-Turn Evaluation of AI Misuse in Fraud and Cybercrime: An Expert-Grounded Framework

Introduction and Motivation

Systematic assessment of AI-enabled misuse in fraud and cybercrime is challenged by the paucity of reliable case data, obfuscated adversarial behavior, and limited operational linkage between incident reporting and specific LLM deployments. The paper "A Multi-Turn Framework for Evaluating AI Misuse in Fraud and Cybercrime Scenarios" (2602.21831) addresses these gaps by introducing a multi-turn, operationally validated evaluation pipeline—Long-Form Tasks (LFTs)—to quantify the extent to which current LLMs can provide actionable support in complex, real-world criminal workflows. The framework is designed to simulate threat actor tactics as they would unfold in practice, incorporating adversarial decomposition and benign/obfuscated queries to test failure modes of current alignment strategies.

Methodology: Structured and Expert-Validated LFT Pipeline

The LFT development process follows a rigorous, five-stage pipeline: model functionality identification, risk modelling, scenario development, prompt engineering, and rubric calibration, each incorporating input from fraud and cybercrime operational experts to ensure realistic threat coverage.

Figure 1: The LFT pipeline aligns each task with real-world misuse patterns via continuous expert input, spanning model analysis to expert-validated grading criteria.

Model selection centered on coverage of reasoning and web-search capabilities, safety alignment variants (including uncensored/fine-tuned LLMs), and actor sophistication levels. The risk model draws from the legislative approach to the fraud offense in the UK, operationalized into a six-stage attack lifecycle: planning, reconnaissance, falsification, engagement, evasion, and scaling. Scenario coverage targets three of the most prevalent fraud typologies—romance scams, CEO impersonation, and identity theft—each with three actor profile variants (non-technical, technical, and collaborative operators). Each LFT consists of twelve sequential prompts (two per stage), with both benign and malicious framing to probe LLM compliance under nuanced prompt decomposition.

Grading is automated via GPT-4.1 as an LLM-as-a-judge, with rubrics co-developed with domain experts and anchored to both actionability and information access beyond baseline search.

Figure 2: The evaluation protocol employs fixed, multi-turn prompts, with auto-grading on actionability and informational novelty, referencing the running conversation context.

Results: Empirical Evaluation Across Models and Scenarios

Distribution and Magnitude of Assistance

A large-scale experimental suite—20,088 LFTs across 14 LLMs—demonstrates minimal practical assistance from well-aligned LLMs for fraud and cybercrime tasks:

• 88.5% of responses scored at or below rubric level 2 on actionability (refusal or only generic information)
• 67.5% of responses similarly failed to synthesize information beyond basic search

The correlation between actionability and information provision was robust, but distributions reveal that the bulk of LLM outputs are confined to generic or refusal-level content, with a clear ceiling effect for higher-order criminal utility.

Figure 3: The overwhelming majority of responses provide only generic or no actionable content, with higher-order assistance rarely observed.

Model Characteristics and Safety Alignment Effects

Model choice—particularly safety alignment—was the primary determinant of assistance quality. Uncensored models (fine-tuned to strip safety guardrails) contributed substantially more practical help: 22.5% of their responses reached ready-to-use, detailed, or higher actionability, compared to low single-digit percentages for aligned counterparts. Notably, even top-performing uncensored models rarely reached level 5/6 assistance, and safety-aligned models rarely exceeded level 2.

Figure 4: Model main effects with 94% credible intervals; uncensored models can exceed aligned baselines by sizeable margins for both actionability and information access.

Furthermore, decomposition attacks markedly increased assistance, with benign task framing yielding average actionability and information access improvements of 0.86 and 1.20 rubric points over explicit malicious requests, respectively. However, the effect was additive rather than multiplicative and bounded by underlying model safety alignment.

Figure 5: Predicted scores for actionability/information access by model and decomposition method; hatched bars denote uncensored models exhibiting dominant assistance rates under benign prompts.

Scenario-Specific Findings and Attack Sensitivity

Scenarios involving CEO impersonation received the highest typical assistance, followed by romance fraud and identity theft. Benign prompt decomposition was especially potent in identity theft, where the delta between benign and malicious framing in model responses was most pronounced. This highlights that technical attack detection is robust in explicit settings, but safety defenses erode under skillful obfuscation.

Figure 6: Scenario-wise assistance, showing differential vulnerability to decomposition between technical and social engineering attacks.

System-level jailbreak attempts (prompting, prompt injection) exerted negligible effects, suggesting current alignment strategies are robust to these attacks relative to prompt decomposition.

Increase in actor sophistication did not meaningfully affect LLM assistance rates—implicating that models respond to the nature and framing of the request, not the apparent expertise of the user.

Extended Reasoning and Search Capabilities

Within the subset of reasoning-capable models, the use of explicit reasoning tokens modestly increased actionability and information access (β ≈ 0.6), with notable variability across repeated runs. Reasoning did not qualitatively override safety features. Integrated search capabilities showed mixed effects, likely confounded by additional alignment measures in those variants.

Figure 7: Model and reasoning effects; increases in reasoning tokens modestly boost assistance, but consistent safety restrictions remain observable.

Implications and Future Directions

This framework demonstrates replicable, expert-driven evaluation of real-world AI misuse risks and establishes important empirical boundaries: current well-aligned LLMs are not a practical force-multiplier for complex fraud or cybercrime through text alone, at least under present capabilities and safety measures. The clear dependency on safety alignment, as opposed to parameter count or release date, underscores the primacy of continuous, dynamic safety fine-tuning and red-teaming for deployed models.

The data expose decomposition attacks as the Achilles' heel of current alignment approaches. Future models must coordinate context-aware, conversational alignment across multi-turn contexts, not just perform shallow prompt-level intent detection. Novel LLM architectures or monitoring frameworks may be required for robust cross-turn harmful intent inference.

Pragmatically, the methodology is extensible to emerging crime modalities and multi-modal LLMs. Comparative baselining with human adversary behavior and non-text media (voice, visual synthetic content) are immediate expansions necessary to track the moving target of AI-assisted cybercrime risk. Integration with open red-teaming and incident-sharing ecosystems will further sharpen LLM risk measurement and mitigation strategies.

Conclusion

The presented multi-turn LFT framework delivers an operationally grounded, statistically robust apparatus for quantifying AI-assisted fraud and cybercrime risk. Results demonstrate that safety alignment is currently an effective limiter for LLM criminal assistance, but decomposition-based prompts and uncensored models circumvent these defenses to a non-trivial extent. Maintaining and refining alignment—especially for open-weight or downloadable models—remains critical. Systematic, expert-in-the-loop evaluations such as this are necessary to distinguish genuine misuse risks from theoretical speculation and should evolve in lockstep with model capabilities and adversarial sophistication.