Synthetic Multi-RADS Benchmark

• Synthetic Multi-RADS Benchmark is a controlled testbed that uses synthetic radiology reports from multiple RADS frameworks to assess model performance.
• It employs expert-validated templates and advanced LLM-driven report synthesis to ensure clinical realism and structured data variability.
• Findings indicate that guided prompting and increased model scale significantly enhance validity and accuracy in diagnostic categorization.

A Synthetic Multi-RADS Benchmark is a rigorously designed testbed for evaluating machine learning and natural language processing models on the automated assignment of categories from standardized radiology reporting frameworks (RADS) using synthetic data. Such benchmarks systematically simulate the diversity and complexity of real-world radiology reports, spanning multiple imaging modalities and RADS schemas, and are characterized by detailed construction protocols, domain-expert validation, and transparent evaluation methodologies. These benchmarks serve as reference standards for comparing models, especially LLMs and deep neural networks, in both multi-class classification and information extraction tasks relevant to diagnostic decision support.

1. Scope and Design Principles

Synthetic Multi-RADS Benchmarks encompass multiple Reporting and Data Systems (RADS) frameworks, unified to represent the complexity of radiologic interpretation and reporting workflows. Prominent examples include the RXL-RADSet, which covers BI-RADS, CAD-RADS, GB-RADS, LI-RADS, Lung-RADS, NI-RADS, O-RADS, PI-RADS, TI-RADS, and VI-RADS across computed tomography (CT), magnetic resonance imaging (MRI), ultrasound (US), and mammography. These datasets are designed to allow for fine-grained, schema-driven testing of model performance on categorization, adherence to reporting lexicons, and robustness to linguistic and structural variation in reports. The synthetic nature enables control over report diversity, ground-truth label assignment, and the inclusion of rare or complex diagnostic scenarios not readily available in naturally occurring corpora (Bose et al., 6 Jan 2026).

2. Dataset Creation and Expert Validation

Construction of a synthetic multi-RADS benchmark typically involves scenario-driven generation of reports, where clinical experts design templates encoding targeted modalities, anatomical site, lesion phenotypes, and desired RADS categories. Model-driven report synthesis is performed using advanced LLMs (e.g., GPT-5.2, Gemini, Claude), with stylistic diversity simulated through multiple radiologist personas ranging in subspecialty and experience.

A two-stage expert verification pipeline ensures dataset validity:

• Primary screening: Senior radiologists audit for clinical realism, section completeness, and adherence to standard section headers (Indication, Technique, Findings, Impression). Revision rates at this stage can range from 5% to 13.5%.
• Subspecialty audit: Domain experts check that generated findings logically support the labeled RADS category, correct lexicon misapplications, and ensure compliance with scoring nuances. Level-2 revision rates are 3–8%, primarily addressing lexicon fidelity and edge-case mislabeling (Bose et al., 6 Jan 2026).

Synthetic mammography-specific benchmarks, such as those described in (Matthews et al., 2020) and (Seyyedi et al., 2020), construct extensive annotated datasets using vendor-supplied algorithms to generate high-resolution “synthetic” 2D projections from DBT volumes, enabling systematic study of pretraining, adaptation, and calibration strategies.

3. Benchmarking Paradigms, Model Evaluation, and Prompting

Evaluation protocols encompass large-scale head-to-head comparisons of open-weight small LLMs (SLMs; e.g., Qwen3, DeepSeek R1, Gemma3, Llama4 MoE, Phi4, Nemotron, Olmo3, SmolLM2, Granite4; spanning 0.135–32B parameters) and proprietary models (e.g., GPT-5.2). Deterministic, schema-guided prompting strategies encode explicit category definitions, tie-breaker rules, and strict output constraints to interrogate both model accuracy and format adherence.

Key prompting modes include:

• Guided prompting: System prompt embeds the scoring rubric, while the user prompt instructs category assignment based on the report text, enforcing a single-token output.
• Zero-shot prompting: Only the user prompt is provided; the system prompt is omitted, exposing the model to greater ambiguity.

Performance is quantified using:

• Validity: Percentage of outputs conforming to required schema (e.g., correct label set).

  $$\text{Validity} = \frac{\text{Number of valid predictions}}{\text{Total predictions}} \times 100\%$$

• Accuracy: Percentage of reports assigned the correct RADS category, counting invalid outputs as errors.

  $$\text{Accuracy} = \frac{\text{Number of correct assignments}}{\text{Total predictions}} \times 100\%$$

• Conditional accuracy: Restricted to the valid subset only.
• In image-based tasks, linearly weighted Cohen’s $\kappa_w$ is common, with bootstrap-derived 95% CIs and statistical significance via z-tests (Matthews et al., 2020).

For multi-view CNNs on synthetic mammography, held-out test accuracy, per-class ROC AUC, recall, and precision are reported, with statistical tests (DeLong) for dataset size and resolution dependencies (Seyyedi et al., 2020).

4. Main Findings and Quantitative Outcomes

Synthetic Multi-RADS Benchmarks have elucidated several core trends:

Model Class / Metric	Validity (%)	Accuracy (%)	κ_w (site/setting)
GPT-5.2 (guided, 1600 cases)	99.8	81.1	—
Top SLMs (20–32B)	~99	75–78	—
All SLMs (pooled)	96.8	61.1	—
Sub-1B SLMs	82.9	27.0	—
Site 1 FFDM (test)	—	—	0.75 [95% CI 0.74–0.76]
Site 2 SM (matrix cal, 500)	—	—	0.79 [0.76–0.81]

• Model scaling: Pronounced performance improvement between sub-1B and ≥10B parameter SLMs, with a validity plateau at ~99% beyond 20B; proprietary LLMs (e.g., GPT-5.2) surpass open models in high-complexity RADS (Bose et al., 6 Jan 2026).
• RADS framework complexity: Accuracy for SLMs degrades as a function of Total Complexity Score (TCS), with high-TCS schemas like LI-RADS and PI-RADS yielding SLM accuracy ~49.4%, while GPT-5.2 maintains ~90%. This suggests that most errors in SLMs are attributable to reasoning failures rather than schema violations.
• Prompting effects: Guided prompting confers strong benefits over zero-shot, raising maximum valid output rates (99.2% vs 96.7%) and effective accuracy (78.5% vs 69.6%).

In vision benchmarks, SCREENet achieves BI-RADS 0 vs not-0 AUC = 0.912 and accuracy = 84.8% when trained on the entire synthetic dataset. Reductions in training size or image resolution produce statistically significant drops in AUC and accuracy. For breast density, adaptation methods such as vector/matrix calibration and FC fine-tuning allow models pretrained on FFDM to achieve strong agreement on SM datasets with minimal target-site data consumption, up to $\kappa_w = 0.79$ with only 500 SM images (Matthews et al., 2020).

5. Methodological Innovations and Adaptations

Synthetic multi-RADS benchmarks enable evaluation under tightly controlled experimental conditions. Innovations include:

• Scenario-driven variety: Systematic simulation of both common and rare diagnostic scenarios, templated lexical variance, and structured section headers.
• Efficient domain adaptation: For BI-RADS breast density, lightweight matrix calibration (20 parameters) achieves substantial adaptation with 100–500 SM images, while larger adaptation (FC fine-tuning) is superior if >1000 images are accessible (Matthews et al., 2020).
• Multi-view modeling: In SCREENet, late fusion of CC and MLO projections (ResNet-50 branches) combines complementary anatomical cues, setting a baseline for high-resolution synthetic mammography (Seyyedi et al., 2020).
• Metric conventions: Deterministic, exact schema validation and bootstrapped uncertainty quantification support reproducible and comparable evaluation across architectures and datasets.

These methodologies facilitate the benchmarking of both transformer-based (text) and CNN-based (vision) models on highly standardized clinical classification tasks while minimizing confounds from label leakage or domain shift.

6. Limitations and Recommendations

Synthetic Multi-RADS Benchmarks provide tractable, realistic, and extensible testbeds but exhibit important constraints:

• Synthetic-to-real domain gap: While expert-generated reports offer control and diagnostic diversity, generalizability to real-world reporting depends on subsequent validation across institutions and languages.
• Ceiling effects: Even with guided prompting and high parameter counts, open-weight SLMs lag behind proprietary LLMs, primarily in complex reasoning-intensive RADS.
• Adaptation challenges: For vision tasks, adaptation from FFDM to SM images is non-trivial; training from scratch on limited SM never matches the adapted FFDM model.
• Augmentation and class balance: Certain benchmarks do not employ extensive augmentation or cost-sensitive learning, potentially affecting rare subclass performance.
• Clinical deployment: Human-in-the-loop governance, schema validation, and abstention strategies are essential for risk mitigation in translational contexts (Bose et al., 6 Jan 2026).

Future directions include targeted instruction-tuning, hybrid rule-based-LLM architectures, and international, multilingual benchmarking to further enhance clinical robustness and fairness.

7. Impact and Benchmarking Standards

Synthetic Multi-RADS Benchmarks, exemplified by RXL-RADSet and major open-access mammography datasets, have become normative for high-fidelity, reproducible assessment of ML and NLP systems in radiology. Their rigor in design, annotation, and evaluation fosters direct comparison of emerging models, stimulates innovation in transfer and adaptation techniques, and provides practical blueprints for multi-site, multi-modality benchmarking frameworks. By adhering to patient-level splits, transparent statistical testing, and carefully curated task definitions, these benchmarks have clarified strengths and limitations across a broad spectrum of model architectures and scales, shaping the development of clinically relevant AI in radiology (Bose et al., 6 Jan 2026, Matthews et al., 2020, Seyyedi et al., 2020).