Deep VRegulome: DNABERT Regulatory Variant Prediction

• Deep VRegulome is a DNABERT-based deep learning framework that aggregates 700 fine-tuned models to predict and score the functional impact of short non-coding variants.
• It employs overlapping 6-mer tokenization and rigorous variant-to-function scoring with motif-level attention visualization for transparent functional interpretation.
• The platform demonstrates high accuracy in TF binding and splice-site prediction and integrates clinical survival analysis to prioritize disease-associated mutations.

Deep VRegulome is a DNABERT-based deep learning platform for the prediction and functional interpretation of short genomic variants in human regulatory regions. By assembling an ensemble of 700 independently fine-tuned DNABERT models, each trained on ENCODE-derived gene regulatory elements or splice junctions, Deep VRegulome enables systematic variant impact prioritization across the non-coding regulome. The framework includes rigorous variant scoring, motif-level attention visualization, and survival analysis pipelines for clinical correlations, and has demonstrated utility in high-throughput prioritization of survival-associated regulatory mutations in whole-genome data from The Cancer Genome Atlas (TCGA) glioblastoma cohort (Dutta et al., 12 Nov 2025).

1. Model Architecture and Task-Specific Fine-Tuning

The core of Deep VRegulome is the DNABERT Transformer encoder, comprising 12 layers and 12 self-attention heads per layer, a hidden dimensionality of 768, and pre-training by masked language modeling (MLM) on the human reference genome. DNABERT uses overlapping 6-mer tokenization (stride 1) with special tokens [CLS] and [SEP]. Deep VRegulome repurposes this architecture for regulatory variant prediction by appending a single linear classification head to the final [CLS] output. The output head computes a sigmoid probability $p(y=1|\text{sequence})$ of functional disruption per input window.

Fine-tuning is performed separately for each of 700 regulatory tasks: 667 transcription factor (TF) binding or histone mark tasks from ENCODE ChIP-seq and 33 splice-site tasks from GENCODE v41. Each model is trained on fixed-length sequences (301 bp for ChIP-seq, 90 bp for splice sites) tokenized into overlapping 6-mers; positive labels are defined by annotated regulatory peaks or junctions, with negative examples drawn from matched background windows (Dutta et al., 12 Nov 2025).

Optimization leverages AdamW with linear warmup and decay, dropout ($p=0.1$ in attention and feed-forward layers), early stopping, and binary cross-entropy loss. Only models reaching at least 85% validation accuracy are retained in the ensemble.

2. Data Sources, Preprocessing, and Task Framing

Training data are constructed from two major sources. The ENCODE ChIP-seq corpus underpins TF and histone mark classifiers, with positive class windows centered at annotated peak summits and negative examples drawn from background. The splice-site corpus uses ±45 bp windows around GENCODE donor or acceptor junctions as positives, with analogous negative controls from mappable regions of the genome.

All sequences are standardized in length (90 bp or 301 bp) and fully tokenized as overlapping 6-mers (yielding 85 or 296 tokens, respectively). For each model, training proceeds with balanced classes and early stopping on a validation set. Ensemble diversity arises from task-specific models sensitive to the individual sequence grammars and length distributions associated with each regulatory factor or junction type.

3. Variant-to-Function Scoring and Aggregation

Deep VRegulome assesses the impact of short variants by mapping each candidate to the set of overlapping reference windows associated with the relevant 700 task-specific models, denoted $M(v)$. For every variant, separate inference is run on reference and alternate sequences using all applicable models. Each model outputs two probabilities: $p_i(\text{ref})$ and $p_i(\text{alt})$.

Impact quantification uses the log-odds ratio per model:

$$f_i = \log_2 \frac{p_i(\text{ref})}{1-p_i(\text{ref})} - \log_2 \frac{p_i(\text{alt})}{1-p_i(\text{alt})}$$

The overall variant impact score $S(v)$ is defined as the mean $f_i$ across all models in $M(v)$:

$S(v) = \frac{1}{|M(v)|} \sum_{i\in M(v)} f_i$

Variants are labeled as functionally "disruptive" if $p(\text{ref}) > 0.5$ and $p(\text{alt}) < 0.5$ within any model. Additional quantitative metrics include score-change $$\Delta p = (p(\text{alt}) - p(\text{ref})) \cdot \max\{ p(\text{ref}), p(\text{alt}) \}$$, facilitating downstream ranking.

4. Model Performance, Validation, and Benchmarking

Performance evaluation demonstrates high accuracy and discrimination for the Deep VRegulome ensemble. Splice-site classifiers achieve 94–96% accuracy, F1 scores of 0.94–0.96, and MCCs of 0.89–0.92. Among 667 TF tasks and 33 histone marks, 421 and 4 models, respectively, exceed 85% accuracy, with the top TF models attaining ROC-AUC > 0.97. Deep VRegulome models outperform random (AUC=0.50) and at minimum match legacy CNNs such as DeepSEA on held-out ChIP-seq peaks (direct numerical comparisons not provided)(Dutta et al., 12 Nov 2025).

Application to TCGA glioblastoma WGS data identified 572 recurrent splice-disrupting and 9,837 regulatory mutations affecting TFBS, with 1,352 mutations and 563 regulatory regions linked to survival through downstream clinical analysis. This illustrates direct utility for large-scale variant prioritization and association studies.

5. Interpretation: Attention-Based Motif Visualization

Deep VRegulome provides motif-level interpretability via attention score mapping. For each input, the final-layer attention matrix $A^{(\ell,h)} \in \mathbb{R}^{T \times T}$ is extracted, and the attention weights from the [CLS] token to every input token are averaged over all heads:

$$\alpha_j = \frac{1}{H} \sum_{h=1}^H A^{(L,h)}_{\mathrm{[CLS]},j}$$

These attention scores are mapped back to nucleotide space (evenly distributed over 6 nucleotides per 6-mer token), yielding nucleotide-resolution heatmaps highlighting sequence regions contributing most to predicted regulatory function or disruption. This supports motif discovery and functional interpretation of variant impacts.

6. Implementation, Computational Requirements, and Code Availability

All models use the HuggingFace Transformers (PyTorch) DNABERT implementation with custom tokenization data-loaders and variant-pair batching. Each DNABERT model has approximately 110 million trainable parameters. Training of a single model requires approximately two hours on NVIDIA V100 GPUs (32 GB); complete suite training for the 700-model ensemble totals about 1,400 GPU-hours (Dutta et al., 12 Nov 2025).

All code, fine-tuned model checkpoints, and an interactive data portal are publicly available for downstream analysis and re-use, facilitating reproducibility.

7. Context, Related Work, and Future Perspectives

Deep VRegulome represents a comprehensive application of foundation-model-based transfer learning to non-coding variant impact prediction in the human regulome. The ensemble paradigm, in which multiple task-specific DNABERT models capture unique regulatory grammars, achieves both sensitivity and interpretability for high-throughput variant annotation. Unlike approaches based on single CNNs or fixed motif scans, this framework leverages transformer self-attention to model broad dependencies and to extract informative sequence features at nucleotide-level (Dutta et al., 12 Nov 2025).

A plausible implication is that similar multi-task, attention-based architectures may generalize to other species and to additional genomics applications if relevant ChIP-seq or regulatory annotations are available. The demonstrated integration of clinical outcome data (e.g., survival analysis) with variant scoring enables functionally grounded, disease-relevant prioritization in human genetics. The requirement for high computational resources—particularly for full ensemble retraining or cross-species expansion—remains a practical consideration.