CodeBERT Embeddings

• CodeBERT embeddings are dense, context-aware vector representations derived from a transformer architecture that jointly encodes programming and natural language.
• They utilize methods like token-level mean-pooling, significantly improving metrics such as accuracy and MCC on tasks like vulnerability detection and code search.
• Recent innovations like LoRA adapters enable efficient fine-tuning, boosting performance in specialized code tasks with minimal parameter overhead.

CodeBERT embeddings are dense, context-aware vector representations of source code tokens and sequences, derived from the CodeBERT Transformer architecture. CodeBERT, introduced by Feng et al. (Feng et al., 2020), extends BERT/RoBERTa to jointly encode programming language (PL) and natural language (NL), enabling bimodal and unimodal tasks such as code search, summarization, and vulnerability detection. CodeBERT embeddings, commonly 768-dimensional per token, leverage pre-training objectives tailored for code, but are limited in capturing deep code structure and semantic logic. Recent advances, notably LoRACode adapters (Chaturvedi et al., 7 Mar 2025), enhance these embeddings via parameter-efficient fine-tuning. Several empirical studies have analyzed extraction strategies, feature reliance, pooling effects, application integration, and adaptation methodologies.

1. CodeBERT Architecture and Embedding Generation

CodeBERT employs the RoBERTa-base architecture: 12 Transformer encoder layers, hidden size 768, 12 self-attention heads, totaling ≈125 M parameters (Feng et al., 2020, Farasat et al., 16 Sep 2025). Input sequences are tokenized with a BPE/WordPiece vocabulary trained on PL and NL. Each input token $t$ at position $p$ in segment $s$ is embedded as

$$E(t,p,s) = E_{token}(t) + E_{pos}(p) + E_{seg}(s) \in \mathbb{R}^{768}$$

Embedding vectors pass through multi-head attention and feed-forward sublayers, yielding per-token contextualized embeddings (shape: sequence length $\times$ 768). The encoder is pre-trained jointly on code and natural language, using Masked Language Modeling (MLM), where 15% of tokens are masked and predicted, and Replaced Token Detection (RTD), which discriminates “real” from synthetic tokens (Feng et al., 2020, Yu et al., 2022).

Key extraction methods include:

• CLS-token embedding: The hidden state corresponding to the [CLS] special token (position 0) is used as a fixed-length representation for the entire snippet (Zhao et al., 2023, Feng et al., 2020).
• Token-level mean-pooling: The mean of all token-wise output embeddings (excluding padding/special tokens) produces a 768-dimensional vector encompassing broader contextual and semantic information (Chaturvedi et al., 7 Mar 2025, Zhao et al., 2023).
• Raw sequence extraction: For downstream sequence models, the complete token-wise last hidden state matrix is retained (shape: $L \times 768$) (Farasat et al., 16 Sep 2025).

2. Pooling Strategies and Semantic Coverage

Empirical research demonstrates that mean-pooling over code tokens systematically outperforms using the [CLS] embedding. Specifically, mean-pooling yields substantial improvements on Accuracy, F1, and MCC across code vulnerability detection, defect prediction, and clone detection benchmarks, with statistical significance (Wilcoxon p-values: $p<0.001$) (Zhao et al., 2023). For example, in Code Vulnerability Detection on CWE119, token-mean vectors improve accuracy by $+56.75\%$ and MCC by $+147.08\%$. The richer representation arises because mean-pooling aggregates fine-grained, identifier-level features, overcoming the underrepresentation of critical code signals in the [CLS] vector, which was primarily optimized for natural language sentence-level summarization.

Recommended practices:

1. Use token-level mean-pooling for semantic tasks.
2. Embed NL and PL sequences independently and then fuse, for mixed-modal applications (Zhao et al., 2023).
3. Evaluate embeddings on multiple metrics to ensure robustness and significance.

3. Surface Feature Dependence and Structural Limitations

CodeBERT embeddings predominantly encode surface-level features, especially identifier names. Controlled anonymization experiments reveal catastrophic declines in task accuracy with systematic removal of variable, method, and invocation names. In NL-to-code search (“top-1 accuracy”), removal of all three categories in Java drops performance from $70.36\%$ to ~$17\%$ (Zhang et al., 2023). Clone detection accuracy similarly plummets. This demonstrates that CodeBERT models do not robustly learn control-flow, data-flow, or algorithmic structures independent of naming. Both random and semantically misleading renaming produce similar degradations, indicating a lack of deep logical abstraction (Zhang et al., 2023).

Authors highlight the need for future objectives that mask basic blocks or data-flow edges and the integration of explicit program analysis structures (ASTs, CFGs) beyond shallow edge-prediction (Zhang et al., 2023). Naively applying CodeBERT embedding models to code with obfuscated, autogenerated, or poor naming conventions is thus ineffective for semantic tasks.

4. Adapter-Based Fine-Tuning: LoRA for CodeBERT

LoRA (Low-Rank Adaptation) adapters inject a trainable, low-rank update $\Delta W = A B$ into frozen attention projection weights $W_0$ of CodeBERT (Chaturvedi et al., 7 Mar 2025). Insertions apply to Query/Value projection matrices in all 12 layers, with key dimensions:

• $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times k}$, $r \ll d, k$.
• Scaling factor $\alpha$ stabilizes training for small $r$ via $\Delta W = \alpha/r A B$.
• Total adapter parameters are <2% of model size ($\sim$2M params for $r=32$).

Empirical results for LoRACode on large-scale retrieval (XLCoST, CodeSearchNet):

• MRR increases by up to 780% for Python Code2Code search ($5.48\% \rightarrow 48.27\%$ MRR) (Chaturvedi et al., 7 Mar 2025).
• Task-specific and language-specific adapters offer substantial gains over multilingual, monolithic adapters, e.g., $+86.7\%$ MRR in Python Text2Code.
• Corpora with larger language subset size yield greater improvements; smaller subsets yield modest gains.

LoRA enables rapid fine-tuning (2M samples, 25 min, 2$\times$H100 GPU), modular adaptation, and efficient deployment, supporting real-time code search at $>$90% throughput of the original model.

5. Practical Applications and Classifier Integration

CodeBERT embeddings serve as features for numerous downstream models:

• Vulnerability Detection: Embeddings (sequence of 768D token vectors) are streamed into BiLSTMs (3$\times$50 units, p=0.2 dropout) or CNNs (Conv1D $\rightarrow$ ReLU $\rightarrow$ MaxPool, filter sizes 32–256) (Farasat et al., 16 Sep 2025). CNN+CodeBERT achieves high precision ($\sim$91.6%) and recall ($\sim$86.9%), while BiLSTM+CodeBERT is less effective than Word2Vec+BiLSTM.
• Comment Generation: Fine-tuned Bash encoder (CodeBERT) produces mean-pooled vectors, normalized and fused with retrieved neighbors, generating comments via a six-layer Transformer decoder (Yu et al., 2022).
• Naturalness Assessment: CodeBERT-nt computes embedding similarity (cosine of flattened token vectors) between original and MLM-predicted code to rank buggy lines, aggregating confidence, similarity, and exact-match metrics (Khanfir et al., 2022).

For feature-based tasks (e.g., defect detection), using untouched sequence embeddings with downstream deep sequence models can outperform classic pooled approaches (Farasat et al., 16 Sep 2025). However, in specific cases, classical embeddings (Word2Vec) paired with sequence models remain competitive, suggesting domain adaptation and embedding-classifier interaction are critical.

6. Extraction, Usage, and Best Practices

Extraction is supported via HuggingFace Transformers API (Feng et al., 2020). For a code snippet (max length 512):

• Tokenize via pretrained CodeBERT tokenizer, producing [CLS], subtokens, [SEP].
• Encode with CodeBERT, returning last_hidden_state ($L \times 768$).
• Recommended to mean-pool all code tokens for fixed-size embeddings unless downstream models require full sequences (Zhao et al., 2023, Farasat et al., 16 Sep 2025).
• Embeddings are applicable to similarity search, clustering, classification, and as input for retrieval or generative decoders.

For adaptation (LoRA), attach adapters to Query/Value layers (r=16–64); fine-tune with contrastive objectives on large code corpora (Chaturvedi et al., 7 Mar 2025). Language and task specialization is essential for optimal retrieval and classification gains.

7. Limitations, Trends, and Research Directions

CodeBERT embeddings, although rich in contextual token information, do not intrinsically encode deeper code logic, limiting robustness in the face of naming variation or obfuscation (Zhang et al., 2023). LoRA adapters significantly ameliorate retrieval performance with minimal parameter cost and compute time, cementing parameter-efficient adaptation as a viable strategy for code search and related tasks (Chaturvedi et al., 7 Mar 2025). Ongoing directions include integrating data-flow or AST encoders, developing multi-task adapters generalizing across application domains, and designing pre-training objectives promoting structural and logical representation.

A plausible implication is that for code intelligence tasks requiring semantic generalization beyond surface features, architectural innovation and targeted objectives are imperative. Future models should jointly optimize for syntactic, semantic, and functional cues and leverage advanced adaptation schemes to maximize generalizability and efficiency.