NL–PL Probing: Bridging Code & Language

• NL–PL probing is a diagnostic technique that evaluates if neural models encode the cross-modal relationships between code and its natural language documentation.
• It employs masked token prediction tasks on paired code-documentation data to quantify token-level alignment and assess model transparency.
• Empirical results, especially with CodeBERT, indicate significantly improved NL and PL token prediction accuracy through bimodal pretraining and RTD objectives.

Natural Language–Programming Language (NL–PL) probing comprises methodologies and experimental paradigms for diagnosing and quantifying the knowledge bridging natural language (NL) and programming language (PL) representations in pretrained neural models. Originating in the context of bimodal representation learning (notably in works such as CodeBERT), NL–PL probing tasks evaluate a model’s ability to map and reason between code and its natural language documentation, thereby illuminating its cross-modal token comprehension, grounding, and linguistic-structural integration properties (Feng et al., 2020). Additionally, probing strategies are used to evaluate whether embeddings encode requisite linguistic or programmatic abstractions, which is crucial for model transparency, interpretability, and downstream transferability.

1. Definition and Conceptual Scope

NL–PL probing is defined as the application of carefully constructed diagnostic tasks—often cast as cloze or classification problems—whose aim is to determine whether specific NL–PL correspondences or abstractions are encoded in model representations. Unlike direct downstream evaluation (e.g., code search, summarization), probing isolates particular phenomena (token alignment, type inference, syntactic mapping) and prevents performance inflation due to unrelated confounds or memorization.

In CodeBERT, NL–PL probing refers both to (a) tasks where a masked token (in either code or documentation) must be predicted given the other modality as context, and (b) broader zero-shot setups where only the pretrained, unadapted model is evaluated, directly quantifying the strength of learned cross-modal ties (Feng et al., 2020).

2. Probing Methodologies and Protocols

2.1 Dataset Construction

Canonical NL–PL probing utilizes paired function-level code-documentation examples (e.g., from the CodeSearchNet benchmark). The probing dataset is constructed by:

• Masking a single token in code (c) or documentation (w).
• Assembling a candidate set of choices (correct solution plus distractors, curated or minimally disambiguated).
• Encoding the input as [CLS] w [[SEP](https://www.emergentmind.com/topics/semantic-entropy-production-sep-metric)] c₁...c_{i-1} [MASK] c_{i+1}...c_m [EOS] (code) or [CLS] w₁...w_{j-1} [MASK] w_{j+1}...w_n [SEP] c [EOS] (doc), reflecting bidirectional and prefix-based contexts.

2.2 Probing Tasks

Two main tasks are standard:

• PL probing: Predict the correct code token masked out of a snippet given natural language context. Two candidates are provided (correct, distractor).
• NL probing: Predict the correct NL word masked out of documentation, with four candidate words.

A variant uses only the preceding context for code, modeling code completion.

2.3 Zero-shot Evaluation

• The neural model parameters remain frozen.
• For each candidate token $t$, the score is computed via the MLM head,

$$s(t) = \log p_{\rm MLM}([\text{MASK}] = t \mid \text{context})$$

and the highest-scoring token is selected.

• No gradient-based adaptation occurs.

3. Loss Functions and Pretraining Objectives

NL–PL models such as CodeBERT are pretrained with dual objectives over concatenated NL–PL sequences:

• Masked Language Modeling (MLM):

$$\mathcal{L}_{\rm MLM}(\theta) = -\sum_{i\in m} \log p^{D_1}_\theta(x_i \mid x_{\setminus m})$$

where $m$ denotes masked positions over the joint vocabulary.

• Replaced Token Detection (RTD):

$$\mathcal{L}_{\rm RTD}(\theta) = -\sum_{i=1}^{|x|} \Big[ \delta(i)\,\log p^{D_2}_\theta(\text{real}\mid i) +(1- \delta(i))\,\log p^{D_2}_\theta(\text{fake}\mid i) \Big]$$

with $\delta(i) = 1$ if $x^{\rm corrupt}_i = x_i$, $0$ otherwise.

The combined loss is minimized:

$$\min_\theta\;[\mathcal{L}_{\rm MLM}(\theta) + \mathcal{L}_{\rm RTD}(\theta)]$$

4. Metrics and Experimental Results

The primary evaluation metric is accuracy:

$$\mathrm{Acc} = \frac{\#\,\text{correctly predicted examples}}{\#\,\text{all examples}}\times 100\%$$

Empirical results (“NL–PL Probing Zero-Shot Accuracy”):

Model	PL (2-way)	PL (prefix)	NL (4-way)
RoBERTa	62.45	52.24	61.21
Pre-train w/ code	74.11	56.71	65.19
CodeBERT (MLM)	85.66	59.12	74.53

Notably, CodeBERT’s bimodal and RTD-augmented pretraining confers substantially improved NL–PL token-level alignment relative to unimodal or code-only pretrained Transformers (Feng et al., 2020).

5. Interpretations, Limitations, and Best Practices

5.1 Interpretations

• CodeBERT’s gains in NL–PL probing are attributable to bimodal MLM (that structurally ties NL and code tokens) and RTD (sharpened discrimination via both unimodal and bimodal training signals).
• Bimodal representation learning induces richer contextualization, enhancing both intra- and cross-modal retrieval under cloze/probe settings.

5.2 Methodological Limitations

• Current probing is limited to token-classification; properties such as symbolic reasoning, type inference, or API-linking are not directly interrogated.
• Distractors are manually curated; open-vocabulary or adversarial candidate selection remains unexplored, which may underestimate model brittleness.
• Structural information such as abstract syntax trees is not explicitly leveraged, although its integration is a suggested direction for future work.

5.3 Recommendations

• Fine-grained, semantics-oriented probe construction (e.g., variable renaming, type prediction) would yield deeper insights into NL–PL transfer.
• Integrating program structure (ASTs) and training joint generators/discriminators may further enhance RTD efficacy.
• Probing should be systematically extended to higher-level semantic features and non-cloze classification settings.

6. Broader Context and Implications

NL–PL probing serves as an essential technique for demystifying what is intrinsically learned by bimodal and code-focused pretraining paradigms. Results from probing analyses not only benchmark cross-modal token understanding but also inform the design of architectures and training objectives, especially for high-precision applications in code search, code synthesis, and documentation generation. The observed improvements via bimodal pretraining and RTD motivate further research into hybrid, structure-aware objectives and fine-grained diagnostic suites tailored to both linguistic and programmatic abstractions (Feng et al., 2020). The approach illustrated by CodeBERT’s NL–PL probing sets a foundation for robust assessment and future methodological advances in cross-modal understanding.