TransfoREM: Transformer Models Across Domains

• TransfoREM is a dual-domain Transformer approach that targets protoform reconstruction in linguistic studies and 3D radio environment mapping in wireless communications.
• It adapts both encoder–decoder and encoder-only models with domain-specific innovations in data representation, training objectives, and architectural configurations.
• The system demonstrates significant improvements over RNN and traditional methods, highlighting the benefits of hybrid learning paradigms and tailored preprocessing techniques.

TransfoREM refers to distinct Transformer-based machine learning architectures targeting two primary scientific problems: (1) protoform reconstruction in historical linguistics, and (2) 3D radio environment mapping for wireless communications. Despite operating in unrelated domains, both systems utilize Transformer models to surpass previous state-of-the-art, demonstrate hybrid or multi-source learning, and introduce domain-specific innovations in representation, training, and deployment (Kim et al., 2023, Reddy et al., 23 Jan 2026).

1. Transformer Architectures Across Domains

Both instantiations of TransfoREM adopt core principles of the Transformer framework [Vaswani et al., 2017], yet adapt its structure and I/O handling for their respective sequence prediction tasks.

Protoform Reconstruction (Kim et al., 2023):

• Implements a "base" encoder–decoder Transformer.
• Romance datasets: 3 encoder/3 decoder layers, model dim = 128, 8 heads, d_ff = 128, dropout ≈ 0.20.
• Chinese: 2 encoder/5 decoder layers, model dim = 128, 8 heads, d_ff = 647, dropout ≈ 0.17.
• Input concatenates individually position-encoded and language-embedded sequences from each daughter language; decoder outputs the protoform autoregressively.

3D Radio Environment Mapping (Reddy et al., 23 Jan 2026):

• Utilizes an encoder-only Transformer.
• Architecture: 6 encoder layers, model dim = 64, 8 attention heads, max sequence length = R_max (radial bins).
• No decoder; prediction is regression (received signal strength) at masked radial positions.

The following table summarizes the core architectures:

Domain	Model Type	Encoder Layers	Decoder Layers	Model Dimension	Heads	Output
Protoform Reconstruction	Encoder-Decoder	2–3	3–5	128	8	Sequence (protoform)
3D REM	Encoder-Only	6	0	64	8	Scalar (signal strength)

2. Data Representation and Preprocessing

Protoform Reconstruction

• Each cognate set comprises multiple token sequences, one per daughter language (phoneme or character tokens, with diacritics merged).
• Per-token language embeddings distinguish source languages.
• Per-daughter positional encoding applied independently; then all sequences concatenated to form encoder input.
• Training on:
  • Romance: 8,799 cognate sets, 5 daughter forms, 1 proto-Latin target (both IPA phonetic and orthographic).
  • Sinitic: 804 sets, 39 modern varieties + Middle Chinese, tokenized as phonetic segments (tone contours as single tokens).
• 70/10/20 train/val/test split.

3D REM

• Each query point $i$ (Cartesian: $[x_i, y_i, z_i]$) mapped to spherical $[\rho_i, \phi_i, \theta_i]$.
• Input feature $\Gamma_i$: $6 \times R_{\max}$ matrix combining log-distance bins, angles, and Cartesian coordinates along the radial direction.
• Target is received signal strength $\Omega_i$ (dBm) at the corresponding radius; masking applies during training to enforce regression at the radial bin of interest.
• Real-world UAV measurement data from the AERPAW campaign (filtered, ≈17,000 points/altitude) with explicit train/val/test spatial splits.

3. Training Objectives and Procedures

Protoform Reconstruction

• Objective: Minimize token-level cross-entropy loss over the protoform sequence,

$$L(\theta) = - \sum_{(X,Y)\in D} \sum_{t=1}^T \log p_\theta(y_t|y_{<t}, X)$$

• No auxiliary losses; teacher-forced cross-entropy.
• Optimizer: Adam with weight decay; learning rate linear warmup (Romance: 50 epochs, lr ≈ $1.3 \times 10^{-4}$; Chinese: 32 epochs, lr ≈ $7.5 \times 10^{-4}$), then decay.
• Early stopping: minimal validation phoneme edit distance.

3D REM

• Stage 1 (Model-based Pretraining): Predict FSPL + antenna gain synthetic signal strengths along radial sequences. Loss: Mean Squared Error,

$$\mathcal{L}_\mathrm{MSE} = \frac{1}{N R_{\max}} \sum_{i=1}^N \sum_{j=1}^{R_{\max}} \left(P^{(j)}_{b,i,\mathrm{dB}} - \hat{P}^{(j)}_{b,i,\mathrm{dB}}\right)^2$$

• Stage 2 (Data-driven Fine-tuning): Predict real UAV-measured $\Omega_i$, masking all but the observed radius. Loss: Smooth L1,

$$\mathcal{L}_{\mathrm{SmoothL1}}(x) = \begin{cases} 0.5 x^2, & |x|<1 \ |x| - 0.5, & |x|\geq 1 \end{cases},\qquad x = \Omega_i - \hat{\Omega}_i$$

• Optimizer: Adam; custom learning rate schedules per stage (linear warmup + sqrt decay and then step-decay).
• Batch size: 16 (stage 1), 4 (stage 2); epochs: 10 (stage 1), 100 (stage 2).

4. Evaluation Methodologies and Results

Protoform Reconstruction

• Metrics:
  • Phoneme/Character Edit Distance (PED/ED)
  • Normalized PED (NPED)
  • Accuracy (% exact sequence match)
  • Feature Error Rate (FER; using PanPhon articulatory vectors)
  • B-Cubed F-Score (BCFS; cluster-based)
• Results (averaged over 10 runs):
  • On Romance phonetic, Transformer: PED $0.9027 \pm 0.0194$, NPED $0.1146 \pm 0.0021$, Accuracy $53.16\% \pm 0.66$, FER $0.0378 \pm 0.0011$, BCFS $0.8421 \pm 0.0029$.
  • On Sinitic, Transformer: PED $0.9814 \pm 0.0437$, NPED $0.2204 \pm 0.0093$, Accuracy $39.50\% \pm 3.02$.
  • Consistently improved over tuned RNN baselines (8.5% PED, 4pp accuracy gains on Chinese; smaller, persistent gains on Romance).

3D REM

• Metrics:
  • Root Mean Squared Error (RMSE)
  • Mean Absolute Error (MAE)
  • R-squared ($R^2$)
• Results:
  • Stage 1 (FSPL pre-train): RMSE $7.49$ dB, MAE $6.20$ dB, $R^2 = 0.33$
  • Stage 2 (fine-tune on real): RMSE $4.57$ dB, MAE $3.13$ dB, $R^2 = 0.77$
  • Outperforms Kriging by $1.9$ dB median absolute error on held-out altitude splits (e.g., test at $90$ m, train at $50,70,110$ m); similar $1$–$2$ dB gains observed on other splits.
  • Matches or slightly betters TripleLayerML on RMSE/MAE at lower model complexity after comparable stages.

5. Integration of Domain Knowledge and Hybrid Paradigms

Both versions of TransfoREM encode domain-specific priors and leverage hybrid learning regimes.

• Protoform Reconstruction: Encodes intricate language relationships using approaches such as individual language embeddings, per-sequence positional encoding, and concatenation schemes to handle multi-source input.
• 3D REM: Hybridizes a deterministic channel model (Free-Space Path Loss + antenna pattern) with data-driven learning: physics-based pretraining imparts foundational propagation behavior, while empirical fine-tuning specializes to irregularities observed (e.g., shadowing, multipath).

This fusion is critical for enabling data-efficient generalization, especially where available training samples are spatially or temporally sparse.

6. Deployment, Applications, and Future Directions

Protoform Reconstruction

• Provides improved reconstructions of ancestral wordforms across both Romance and Sinitic datasets.
• Learned language embeddings, as shown by hierarchical clustering and Generalized Quartet Distance (GQD), recover phylogenetic relationships more accurately than previous RNN models (GQD = 0.4 vs. 0.8 for earlier baseline).
• Limitations: requires hundreds to thousands of annotated cognate-sets; potential scalability issues with increasing number of daughter languages.
• Future directions: knowledge transfer to under-attested families, unsupervised/weakly-supervised settings, deeper analysis of phonological/historical abstractions (Kim et al., 2023).

3D REM

• Designed for real-time integration at base stations:
  • Ingests ongoing coverage/KPI samples (including UAV data), updates 3D coverage maps.
  • Enables enhanced resource allocation (e.g., beamforming), interference anticipation, dynamic spectrum sharing (identifying vacant 3D "Radio Dynamic Zones").
• Delivers end-to-end interpolation with minimal inference overhead, outperforming Kriging and multi-stage ML pipelines, and can be retrained or updated incrementally in the field (Reddy et al., 23 Jan 2026).

7. Domain-Specific Significance and Broader Implications

TransfoREM exemplifies the extension of Transformer models into structured, domain-informed sequence prediction tasks outside NLP:

• In historical linguistics, it facilitates finer-grained and phylogenetically aware reconstructions, provides a framework for representation analysis, and challenges prevailing assumptions about data requirements for state-of-the-art neural models.
• In wireless communications, its hybrid paradigm sets a precedent for leveraging physics-informed ML pretraining and shows practical gains in network coverage mapping and UAV connectivity management.

A plausible implication is that radial-sequence abstractions (as introduced for REM) or input representation modularization (as for multi-language reconstruction) may inform sequence modeling strategies in analogous scientific domains. Both models are publicly documented and can serve as baselines for future empirical and methodological explorations in their respective fields.

References:

• "Transformed Protoform Reconstruction" (Kim et al., 2023)
• "TransfoREM: Transformer aided 3D Radio Environment Mapping" (Reddy et al., 23 Jan 2026)