LingBot-VA: Multilingual Virtual Assistant

• LingBot-VA is an integrated multilingual virtual assistant that enables robust cross-lingual natural language understanding and adaptive speech synthesis for diverse human-robot interaction scenarios.
• It leverages instruction-tuned sequence-to-sequence models and dynamic fusion of multilingual inputs to excel in intent classification, slot tagging, and vision-language navigation tasks.
• The system demonstrates sample-efficient transfer in zero/few-shot settings, achieving competitive navigation and dialog performance while adapting to varied environmental and speech clarity requirements.

LingBot-VA is an integrated multilingual virtual assistant system, designed for language-aware human-robot interaction (HRI) and robust cross-lingual dialog understanding, leveraging recent advances in neural natural language understanding (NLU), text-to-speech (TTS), and adaptive prosody. Its architecture encompasses intent classification and slot tagging (IC+ST) via instruction-tuned sequence-to-sequence models, highly expressive, resource-efficient speech synthesis, and dynamic environmental adaptation, targeting applications such as L2 (second language) education, virtual agents, and situated dialog in diverse physical settings.

1. Cross-Lingual NLU: Task Formulation and Data

LingBot-VA’s core requirement is robust cross-lingual understanding, enabling operation in multiple spoken and written languages, including scenarios with little or no annotated data in target languages. The relevant task constructs are:

• Intent Classification and Slot Tagging (IC+ST): The system extracts user intent and slot arguments (e.g., "GetWeather", location=Paris) from utterances across languages.
• Vision-Language Navigation (VLN): For grounded HRI tasks, LingBot-VA uses the Bilingual Room-to-Room (XL-R2R/BL-R2R) dataset: 5,798 navigation paths with paired English and Mandarin natural language instructions. Each path is annotated by three speakers per language, resulting in 17,394 instructions per language. The English:Chinese vocabulary ratio (1,583:1,134) reflects Chinese instructions’ mean brevity (33 tokens vs. 48 English), different POS distributions (Chinese: 32.9% nouns, 29.0% verbs; English: 24.3%, 13.7%).
• Zero-Shot Cross-Lingual Setting: In structural evaluation, the system is trained only on English data and tested on human-written Chinese instructions (ε=0), relying solely on multilingual pretraining and MT for transfer (Yan et al., 2019).

2. Multilingual NLU Model Architecture

LingBot-VA leverages cutting-edge neural architectures for NLU in both IC+ST and VLN:

• IC+ST Generation: An instruction-fine-tuned sequence-to-sequence Transformer (AlexaTM 5B, ~5B params: 29 encoder and 24 decoder layers, 2560 hidden units, 32 heads), pre-trained on mC4 (12 languages) and Wikipedia (Rosenbaum et al., 2022). Generation is guided by structured prompts encoding target language, intent, slots to be copied or sampled, slot labels, and in-intent exemplars.
• Cross-Lingual VLN: Input encoding by XLM-RoBERTa-base (12 transformer layers, hidden size 768, multilingual MLM pretraining). Visual observations use ResNet-152 feature extractors. The action policy is decoded by an LSTM-attention network. A Cross-Lingual Language Instructor (XLI) dynamically fuses decision states from human-written and MT-translated instructions. The fusion at step $t$ is accomplished as:

$$h_t^* = Q_t \cdot h_t^{en} + (1-Q_t) \cdot h_t^{zh}, \quad Q_t = \mathrm{softmax}(U[h_t^{en}; h_t^{zh}])$$

where $Q_t$ encodes the mix between English and Chinese evidence.

No additional contrastive or unsupervised alignment loss is employed beyond the cross-entropy navigation loss $L_{nav}$.

3. Synthetic Multilingual Data Generation

To rapidly extend LingBot-VA to new intents, slots, and languages—critical in production environments—the LINGUIST method is adopted (Rosenbaum et al., 2022):

• Prompt Structure: Each prompt contains language (<language>), intent (<intent>), slots to copy/sample (<include>), slot label-index mapping (<labels>), and up to 10 in-intent examples (<examples>).
• Controlled Generation: Slot values in prompts may be force-copied or wild-carded ("[k *]") to let the model sample new values. Label name dropout (20%) regularizes model reliance.
• Decoding and Filtering: Outputs are generated using top-k sampling (k=50, T=0.3), then filtered for structural validity, forced slot inclusion, and intent match (optionally using an English-only IC model for cross-lingual outputs).
• Zero-Shot Transfer: In the absence of annotated target language examples, slot values are automatically machine-translated in the prompt, and generation proceeds in the target language. Cross-lingual intent and slot tagging models are then retrained using the synthetic utterances.

4. Robustness Across Languages and Data Regimes

LingBot-VA achieves competitive performance in low- and zero-resource settings for both navigation and dialog tasks:

• VLN Zero-Shot (ε=0): In the XL-R2R val-seen split for Chinese instructions, the XLI model obtains Navigation Error (NE) = 5.12 m, Success Rate (SR) = 48.4%, and SPL = 41.9%—approaching the upper bound with full Chinese supervision (SR = 50.1%, SPL = 43.2%). Compared to naïve MT-only transfer, XLI closes 80–90% of the gap (Yan et al., 2019).
• VLN Few-Shot Gains: With only 10–20% gold Chinese data, LingBot-VA’s XLI surpasses models trained on 100% Chinese, indicating highly sample-efficient transfer. SPL on val-unseen rises from 23.4 (ε=0) to 38.3 (ε=100%) for XLI, far outpacing baselines.
• IC+ST Few/Zero-Shot: On SNIPS (10-shot novel intent), LINGUIST improves local intent recall from 90.0 to 92.0 and slot F1 from 79.8 to 82.3. For mATIS++ zero-shot transfer (6 target languages), slot F1 increases by +4.14 points over the MT+soft-align baseline; intent accuracy parity is maintained.

5. Speech Synthesis: Expressivity, Adaptation, and L2 Clarity

The TTS subsystem is a lightweight, expressive, adaptive Matcha-TTS variant, optimized for HRI and L2 contexts (Tuttösí, 18 Jun 2025):

• Architecture: Matcha-TTS (20.9M params, 78MB checkpoint) uses Optimal-Transport Conditional Flow Matching. Supports real-time CPU-only inference (RTF ≈ 0.3 on Jetson AGX Orin; <1 on common CPUs).
• Emoji-Driven Expressivity: Eleven emoji styles (“😊,” “😡,” etc.) are encoded by appending special tokens to the input; the system learns prosody bases (pitch, energy, duration) per style via embeddings $e_n$. At inference, prosodic parameters are shifted according to learned gains $(\alpha_n, \beta_n, \gamma_n)$.
• Environmental Adaptation: SNR and RMS/zero-crossing are measured via on-board microphones. Prosody adjusts with pitch shift $$\Delta F_0 = k_p\cdot \mathrm{clamp}(\mathrm{SNR}-\theta_{\mathrm{snr}}, 0, \max)$$ and rate change $$r_{\mathrm{rate}}=1 + k_r \cdot \mathrm{clamp}(\mathrm{SNR}-\theta_{\mathrm{snr}},0, \max)$$. Smoothing prevents abrupt changes.
• L2 Clarity Mode: Using a lexicon or grapheme-to-phoneme, tense vowels {i, e, u, o, ɑ, …} are elongated by factor $\alpha=1.6$ for stressed syllables: $$d_{\mathrm{tense}}(i) = \alpha \cdot d_{\mathrm{base}}(i)$$. This reduces WER for L2 listeners on minimal pairs by >50% (e.g., WER_tense: 60%→29%).

6. Implementation and Practical Deployment

A typical LingBot-VA deployment follows:

• Data Pipeline: ASR → LLM-based IC/ST and emotion detection → LINGUIST prompt generation → TTS with emoji/environment/L2-clarity tags.
• Hardware: ARMv8 CPU ([email protected]), 512MB RAM suffices. No GPU required for real-time operation; optional microphone array for adaptive feedback.
• Quality Control: Auto-filtering of bracket and intent mismatches; periodic WER and MOS checks; human-in-the-loop spot-checks of 1–2% outputs.
• Scalability: AlexaTM 5B generation achieves ≈10 utterances/second per GPU; 500K synthetic utterances per intent within hours. For IC+ST fine-tuning, standard 16GB GPUs suffice; TTS operates at batch size 1 with ≤300ms latency per sentence. ONNX-runtime or TorchScript can be used for optimization.

7. Limitations and Prospects

LingBot-VA’s principal limitations are tied to data regimes and transfer coverage:

• For VLN, only English↔Chinese transfer is evaluated; extension to many-to-many transfer remains an open quantitative challenge (Yan et al., 2019).
• No explicit cross-lingual contrastive/grounding loss is employed beyond $L_{nav}$. Introduction of stronger multilingual alignment losses could further improve zero-shot transfer.
• TTS L2 clarity mode targets minimal tense/lax contrasts in English; generalized mechanisms for L2 clarity across additional phonological phenomena or languages require further research (Tuttösí, 18 Jun 2025).
• All evaluations remain in simulation or controlled environments; generalization to real-world, noisy, and variable HRI deployments will require extensive empirical validation.

A plausible implication is that instruction-tuning for both NLU and TTS, coupled with dynamic adaptation to user and environmental context, positions LingBot-VA as a reference architecture for next-generation cross-lingual dialog assistants.