ENWAR: Multi-Modal LLM for 6G Networks

• ENWAR is a framework that integrates multi-modal sensory data with retrieval-augmented generation to enhance real-time wireless environment perception.
• It employs a multi-stage pipeline—including semantic indexing, FAISS retrieval, and top-p decoding—to significantly improve accuracy and contextual relevance over standard LLMs.
• The framework leverages off-the-shelf LLMs with optimized prompt engineering and fusion methods to deliver precise obstacle analysis and spatial reasoning in dynamic network settings.

ENWAR (ENvironment-aWARe Retrieval-Augmented Generation-empowered Multi-Modal LLM) is a framework designed for real-time wireless environment perception through deep integration of multi-modal sensory data and retrieval-augmented generation (RAG) techniques. Targeting the challenges faced by conventional LLMs in handling domain-specific and multi-modal data required for 6G and beyond networks, ENWAR enables precise situational awareness, spatial reasoning, obstacle analysis, and human-interpretable scene description by leveraging GPS, LiDAR, and vision signals (Nazar et al., 2024).

1. System Architecture

ENWAR operationalizes a multi-stage processing pipeline that unifies raw sensor data, semantic indexing, and autoregressive reasoning.

High-Level Pipeline (Textual Block Diagram):

• Multi-Modal Encoders: Parallel ingestion for GPS, LiDAR, and Camera inputs.
  • GPS: Converts raw trajectory and bearing into structured textual data (e.g., “Unit 1 at (37.77, –122.41), bearing 45° NE, 15m from Unit 2”).
  • Camera: Feeds frames into InstructBLIP (Vicuna-7b backbone) and produces natural language scene summaries.
  • LiDAR: Processes point clouds via SFA3D (ResNet-KFPN + BEV detection), yielding structured object-centric statements.
• Textual Concatenation: Modality-specific outputs are merged into a single description per scene.
• Chunking and Embedding: Scene text is partitioned (1,024 chars/chunk with 100 char overlap) and embedded using Alibaba-NLP gte-large-en-v1.5, yielding dense vectors in $\mathbb{R}^{d}$.
• FAISS Indexing: Embeddings are organized as a domain-specific knowledge base supporting efficient similarity retrieval.
• Retrieval Module: Semantic search and ranking identify the most relevant scene chunks via cosine similarity.
• Generation Module: An LLM (e.g., LLaMA3.1 or Mistral variants) combines prompt, retrieved context, and domain knowledge through top-$p$ sampling decoding.

2. Retrieval-Augmented Generation (RAG) Implementation

ENWAR’s RAG mechanism underpins context-sensitive reasoning by grounding LLM outputs in scene-relevant data.

• Embedding and Indexing: All scene chunks are stored as vectorized documents $\{\vec{d}_j\}$ in a FAISS index.
• Semantic Similarity Search: A query vector $\vec{q}$—the embedding of the synthesized prompt plus current sensory data—is matched to indexed documents. Relevance is scored via cosine similarity:

$$\mathrm{score}(q, d_j) = \frac{\vec{q}\cdot \vec{d}_j}{\|\vec{q}\|\|\vec{d}_j\|}$$

• Retrieval Ranking: Top $p$ percentile (typically $p = 95\%$) are shortlisted; additional ranking by chunk section headers.
• Context Injection: The highest-ranked $K$ chunks (above $95\text{th}$ percentile) populate the LLM prompt preamble as context.
• Prompt Structure Example:

[BEGIN CONTEXT] GPS: Unit1 at (37.77,-122.41), NE 45°, 15m from Unit2; LiDAR: Car at (3m,2m), Pedestrian at ( -1m,4m ); Vision: “Two cars waiting, cyclist on right”; [END CONTEXT] Task: From Unit1’s viewpoint, 1) summarize spatial layout; 2) identify obstacles to Unit2; 3) assess line-of-sight. Answer:

• Generation: The LLM deploys autoregressive decoding using top-$p$ sampling (e.g., $p=0.9$), yielding a multi-sentence human-readable response.

3. Multi-Modal Fusion and Reasoning Dynamics

ENWAR employs early fusion at the textual stage, transforming input modalities into a unified stream prior to embedding. This approach enables:

• Holistic scene understanding, as all physical and semantic cues are provided jointly to the LLM.
• Downstream transformer attention to focus over the concatenated prompt-context sequence, facilitating integrated spatial computations (e.g., distance calculations: “Unit 1 sees obstacle at 12 m”), obstacle identification, and line-of-sight inference.

No additional late-fusion neural layers are employed; instead, fusion is entrusted to the LLM’s multi-head attention mechanisms. The framework does not re-train its LLM backbones; adaptation only occurs at the level of embedding models (e.g., GTE, SFA3D) and through prompt engineering or retrieval parameter tuning. All reasoning complexity is absorbed within the LLM via context conditioning.

4. Model Adaptation: Prompting and LLM Utilization

ENWAR exclusively utilizes off-the-shelf LLMs (Mistral-7B, Mistral-8×7B, LLaMA3.1-8B, -70B, -405B) without in-framework gradient-based fine-tuning. Adaptation for domain specificity leverages:

• Instructional Prompt Templates: Carefully structured prompts define tasks, context, and response style, ensuring scene relevance and task orientation.
• RAG-Driven Contextualization: Scene context is supplied explicitly to steer the LLM’s outputs towards precise environmental analysis.
• Top-$p$ Decoding: Sampling hyperparameters are tuned (typically $p=0.9$) to balance fluency and faithfulness.

The system’s adaptation focus is on robust prompt template design and retrieval optimization, eschewing direct supervised LLM training.

5. Evaluation Methodologies and Empirical Results

Performance is rigorously evaluated using the RAGAS KPI suite, measuring semantic and factual alignment between generated and reference answers.

KPI	Vanilla LLaMA-8B	ENWAR
Relevancy (AR)	70.3%	81.2%
Correctness	54.3%	76.9%
Faithfulness	42.2%	68.6%

Key Performance Indicators:

1. Answer Relevancy (AR): Semantic cosine similarity between predicted ($\vec{E}_{p_i}$) and target ($\vec{E}_{t_i}$) embedding spaces.
2. Context Recall (CR): Fraction of ground-truth sentences correctly retrieved/used.
3. Correctness (Corr): Weighted combination ($\omega=0.25$) of embedding similarity and $F_1$ metric.
4. Faithfulness (Fth): Proportion of answer claims directly supported by retrieved context.

Aggregate Results (30 test scenes, tri-modal): $AR \approx 70\%$, $CR \approx 55\%$, $Corr \approx 80\%$, $Fth \approx 86\%$. Notably, ENWAR surpasses vanilla LLMs by 10–30 points across all KPIs (Nazar et al., 2024).

6. Modalities, Model Scaling, and Observed Trends

ENWAR’s efficacy varies with input modality configuration and LLM scale:

• Single Modality: Performance ranks as Camera $>$ LiDAR $>$ GPS.
• Dual Modality: Camera + LiDAR outperforms GPS + LiDAR.
• Tri-modal Fusion: Highest overall KPI scores are achieved with all three modalities.
• Model Size: Larger LLMs yield higher absolute KPI values, yet benefits plateau when normalized by parameters—a plausible implication is diminishing marginal returns beyond a certain LLM scale for this multi-modal task setup.

7. Application Domain and Significance

ENWAR is evaluated on the DeepSense6G dataset comprising GPS, LiDAR, and camera streams, targeting real-time, human-interpretable situational awareness in highly dynamic, multi-agent wireless environments. Its capacity for granular spatial reasoning, accurate obstacle identification, and line-of-sight assessment demonstrates the viability of RAG-empowered LLMs in mission-critical domains where multi-modal perception is essential. Results position ENWAR as a methodological advance over general-purpose LLM deployments, specifically for the network management, orchestration, and cognitive perception requirements anticipated in 6G and beyond wireless infrastructures (Nazar et al., 2024).