Development of a Retrieval-Augmented Generation Virtual Assistant for Enhanced Information Discovery at Rubin Observatory

Abstract: The NSF-DOE Vera C. Rubin Observatory will generate petabytes of data through the Legacy Survey of Space and Time (LSST) over the next decade, enabling discoveries across a broad range of astrophysical fields. Alongside these data products, Rubin maintains a large but heterogeneous collection of supporting documentation, including operational guides, technical notes, and scientific papers. Because this material is distributed across multiple platforms and formats, staff and scientists often struggle to efficiently locate accurate, up-to-date information. Many resources also reside on internal systems, limiting the ability of general-purpose LLMs to provide reliable answers to Rubin-specific questions. To address these challenges, we explore the use of Retrieval Augmented Generation (RAG) to improve information discovery. We present a prototype RAG-based virtual assistant that delivers context-aware, factual, conversational access to Rubin's vast and heterogenous documentation ecosystem. The system integrates material from multiple sources and enables semantic search through a conversational interface, using Weaviate for embeddings, LangChain for query orchestration, and an OpenAI GPT model as the LLM backend. By grounding responses in domain-specific knowledge, the assistant reduces hallucinations, improves accuracy, and demonstrates the potential of RAG to enhance access to distributed knowledge, streamline workflows, and support effective use of LSST data products.

• The paper introduces a RAG system that integrates state-of-the-art LLMs, vector storage, and dynamic retrieval to reduce hallucinations.
• It employs a modular architecture with OpenAI embeddings, Weaviate storage, and LangChain orchestration to enable scalable access to diverse documentation.
• Practical evaluations show improved knowledge validation and operational efficiency, guiding future enhancements in retrieval precision.

Retrieval-Augmented Generation Virtual Assistant for Rubin Observatory: System Overview, Challenges, and Future Directions

Introduction

The paper "Development of a Retrieval-Augmented Generation Virtual Assistant for Enhanced Information Discovery at Rubin Observatory" (2607.01659) introduces a domain-adapted Retrieval-Augmented Generation (RAG) virtual assistant, tailored for the Vera C. Rubin Observatory. The observatory’s Legacy Survey of Space and Time (LSST) generates unprecedented data volumes—expected to reach 500 PB of images and 15 PB of catalogs over a decade. Accompanying this scientific output is a highly fragmented and continually evolving corpus of technical, operational, and scientific documentation distributed across diverse platforms and formats, many of which are inaccessible to general-purpose LLMs due to network restrictions and proprietary data.

The RAG virtual assistant addresses acute challenges in scientific information discovery, supporting both observatory operations and the global LSST user community. By leveraging RAG rather than static model fine-tuning, the system mitigates LLM hallucinations, grounds answers in current, trusted documentation, and is engineered for maintainability as the knowledge base evolves. The prototype integrates multiple state-of-the-art components—OpenAI embeddings, Weaviate vector storage, LangChain orchestration, and a Streamlit conversational frontend—deployed within the Rubin Science Platform (RSP) production environment.

Motivation and Problem Scope

The observatory’s documentation ecosystem includes Confluence wikis, Jira tickets, the Rubin Community Forum, Slack archives, DocuShare repositories, software documentation in GitHub, and numerous PDFs. This heterogeneity impedes efficient knowledge retrieval and threatens decision-making with staleness, inconsistencies, or access limitations. Traditional search and legacy support models are inadequate, especially given the highly specialized, context-dependent vocabulary and the scale of the Rubin user community (>10,000 scientists).

General-purpose LLMs lack necessary domain knowledge, are vulnerable to hallucinations, and cannot access internal documents. RAG offers a principled solution by dynamically exploiting external, curated knowledge at query time, with generation grounded in recently indexed authoritative content.

System Architecture

The architecture is modular, enabling independent replacement of retrieval, augmentation, and LLM components. Document ingestion, embedding, vector storage, retrieval, and generation are orchestrated via LangChain, facilitating flexible experimentation.

Document Ingestion and Storage: Documents from diverse sources (Confluence, Jira, GitHub, Community Forum, DocuShare, Slack) are subject to format-aware loading, cleaning, and deduplication preprocessing. Chunking employs character-based splits with overlap to balance retrieval granularity and context coherence, though limitations of fixed-size chunking are acknowledged.

Embeddings and Vector Storage: Textual content is embedded using OpenAI’s text-embedding-3-large model; these vectors, together with metadata, are stored in Weaviate, supporting efficient similarity search and facilitating provenance-aware retrieval.

Retrieval-Augmented Generation Pipeline: At query time, user prompts are also embedded and matched via cosine similarity against indexed vectors. Top-ranked chunks and their metadata are combined with the query in an augmented prompt, which is fed to the generative LLM (OpenAI GPT for the prototype), yielding an extracted, context-grounded response output with transparent source citations.

Frontend and Deployment: An interactive Streamlit UI provides a conversational interface, filterable source domains, and real-time answer streaming. The system is containerized and deployed using Rubin’s Phalanx Kubernetes-based infrastructure, inheriting RSP’s authentication, monitoring, and resource control.

Figure 1: Architecture of the Rubin AI virtual assistant from document ingestion, embedding, and vector storage to retrieval and response generation within the RSP ecosystem.

Figure 2: Streamlit-based conversational frontend with source selection, live retrieval, and response streaming, integrated into the RSP user environment.

Data Pipeline and Indexing

The ingestion pipeline critically shapes downstream retrieval quality.

• Sources: Textual and semi-structured data from all observatory-relevant systems are ingested, with plans for automated synchronization with upstream content (GitHub webhooks, REST APIs).
• Preprocessing: Lossy PDF conversions, non-text diagrams, OCR for scanned materials, and deduplication across sources are noted as ongoing issues.
• Chunking Strategy: Character-based chunking presently results in suboptimal boundaries—breaking context or splitting code blocks—necessitating future transition to hierarchical, semantic, or structure-informed chunking (e.g., cell-wise for Jupyter, header-based for technical docs).

Retrieval and Generation Details

Two separate OpenAI models are utilized:

• Embeddings: text-embedding-3-large produces dense representations of both queries and document chunks.
• LLM: gpt-3.5-turbo (as a baseline, with modularity to replace) generates answers by conditioning on the augmented prompt containing the most relevant context snippets.

Weaviate’s hybrid search capabilities (dense + BM25 sparse keyword) are slated for production, improving result quality for identifiers and lexically precise queries where dense semantic similarity alone is insufficient.

Evaluation and Metrics

Standard LLM benchmarks are inadequate for RAG-specific assessment. A validation query set probes:

• LLM baseline knowledge,
• RAG-specific improvements on domain queries,
• Failure modes for ambiguous and edge-case questions.

Automated evaluation leverages the RAGAS framework (Es et al., 2023), combining metrics for faithfulness, answer relevancy, context precision, and recall—enabling granular diagnosis of pipeline weaknesses and guiding iterative tuning.

Qualitative assessments show non-trivial performance limitations, often traceable to chunking granularity, document coverage, and prompt template engineering. Current response quality is not production-ready, underscoring the need for systematic improvements and robust corpus management.

Key Numerical/Empirical Highlights

While the implementation stage precludes large-scale quantitative results, the assistant’s roll-out in real production infrastructure demonstrates technical feasibility. Response quality is explicitly evaluated on targeted validation queries with systematic failure diagnosis. Benchmarking against baseline LLM performance, particularly for schema-sensitive queries or those requiring up-to-date documentation, evidences that RAG substantially reduces plausible hallucinations (false API calls, column names, or outdated procedural advice).

Practical Implications and Theoretical Impact

Operational Efficiency: The assistant delivers unified, conversationally accessible retrieval across otherwise siloed sources, essential for time-critical operations and large-scale community support.

Knowledge Validation: By systematically surfacing documentation gaps and ambiguities, the tool drives higher documentation rigor and consistency—serving as an integral piece of the observatory’s knowledge infrastructure.

Scalability and Extensibility: Modular design facilitates scaling across new LLMs, embedding models, and evolving domain-specific corpora.

Reference for Other Scientific Domains: The design and lessons herein generalize to other complex, documentation-rich scientific facilities.

Future Directions and Research Frontiers

Research priorities focus on:

• Advanced Chunking: Adoption of semantic and hierarchical retrieval (child-parent models, cell-based for notebooks) to increase context retention and retrieval precision.
• Hybrid Retrieval: Full deployment of Weaviate’s hybrid search (dense + sparse), empirically optimizing recall for terminology-heavy and exact-match queries.
• Prompt Engineering: Systematic instruction fine-tuning for improved faithfulness, explicit uncertainty handling, and code-specific response formatting.
• Open-Source and Model Diversity: Replacement of proprietary LLMs and embeddings with high-quality open-source alternatives where feasible, benchmarking for cost-effectiveness, latency, and code reasoning fidelity.
• Agentic RAG: Transition to agentic paradigms where query reformulation, iterative reasoning, and real-time validation are possible—enabling stepwise decomposition and tool invocation during complex workflows.
• Automated Ingestion and Monitoring: Trigger-based, continuous synchronization with source platforms, with logging, audit trails, and re-ingestion upon content updates.

Conclusion

This work delineates the architecture and operationalization of a purpose-built RAG-based virtual assistant for the Rubin Observatory, providing a blueprint for LLM-enabled, domain-adapted information discovery in data-intensive science. The system’s practical impact is anticipated to increase in step with LSST data releases and documentation complexity. The assistant is positioned not as a generic chatbot, but as a precision tool for science enablement—reducing the gap between documentation and executable, validated scientific workflows in an evolving technical regime. The ongoing transition from isolated retrieval and generation to agentic reasoning and schema awareness remains a key research trajectory for the near future.