NotebookLM as a Socratic physics tutor: Design and preliminary observations of a RAG-based tool

Abstract: This study explores NotebookLM, a Google Gemini - powered AI platform that integrates Retrieval-Augmented Generation (RAG) as a Socratic tutor for physics education. In this implementation, NotebookLM was configured to support students in solving conceptually oriented physics problems through a guided, questioning-based dialogue. When deployed as a collaborative tutor, the system restricts student interaction to a chat-only interface, promoting controlled and guided engagement. By grounding its responses in teacher-provided source documents, the AI tutor helps mitigate one of the major shortcomings of standard LLMs - hallucinations - thereby ensuring more traceable and reliable answers. This work details the methodological design of the tutor, including the iterative development of a pedagogical "Training Manual", and presents preliminary qualitative observations from demonstrations with pre-service and in-service teachers. These observations highlight both the promising potential of the tool and key pedagogical challenges, such as managing user motivation. While limitations remain, this work offers a promising and replicable model for educators seeking to implement grounded AI tutors in their own teaching contexts.

• The paper demonstrates a RAG-based AI tool that uses curated instructional content to offer reliable and traceable physics tutoring.
• It employs a Socratic method to guide learners through physics problems while minimizing LLM hallucination risks.
• Preliminary observations reveal both the system's potential for deep conceptual engagement and challenges in maintaining student motivation.

NotebookLM as a Socratic Physics Tutor: Design and Implementation

Introduction

The paper, "NotebookLM as a Socratic Physics Tutor: Design and Preliminary Observations of a RAG-based Tool" (2504.09720), explores the use of the Retrieval-Augmented Generation (RAG) framework in physics education via NotebookLM—a Google Gemini-powered AI platform. The study investigates how NotebookLM can be configured to facilitate a Socratic approach to teaching physics, addressing a critical challenge of LLMs: the generation of unreliable information, known as hallucinations. By utilizing curated instructional content, this platform aims to provide traceable and verifiable educational guidance.

Design and Methodology

NotebookLM, designed around a RAG framework, leverages external, verified sources to enhance the generation of reliable and contextually relevant answers (Figure 1). The implementation discussed in the paper involved creating a detailed "Training Manual" to shape the AI's behavior, encouraging a Socratic dialogue that guides students through physics problem-solving processes without directly providing solutions.

Figure 1: Screenshot of the NotebookLM interface showing the three panels: Sources for storing and indexing diverse teaching materials with traceable citations; chat for dialogue; a Studio for automatically generating structured learning aids such as summaries, study guides, mind maps, and podcast-style audio summaries.

The tutor's capabilities are structured around two modes of operation: when curated solutions are available, it strictly adheres to provided instructional content to ensure responses are grounded and backed by authoritative sources. When solutions are not pre-loaded, the AI offers provisional guidance based on its own reasoning capabilities, although it acknowledges its potential inaccuracies to promote critical student evaluations.

Implementation Environment and Features

The deployment involved a configuration that restricts student interactions to a chat-only interface, facilitated by a premium NotebookLM Plus subscription (Figure 2). This setup allows educators to pre-load specific educational materials and behavioral guidelines into the tutor, ensuring a controlled and focused learning environment.

Figure 2: NotebookLM interface: (a) Sharing options configuration available to teachers with NotebookLM Plus, now including public link sharing that allows chat-only access for students without email requirements. (b) The student chat interface with a sample welcome message.

A crucial aspect discussed is the choice of file formats for embedding educational materials. The versatility of Google Docs is emphasized due to its superior compatibility with NotebookLM for accurately interpreting visual content required in physics problem-solving (Figure 3).

Figure 3: NotebookLM's graph interpretation across formats. (a) Velocity--time graph for the bouncing ball. (b) Native Google Doc: system correctly reads the graph and returns the requested numerical values. (c) PDF exported from Google Docs: the figure is not accessible; the model reports that the graph is not present in the source and cannot extract axes or numeric values.

Observations and Challenges

The qualitative observations from interactions with pre-service and in-service teachers indicate a positive reception toward the Socratic dialogue facilitated by NotebookLM. However, several challenges were identified, particularly regarding user motivation. In scenarios where the AI consistently refrained from providing direct answers, some students expressed frustration, emphasizing the need for an adaptive approach in scaffolding.

Illustrative dialogues exemplify the AI's strategies in cases with and without pre-configured guidance, highlighting the system’s attempts to engage learners through a structured questioning method. The pedagogical tension between fostering deep conceptual understanding and ensuring student motivation remains a focal area for future exploration.

Conclusions

The study presents a replicable model for integrating AI tutors into physics education using NotebookLM, showcasing its potential to support Socratic teaching methodologies through structured AI-human interaction. The capability to tailor the AI's responses with teacher-curated content offers educators a powerful tool for fostering active learning environments.

Future studies should address the challenges of balancing conceptual rigor with motivational strategies, possibly through the implementation of dynamic scaffolding protocols that adjust guidance based on real-time student needs. The continuous evolution of platforms like NotebookLM promises to expand these capabilities, making them more robust and adaptable to diverse educational contexts.