Stay Hungry, Stay Foolish: On the Extended Reading Articles Generation with LLMs

Abstract: The process of creating educational materials is both time-consuming and demanding for educators. This research explores the potential of LLMs to streamline this task by automating the generation of extended reading materials and relevant course suggestions. Using the TED-Ed Dig Deeper sections as an initial exploration, we investigate how supplementary articles can be enriched with contextual knowledge and connected to additional learning resources. Our method begins by generating extended articles from video transcripts, leveraging LLMs to include historical insights, cultural examples, and illustrative anecdotes. A recommendation system employing semantic similarity ranking identifies related courses, followed by an LLM-based refinement process to enhance relevance. The final articles are tailored to seamlessly integrate these recommendations, ensuring they remain cohesive and informative. Experimental evaluations demonstrate that our model produces high-quality content and accurate course suggestions, assessed through metrics such as Hit Rate, semantic similarity, and coherence. Our experimental analysis highlight the nuanced differences between the generated and existing materials, underscoring the model's capacity to offer more engaging and accessible learning experiences. This study showcases how LLMs can bridge the gap between core content and supplementary learning, providing students with additional recommended resources while also assisting teachers in designing educational materials.

• The paper proposes an automated LLM-driven framework to generate extended reading articles from video transcripts, streamlining content creation.
• It employs a multi-stage process that drafts a preliminary article, ranks TED-Ed lessons using semantic similarity, and refines the text to integrate recommendations.
• Evaluation shows Llama-3.1-405b outperforms alternatives, achieving a balance of relevance, coherence, and accurate supplemental course suggestions.

This research explores using LLMs to automatically generate extended reading materials and suggest relevant supplementary courses, aiming to assist educators and enhance student learning experiences. The study uses TED-Ed lessons, specifically their video transcripts and "Dig Deeper" sections, as a case study.

Core Problem Addressed

Creating comprehensive and engaging educational materials, including supplementary readings and resource recommendations, is a time-intensive task for educators. This paper proposes an LLM-based system to automate parts of this process.

Methodology and Implementation

The proposed system operates in three stages:

1. Initial Article Generation (Stage 1):
   • An LLM (termed "Dig Deeper Generator") takes a video transcript as input.
   • It generates an initial draft of an extended reading article ("Dig Deeper").
   • The LLM is prompted to enrich the article with content types commonly found in TED-Ed's Dig Deeper sections, such as historical facts, relevant dates/events, terminology explanations, cultural context, examples, case studies, or anecdotes.
2. Relevant Lesson Recommendation (Stage 2):
   • The generated article from Stage 1 is compared against a database of 2,930 TED-Ed lessons using a sentence transformer model to calculate semantic similarity scores.
   • The top 100 most similar lessons are selected as candidates.
   • These candidates, along with the generated article, are fed into an LLM-based recommendation ranking model. This model evaluates the relationship based on:
     • Presence of related keywords from the article in the lesson.
     • Overall relevance of the lesson to the article's topic.
     • Contextual alignment of keywords between the article and the lesson.
   • Based on this evaluation, the system selects the most relevant lessons to recommend.
3. Final Article Refinement (Stage 3):
   • The system identifies the locations of keywords within the initial article that justify the selection of the recommended lessons.
   • The initial article is rewritten by another LLM ("Final Dig Deeper Generator").
   • This rewriting process integrates the recommended lessons and associated keywords more seamlessly, aiming to enhance the article's connection to the recommendations while maintaining coherence and depth relative to the original transcript's topic.

The overall framework can be visualized as:

graph LR A[Video Transcript] --> B(Stage 1: LLM Dig Deeper Generator); B --> C{Initial Dig Deeper Article}; C --> D(Stage 2: Recommendation); D -- Top N Lessons & Keywords --> E(Stage 3: LLM Final Dig Deeper Generator); E --> F[Final Dig Deeper Article with Recommendations]; subgraph Stage 2: Recommendation direction LR G[Sentence Transformer] --> H{Similarity Scoring}; I[LLM Ranker] --> J{Lesson Selection}; C --> G; K[TED-Ed Lesson Database] --> G; C --> I; H -- Top 100 Candidates --> I; end style F fill:#ccf,stroke:#333,stroke-width:2px

Data and Evaluation

• Dataset: Transcripts, Dig Deeper articles, and recommended links from TED-Ed lessons. Only lessons recommending other on-site lessons were included. Transcripts were summarized to a uniform length before processing.
• Models Tested: Llama-3.1-405b (via SambaNova API) and Gemma-2-27b (run locally on an NVIDIA RTX 4090).
• Metrics:
  • Hit Rate: Measures how often the system's recommended lessons match the original TED-Ed recommendations.
  • Relevance: Assessed using BERTScore, BM25, and Cosine Similarity between the generated article and the original transcript/Dig Deeper content.
  • Coherence: Evaluated using an LLM to score the structural quality and readability of the generated articles on a scale of 1-10.

Key Findings

• Model Performance: Llama-3.1-405b generally outperformed Gemma-2-27b across most metrics, achieving higher scores for Hit Rate (0.320), BERTScore (0.642), BM25 (2.923), Cosine Similarity (0.476), and Coherence (8.469).
• Ablation Studies:
  • Removing the initial article generation (Stage 1) and recommending directly from the transcript significantly increased the Hit Rate (0.515) but slightly reduced coherence. This suggests the LLM's exploratory generation in Stage 1 introduces diversity that lowers the direct match rate but potentially improves article structure.
  • Removing the final refinement (Stage 3) yielded higher relevance scores (BERTScore, BM25, Cosine Similarity) but lower coherence, indicating Stage 3 successfully integrates recommendations smoothly, albeit sometimes at the cost of direct topical relevance.
• Content Analysis: The study categorized existing TED-Ed Dig Deeper sections:
  • Category 1 (Links Only): Low coherence scores.
  • Category 2 (Text Only): High relevance and coherence, but lacks recommendation links.
  • Category 3 (Text with Links): Target style; balanced but slightly lower coherence due to topic shifts for recommendations.
  • The proposed system ("Ours") achieved scores comparable to Category 2 in relevance and coherence, demonstrating its ability to generate well-structured, relevant articles that also incorporate recommendations.

Practical Implications

• Educator Tool: This system provides a practical framework for educators to automatically generate supplementary reading materials based on core content (like a lecture transcript or video). It also suggests relevant additional resources, saving significant preparation time.
• Student Resource: Learners can benefit from automatically generated extended readings that provide deeper context (history, examples) and guide them towards related lessons for self-directed learning.
• Content Enrichment: The method shows how LLMs can bridge core educational content with supplementary learning by enriching articles with contextual details and integrating recommendations seamlessly.
• Implementation Considerations: The choice of LLM impacts performance. The trade-off between recommendation accuracy (Hit Rate) and article coherence/relevance needs consideration, potentially adjustable through prompting or stage weighting. The system relies on a database of potential courses/lessons for the recommendation stage.

This work demonstrates a viable approach using LLMs to automate the creation of enriched educational content, specifically extended reading articles coupled with relevant course recommendations, based on the TED-Ed model (2504.15013).