Evontree: Ontology Rule-Guided Self-Evolution of Large Language Models

Abstract: LLMs have demonstrated exceptional capabilities across multiple domains by leveraging massive pre-training and curated fine-tuning data. However, in data-sensitive fields such as healthcare, the lack of high-quality, domain-specific training corpus hinders LLMs' adaptation for specialized applications. Meanwhile, domain experts have distilled domain wisdom into ontology rules, which formalize relationships among concepts and ensure the integrity of knowledge management repositories. Viewing LLMs as implicit repositories of human knowledge, we propose Evontree, a novel framework that leverages a small set of high-quality ontology rules to systematically extract, validate, and enhance domain knowledge within LLMs, without requiring extensive external datasets. Specifically, Evontree extracts domain ontology from raw models, detects inconsistencies using two core ontology rules, and reinforces the refined knowledge via self-distilled fine-tuning. Extensive experiments on medical QA benchmarks with Llama3-8B-Instruct and Med42-v2 demonstrate consistent outperformance over both unmodified models and leading supervised baselines, achieving up to a 3.7% improvement in accuracy. These results confirm the effectiveness, efficiency, and robustness of our approach for low-resource domain adaptation of LLMs.

• The paper demonstrates that minimal ontology rules can effectively extract and inject implicit domain knowledge in data-scarce environments.
• The methodology leverages subclass transitivity and synonym-subclass propagation to reliably validate hierarchical ontologies, achieving up to 3.7% improvement in medical QA tasks.
• The framework explores explicit, implicit, and mixed injection strategies, ensuring robust domain adaptation with minimal degradation in general performance and safety.

Evontree: Ontology Rule-Guided Self-Evolution of LLMs

Introduction

Evontree introduces a principled framework for domain adaptation of LLMs in data-scarce environments, leveraging a minimal set of high-quality ontology rules to extract, validate, and inject domain knowledge. The approach is motivated by the observation that LLMs encode substantial implicit knowledge through pre-training, yet often lack the fine-grained, structured relationships required for specialized domains such as medicine. By systematically mining and refining the model's internal ontology using only two core rules—subclass transitivity and synonym-subclass propagation—Evontree circumvents the need for large external corpora, addressing privacy and data scarcity constraints inherent to sensitive domains.

Figure 1: The overview of Evontree, illustrating the pipeline from ontology extraction to rule-driven validation and knowledge injection.

Methodology

Ontology Knowledge Extraction

Evontree begins by eliciting the model's implicit domain ontology via structured prompts, focusing on subclass and synonym relationships. For each root concept (e.g., "Cell", "Antibiotic"), the model generates a tree-structured ontology spanning at least three hierarchical layers. To mitigate hallucination, the framework introduces the $\mathtt{ConfirmValue}$ metric, a perplexity-based confidence score computed over multiple paraphrased prompts. Only triples consistently exceeding a model-specific threshold $\tau^*$ are considered "confirmed".

Rule-Driven Ontology Examination

Two ontology rules are central:

• R1 (Synonym-Subclass Propagation): If $x$ is a synonym of $y$ and $y$ is a subclass of $z$, then $x$ is a subclass of $z$.
• R2 (Subclass Transitivity): If $x$ is a subclass of $y$ and $y$ is a subclass of $z$, then $x$ is a subclass of $z$.

Reliable triples are selected by identifying closed triangles in the ontology graph, minimizing the risk of propagating erroneous knowledge. Extrapolated triples are generated via rule application and filtered by $\mathtt{ConfirmValue}$; those below threshold are labeled as "gap triples"—highly reliable but unfamiliar to the model.

Figure 2: Ontology path statistics and quality evaluation, demonstrating the accuracy and confidence distribution across triple types.

Knowledge Injection Strategies

Evontree explores three injection paradigms:

• Explicit Injection: Directly synthesizes QA pairs from gap triples and their derivation chains.
• Implicit Injection: Appends ontology chains as hints to concept-centric question templates, promoting more natural and diverse self-distilled training data.
• Mixed Injection: Combines both strategies.

Fine-tuning is performed using LoRA adapters, reinforcing the model's domain knowledge without external supervision.

Experimental Results

Medical QA Benchmarks

Evontree is evaluated on MedMCQA, MedQA, and PubMedQA using Llama3-8B-Instruct and Med42-v2. The implicit and mixed injection variants consistently outperform both raw models and supervised baselines, with up to 3.7% accuracy improvement over Med42-v2 and 1.1% over the best baseline. Explicit injection is less effective, corroborating findings from prior ontology-injection studies.

Generalization and Safety

On general capability benchmarks (MMLU, ARC, TriviaQA), Evontree variants exhibit negligible degradation, with the best variant showing only a 0.15% drop. Safety evaluation using AdvBench indicates no significant increase in unsafe outputs for implicit and mixed variants; explicit injection, however, can negatively impact safety, likely due to rigid template-based QA formats.

Ablation Studies

Ablation experiments confirm the necessity of reliable triple selection, gap triple filtering, and ontology-based injection. Removing any component results in substantial drops in both triple accuracy and downstream task performance.

Figure 3: Ablation Study of Med42-v2, quantifying the impact of each module on medical QA performance.

Hyperparameter Sensitivity

Threshold selection for gap triples is robust; all tested variants outperform the raw model, though optimal performance depends on balancing factual accuracy, prior familiarity, and injection set size.

Figure 4: Hyperparameter Analysis, showing the effect of varying the confirmation threshold on downstream performance.

Theoretical and Practical Implications

Evontree demonstrates that ontology rule-guided supervision is a viable alternative to data-centric domain adaptation, especially in privacy-constrained or low-resource settings. The framework's reliance on internal model knowledge and minimal external rules enables efficient, targeted evolution of LLMs, reducing the risk of catastrophic forgetting and knowledge conflicts. The approach is extensible to other domains with well-defined ontologies and can be integrated with existing PEFT methods for scalable deployment.

Future Directions

Potential avenues for further research include:

• Extension to multi-modal and cross-domain ontologies.
• Automated discovery of additional ontology rules for richer knowledge editing.
• Integration with continual learning and knowledge conflict resolution frameworks.
• Application to other professional domains (e.g., finance, law) with stringent data requirements.

Conclusion

Evontree provides a systematic, ontology rule-driven framework for self-evolution of LLMs in specialized domains, achieving robust performance gains without external annotated corpora. The paradigm prioritizes quality and consistency in knowledge supplementation, offering a practical solution for domain adaptation under data scarcity and privacy constraints. The results suggest that ontology rule-guided editing can serve as a foundation for future advances in efficient, reliable LLM specialization.