Human-In-The-Loop Workflow for Neuro- Symbolic Scholarly Knowledge Organization

Abstract: As the volume of scientific literature continues to grow, efficient knowledge organization is an increasingly challenging task. Traditional structuring of scientific content is time-consuming and requires significant domain expertise, increasing the need for tool support. Our goal is to create a Human-in-the-Loop (HITL) workflow that supports researchers in creating and structuring scientific knowledge, leveraging neural models and knowledge graphs, exemplified using the Open Research Knowledge Graph (ORKG). The workflow aims to automate key steps, including data extraction and knowledge structuring, while keeping user oversight through human validation. We developed a modular framework implementing the workflow and evaluated it along the Quality Improvement Paradigm (QIP) with participants from the ORKG user community. The evaluation indicated that the framework is highly usable and provides practical support. It significantly reduces the time and effort required to transition from a research interest to literature-based answers by streamlining the import of information into a knowledge graph. Participants evaluated the framework with an average System Usability Scale (SUS) score of 84.17, an A+ -- the highest achievable rating. They also reported that it improved their time spent, previously between 4 hours and two weeks, down to an average of 24:40 minutes. The tool streamlines the creation of scientific corpora and extraction of structured knowledge for KG integration by leveraging LLMs and user-defined models, significantly accelerating the review process. However, human validation remains essential throughout the extraction process, and future work is needed to improve extraction accuracy and entity linking to existing knowledge resources.