SciConNav: Knowledge navigation through contextual learning of extensive scientific research trajectories

Abstract: New knowledge builds upon existing foundations, which means an interdependent relationship exists between knowledge, manifested in the historical development of the scientific system for hundreds of years. By leveraging natural language processing techniques, this study introduces the Scientific Concept Navigator (SciConNav), an embedding-based navigation model to infer the "knowledge pathway" from the research trajectories of millions of scholars. We validate that the learned representations effectively delineate disciplinary boundaries and capture the intricate relationships between diverse concepts. The utility of the inferred navigation space is showcased through multiple applications. Firstly, we demonstrated the multi-step analogy inferences within the knowledge space and the interconnectivity between concepts in different disciplines. Secondly, we formulated the attribute dimensions of knowledge across domains, observing the distributional shifts in the arrangement of 19 disciplines along these conceptual dimensions, including "Theoretical" to "Applied", and "Chemical" to "Biomedical', highlighting the evolution of functional attributes within knowledge domains. Lastly, by analyzing the high-dimensional knowledge network structure, we found that knowledge connects with shorter global pathways, and interdisciplinary knowledge plays a critical role in the accessibility of the global knowledge network. Our framework offers a novel approach to mining knowledge inheritance pathways in extensive scientific literature, which is of great significance for understanding scientific progression patterns, tailoring scientific learning trajectories, and accelerating scientific progress.