Reproducible Research is more than Publishing Research Artefacts: A Systematic Analysis of Jupyter Notebooks from Research Articles

Abstract: With the advent of Open Science, researchers have started to publish their research artefacts (i. e., data, software, and other products of the investigations) in order to allow others to reproduce their investigations. While this publication is beneficial for science in general, it often lacks a comprehensive documentation and completeness with respect to the artefacts. This, in turn, prevents the successful reproduction of the analyses. Typical examples are missing scripts, incomplete datasets or specification of used software. Moreover, issues about licences often create legal concerns. This is true for the use of commercial software but also for the publication of research artefacts without proper sharing licence. As a result, the sole publication of research artefacts does not automatically result in reproducible research. To empirically confirm this, we have been systematically analysing research publications that also published their investigations as Jupyter notebooks. In this paper, we present preliminary results of this analysis for five publications. The results show, that the quality of the published research artefacts must be improved in order to assure reproducibility.