DevOps and Microservices in Scientific System development

Abstract: There is a gap in scientific information systems development concerning modern software engineering and scientific computing. Historically, software engineering methodologies have been perceived as an unwanted accidental complexity to computational scientists in their scientific systems development. More recent trends, like the end of Moore's law and the subsequent diversification of hardware platforms, combined with the increasing multidisciplinarity of science itself have exacerbated the problem because self-taught "end user developers" are not familiar with the disciplines needed to tackle this increased complexity. On a more positive note, agile programming methods have approached software development practices to the way scientific software is produced. In this work, we present the experience of a multi-year industry research project where agile methods, microservices and DevOps were applied. Our goal is to validate the hypothesis that the use of microservices would allow computational scientists to work in the more minimalistic prototype-oriented way that they prefer while the software engineering team would handle the integration. Hence, scientific multidisciplinary systems would gain in a twofold way: (i) Subject Matter Experts(SME) use their preferable tools to develop the specific scientific part of the system; (ii) software engineers provide the high quality software code for the system delivery.