Non-Functional Requirements for Machine Learning: An Exploration of System Scope and Interest

Abstract: Systems that rely on Machine Learning (ML systems) have differing demands on system quality compared to traditional systems. Such quality demands, known as non-functional requirements (NFRs), may differ in their definition, scope, and importance from NFRs for traditional systems. Despite the importance of NFRs for ML systems, our understanding of their definitions and scope -- and of the extent of existing research in each NFR -- is lacking compared to our understanding in traditional domains. Building on an investigation into importance and treatment of ML system NFRs in industry, we make three contributions towards narrowing this gap: (1) we present clusters of ML system NFRs based on shared characteristics, (2) we use Scopus search results -- as well as inter-coder reliability on a sample of NFRs -- to estimate the number of relevant studies on a subset of the NFRs, and (3), we use our initial reading of titles and abstracts in each sample to define the scope of NFRs over parts of the system (e.g., training data, ML model, or other system elements). These initial findings form the groundwork for future research in this emerging domain.