Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology

Abstract: Performance metrics (error measures) are vital components of the evaluation frameworks in various fields. The intention of this study was to overview of a variety of performance metrics and approaches to their classification. The main goal of the study was to develop a typology that will help to improve our knowledge and understanding of metrics and facilitate their selection in machine learning regression, forecasting and prognostics. Based on the analysis of the structure of numerous performance metrics, we propose a framework of metrics which includes four (4) categories: primary metrics, extended metrics, composite metrics, and hybrid sets of metrics. The paper identified three (3) key components (dimensions) that determine the structure and properties of primary metrics: method of determining point distance, method of normalization, method of aggregation of point distances over a data set.

• The paper introduces a methodological framework that systematically categorizes over 40 error measures into primary, extended, composite, and hybrid sets using point distance, normalization, and aggregation methods.
• It addresses limitations in existing classifications by providing a multi-dimensional approach, ensuring accurate and robust performance evaluation for regression, forecasting, and prognostics tasks.
• The framework supports the development of new metrics and enhances pedagogical methods, offering actionable insights for both practical applications and further research in machine learning model assessment.

An Analytical Framework for Performance Metrics in Machine Learning: A Critical Evaluation

The rigorous evaluation of machine learning models is predicated upon the judicious selection and application of performance metrics. The paper "Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology" by Alexei Botchkarev contributes a methodological advancement by presenting a comprehensive framework and a detailed typology for performance metrics. This analytical framework is outlined with the objective of optimizing the selection and utilization of metrics in machine learning regression, forecasting, and prognostics tasks.

Framework and Typology of Performance Metrics

The paper proposes a typology designed to systematically classify over 40 commonly used primary metrics into four distinct categories: primary metrics, extended metrics, composite metrics, and hybrid sets of metrics. Three principal components define the structure and properties of primary metrics:

1. Point Distance Calculation (𝔻): The methodology for determining the discrepancy between actual and predicted values. Metrics utilize various methods such as error, absolute error, squared error, and logarithmic quotient error. These are foundational to all primary metrics and determine much of their theoretical and applied properties.
2. Normalization Methods (ℕ): Techniques applied to standardize point distances. Normalization varies from unitary normalization to more complex variations involving actual and predicted values. This step is critical in facilitating comparisons across datasets with differing scales.
3. Aggregation Methods (𝔾): Techniques for summarizing point distances across datasets. Common methods include arithmetic mean, median, geometric mean, and sum aggregation, each of which influences the sensitivity and robustness of the resultant metric, particularly in the presence of outliers.

Evaluation of Existing Classifications

The typology extends previous classifications by moving beyond a one-level structure and avoiding overlapping group assignments. It addresses deficiencies in existing classifications, such as those by Hyndman and others, by incorporating a multi-dimensional approach that explicates the internal structures of metrics. This allows a more granular analysis which can be particularly beneficial when diverging properties across absolute and squared error metrics are critical for specific applications.

Implications and Future Directions

By elucidating the structural components of performance metrics, this typology is a practical guide for selecting appropriate metrics based on specific research or business objectives. The sequential approach of determining point distance, applying normalization, and aggregating results streamlines metric selection and provides clarity on metric design and application.

Furthermore, the proposed generic metric formula and visualization chart serve as tools not only for assessment but also for the creation of novel metrics. This offers significant pedagogical value in academic settings, improving education around metric selection, and provides a scaffold for future research focused on comprehensive comparisons and empirical validations using platforms like R Studio.

In future explorations, extending this conceptual research to evaluate other categories of metrics - extended, composite, and hybrid sets - will further empower the selection process in complex machine learning tasks. Empirical studies examining the behavior of these metrics in diverse data environments and task-specific requirements can also provide deeper insights into their practical applicability and effectiveness.

The paper’s contribution lies in its systematic approach to metric classification, promoting a nuanced understanding of performance evaluation in machine learning and fostering a solid foundation for advancements in model assessment and metric innovation. This work is pivotal for both enhancing existing methodologies and fostering new developments in the evaluation of machine learning models.