FlakeRanker: Automated Identification and Prioritization of Flaky Job Failure Categories

Abstract: This document presents the artifact associated with the ICSE SEIP 25 paper titled On the Diagnosis of Flaky Job Failures: Understanding and Prioritizing Failure Categories. The original paper identifies and analyzes 46 distinct categories of flaky job failures that developers encounter, using Recency (R), Frequency (F), and Monetary (M) measures. In addition, it uses an RFM clustering model to identify and prioritize the most wasteful and persistent. The original paper only discusses the rankings and evolution of the top 20 categories in the results. This artifact contains (1) the regex and scripts used to automate the labeling process for RQ1, (2) complete analysis results, including the ranking of all 46 categories by cost in RQ2 and the evolution of these categories over time in RQ3, and (3) the RFM dataset and scripts used to create the RFM clustering model for prioritization in RQ4. In addition, we engineered the labeling tool and the RFM-based prioritization methodology in a command-line interface (CLI) called FLAKERANKER to facilitate reuse and repurposing in future studies.