Refugee status determination: how cooperation with machine learning tools can lead to more justice

Abstract: Previous research on refugee status adjudications has shown that prediction of the outcome of an application can be derived from very few features with satisfactory accuracy. Recent research work has achieved between 70 and 90% accuracy using text analytics on various legal fields among which refugee status determination. Some studies report predictions derived from the judge identity only. Additionally most features used for prediction are non-substantive and external features ranging from news reports, date and time of the hearing or weather. On the other hand, literature shows that noise is ubiquitous in human judgments and significantly affects the outcome of decisions. It has been demonstrated that noise is a significant factor impacting legal decisions. We use the term "noise" in the sense described by D. Kahneman, as a measure of how human beings are unavoidably influenced by external factors when making a decision. In the context of refugee status determination, it means for instance that two judges would take different decisions when presented with the same application. This article explores ways that machine learning can help reduce noise in refugee law decision making. We are not suggesting that this proposed methodology should be exclusive from other approaches to improve decisions such as training of decision makers, skills acquisition or judgment aggregation, but rather that it is a path worth exploring. We investigate how artificial intelligence and specifically data-driven applications can be used to benefit all parties involved in refugee status adjudications. We specifically look at decisions taken in Canada and in the United States. Our research aims at reducing arbitrariness and unfairness that derive from noisy decisions, based on the assumption that if two cases or applications are alike they should be treated in the same way and induce the same outcome.