Empowering Prior to Court Legal Analysis: A Transparent and Accessible Dataset for Defensive Statement Classification and Interpretation
Abstract: The classification of statements provided by individuals during police interviews is a complex and significant task within the domain of NLP and legal informatics. The lack of extensive domain-specific datasets raises challenges to the advancement of NLP methods in the field. This paper aims to address some of the present challenges by introducing a novel dataset tailored for classification of statements made during police interviews, prior to court proceedings. Utilising the curated dataset for training and evaluation, we introduce a fine-tuned DistilBERT model that achieves state-of-the-art performance in distinguishing truthful from deceptive statements. To enhance interpretability, we employ explainable artificial intelligence (XAI) methods to offer explainability through saliency maps, that interpret the model's decision-making process. Lastly, we present an XAI interface that empowers both legal professionals and non-specialists to interact with and benefit from our system. Our model achieves an accuracy of 86%, and is shown to outperform a custom transformer architecture in a comparative study. This holistic approach advances the accessibility, transparency, and effectiveness of statement analysis, with promising implications for both legal practice and research.
- J. M. Salerno and B. L. Bottoms, “Emotional evidence and jurors’ judgments: The promise of neuroscience for informing psychology and law,” Behavioral sciences & the law, vol. 27, no. 2, pp. 273–296, 2009.
- A. Deeks, “The judicial demand for explainable artificial intelligence,” Columbia Law Review, vol. 119, no. 7, pp. 1829–1850, 2019.
- G. Vilone and L. Longo, “Explainable artificial intelligence: a systematic review,” arXiv preprint arXiv:2006.00093, 2020.
- J. Lawrence and C. Reed, “Argument mining: A survey,” Computational Linguistics, vol. 45, no. 4, pp. 765–818, 2020.
- W. C. Thompson and E. L. Schumann, “Interpretation of statistical evidence in criminal trials: The prosecutor’s fallacy and the defense attorney’s fallacy,” in Expert Evidence and Scientific Proof in Criminal Trials. Routledge, 2017, pp. 371–391.
- R. M. Re and A. Solow-Niederman, “Developing artificially intelligent justice,” Stan. Tech. L. Rev., vol. 22, p. 242, 2019.
- J. Cui, X. Shen, and S. Wen, “A survey on legal judgment prediction: Datasets, metrics, models and challenges,” IEEE Access, 2023.
- H. Zhong, C. Xiao, C. Tu, T. Zhang, Z. Liu, and M. Sun, “How does nlp benefit legal system: A summary of legal artificial intelligence,” arXiv preprint arXiv:2004.12158, 2020.
- E. Shushkevich and J. Cardiff, “Detecting fake news about covid-19 on small datasets with machine learning algorithms,” in 2021 30th Conference of Open Innovations Association FRUCT. IEEE, 2021, pp. 253–258.
- K. Merchant and Y. Pande, “Nlp based latent semantic analysis for legal text summarization,” in 2018 international conference on advances in computing, communications and informatics (ICACCI). IEEE, 2018, pp. 1803–1807.
- A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
- S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr et al., “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 6881–6890.
- S. Abnar and W. Zuidema, “Quantifying attention flow in transformers,” arXiv preprint arXiv:2005.00928, 2020.
- Y. Tay, D. Bahri, D. Metzler, D.-C. Juan, Z. Zhao, and C. Zheng, “Synthesizer: Rethinking self-attention for transformer models,” in International conference on machine learning. PMLR, 2021, pp. 10 183–10 192.
- T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al., “Transformers: State-of-the-art natural language processing,” in Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38–45.
- Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler, “Long range arena: A benchmark for efficient transformers,” arXiv preprint arXiv:2011.04006, 2020.
- D. Song, A. Vold, K. Madan, and F. Schilder, “Multi-label legal document classification: A deep learning-based approach with label-attention and domain-specific pre-training,” Information Systems, vol. 106, p. 101718, 2022.
- G. Ratnayaka, T. Rupasinghe, N. de Silva, M. Warushavithana, V. S. Gamage, M. Perera, and A. S. Perera, “Classifying sentences in court case transcripts using discourse and argumentative properties,” The International Journal on Advances in ICT for Emerging Regions, vol. 12, no. 1, 2019.
- I. Chalkidis, I. Androutsopoulos, and N. Aletras, “Neural legal judgment prediction in english,” arXiv preprint arXiv:1906.02059, 2019.
- S. Rönnqvist, A.-J. Kyröläinen, A. Myntti, F. Ginter, and V. Laippala, “Explaining classes through stable word attributions,” in Findings of the Association for Computational Linguistics: ACL 2022, 2022, pp. 1063–1074.
- A. Sekhon, H. Chen, A. Shrivastava, Z. Wang, Y. Ji, and Y. Qi, “Improving interpretability via explicit word interaction graph layer,” arXiv preprint arXiv:2302.02016, 2023.
- D. Harbecke and C. Alt, “Considering likelihood in nlp classification explanations with occlusion and language modeling,” arXiv preprint arXiv:2004.09890, 2020.
- T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Huggingface’s transformers: State-of-the-art natural language processing,” 2020.
- D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.