Papers
Topics
Authors
Recent
Search
2000 character limit reached

Legally Binding but Unfair? Towards Assessing Fairness of Privacy Policies

Published 12 Mar 2024 in cs.CY, cs.AI, and cs.CL | (2403.08115v2)

Abstract: Privacy policies are expected to inform data subjects about their data protection rights and should explain the data controller's data management practices. Privacy policies only fulfill their purpose, if they are correctly interpreted, understood, and trusted by the data subject. This implies that a privacy policy is written in a fair way, e.g., it does not use polarizing terms, does not require a certain education, or does not assume a particular social background. We outline our approach to assessing fairness in privacy policies. We identify from fundamental legal sources and fairness research, how the dimensions informational fairness, representational fairness and ethics / morality are related to privacy policies. We propose options to automatically assess policies in these fairness dimensions, based on text statistics, linguistic methods and artificial intelligence. We conduct initial experiments with German privacy policies to provide evidence that our approach is applicable. Our experiments indicate that there are issues in all three dimensions of fairness. This is important, as future privacy policies may be used in a corpus for legal artificial intelligence models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (75)
  1. European Union, “REGULATION (EU) 2016/679 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation),” Official Journal of the European Union, vol. L119/1, 2016.
  2. R. N. Zaeem and K. S. Barber, “The effect of the gdpr on privacy policies: Recent progress and future promise,” ACM Trans. Manage. Inf. Syst., vol. 12, no. 1, dec 2020.
  3. C. Starke, J. Baleis, B. Keller, and F. Marcinkowski, “Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature,” Big Data & Society, vol. 9, no. 2, 2022.
  4. A. Boudjella, M. Sharma, and D. Sharma, “Non-native english speaker readability metric: Reading speed and comprehension,” Journal of Applied Mathematics and Physics, vol. 5, no. 6, pp. 1257–1268, 2017.
  5. S. Trzepla, “Ux accessibility for elderly - 12 principles,” UX Planet, 07 2019, accessed Feb 2024. [Online]. Available: https://uxplanet.org/ux-accessibility-for-elderly-12-principles-9708289b6f78
  6. L. Rello and R. Baeza-Yates, “How to present more readable text for people with dyslexia,” Universal Access in the Information Society, vol. 16, pp. 29–49, 2017.
  7. R. Evans, C. Orăsan, and I. Dornescu, “An evaluation of syntactic simplification rules for people with autism,” in Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), 2014, pp. 131–140.
  8. N. L. Aikens and O. Barbarin, “Socioeconomic differences in reading trajectories: The contribution of family, neighborhood, and school contexts.” Journal of educational psychology, vol. 100, no. 2, p. 235, 2008.
  9. D. Rice, J. H. Rhodes, and T. Nteta, “Racial bias in legal language,” Research & Politics, vol. 6, no. 2, p. 2053168019848930, 2019.
  10. N. Baker Gillis, “Sexism in the judiciary: The importance of bias definition in NLP and in our courts,” in Proceedings of the 3rd Workshop on Gender Bias in Natural Language Processing, 2021, pp. 45–54.
  11. E. Gumusel, V. Q. Malic, D. R. Donaldson, K. Ashley, and X. Liu, “An annotation schema for the detection of social bias in legal text corpora,” in Information for a Better World: Shaping the Global Future, M. Smits, Ed.   Springer International Publishing, 2022, pp. 185–194.
  12. B. Hooker, “Fairness,” Ethical Theory and Moral Practice, vol. 8, no. 4, pp. 329–352, 08 2005.
  13. P. Schwöbel and P. Remmers, “The long arc of fairness: Formalisations and ethical discourse,” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, p. 2179–2188.
  14. S. Verma and J. Rubin, “Fairness definitions explained,” in 2018 ACM/IEEE International Workshop on Software Fairness, 2018, pp. 1–7.
  15. N. Mehrabi, F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan, “A survey on bias and fairness in machine learning,” ACM Computing Surveys, vol. 54, no. 6, pp. 1–35, 2021.
  16. A. Chouldechova, “Fair prediction with disparate impact: A study of bias in recidivism prediction instruments,” Big Data, vol. 5, no. 2, pp. 153–163, 2017.
  17. M. Defrance and T. De Bie, “Maximal fairness,” in Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 2023, p. 851–880.
  18. R. N. Landers and T. S. Behrend, “Auditing the ai auditors: A framework for evaluating fairness and bias in high stakes ai predictive models,” American Psychologist, 2022.
  19. J. Greenberg, “Organizational justice: Yesterday, today, and tomorrow,” Journal of management, vol. 16, no. 2, pp. 399–432, 1990.
  20. J. A. Colquitt and K. P. Zipay, “Justice, fairness, and employee reactions,” Annual Review of Organizational Psychology and Organizational Behavior, vol. 2, no. 1, pp. 75–99, 2015.
  21. J. A. Colquitt and J. B. Rodell, “187Measuring Justice and Fairness,” in The Oxford Handbook of Justice in the Workplace.   Oxford University Press, 2015.
  22. D. Doherty and J. Wolak, “When do the ends justify the means? evaluating procedural fairness,” Political Behavior, vol. 34, pp. 301–323, 2012.
  23. J. Schoeffer, N. Kuehl, and Y. Machowski, ““there is not enough information”: On the effects of explanations on perceptions of informational fairness and trustworthiness in automated decision-making,” in Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022, p. 1616–1628.
  24. M. K. Lee, A. Jain, H. J. Cha, S. Ojha, and D. Kusbit, “Procedural justice in algorithmic fairness: Leveraging transparency and outcome control for fair algorithmic mediation,” Proceedings of the ACM on Human-Computer Interaction, vol. 3, no. CSCW, pp. 1–26, 2019.
  25. E. Ntoutsi, P. Fafalios, U. Gadiraju, V. Iosifidis, W. Nejdl, M.-E. Vidal et al., “Bias in data‐driven artificial intelligence systems—an introductory survey,” Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, vol. 10, no. 3, pp. 1–14, 2020.
  26. S. L. Blodgett, S. Barocas, H. Daumé III, and H. Wallach, “Language (technology) is power: A critical survey of “bias” in nlp,” in Proceedings of the 58th annual meeting of the association for computational linguistics, 2020, pp. 5454–5476.
  27. M. Abbasi, S. A. Friedler, C. Scheidegger, and S. Venkatasubramanian, “Fairness in representation: quantifying stereotyping as a representational harm,” in Proceedings of the 2019 SIAM International Conference on Data Mining.   SIAM, 2019, pp. 801–809.
  28. E. Edenberg and A. Wood, “Disambiguating algorithmic bias: From neutrality to justice,” in Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, 2023, pp. 691–704.
  29. A. Caliskan, J. J. Bryson, and A. Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science, vol. 356, no. 6334, pp. 183–186, 2017.
  30. O. Papakyriakopoulos, S. Hegelich, J. C. M. Serrano, and F. Marco, “Bias in word embeddings,” in Proceedings of the 2020 conference on fairness, accountability, and transparency, 2020, pp. 446–457.
  31. UN, “Universal declaration of human rights,” General Assembly resolution 217 A, 1948. [Online]. Available: https://www.un.org/sites/un2.un.org/files/2021/03/udhr.pdf
  32. Council of Europe, “European convention for the protection of human rights and fundamental freedoms,” no. ETS 5, 1950. [Online]. Available: https://www.refworld.org/docid/3ae6b3b04.html
  33. E. Awad, S. Dsouza, R. Kim, J. Schulz, J. Henrich, A. Shariff, J.-F. Bonnefon, and I. Rahwan, “The moral machine experiment,” Nature, vol. 563, no. 7729, pp. 59–64, 2018.
  34. Y. Liu, A. Medlar, and D. Głowacka, “Lexical ambiguity detection in professional discourse,” Information Processing & Management, vol. 59, no. 5, p. 103000, 2022.
  35. R. Flesch, “A new readability yardstick.” Journal of applied psychology, vol. 32, no. 3, p. 221, 1948.
  36. S. I. Becher and U. Benoliel, “Law in books and law in action: The readability of privacy policies and the gdpr,” in Consumer law and economics.   Springer, 2021, pp. 179–204.
  37. S. A. Crossley, S. Skalicky, M. Dascalu, D. S. McNamara, and K. Kyle, “Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas,” Discourse Processes, vol. 54, no. 5-6, pp. 340–359, 2017.
  38. K. Kyle, A. T. Choe, M. Eguchi, G. LaFlair, and N. Ziegler, “A comparison of spoken and written language use in traditional and technology-mediated learning environments,” ETS Research Report Series, vol. 2021, no. 1, pp. 1–29, 2021.
  39. W. Zhao, M. Strube, and S. Eger, “DiscoScore: Evaluating text generation with BERT and discourse coherence,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, 2023, pp. 3865–3883.
  40. T. G. Cocklin, N. J. Ward, H.-C. Chen, and J. F. Juola, “Factors influencing readability of rapidly presented text segments,” Memory & Cognition, vol. 12, pp. 431–442, 1984.
  41. R. Power, D. Scott, and N. Bouayad-Agha, “Document structure,” Computational Linguistics, vol. 29, no. 2, pp. 211–260, 2003.
  42. E. M. Smith, M. Hall, M. Kambadur, E. Presani, and A. Williams, ““I’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 9180–9211.
  43. N. Garg, L. Schiebinger, D. Jurafsky, and J. Zou, “Word embeddings quantify 100 years of gender and ethnic stereotypes,” Proceedings of the National Academy of Sciences, vol. 115, no. 16, pp. E3635–E3644, 2018.
  44. S. Schröder, A. Schulz, P. Kenneweg, R. Feldhans, F. Hinder, and B. Hammer, “Evaluating metrics for bias in word embeddings,” arXiv preprint arXiv:2111.07864, 2021.
  45. J. Risch, A. Stoll, L. Wilms, and M. Wiegand, “Overview of the germeval 2021 shared task on the identification of toxic, engaging, and fact-claiming comments,” in Proceedings of the GermEval 2021 Shared Task on the Identification of Toxic, Engaging, and Fact-Claiming Comments, 2021, pp. 1–12.
  46. T. Garg, S. Masud, T. Suresh, and T. Chakraborty, “Handling bias in toxic speech detection: A survey,” ACM Computing Surveys, vol. 55, no. 13s, pp. 1–32, 2023.
  47. I. Wallimann-Helmer, L. Terán, E. Portmann, H. Schübel, and J. Pincay, “An integrated framework for ethical and sustainable digitalization,” in 2021 Eighth International Conference on eDemocracy & eGovernment (ICEDEG).   IEEE, 2021, pp. 156–162.
  48. M. C. Pavan, V. G. Dos Santos, A. G. Lan, J. Martins, W. R. Santos, C. Deutsch, P. B. Costa, F. C. Hsieh, and I. Paraboni, “Morality classification in natural language text,” IEEE Transactions on Affective Computing, 2020.
  49. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088, 2024.
  50. OpenAI, “Gpt-4 turbo,” 11 2023, accessed Feb 2024. [Online]. Available: https://openai.com/blog/new-models-and-developer-products-announced-at-devday
  51. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020.
  52. S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg et al., “Sparks of artificial general intelligence: Early experiments with gpt-4,” arXiv preprint arXiv:2303.12712, 2023.
  53. M. Pock, A. Ye, and J. Moore, “Llms grasp morality in concept,” arXiv preprint arXiv:2311.02294, 2023.
  54. H. Nori, Y. T. Lee, S. Zhang, D. Carignan, R. Edgar, N. Fusi, N. King, J. Larson, Y. Li, W. Liu et al., “Can generalist foundation models outcompete special-purpose tuning? case study in medicine,” arXiv preprint arXiv:2311.16452, 2023.
  55. B. Bartelt and E. Buchmann, “Transparency in privacy policies,” in 12th International Conference on Building and Exploring Web Based Environments (to appear), 2024.
  56. R. Nokhbeh Zaeem, S. Anya, A. Issa, J. Nimergood, I. Rogers, V. Shah, A. Srivastava, and K. S. Barber, “Privacycheck v2: A tool that recaps privacy policies for you,” in Proceedings of the 29th ACM international conference on information & knowledge management, 2020, pp. 3441–3444.
  57. W. B. Tesfay, P. Hofmann, T. Nakamura, S. Kiyomoto, and J. Serna, “Privacyguide: Towards an implementation of the eu gdpr on internet privacy policy evaluation,” in Proceedings of the Fourth ACM International Workshop on Security and Privacy Analytics, 2018, pp. 15–21.
  58. O. Amaral, S. Abualhaija, D. Torre, M. Sabetzadeh, and L. C. Briand, “Ai-enabled automation for completeness checking of privacy policies,” IEEE Transactions on Software Engineering, vol. 48, no. 11, pp. 4647–4674, 2021.
  59. W. B. Tesfay, P. Hofmann, T. Nakamura, S. Kiyomoto, and J. Serna, “I read but don’t agree: Privacy policy benchmarking using machine learning and the eu gdpr,” in Companion Proceedings of the The Web Conference 2018, 2018, p. 163–166.
  60. D. Sánchez, A. Viejo, and M. Batet, “Automatic assessment of privacy policies under the gdpr,” Applied Sciences, vol. 11, no. 4, p. 1762, 2021.
  61. I. Pollach, “A typology of communicative strategies in online privacy policies: Ethics, power and informed consent,” Journal of Business Ethics, vol. 62, pp. 221–235, 2005.
  62. APEC. (2015) Asia-pacific economic cooperation privacy framework. [Online]. Available: https://www.apec.org/docs/default-source/Publications/2017/8/APEC-Privacy-Framework-%282015%29/217_ECSG_2015-APEC-Privacy-Framework.pdf
  63. G. Malgieri, “The concept of fairness in the gdpr: A linguistic and contextual interpretation,” in Proceedings of the 2020 Conference on fairness, accountability, and transparency, 2020, pp. 154–166.
  64. R. Schäfer, “Processing and querying large web corpora with the COW14 architecture,” in Proceedings of Challenges in the Management of Large Corpora 3 (CMLC-3), 2015.
  65. R. Schäfer and F. Bildhauer, “Building large corpora from the web using a new efficient tool chain,” in Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), 2012, pp. 486–493.
  66. B. Savoldi, M. Gaido, L. Bentivogli, M. Negri, and M. Turchi, “Gender bias in machine translation,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 845–874, 2021.
  67. T. Kenter and M. De Rijke, “Short text similarity with word embeddings,” in Proceedings of the 24th ACM international on conference on information and knowledge management, 2015, pp. 1411–1420.
  68. M. Han, X. Zhang, X. Yuan, J. Jiang, W. Yun, and C. Gao, “A survey on the techniques, applications, and performance of short text semantic similarity,” Concurrency and Computation: Practice and Experience, vol. 33, no. 5, p. e5971, 2021.
  69. M. Honnibal and I. Montani, “spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing,” 2017, to appear.
  70. Y. J. Choe, K. Park, and D. Kim, “word2word: A collection of bilingual lexicons for 3,564 language pairs,” in Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020), 2020.
  71. J. Musch and P. Rösler, “Schnell-lesen: Was ist die grenze der menschlichen lesegeschwindigkeit?” Kognitive Leistungen: Intelligenz und mentale Fähigkeiten im Spiegel der Neurowissenschaften, pp. 89–106, 2011.
  72. M. Martelli, M. De Luca, L. Lami, C. Pizzoli, M. Pontillo, D. Spinelli, and P. Zoccolotti, “Bridging the gap between different measures of the reading speed deficit in developmental dyslexia,” Experimental brain research, vol. 232, pp. 237–252, 2014.
  73. B. Muller, B. Alastruey, P. Hansanti, E. Kalbassi, C. Ropers, E. M. Smith, A. Williams, L. Zettlemoyer, P. Andrews, and M. R. Costa-jussà, “The gender-gap pipeline: A gender-aware polyglot pipeline for gender characterisation in 55 languages,” arXiv preprint arXiv:2308.16871, 2023.
  74. N. Baccouri, “deep-translator: A python library for language translation,” https://deep-translator.readthedocs.io/en/latest/README.html, 2020, accessed Feb 2024.
  75. O. Guhr, A.-K. Schumann, F. Bahrmann, and H. J. Böhme, “Training a broad-coverage german sentiment classification model for dialog systems,” in Proceedings of The 12th Language Resources and Evaluation Conference, 2020, pp. 1620–1625.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free