Objective Measurement of AI Literacy: Development and Validation of the AI Competency Objective Scale (AICOS)

Abstract: As AI becomes more pervasive in various aspects of life, AI literacy is becoming a fundamental competency that enables individuals to move safely and competently in an AI-pervaded world. There is a growing need to measure this competency, e.g., to develop targeted educational interventions. Although several measurement tools already exist, many have limitations regarding subjective data collection methods, target group differentiation, validity, and integration of current developments such as Generative AI Literacy. This study develops and validates the AI Competency Objective Scale (AICOS) for measuring AI literacy objectively. The presented scale addresses weaknesses and offers a robust measurement approach that considers established competency and measurement models, captures central sub-competencies of AI literacy, and integrates the dimension of Generative AI Literacy. The AICOS provides a sound and comprehensive measure of AI literacy, and initial analyses show potential for a modular structure. Furthermore, a first edition of a short version of the AICOS is developed. Due to its methodological foundation, extensive validation, and integration of recent developments, the test represents a valuable resource for scientific research and practice in educational institutions and professional contexts. The AICOS significantly contributes to the development of standardized measurement instruments and enables the targeted assessment and development of AI skills in different target groups.