Dazed & Confused: A Large-Scale Real-World User Study of reCAPTCHAv2

Abstract: Since about 2003, captchas have been widely used as a barrier against bots, while simultaneously annoying great multitudes of users worldwide. As their use grew, techniques to defeat or bypass captchas kept improving, while captchas themselves evolved in terms of sophistication and diversity, becoming increasingly difficult to solve for both bots and humans. Given this long-standing and still-ongoing arms race, it is important to investigate usability, solving performance, and user perceptions of modern captchas. In this work, we do so via a large-scale (over 3, 600 distinct users) 13-month real-world user study and post-study survey. The study, conducted at a large public university, was based on a live account creation and password recovery service with currently prevalent captcha type: reCAPTCHAv2. Results show that, with more attempts, users improve in solving checkbox challenges. For website developers and user study designers, results indicate that the website context directly influences (with statistically significant differences) solving time between password recovery and account creation. We consider the impact of participants' major and education level, showing that certain majors exhibit better performance, while, in general, education level has a direct impact on solving time. Unsurprisingly, we discover that participants find image challenges to be annoying, while checkbox challenges are perceived as easy. We also show that, rated via System Usability Scale (SUS), image tasks are viewed as "OK", while checkbox tasks are viewed as "good". We explore the cost and security of reCAPTCHAv2 and conclude that it has an immense cost and no security. Overall, we believe that this study's results prompt a natural conclusion: reCAPTCHAv2 and similar reCAPTCHA technology should be deprecated.

• The paper finds that repeated exposure to reCAPTCHAv2 leads to significant improvements in user solving efficiency.
• The paper reveals that task context, such as password recovery versus account creation, markedly influences solving times.
• The paper highlights critical security vulnerabilities and high resource costs, urging a shift toward more effective alternatives.

An Analysis of Captcha Usability: A Study on reCAPTCHAv2

The paper, "Dazed content Confused: A Large-Scale Real-World User Study of reCAPTCHAv2" by Searles et al., reports an extensive empirical study examining the efficacy, usability, and user perceptions of reCAPTCHAv2. The investigation is motivated by the widespread use of captchas as a barrier against bots, juxtaposed with the persistent evolution of bot technology capable of overcoming traditional captcha methods.

Methodology and Data Collection

Conducted at UC Irvine, the 13-month study involved over 3,600 distinct participants interacting with reCAPTCHAv2 on a real-world account management service as part of regular university operations. This user base consisted primarily of university students, providing authentic usage data devoid of bias often introduced in controlled studies. Participants were unaware of their involvement in the study, resulting in natural interaction data concerning account creation and password recovery tasks.

Key Findings

Usability Analysis:

1. Improvement Over Attempts: A significant finding from the study is that users' efficiency in solving checkbox captcha challenges improves with multiple attempts, suggesting a learning effect with repeated exposure.
2. Contextual Influence: The study revealed context-specific differences in solving times, noting that password recovery tasks were completed faster than account creations. This has implications for website developers and indicates that the task context can affect user interaction.
3. Educational and Disciplinary Impact: Analysis showed a correlation between participants' major and education level with captcha-solving performance. Technical disciplines exhibited better performance, and seniors performed faster than freshmen.

User Experience Insights:

The study's post-experience surveys demonstrated that image-based captchas were perceived as annoying, whereas checkbox challenges were regarded as easier and rated higher on the System Usability Scale (SUS). This aligns with the increased solving times for image challenges and suggests a potential shift or consideration for alternative methods that prioritize user satisfaction.

Security and Cost Concerns

The authors conducted a thorough examination of reCAPTCHAv2's security efficacy, concluding that current implementations offer little resistance against sophisticated automated attacks. The exploration of checkbox and image challenges revealed vulnerabilities that compromise security, raising questions about the true utility of reCAPTCHAv2 beyond its superficial role.

In terms of cost, the paper offers an analysis of the cumulative resource consumption associated with user interactions, highlighting substantial time and environmental costs. They estimated at least 512 billion reCAPTCHAv2 sessions resulting in 819 million hours — translating into approximately \$6.1 billion in unpaid wages — and significant energy consumption leading to CO2 emissions.

Implications and Recommendations

The conclusions drawn underscore a crucial recommendation: the deprecation of reCAPTCHA in favor of more effective and user-friendly alternatives. The stark juxtaposition between costs — both human and environmental — and security vulnerabilities challenges the continued reliance on reCAPTCHAv2.

Future Directions

Researchers and practitioners engaged in the domain of human-computer interaction and digital security may derive key learnings from this study, as it furnishes a robust dataset and analysis that could inform the development of next-generation captcha mechanisms. Further exploration could explore enhancing user perception while maintaining robust security measures.

In summary, this paper contributes a detailed empirical understanding of the usability dynamics and contextual dependencies of reCAPTCHAv2, inviting a reevaluation of current practices in utilizing such systems as security measures on the web.