Crowdsourcing: A Framework for Usability Evaluation

Abstract: Objective: This research explores using crowdsourcing for software usability evaluation. Background: Usability studies are essential for designing user-friendly software, but traditional methods are often costly and time-consuming. Crowdsourcing offers a quicker, cost-effective alternative for remote usability evaluation, though ensuring quality feedback remains a challenge. Method: A systematic mapping study was conducted to review current usability evaluation research. Subsequently, multi-experiments were performed, comparing novice crowd usability inspectors to experts using expert heuristic evaluation as a benchmark. These results were used to create and validate a framework for crowd usability inspection through a case study. Results: The mapping study identified expert heuristic evaluation as a prevalent method, especially for websites. Experimental findings showed that novice crowd usability inspections, guided by expert heuristics, can match experts in identifying usability issues in content, quality, severity, and time efficiency. The case study demonstrated that the framework allows effective usability inspections, leading to successful software redesigns. Iterations of 3-5 novice inspections effectively resolved key usability issues within three cycles. Conclusion: Crowdsourcing is an effective alternative to expert heuristic evaluation for usability assessment. The proposed framework for crowd usability inspection is a viable solution for budget-constrained software companies. Keywords: crowdsourcing, crowd usability evaluation, expert heuristic evaluation, framework.