Working in Pairs: Understanding the Effects of Worker Interactions in Crowdwork

Abstract: Crowdsourcing has gained popularity as a tool to harness human brain power to help solve problems that are difficult for computers. Previous work in crowdsourcing often assumes that workers complete crowdwork independently. In this paper, we relax the independent property of crowdwork and explore how introducing direct, synchronous, and free-style interactions between workers would affect crowdwork. In particular, motivated by the concept of peer instruction in educational settings, we study the effects of peer communication in crowdsourcing environments. In the crowdsourcing setting with peer communication, pairs of workers are asked to complete the same task together by first generating their initial answers to the task independently and then freely discussing the tasks with each other and updating their answers after the discussion. We experimentally examine the effects of peer communication in crowdwork on various common types of tasks on crowdsourcing platforms, including image labeling, optical character recognition (OCR), audio transcription, and nutrition analysis. Our experiment results show that the work quality is significantly improved in tasks with peer communication compared to tasks where workers complete the work independently. However, participating in tasks with peer communication has limited effects on influencing worker's independent performance in tasks of the same type in the future.