Add new comment

Mechanical Turk, Amazon's popular crowdsourcing platform, is used by many people and organizations who need repetitive tasks completed. For rapid completion of unskilled (easy) tasks online, crowdsourcing can be an easy and affordable solution.

But what if you need people to complete a skilled task? How do you filter the crowd to find qualified people? Researchers at ICSI decided to find out, and lead author Luke Gottlieb presented their results at the ACM Multimedia conference which was held in October in Nara, Japan.

In their experiment, they tried to find people who were qualified to determine the location of video clips they were sent. ICSI is active in multimedia and geolocation research, so the ability to compare the accuracy of skilled humans to that of machine geolocation for video will be useful. The experiment, in addition to providing valuable insight into the potential use of Mechanical Turk for skilled tasks, provides a benchmark for how well skilled humans can perform geolocation compared with the ICSI Multimodal Location Estimation (MMLE) system. To learn more about the MMLE system, see this recent post.

Prior to screening users on Mechanical Turk, the team presented video clips to other researchers to narrow down a test set of clips they believed could be accurately located within 5 minutes by a skilled human.

In the qualifying test, it was important that the videos didn't require specific cultural knowledge, and that it would be an entertaining enough task for many users to want to try it. The researchers set a standard of 80% accuracy in order for a user to qualify as skilled for the task, and provided detailed instructions for completing the task. They found that 20% of mechanical turk users who tried the test set proved to be skilled enough for the task.

Once the qualified users were selected from the test set, the pay rate was increased so that users would still be motivated to complete the tasks even if they weren't quite as entertaining as the tasks in the training set. Keeping users motivated is an important part of crowdsourcing, in order to have enough of a crowd of users who want to complete the tasks. By keeping the tasks entertaining in some way, enough users will complete the tasks despite a relatively low pay rate. This enables researchers to recruit a large team of skilled users affordably.

More information on the experiment, including an illustrated example, is available on the MMLE web page. You can also read the paper that was presented at ACM Multimedia 2012. Because the research presented in this paper impressed the conference organizers, Gottlieb and his colleagues have been invited to write an expanded follow-up article for the journal IEEE Transactions on Multimedia, which they expect to submit for publication early in 2013.