Toward Automatic Task Design: A Progress Report

DSpace/Manakin Repository

Toward Automatic Task Design: A Progress Report

Citable link to this page


Title: Toward Automatic Task Design: A Progress Report
Author: Huang, Eric; Zhang, Haoqi; Parkes, David C.; Gajos, Krzysztof Z.; Chen, Yiling

Note: Order does not necessarily reflect citation order of authors.

Citation: Eric, Huang, Haoqi Zhang, David C. Parkes, Krzysztof Z. Gajos, and Yiling Chen. 2010. Toward automatic task design: a progress report. In Proceedings of the ACM SIGKDD Human Computation Workshop: July 25-28, 2010, Washington DC, 77-85. New York, NY: Association for Computing Machinery.
Full Text & Related Files:
Abstract: A central challenge in human computation is in understanding how to design task environments that effectively attract participants and coordinate the problem solving process. In this paper, we consider a common problem that requesters face on Amazon Mechanical Turk: how should a task be designed so as to induce good output from workers? In posting a task, a requester decides how to break down the task into unit tasks, how much to pay for each unit task, and how many workers to assign to a unit task. These design decisions affect the rate at which workers complete unit tasks, as well as the quality of the work that results. Using image labeling as an example task, we consider the problem of designing the task to maximize the number of quality tags received within given time and budget constraints. We consider two different measures of work quality, and construct models for predicting the rate and quality of work based on observations of output to various designs. Preliminary results show that simple models can accurately predict the quality of output per unit task, but are less accurate in predicting the rate at which unit tasks complete. At a fixed rate of pay, our models generate different designs depending on the quality metric, and optimized designs obtain significantly more quality tags than baseline comparisons.
Published Version: doi:10.1145/1837885.1837908
Terms of Use: This article is made available under the terms and conditions applicable to Other Posted Material, as set forth at
Citable link to this page:

Show full Dublin Core record

This item appears in the following Collection(s)


Search DASH

Advanced Search