Composing and Analyzing Crowdsourcing Workflows

When trying to crowdsource complex tasks, it is often better to break the big task into a set of smaller tasks and connect them into a production workflow.  Such workflows often allow a set of workers to work together in a task, creating an “assembly line for knowledge work” and allow the completion of tasks that are too complex or difficult for any participating individual to accomplish.  Recent research has focused on analyzing and optimizing existing workflows, using Partially Observed Markov Decision Processes (POMDPs).  The results indicate that using decision-theoretic approaches for dynamically adjusting the settings of crowdsourcing workflows and exchanging among multiple, existing workflows can lead to substantial improvements in both the quality and the overall cost of executing a workflow.

 

In this work, we present a method for creating and analyzing arbitrary workflows, and generating predictions about the resulting quality and cost of a given workflow.  Specifically, we focus on the following questions:  (1) Given a workflow for a task, can we predict the cost and time for completing the task, together with the resulting quality of the generated product?  (2) Given a workflow for a task, can we generate alternative workflows that are expected to be better than the current one?