Empirical Methods in Peer Prediction

Abstract

Human computation system, often popularly referred to as crowdsourcing,requires the alignment of the incentives of human participants to report truthfully and an effective mean to deal with noise in the human-generated data. The main objective of this thesis is to introduce a new class of peer prediction mechanisms called empirical peer prediction mechanisms that represent an unified approach to resolving the incentive alignment and noisy-data challenges in human computation systems. In the information elicitation literature, existing peer prediction mechanisms provide theoretical solutions to the incentive alignment problems; however, implementing them in practice has been challenging due to restrictive assumptions. On the other hand, in the machine learning literature, researchers have proposed models and algorithms to estimate the error-rates of workers in human computation systems in an effort to reduce noise in the system; however, these models have largely ignoredthe incentive problem. While they have developed independently, these two disciplines ultimately share the same goal of improving human computation systems. In this thesis, I bring together the mechanisms and the algorithms from these two disciplines to introduce three new peer prediction mechanisms - the EmpiricalPeer Prediction Method, the k-Means Peer Prediction Method, and the EmpiricalScoring Rule Mechanism. I empirically demonstrate that these mechanisms align the incentives of the self-interested agents such that their utilities are maximized by reporting their signals truthfully. Moreover, I also show that the three mechanisms are robust against various reporting strategies including collusion.