Information Elicitation and Aggregation: Theory, Behavior, and Application

Abstract

Recent decades have witnessed a broadening application of the wisdom of crowds, ranging from forecasting geopolitical events to estimating business variables to predicting the replicability of social science studies. In these applications, information is elicited from a crowd of human participants and then aggregated to generate final judgments or informed decisions. Two central problems in this process are the information elicitation problem, i.e., how we can elicit authentic and high-quality information from selfish information holders, and the information aggregation problem, i.e., how we can aggregate the noisy or even biased information that we collect into more accurate judgments or decisions.

The challenges in solving the above two problems vary with the concrete application scenarios. At a high level, these scenarios can be divided into two categories, the with-verification setting and the without-verification setting. In the with-verification setting, the principal can access (historical) ground truth information to verify the information quality of each participant and design elicitation and aggregation mechanisms accordingly. In contrast, the principal cannot access such ground truth information in the without-verification setting. In this thesis, I explore and make progress on information elicitation and aggregation problems under several specific scenarios in both settings.

In the with-verification setting, I study the betting scenario and propose the randomized wagering mechanisms for prediction elicitation. These mechanisms overcome an impossibility result for deterministic mechanisms that four desirable properties cannot be satisfied simultaneously. In the without-verification setting, I study both the probabilistic prediction elicitation and aggregation problems. I first extend the strictly proper scoring rules, the most prevalent information elicitation solution in the with-verification setting, into the without-verification setting and derive the surrogate scoring rules. These rules not only provide strong incentives for participants to report true beliefs but also characterize their prediction accuracy. I further develop a forecast aggregation framework using the elicitation without-verification schemes such as the surrogate scoring rules to improve the aggregation accuracy consistently. This improvement is examined and verified on a diverse set of real-world human forecasting datasets.

Human behavior, involving how people understand elicitation questions, generate beliefs, and react to elicitation schemes, is overlooked in the literature on information elicitation and aggregation. This thesis also explores human behavior and its influence within this domain. In particular, I develop auction mechanisms to elicit truthful reporting of private signals when human players have bounded rationality in interdependent valuation auctions. I also conduct real-world elicitation and aggregation experiments that use laypeople to predict the replicability of social science studies. Several interesting experimental findings of laypeople's behaviors are analyzed and discussed.