Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics
View/ Open
Published Version
https://www.ijcai.org/Proceedings/16/Papers/371.pdfMetadata
Show full item recordCitation
Shnayder, Victor, Rafael M. Frongillo, and David C. Parkes. 2016. Measuring Performance of Peer Prediction Mechanisms Using Replicator Dynamics. In Proceedings of the 25th International Joint Conference on Artificial Intelligence (IJCAI 2016), New York, NY, July 9-15, 2016.Abstract
Peer prediction is the problem of eliciting private, but correlated, information from agents. By rewarding an agent for the amount that their report "predicts" that of another agent, mechanisms can promote effort and truthful reports. A common concern in peer prediction is the multiplicity of equilibria, perhaps including high-payoff equilibria that reveal no information. Rather than assume agents counter-speculate and compute an equilibrium, we adopt replicator dynamics as a model for population learning. We take the size of the basin of attraction of the truthful equilibrium as a proxy for the robustness of truthful play. We study different mechanism designs, using models estimated from real peer evaluations in several massive on-line courses. Among other observations, we confirm that recent mechanisms present a significant improvement in robustness over earlier approaches.Terms of Use
This article is made available under the terms and conditions applicable to Open Access Policy Articles, as set forth at http://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#OAPCitable link to this page
http://nrs.harvard.edu/urn-3:HUL.InstRepos:32220916
Collections
- FAS Scholarly Articles [18292]
Contact administrator regarding this item (to report mistakes or request changes)