Opening the black box of evaluation: How quality is recognized by peer review panels

Abstract

In two recent contributions to the study of peer review we went beyond stating the obvious that peer review produces valid judgments. We draw on in-depth analyses of five fellowship competitions in the United States, and of four grant panels organized by the Academy of Finland. We analyze and compare the intersubjective understandings academic experts create and maintain in making collective judgments on research quality. More specifically, we analyze the social conditions that lead panelists to an understanding of their choices as fair and legitimate, and to a belief that they are able to identify the best and less good proposals.