Improving Anchoring Vignettes: Designing Surveys to Correct Interpersonal Incomparability

Abstract

We report the results of several randomized survey experiments designed to evaluate two intended improvements to anchoring vignettes, an increasingly common technique used to improve interpersonal comparability in survey research. This technique asks for respondent self-assessments followed by assessments of hypothetical people described in vignettes. Variation in assessments of the vignettes across respondents reveals interpersonal incomparability, allowing researchers to improve comparability by rescaling self-assessments relative to vignette responses. Our experiments show, first, that switching the question order so that self-assessments follow the vignettes primes respondents to define the response scale in a common way. In this case, priming is not a bias to avoid but a means of better communicating the question’s meaning. Second, we demonstrate that combining vignettes and self-assessments in a single direct comparison induces inconsistent and considerably less informative responses. Since similar combined strategies are widely employed for related purposes, our results suggest that anchoring vignettes could reduce measurement error in many applications where they are not currently used. Data for our experiments come from a national telephone survey and a separate online survey.