Today I attended a World Bank training on survey methods in developing economies. (I would just like to say, the World Bank has great food, and where do they get their papaya, because when I buy papaya here it always tastes funny.)
One interesting bit of the training seminar had to do with calibrating surveys to adjust for differences in perceptions or attitudes among respondents. To get a more concrete idea of the problem, imagine asking two people to rank their health from 1 to 10. The first person is a young, Olympic athlete with a cold and sprained ankle, who rates her health as 5. The second is an old, barely mobile man who had a good day and was able to do a lap around his building with his walker. He ranks his health as 8. Is he really more healthy than the athlete? No, but the respondent's relative baseline matters.
An example from actual research: Gary King, Christopher Murray, Joshua Salomon, and Ajay Tandon asked survey respondents in China and Mexico to rate their ability to participate in governance. Chinese respondents, on average, ranked their political efficacy higher than Mexican respondents.
Few people would really believe that citizens have more political efficacy in China than Mexico. More likely, what Chinese citizens considered to be a "high" level of efficacy was different from what Mexican citizens considered to be a "high" level of efficacy. The researchers devised a way to account for differences in the respondents' relative baselines. They used vignettes, or stories illustrating a particular level of political participation, and asked respondents to compare themselves to the hypothetical. Using this method, Chinese respondents ranked their political efficacy lower than Mexican respondents, on average.
One interesting bit of the training seminar had to do with calibrating surveys to adjust for differences in perceptions or attitudes among respondents. To get a more concrete idea of the problem, imagine asking two people to rank their health from 1 to 10. The first person is a young, Olympic athlete with a cold and sprained ankle, who rates her health as 5. The second is an old, barely mobile man who had a good day and was able to do a lap around his building with his walker. He ranks his health as 8. Is he really more healthy than the athlete? No, but the respondent's relative baseline matters.
An example from actual research: Gary King, Christopher Murray, Joshua Salomon, and Ajay Tandon asked survey respondents in China and Mexico to rate their ability to participate in governance. Chinese respondents, on average, ranked their political efficacy higher than Mexican respondents.
Few people would really believe that citizens have more political efficacy in China than Mexico. More likely, what Chinese citizens considered to be a "high" level of efficacy was different from what Mexican citizens considered to be a "high" level of efficacy. The researchers devised a way to account for differences in the respondents' relative baselines. They used vignettes, or stories illustrating a particular level of political participation, and asked respondents to compare themselves to the hypothetical. Using this method, Chinese respondents ranked their political efficacy lower than Mexican respondents, on average.
From King, et al, 2004. Left are initial responses; right are responses calibrated with vignettes
Similar use of hypothetical vignettes can be used in surveys on other topics, including health, access to resources, education, etc., to ensure that respondents interpret questions similarly, and that their responses have the same meaning.
RSS Feed