The Survey Chronicles: Health Expectations

One thing my survey sought to measure was the respondent’s perceived risk of falling ill.  We expected this to be tricky, because many of the respondents are not highly number literate, and most have no concept of probability.  This was easily the most challenging part of our survey to design.

We tried to model our question after surveys that had been done in other developing economies, as detailed in Delevande, Gine and McKenzie.  These surveys asked respondents to allocate piles of stones to different pots, depending on how likely they thought different amounts of rainfall were.  The advantage of this is that the respondent does not even need to be able to count, he or she can just compare quantities, and see that one is larger.  Also, the respondent can easily make marginal changes, moving a few stones to a different pot, if you change the conditions of the questions slightly.  We adapted our questions to ask respondents the probability they would get sick over the next month.

What sounded like a simple and elegant model turned out to be an incredible struggle.  The responses we got failed basic logic tests as often as not; for example, respondents would predict a higher risk of getting sick once in the next month than once in the next year. After rounds and rounds of piloting, and consulting everyone we could think of, we identified the major problems, and attempted to address them:

1.       People don’t like to say they will get sick, because they think it will then happen.  We addressed this by changing our question to ask about the probability someone as healthy as them would get sick over the next month.  This seemed to help.

2.       “Chance” and “likelihood” don’t translate well in Dagbani.  We did our best to write the questions in English so that they would be as simple as possible, and then carefully worked through the Dagbani to find the clearest translation.

3.       People have little concept of marginal changes in probability.  In northern Ghana, something is either certain to happen, certain not to happen, or may happen (50-50).  Our weather reports, forecasting 40% chance of rain, would make no sense here.  Conceptualizing one thing as more likely than another, or conceptualizing small changes in risk, is completely counter-intuitive.  We never completely solved this problem.

4.       People don’t like stones.  Natural objects, like stones or beans, are associated with witches’ fortune-telling, which people don’t like.  And here we were asking them to predict future sickness with stones.  We switched to bottle caps, which as a man-made object, are less threatening.  My friends in Tamale drank beer very diligently and enthusiastically in the weeks leading up to my survey in order to provide the 720 bottle tops I needed. 

As a result of continuing problems with #3, this section proved to be the most difficult in our survey.  It was only a few questions long, but the time it took to explain the concept, go through examples with the respondent, and work through the questions added up to nearly 30 minutes, almost a third of the total survey.