Figures
Abstract
Background
Assessing socio-economic position can be difficult, particularly in developing countries. Collection of socio-economic data usually relies on interviewer-administered questionnaires, but there is little research exploring how questionnaire delivery mode (QDM) influences reporting of these indicators. This paper reports on results of a trial of four QDMs, and the effect of mode on poverty reporting.
Methods
This trial was nested within a community-randomised trial of an adolescent reproductive health intervention conducted in rural Zimbabwe. Participants were randomly allocated to one of four QDMs (three different self-administered modes and one interviewer-administered mode); a subset was randomly selected to complete the questionnaire twice. Questions covered three socio-economic domains: i) ownership of sellable and fixed assets; ii) ability to afford essential items; and iii) food sufficiency. Statistical analyses assessed the association between QDM and reporting of poverty, and compared the extent of response agreement between questionnaire rounds.
Results
96% (n = 1483) of those eligible took part; 395 completed the questionnaire twice. Reported levels of poverty were high. Respondents using self-administered modes were more likely to report being unable to afford essential items and having insufficient food. Among those completing the questionnaire twice using different modes, higher levels of poverty and food insufficiency were reported when they completed the questionnaire using a self-administered mode.
Citation: Pascoe SJS, Hargreaves JR, Langhaug LF, Hayes RJ, Cowan FM (2013) ‘How Poor Are You?’ – A Comparison of Four Questionnaire Delivery Modes for Assessing Socio-Economic Position in Rural Zimbabwe. PLoS ONE 8(9): e74977. https://doi.org/10.1371/journal.pone.0074977
Editor: Thomas Eisele, Tulane University School of Public Health and Tropical Medicine, United States of America
Received: January 21, 2013; Accepted: August 12, 2013; Published: September 20, 2013
Copyright: © 2013 Pascoe et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The funding for this study was provided through a grant plus a competitively awarded supplement from National Institutes of Mental Health (NIMH) RO1 MH 65570, “Randomised trial of HIV/STI prevention in Zimbabwean Youth.” (URL: www.nimh.nih.gov/.) The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Health inequalities in low and middle-income countries are a public health problem of growing importance. Monitoring trends in health inequalities and comparing their magnitude between settings should help identify strategies to reduce them and requires accurate data on economic status [1]–[4] If the methods used to ascertain economic status do not accurately measure the poverty or wealth status of those living within a community, then this may distort observed relationships, possibly leading to incorrect policy recommendations.
Assessing socio-economic position (SEP) within developing countries can be difficult. The term ‘poverty’ covers a wide range of concepts and constructs and can vary across context and time. Income and/or consumption expenditure data are generally considered gold standard quantitative measures of SEP; there are however, disadvantages and difficulties with using these indicators. Data can be complex and time-consuming to collect and subject to potential reporting biases [2], [5] Many studies have instead assessed SEP using an asset-based approach, combining data on ownership of fixed and sellable assets such as type of toilet, plough, bicycle as a single measure of SEP. [6]–[9] Others have included data on food sufficiency or ability to afford essential items (such as health care and education). [6], [8], [10]–[13].
Data to inform an asset-based measure of SEP are almost always collected using interviewer-administered questionnaires. [6], [8], [14], [15] This allows interviewers to clarify questions, probe for more information and provide reassurance around sensitive or stigmatizing questions. Reporting level of wealth is sensitive, and may be affected by social desirability bias. Recent comparative research on the effect of questionnaire delivery mode (QDM) on collection of data on sensitive variables, such as sexual behaviour and mental health, indicate that self-administered methods (which includes paper and computer self-administered questionnaires) generally lead to higher, and possibly more accurate, rates of reporting. [16]–[20] Tourangeau et al. found reporting of income and SEP to be an equally sensitive experience, yet in our search of the literature we found no comparative research exploring the influence of survey method on reporting of SEP within developing countries. [16].
The Regai Dzive Shiri project was a community-randomized trial of a multi-component adolescent reproductive health intervention conducted in rural Zimbabwe between 2003–2007. [21] In 2006 we nested a randomised controlled trial of four QDM into the interim survey in order to compare reporting of various stigmatized behaviours and responses to sensitive questions by mode. We report here on differences in prevalence of indicators of SEP between the different modes.
Methods
Ethical Approval
This study received approval from the Medical Research Council of Zimbabwe and the ethics committees at University College London and the London School of Hygiene and Tropical Medicine.
In 2003, the Regai Dzive Shiri baseline survey was conducted in 30 rural communities in three provinces in eastern Zimbabwe. Eighty-six percent (n = 6791) of pupils in their ninth year of schooling (mean age 15yrs) attending 82 secondary schools in the trial communities took part. [21] In 2006, we conducted a further survey in 12 of the 30 communities that were randomly selected. Survey respondents were eligible if they were members of the trial cohort and were still resident in these communities. Using a permuted random block design, survey respondents were randomly allocated to one of four QDM: i) interviewer-administered questionnaire (IAQ), ii) self-administered questionnaire (SAQ) using paper and pen, iii) audio self-administered questionnaire (AudioSAQ) - SAQ accompanied by an audio soundtrack, and; iv) audio computer-assisted self-interview (ACASI) - completed on a laptop computer accompanied by an audio soundtrack (Figure 1).
In order to allow for some internal comparison, two rounds of the questionnaire were administered to just under 30% randomly selected survey respondents. Round 2 was a shortened version of the first questionnaire. Respondents assigned to complete the questionnaire twice were randomly allocated a QDM for Round 2 and asked to return to the survey site one week later to complete Round 2.
The questionnaire covered a wide range of domains including demographics, poverty, mobility, reproductive health knowledge and sexual behaviour. SEP was examined through a series of questions prefaced by ‘Think about yourself and your household now’ and covering three socio-economic domains: i) ownership of sellable assets (e.g. oxcart, blankets) and fixed assets which related to physical household structure (i.e. roof and building materials); ii) ability to afford essential items (e.g. cooking oil, soap, clinic fees); and iii) food sufficiency in the past week (e.g. days without food or skipping meals). These data referred to characteristics of the respondent’s household. Questions were translated into Shona (indigenous language) and cognitive interviewing (a technique used to pre-test question wording) was used to ensure comprehension. [22], [23].
Data Handling and Analysis
All questionnaire data completed on IAQ, SAQ, and AudioSAQ were double-entered onto a password-protected database. Data from ACASI were downloaded and imported directly into the database.
Chi-square tests were used and risk ratios calculated to compare reporting of each socio-economic indicator between the four QDMs (taking IAQ as the reference group).
A summary variable was then created for each socio-economic domain. For asset ownership and essential items, summed-score indices were created. If a respondent owned an asset or was able to afford an item, this conferred a score of 1; not owning the asset or being able to afford the item was given a score of 0. Indices were created by adding variable scores together in each domain (i.e. the essential item summed-score included 7 variables; asset ownership score 13 variables). These indices were then split into three approximately equal-sized groups (terciles) for the whole population, creating three categories of SEP for each domain (where SEP1 was the poorest category and SEP3 the wealthiest). Chi-square tests and ordinal logistic regression models were used to assess the association between QDM and SEP. One-way analysis of variance (ANOVA) was performed to compare the mean summed-score of each index between QDM. Food sufficiency status was determined based on 4 question items (see tables); food insufficiency was defined by an affirmative response to one or more of these questions. Chi-square tests and logistic regression were used to assess the association between QDM and reported food insufficiency.
For respondents who completed both rounds of the survey, responses were compared between rounds 1 and 2 for all economic indicators included on both questionnaires (this excluded asset-based questions). McNemar’s chi-square test was used to test the null hypothesis of no difference in reporting between rounds. The extent of agreement between rounds was assessed using the kappa statistic. For these analyses, responses were compared between self-administered modes of delivery (i.e. SAQ, AudioSAQ and ACASI) and IAQ; and also compared where respondents completed IAQ in both rounds or a self-administered mode in both rounds. Self-administered modes were combined into one category to restrict the number of comparisons made and allow a more general assessment of differences in reporting between self-administered and interview-administered QDMs. The correlation between essential item scores was explored for those with two rounds of data using different modes in each round. All statistical analyses were performed using Stata 10 (College Station, TX).
Results
Of 1,557 cohort respondents still living in the study communities at the time of the survey, 96% (n = 1495; 827 males) took part in the interim survey (mean age 18.2yrs; range 15–23); 12 respondents failed to complete the questionnaire. Table 1 presents demographic data; randomization was successful in achieving relative balance of key characteristics across the four arms of the study. The majority of respondents (87%; 1277) had completed at least four years of secondary school (ie completed Form 4); just over 4% (63) were married; and levels of orphaning were high with 40% (573/1432; 95% CI: 37.5–42.6) reporting the loss of one or both parents (11% had lost both parents). [18], [24].
Socio-economic Indicators and Questionnaire Delivery Mode
For the 1483 that completed the questionnaire, completion of socio-economic questions was high with less than 2% item non-response rate. Table 2 presents the proportion of respondents reporting each item in each socio-economic domain by QDM. Overall levels of reported poverty were high. Most households appeared to own everyday items such as tables, chairs and sufficient blankets, although 10% of participants reported that their household did not own these items. Only 10% of participants reported owning luxury items such as a motorcycle, car, or fridge, and less than half owned an oxcart or bicycle. Households of over 55% of participant were unable to afford cooking oil, 21% unable to afford soap to wash clothes; 41% unable to afford clinic fees; and 55% reported school absenteeism due to lack of money for fees. Reports of food insufficiency were also high with over 18% reporting going a day without food in the past week as there was no food in the household, and 28% unable to afford more than one meal a day. Overall, 45% of respondents reported one or more indicators of food insufficiency.
The data show relatively little difference in the proportion of households reporting ownership of sellable assets between QDMs, but substantial differences for the other domains. Higher rates of reporting an inability to afford essential items and food insufficiency were seen among participants who used self-administered modes of questionnaire delivery.
The right half of Table 2 presents risk ratios to compare the proportion reporting each indicator between the four QDMs. Those using self-administered modes were between two and four times as likely to report not owning blankets, tables and chairs, and were more likely to report that their house was built of traditional materials (poles and dagga and grass roof), than those using IAQ. Likewise, in the domain ‘ability to afford essential items’, respondents using self-administered modes were more likely to report being unable to afford certain items than IAQ users. For example, SAQ users were 1.9 times more likely to report not being able to afford clothes soap as IAQ users; AudioSAQ and ACASI users were 1.6 and 1.5 times more likely to report this item respectively. There was also a strong association between QDM and reporting food insufficiency; respondents using self-administered modes were twice as likely to report adults skipping meals and going a day without food as respondents who were interviewed.
Association between Socio-economic Position and Questionnaire Delivery Mode
Table 3 shows the proportion of respondents categorised in each SEP tercile for each socio-economic domain by QDM; the mean summed-score for both the asset wealth and essential item index by delivery mode; and odds ratios to demonstrate the relationship between SEP and QDM. Where SEP was defined by ownership of fixed and sellable assets, whilst there was some evidence of a difference in reported asset wealth between the different QDMs (p = 0.016); the mean index score was similar irrespective of which mode was used (p = 0.218); and odds ratios indicated that the odds of being categorised in the wealthier SEP categories versus the poorest category were similar for all QDMs.
The latter section of the table though, presents strong evidence of an association between QDM and SEP defined by ability to afford essential items, and with reporting food sufficiency. The data showed that those using self-administered modes were more likely to be categorised in the poorest SEP tercile, and to report having insufficient food, than IAQ users. ANOVA showed a significant difference in the mean essential item score between the QDMs (p<0.001) with higher mean sum-scores among IAQ users. Odds ratios indicated that for participants using SAQ to report ability to afford essential items, the odds of being categorised in SEP3 or SEP2 rather than SEP1, were 0.5 times the odds of being categorised in these wealthier categories when using IAQ. Those using AudioSAQ and ACASI were 0.7 times as likely to be categorised in the higher SEP terciles as those using IAQ. The odds of reporting food insufficiency among SAQ users were 1.9 times those of participants who used IAQ; AudioSAQ and ACASI users were also significantly more likely to report food insufficiency (OR = 1.7 and 1.6 respectively).
Comparison between Round 1 and Round 2
Of the 395 respondents who completed the questionnaire twice, 131 completed the two rounds using IAQ in one round and a self-administered mode in the other (SAQ (n = 45), AudioSAQ (n = 46), ACASI (n = 40)); 30 completed IAQ in both rounds; and 234 completed a self-administered mode in both rounds (27 completed SAQ-SAQ; 23 AudioSAQ-AudioSAQ; 31 ACASI-ACASI; 50 SAQ-AudioSAQ or vice versa; 57 SAQ-ACASI or vice versa; and 46 ACASI-AudioSAQ or vice versa).
Tables 4 and 5 show the proportion that reported each socio-economic indicator by each survey round. Table 4 presents data for respondents who used a different mode in each round i.e. IAQ in round 1 and self-administered in Round 2 or vice versa; Table 5 presents data for respondents who used IAQ or a self-administered mode in both rounds. A greater proportion of respondents reported not being able to afford essential items or aspects of food insecurity when they completed the questionnaire using a self-administered mode, irrespective of whether the self-administered mode was completed in the first or second round. For example, 11% of respondents who used IAQ in Round 1 reported having gone a day without food in the last week (Table 4). However, when these respondents completed the questionnaire using a self-administered mode in Round 2, 22% reported a day without food. Among respondents who first completed the questionnaire using a self-administered mode, 22% reported a day without food; yet only 6% of these respondents reported the same indicator when the questionnaire was interviewer-administered in Round 2. The Kappa statistic (κ) for several indicators (particularly food sufficiency indicators) in Table 4 are less than or equal to 0.4, which suggests that there was generally only poor to moderate agreement in the answers given between the two rounds and two modes of questionnaire.
Among respondents who completed either IAQ or a self-administered mode in both rounds, there was, for most socio-economic indicators, no statistically significant difference (p>0.05) in the proportion reporting each indicator in each round (Table 5), indicating that there was more consistency between data across the two rounds when the same or similar QDMs were used in each round.
Figure 2 shows the association between IAQ mean essential item score and self-administered mode score for those who completed two rounds of data using the different modes in each round (n = 131). This demonstrates an association between the scores (r = 0.77; 95% CI: 0.61–0.92; p<0.001), indicating that whilst relative rankings of poverty are preserved with each method, lower scores were obtained using self-administered modes as participants were more likely to report being unable to afford essential items.
Discussion
We compared the effect of questionnaire delivery mode on reporting of socio-economic indicators in Zimbabwe. Data presented here suggest that QDM is associated with how different socio-economic indicators are reported. While non-item response rate was low for all methods, we found significant differences in prevalence of indicators of economic status between interviewer-administered and self-administered modes. The IAQ was most often associated with the lowest prevalence of reporting items thought to indicate lower SEP, although QDM did not appear to influence reporting of asset wealth to the same degree as reporting ability to afford essential items and food sufficiency. This is reassuring given that many researchers restrict the measurement of SEP to the collection of asset-based indicators, but it does highlight the importance of considering the mode of collection when other indicators are to be measured and recorded. Data comparing Rounds 1 and 2 also showed that, irrespective of the order in which the questionnaires were completed, respondents were more likely to report a higher degree of poverty using self-administered modes, than when questioned by an interviewer. Although there was some inconsistency in reporting of poverty indicators between rounds there was no significant difference in reporting when round alone was taken into account.
This study had a number of strengths, in particular a high coverage within study communities; a large sample size; randomisation of QDM, which enabled us to ensure unbiased comparisons; and repeated rounds, which allowed us to look at the consistency of reporting. However, the context in which this study was undertaken could influence generalisability of the results to other populations. Between 2001 and 2009 Zimbabwe experienced extreme political and economic challenges; hyperinflation and high rates of unemployment and internal displacement may all have impacted on reporting socio-economic status. The restricted age group of the study population, the very high literacy rate in Zimbabwe and the fact that all participants had completed at least some years of secondary education may also have influenced the results, particularly if these younger individuals were less aware of the economic status within their household.
The majority of surveys that examine SEP use a trained interviewer to administer survey tools. One of the main considerations when conducting research on sensitive issues and behaviours is to try and improve validity and reliability of self-reporting by limiting the effects of social desirability bias. [20], [25]–[27] It is generally assumed that higher levels of reporting of socially undesirable attributes, such as lower SEP, indicate more accurate levels of self-reporting. [28]–[31] In this study the prevalence of items indicative of lower SEP appeared to be higher among those using self-administered modes than those being interviewed. However, the expected direction of social desirability bias in this situation is not clear. One possible explanation is that respondents were embarrassed to report their wealth status when asked by an interviewer directly. Qualitative data collected from some of these survey respondents supports this. One female respondent stated: ‘For example, the question ‘Have you ever gone to bed hungry in your household?’ is difficult to answer…because you may just feel embarrassed to say it’s true that in our household we sometimes go to bed hungry.’ [32] Another possibility is that those using self-administered modes of completion felt more able to exaggerate the severity of their economic situation and less censured in their answers. Data from participatory research exploring the issue of wealth, conducted at the same time in these communities indicated that participants censured each other when people tried to claim they were poorer than they actually were (data available from the author). Furthermore, the fact that questionnaire method did not influence reporting of asset wealth could also suggest that people are more honest about items that may be observed or that are objective, but may misreport items that are more subjective or hidden.
These data suggest that there needs to be further exploration of the role of questionnaire delivery mode on self-reported measures of poverty, validated against objective measures where possible, in order to ensure that the measures used to ascertain socio-economic position within a country or community are a true and accurate reflection of actual status.
Author Contributions
Conceived and designed the experiments: SJSP JRH LFL RJH FMC. Performed the experiments: SJSP LFL FMC. Analyzed the data: SJSP JRH RJH FMC. Wrote the paper: SJSP JRH LFL RJH FMC.
References
- 1. Barros AJ, Ronsmans C, Axelson H, Loaiza E, Bertoldi AD, et al. (2012) Equity in maternal, newborn, and child health interventions in Countdown to 2015: a retrospective review of survey data from 54 countries. Lancet 379: 1225–1233.
- 2.
Howe LD, Galobardes B, Matijasevich A, Gordon D, Johnston D, et al.. (2012) Measuring socio-economic position for epidemiological studies in low- and middle-income countries: a methods of measurement in epidemiology paper. International Journal of Epidemiology.
- 3.
Gwatkin D, Rutstein S, Johnson K, Suliman E, Wagstaff A, et al.. (2007) Socio-Economic Differences in Health, Nutrition and Population in Developing Countries: An Overview.. Washington DC, USA: The World Bank.
- 4.
Health CftSDo (2008) Closing the Gap in a Generation: Health Equity through Action on the Social Determinants of Health. Final Report of the Commission on Social Determinants of Health. Geneva: World Health Organization.
- 5. Morris SS, Carletto C, Hoddinott J, Christiaensen LJM (2000) Validity of rapid estimates of household wealth and income for health surveys in rural Africa. J Epidemiol Community Health 54: 381–387.
- 6.
Central Statistical Office (CSO) [Zimbabwe, Macro International Inc (2007) Zimbabwe Demographic and Health Survey 2005–06. Calverton, Maryland: CSO and Macro International Inc.
- 7. Gillespie S, Kadiyala S, Greener R (2007) Is poverty or wealth driving HIV transmission? AIDS 21 Suppl 7S5–S16.
- 8.
Ministry of Health and Child Welfare (Zimbabwe), Zimbabwe National Family Planning Council, National AIDS Council (Zimbabwe), US Centers for Disease Control and Prevention (2004) The Zimbabwe Young Adult Survey (YAS) 2001–2002 Final Report. Harare, Zimbabwe: Ministry of Health and Child Welfare and U.S. Centres for Disease Control and Prevention.
- 9. Wojcicki JM (2005) Socioeconomic status as a risk factor for HIV infection in women in East, Central and Southern Africa: a systematic review. J Biosoc Sci 37: 1–36.
- 10.
Cohen D (1998) Poverty and HIV/AIDS in sub-Saharan Africa. In: SEPED Conference Paper Series - Contribution to a five year review of the Fourth World Conference on Women (Beijing) and the World Summit for Social Development (Copenhagen). 1–12.
- 11. Weiser SD, Leiter K, Bangsberg DR, Butler LM, Percy-de Korte F, et al. (2007) Food insufficiency is associated with high-risk sexual behavior among women in Botswana and Swaziland.[see comment]. PLoS Medicine 4: 1589–1597.
- 12. Rollins N (2007) Food insecurity–a risk factor for HIV infection. PLoS Med 4: 1576–1577.
- 13.
Central Statistical Office (2000) Zimbabwe Demographic and Health Survey 1999. Central Statistical Office, Harare, Zimbabwe; MEASURE; Macro International Inc.
- 14.
UNAIDS (2008) 2008 Report on the global AIDS epidemic.
- 15.
UNDP (2007) Human Development Report 2007/2008. UNDP.
- 16. Tourangeau R, Yan T (2007) Sensitive questions in surveys. Psychol Bull 133: 859–883.
- 17. Newman JC, Des Jarlais DC, Turner CF, Gribble J, Cooley P, et al. (2002) The differential effects of face-to-face and computer interview modes. Am J Public Health 92: 294–297.
- 18. Langhaug LF, Cheung YB, Pascoe S, Hayes R, Cowan FM (2009) Difference in prevalence of common mental disorder as measured using four questionnaire delivery methods among young people in rural Zimbabwe. J Affect Disord 118: 220–223.
- 19. Gregson S, Zhuwau T, Ndlovu J, Nyamukapa CA (2002) Methods to reduce social desirability bias in sex surveys in low-development settings: experience in Zimbabwe. Sex Transm Dis 29: 568–575.
- 20. Fenton KA, Johnson AM, McManus S, Erens B (2001) Measuring sexual behaviour: methodological challenges in survey research. Sexually Transmitted Infections 77: 84–92.
- 21. Cowan FM, Pascoe SJ, Langhaug LF, Dirawo J, Chidiya S, et al. (2008) The Regai Dzive Shiri Project: a cluster randomised controlled trial to determine the effectiveness of a multi-component community-based HIV prevention intervention for rural youth in Zimbabwe–study design and baseline results. Trop Med Int Health 13: 1235–1244.
- 22. Mavhu W, Langhaug L, Manyonga B, Power R, Cowan F (2008) What is 'sex' exactly? Using cognitive interviewing to improve the validity of sexual behaviour reporting among young people in rural Zimbabwe. Cult Health Sex 10: 563–572.
- 23.
Willis GB (2004) Cognitive interviewing: a tool for improving questionnaire design: SAGE. 335 p.
- 24. Langhaug LF, Pascoe SJS, Mavhu W, Woelk G, Sherr L, et al. (2010) High prevalence of mental ill-health among adolescents living in rural Zimbabwe. Journal of Community Health 35: 355–364.
- 25. Catania JA, Gibson DR, Chitwood DD, Coates TJ (1990) Methodological problems in AIDS behavioral research: influences on measurement error and participation bias in studies of sexual behavior. Psychological Bulletin 108: 339–362.
- 26.
Klimes-Dougan B (1998) Screening for suicidal ideation in children and adolescents: methodological considerations. Journal of Adolescence 21 435–444.
- 27.
Moum T (1998) Mode of administration and interviewer effects in self-reported symptoms of anxiety and depression. Social Indicators Research 45 279–318.
- 28.
Brener ND, Billy J, Grady WR (2003) Assessment of factors affecting the validity of self-reported health-risk behaviour among adolescents: evidence from the scientific literature. Journal of Adolescent Health 33 436–457.
- 29. Kreuter F, Presser S, Tourangeau R (2008) Social desirability bias in CATI, IVR, and web surveys: the effects of mode and question sensitivity. Public Opinion Quarterly 72: 847–865.
- 30. Durant LE, Carey MP (2000) Self-administered questionnaires versus face-to-face interviews in assessing sexual behavior in young women. Archives of Sexual Behavior 29: 309–322.
- 31.
Weinhardt LS, Forsyth AD, Carey MP, Jaworski BA, Durant LE (1998) Reliability and validity of self report measures of HIV-related sexual behavior: progress since 1990 and recommendations for research and practice. Archives of Sexual Behavior 27 155–180.
- 32.
Langhaug LF, Cheung YB, Pascoe SJ, Mavhu W, Chirawu P, et al.. (2007) Comparing four questionnaire delivery methods for collection of self-reported sexual behaviour data in rural Zimbabwean youth. In Conference Proceedings of the 17th Biennial meeting of The International Society for Sexually Transmitted Diseases, Seattle, USA. Abstract O−038.