Correction
8 Nov 2013: Hsiao M, Morris SK, Bassani DG, Montgomery AL, Thakur JS, et al. (2013) Correction: Factors Associated with Physician Agreement on Verbal Autopsy of over 11500 Injury Deaths in India. PLOS ONE 8(11): 10.1371/annotation/7e655300-7920-4c75-b999-b2e32bfc2e8c. https://doi.org/10.1371/annotation/7e655300-7920-4c75-b999-b2e32bfc2e8c View correction
Figures
Abstract
Introduction
Worldwide, injuries account for 9.8% of all deaths. The majority of these deaths occur in low- and middle-income countries where vital registration systems are often inadequate. Verbal autopsy (VA) is a tool used to ascertain cause of death in such settings. Validation studies for VA using hospital diagnosed causes of death as comparisons have shown that injury deaths can be reliably diagnosed by VA. However, no study has assessed the factors that may affect physicians' abilities to code specific causes of injury death using VA.
Method/Principal Findings
This study used data from over 11 500 verbal autopsies of injury deaths from the Million Death Study (MDS) in which 6.3 million people in India were monitored from 2001–2003 for vital events. Deaths that occurred in the MDS were coded by two independent physicians. This study focused on whether physician agreement on the classification of injury deaths was affected by characteristics of the deceased and respondent. Agreement was analyzed using three primary methods: 1) kappa statistic; 2) sensitivity and specificity analysis using the final VA diagnosed category of injury death as gold standard; and 3) multivariate logistic regression using a conceptual hierarchical model. The overall agreement for all injury deaths was 77.9% with a kappa of 0.74 (99% CI 0.74–0.75). Deaths in the injury categories of “transport”, “falls”, “drowning” and “other unintentional injury” occurring outside the home were associated with greater physician agreement than those occurring at home. In contrast, self-inflicted injury deaths that occurred outside the home were associated with lower physician agreement.
Conclusions/Significance
With few exceptions, most characteristics of the deceased and the respondent did not influence physician agreement on the classification of injury deaths. Physician training and continued adaptation of the VA tool should focus on the reasons these factors influenced physician agreement.
Citation: Hsiao M, Morris SK, Bassani DG, Montgomery AL, Thakur JS, Jha P (2012) Factors Associated with Physician Agreement on Verbal Autopsy of over 11500 Injury Deaths in India. PLoS ONE 7(1): e30336. https://doi.org/10.1371/journal.pone.0030336
Editor: Thomas Behrens, University of Bochum, Germany
Received: August 30, 2011; Accepted: December 14, 2011; Published: January 17, 2012
Copyright: © 2012 Hsiao et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Worldwide, injuries account for about 10% of all deaths and about 12% of the total burden of diseases measured in disability-adjusted life years [1]. The majority of these injuries occur in low and middle-income countries [1]. In such settings with limited resources, deaths often occur outside the formal healthcare system and are not recorded in the country's vital registration system. The lack of accurate vital statistical data hampers public health action, policy making, and the implementation of evidence-based interventions.
Verbal autopsy (VA) is a cost-effective tool for ascertaining cause of death in low-resource settings with incomplete vital registration [2]. The VA relies on the assumption that each cause of death has a unique set of signs and symptoms that can be used to distinguish between different causes of death. The VA also assumes that the signs and symptoms leading up to a death can be accurately reported by the deceased's family or acquaintances during a standardized interview.
Various methodologies exist to determine a cause of death based on the signs and symptoms collected on the VA questionnaire [3]. These include data- or expert-derived algorithms [4], symptom pattern methodology [5], and probabilistic models [6], [7]. Another common methodology utilizes two or more trained physicians to review the VA form to determine a cause of death [2].
Studies using VA with physician coders have been externally validated by comparing the VA cause of death to hospital-based registries or death certificates [8]–[13]. However, in low-resource settings, this method of validation may be biased because people with access to healthcare facilities, death certificates, or death registration are often not representative of the entire population.
Injury as a broad category of death can be diagnosed with high sensitivity, specificity, and positive predictive value by VA [11], [12], [14]. However, no study has assessed the factors that may influence the ability of physicians to distinguish between different injuries using VA. We used physician agreement as a metric in our studies to measure the performance characteristics of VA. Although no inference can be made on accuracy or external validity, higher physician agreement should be correlated with better reliability of VA assigned cause of death. We previously reported that younger age and female gender were associated with lower agreement between physicians on the cause of childhood deaths in India [15]. The purpose of our current study is to investigate whether characteristics of the respondent and deceased are associated with physician agreement on the classification of injury deaths.
Methods
This study uses data from the Million Death Study (MDS); the methodology of the MDS is published in detail elsewhere [2]. In brief, the MDS monitored the vital statistics of 6.3 million people in 1.1 million nationally representative households in India from 2001–2003. A total of 6671 sampling units, each with an average of 150 households, were selected by stratified random sampling based on urban or rural setting, village or city population size, and female literacy rate according to the 1991 national census. For every death occurring in these households, a verbal autopsy (VA) questionnaire called RHIME (Routine, Reliable, Representative and Re-sampled Household Investigation of Mortality with Medical Evaluation; available at http://www.cghr.org/index.php/training/verbal-autopsy-forms-2/) was completed by a trained fieldworker based on a structured interview with a family member or close acquaintance of the deceased. RHIME includes both preset close-ended questions and an open-ended narrative. Each completed VA questionnaire was independently reviewed by two physicians trained to use the VA questionnaire to determine cause of death in World Health Organization International Classification of Disease 10 (ICD-10) code [16]. The physicians also provided a set of keywords that were important in determining the cause of death. If the two physicians initially agreed on the same cause of death, the corresponding ICD-10 code was assigned as the final cause of death. If the two physicians initially disagreed, they were required to anonymously reconcile by exchanging ICD-10 codes and keywords. If they agreed in this second reconciliation stage, the final cause of death was assigned. However, if there was still disagreement after the reconciliation stage, a third, senior physician adjudicated and assigned the final cause of death based on the VA questionnaire along with the ICD-10 codes and keywords provided by the first two physicians. A web-based system managed the coding process with more than 130,000 deaths coded by 130 physicians.
We used MDS data from 2001–2003 for this study. All deaths due to injury were grouped into 9 broad categories: transport injuries; falls; fire; drowning; venomous snakes, animals and plants; poisonings and other unintentional injuries; self-inflicted injuries; war, violence and other intentional injuries; and injuries of undetermined intent. The ICD-10 codes of these 9 injury categories are listed in Table 1, along with the number of deaths in each category stratified by age groups: children (≤14 years), adults(15 to 69 years), and elderly (≥70 years). For this study, physician agreement was defined as agreement of the first ICD-10 codes from the two physicians (i.e. the same injury category for both physicians at the first coding stage prior to the second reconciliation stage). The extent of agreement between two physicians beyond chance was assessed using the kappa statistic [17]. The 99% confidence interval for the kappa statistic was calculated using the bootstrap method for polychotomous variables with bias correction [18]. The Landis and Koch classification of inter-rater reliability was used to interpret the kappa statistic: ≤0.4 – poor to fair agreement; >0.4 to ≤0.6 – moderate agreement; >0.6 to ≤0.8 – substantial agreement; >0.8 – high agreement [17]. Analysis was conducted on the entire dataset and also stratified by age. Sensitivity and specificity of each physician's initial cause of death diagnosis was performed using the final RHIME cause of death diagnosis as the gold standard. This analysis should only be interpreted with the understanding that the gold standard (final cause of death diagnosis) was not independent of the test itself (initial physician cause of death diagnosis). The methodology used in this study to assess physician agreement and factors affecting agreement has been described in detail previously [15].
We used a multivariate logistic regression model to study the association between physician agreement and the geographic, socioeconomic, demographic, and other characteristics of the deceased and the respondent. The model was constructed using an a priori hierarchical conceptual framework that grouped variables into three blocks: distal, middle, and proximal [19]. The distal block included geographic factors (Empowered Action Group region including Assam (EAGA), geographic region, and urban or rural setting). The middle block included socio-demographic factors of the respondent (whether the respondent lived with the deceased, respondent's gender, education, religion, language, and relation to the deceased). The proximal block included characteristics of the deceased (deceased's gender, education, location of death, and whether the death was registered). Variable selection was conducted using backward elimination. A variable was retained in the final model if its likelihood ratio test resulted in p-value lower than 0.20. Respondents' education was predefined to remain in the final model regardless of p-value from the likelihood ratio test because of its expected influence on the quality of VA questionnaire. The final model for adults was also adjusted for the deceased's age as a continuous variable given the finding that increasing age was correlated linearly with lower physician agreement between injury death categories. Adjustment for age was not necessary in the children and elderly groups because the correlation between age and physician agreement within the group was not found to be significant. A factor was considered to be significantly associated with physician agreement if the likelihood ratio test resulted in p<0.05. All statistical analysis was performed using Stata SE version 10.0 [20].
Ethics approval for the MDS was obtained from the Postgraduate Institute of Medical Education and Research, Chandigarh, India and St. Michael's Hospital, Toronto, Ontario, Canada. Enrollment in the MDS was on a voluntary basis and the study posed no or minimal risks to enrolled subjects. Families were free to withdraw from the study. Verbal consent was obtained. All personal identifiers present in the raw data are anonymized before analysis. MDS data storage and usage protocols were described in detail elsewhere [2].
Results
There were 11 543 injury deaths in the MDS from 2001–2003. Of these, 11 509 deaths (99.7%) were coded by two physicians and were included in this study. The process of determining the final cause of death is shown in Figure 1.
The summary of the kappa statistic analysis, stratified by age group, is shown in Table 2. For all age groups, non-overlapping 99% confidence intervals between one or more sub-categories of a predictor variable suggest statistical significance and require multivariate logistic regression modeling to rule out confounding effects. The overall agreement for all injury deaths was 77.9% with a kappa of 0.74 (99% CI 0.74–0.75). For children (≤14 years), there was 84.1% agreement and a kappa of 0.81 (99% CI 0.79–0.82). In this group, the 99% confidence intervals for kappa did not overlap between one or more sub-categories of the following variables: whether the respondent lived with the deceased, religion, language, deceased's gender, deceased's education, location of death, and whether the death was registered. For adults (15–69 years), there was 80.9% agreement overall and a kappa of 0.77 (99% CI 0.76–0.78). In this group, the 99% confidence intervals for kappa of all the variables did not overlap, except for respondent education and death registration. For the elderly (≥70 years), there was 58.2% agreement overall and a kappa of 0.44 (99% CI 0.40–0.45). In this age group, the variables where the 99% confidence interval for kappa did not overlap were whether the respondent lived with the deceased, respondent education, respondent relationship to the deceased, religion, language, deceased gender, location of death, and EAGA.
The sensitivity and specificity of the initial physician coded cause of death was calculated using the final cause of death as gold standard (Table 3). There were 3 200 child, 16 346 adult, and 3 506 elderly initial physician coded cause of death (two codes per death). Excluding injuries of undetermined intent, the sensitivity of initial physician coding for all injury categories was above 82.0% in children, above 81.3% in adults, and above 72.7% in the elderly. Sensitivity for injuries of undetermined intent was the lowest in all three age groups – 61.8% in children, 59.7% in adults, and 53.8% in the elderly. The specificity of initial physician coding for all age groups was greater than 98.3% for all injury causes.
The results of the multivariate logistic regression on factors influencing physician agreement are summarized in Table 4. For adults, physicians were more likely to agree on the classification of injury deaths if deaths occurred at a health facility (OR 1.19, 95% CI 1.01–1.39) and at a location outside of home or health facility (OR 1.50, 95% CI 1.30–1.73) compared to deaths at home. We postulated that injury deaths that are more likely to occur outside of home or health facility, such as transport injuries and drowning, may be easier to classify and would therefore have greater physician agreement. When we stratified the analysis by injury category, death location outside of home was associated with greater physician agreement only for deaths due to transport injury, drowning, and other unintentional injury (Table 5). In adult transport injury deaths, the odds ratio for physician agreement was 5.41 (95% CI 3.18–9.22) for deaths occurring at a health facility and 5.06 (95% CI 3.23–7.92) for deaths at a location outside of home or health facility. For both drowning and other unintentional injury deaths among adults, physician agreement was greater only for deaths that occurred at a location outside of the home or health facility (drowning - OR 2.07, 95% CI 1.08–3.99; other unintentional injury - OR 1.76, 95% CI 1.21–2.56) compared to deaths that occurred at home. In this age group, there was lower physician agreement for deaths due to self-inflicted injury that occurred at locations outside of the home or health facility (OR 0.60; 95% CI 0.44–0.81) compared to self-inflicted deaths at home.
Also for adults, physicians were more likely to agree on the classification of injury deaths if the respondent did not live with the deceased (OR 1.33; 95% CI 1.18–1.51) compared to those who did live with the deceased. This association was similar in all injury categories (data not shown).
For the elderly, physician agreement was greater for deaths that occurred outside the home, whether at a health facility (OR 2.31, 95% CI 1.58–3.36) or at a location other than the home or health facility (OR 3.28, 95% CI 2.22–4.83). When stratified by injury category, greater physician agreement for elderly deaths that occurred outside the home was apparent only for transport injury, fall, and other unintentional injury deaths (Table 5). In transport injury deaths, physicians were more likely to agree for deaths that occurred at a health facility (OR 4.92, 95% CI 1.55–15.69) and at a location outside of the home or health facility (OR 4.18, 95% CI 1.51–11.58) compared to deaths that occurred at the home. Similarly in elderly deaths due to falls, physicians were more likely to agree for deaths that occurred at a health facility (OR 3.12, 95% CI 1.73–5.63) and at a location outside of the home or health facility (OR 2.42, 95% CI 1.13–5.15) compared to at the home. For other unintentional injury deaths in this age group, physician agreement was greater only for deaths that occurred outside of the home or health facility (OR 5.96, 95% CI 1.90–18.69) compared to at the home. Physicians were also more likely to agree on cause of deaths among the elderly if the respondent was a neighbor rather than a relative (OR 1.52, 95% CI 1.03–2.23). This association was similar among all injury categories.
For children, the only variable significantly associated with physician agreement was the location of death. Physician agreement was more common for deaths that occurred outside of the home or health facility (OR 2.18, 95% CI 1.40–3.40), regardless of the injury category.
Discussion
The purpose of this analysis was to assess factors that may influence physicians' ability to correctly classify injury deaths based on VA questionnaires. We used physician agreement as the metric in our analysis under the assumption that poor agreement is correlated with difficulty in classifying injury deaths. Several validation studies have shown that VA tools perform well for injury deaths as one broad category with high sensitivity, specificity, and positive predictive value when compared to hospital records and death certificates [11], [12], [14]. However, to the best of our knowledge, no studies have assessed the factors that influence physician's ability to further classify injury deaths into specific categories using VA. From a public health injury prevention perspective, differentiating between injury categories is essential to priority setting and formulating targeted interventions. Using kappa statistic, sensitivity and specificity analysis, and multivariate logistic regression, this study specifically assessed factors that may influence physician agreement in injury deaths.
Using childhood deaths (age ≤14) of all causes, we previously reported that physician agreement was not affected by features of the death itself or by most geographic, socioeconomic, or demographic characteristics of the respondent and/or deceased. The exceptions were with the gender and age of the deceased [15]. This study of adult and child injury deaths also suggests that, with few exceptions, physician agreement on category of injury death was not affected by these factors. Specifically, we did not find consistent significant differences in physician agreement based on geographic factors, respondents' gender, education, religion, language, or on the gender, education, and death registration of the deceased. The similar level of agreement across all these variables is reassuring to all VA based studies.
Injury deaths are prevalent in the elderly population in India (manuscript in preparation). While the ability of VA to yield a broad classification of the underlying cause of death diminishes for deaths that occur over age 70 [2], we have shown that, even in the elderly population, VA performed reasonably well in distinguishing between different injury categories.
Location of Death
For adult and elderly deaths in the categories of transport injury, other unintentional injury, fall (elderly only), and drowning (adults only), the likelihood of physician agreement increased if the location of death was outside of home. This association suggests that location of death provides an important clue for physicians in determining the cause of death for these four injury categories. One hypothesis is that for these categories, deaths occurring outside of home may be due to injuries of higher severity and acuity, resulting in a more immediate death that may be easier to classify. On the contrary, deaths occurring at home may have longer time lag between injury and death (for example, an elderly person becomes injured in a fall or car accident, undergoes treatment in hospital for several months, is eventually discharged and dies at home shortly after). When there is a longer lag time and opportunity for more intervening events between initial injury and death, as we hypothesize for injury deaths at home, physicians may have more difficulty determining cause of death. In contrast to these four categories, however, deaths due to self-inflicted injury were associated with lower physician agreement if the death occurred outside of home or health facility. One possible explanation for this finding is that the circumstances around a self-inflicted injury death is likely not as well known or reported by the respondent if the death occurred outside of home or health facility. On the other hand, detailed information on self-inflicted deaths may be more available if the death occurred at home or at health facilities. Additional studies exploring the open-ended narrative section of the VA may provide support for these hypotheses and generate additional explanations for the association between location of death and physician agreement.
In children, the association between increased physician agreement and death outside of home or health facility was also present. However, due to the smaller sample size we were unable to identify associations with particular injury categories. We believe that the association in children may be explained by the same rationale as adults, but the association will need to be assessed with a larger sample.
Whether Respondent Lived with Deceased & Respondent Relationship
For adult deaths, physicians were more likely to agree on the classification of injury deaths if the respondent did not live with the deceased. Similarly for elderly deaths, physicians were more likely to agree if the respondent was a neighbour rather than a relative. We believe that this association may be reflective of the less detailed narratives given by the respondents who did not live with the deceased. For injury deaths, paradoxically, physicians may find narratives with fewer details easier to determine the cause of death. The less detailed narratives often comprise of only a few key words indicative of the cause of death such as “traffic accident” or “fall”. Detailed narratives, however, may describe multiple events during a longer intervening time gap between injury and death that, as hypothesized above for deaths occurring at home, may make determining cause of death more difficult.
Limitations
Although we were able to assess the association between physician agreement and various characteristics of the respondent and deceased, we did not analyze the impact of factors related to the trained physician coders. Our analysis was performed under the assumption that the clinical experience, knowledge, medical specialty, and other factors that may affect the physician's ability to determine the cause of death based on VA were uniform among all physician coders. This assumption was likely not true. Nevertheless, we believe it is unlikely that this limitation would have led to a systematic bias in the results as the deaths were randomly assigned to the physician coders. Future studies should be performed to assess physician characteristics that affect their ability to determine cause of death using a VA instrument.
The sensitivity and specificity analysis of the cause of death determined in the first stage by physicians must be interpreted with the understanding that the gold standard used (RHIME-determined cause of death) was not independent from the initial physician assigned cause of death. Nonetheless, this analysis uncovered that physicians have more difficulty arriving at a diagnosis for deaths caused by injury of undetermined intent. Further interviewer and physician training to improve both the gathering and identification of details to arrive at a more specific category of injury death may help improve the VA tool.
The degree of physician agreement is contingent on the number of injury categories used in the analysis. Fewer categories would result in higher physician agreement and vice versa. We decided on the 9 injury categories based on a public health injury prevention perspective. Ideally, VA should be able to distinguish between injury categories such that the final output can be informative for public health priority setting, strategic planning, and disease monitoring. A limitation to our categorization is that no inference can be made on physician ability to diagnose specific causes of injury death within each injury category. Using transport injury deaths as example, the study cannot comment on whether VA is specific enough to determine the deceased's mode of transportation and the mechanism of collision (i.e. three character ICD-10 “V” codes). This limitation is even more important for the more heterogeneous injury categories of “other unintentional injuries” and “injuries of undetermined intent”.
In summary, physician agreement on the injury category cause of death was not affected by most characteristics of the deceased or respondent, with the exceptions of location of death, respondent relationship, and whether the respondent lived with the deceased. Specifically, for transport injury, fall, drowning, and other unintentional injury deaths, the location of death outside of the home was associated with greater physician agreement. In contrast, self-inflicted injury deaths that occurred outside of the home were associated with lower physician agreement. In addition, physicians were more likely to be in agreement if the respondent did not live with the deceased or if the respondent was a neighbour instead of a relative. Recognition of these factors that influence the physician's ability to determine cause of injury death is essential for continued adaptation and improvement of the VA tool.
Acknowledgments
We thank Brendon Pezzack for help with graphics and Ansely Wong for editing support.
Author Contributions
Conceived and designed the experiments: MH SKM DGB ALM JST PJ. Performed the experiments: MH SKM DGB. Analyzed the data: MH SKM DGB ALM JST PJ. Contributed reagents/materials/analysis tools: MH SKM DGB ALM JST PJ. Wrote the paper: MH SKM DGB ALM JST PJ.
References
- 1.
World Health Organization, editor (2008) The global burden of disease: 2004 update.
- 2. Jha P, Gajalakshmi V, Gupta PC, Kumar R, Mony P, et al. (2006) Prospective study of one million deaths in India: Rationale, design, and validation results. PLoS Med 3(2): e18.
- 3. Soleman N, Chandramohan D, Shibuya K (2006) Verbal autopsy: Current practices and challenges. Bull World Health Organ 84(3): 239–245.
- 4. Quigley MA, Chandramohan D, Rodrigues LC (1999) Diagnostic accuracy of physician review, expert algorithms and data-derived algorithms in adult verbal autopsies. Int J Epidemiol 28(6): 1081–1087.
- 5. Murray CJ, Lopez AD, Feehan DM, Peter ST, Yang G (2007) Validation of the symptom pattern method for analyzing verbal autopsy data. PLoS Med 4(11): e327.
- 6. Byass P, Huong DL, Minh HV (2003) A probabilistic approach to interpreting verbal autopsies: Methodology and preliminary validation in Vietnam. Scand J Public Health Suppl 62: 32–37.
- 7. Byass P, Fottrell E, Dao LH, Berhane Y, Corrah T, et al. (2006) Refining a probabilistic model for interpreting verbal autopsy data. Scand J Public Health 34(1): 26–31.
- 8. Gajalakshmi V, Peto R (2004) Verbal autopsy of 80,000 adult deaths in Tamilnadu, south India. BMC Public Health 4: 47.
- 9. Gajalakshmi V, Peto R, Kanaka S, Balasubramanian S (2002) Verbal autopsy of 48 000 adult deaths attributable to medical causes in Chennai (formerly Madras), India. BMC Public Health 2: 7.
- 10. Kahn K, Tollman SM, Garenne M, Gear JS (2000) Validation and application of verbal autopsies in a rural area of South Africa. Trop Med Int Health 5(11): 824–831.
- 11. Khademi H, Etemadi A, Kamangar F, Nouraie M, Shakeri R, et al. (2010) Verbal autopsy: Reliability and validity estimates for causes of death in the Golestan cohort study in Iran. PLoS One 5(6): e11183.
- 12. Setel PW, Whiting DR, Hemed Y, Chandramohan D, Wolfson LJ, et al. (2006) Validity of verbal autopsy procedures for determining cause of death in Tanzania. Trop Med Int Health 11(5): 681–696.
- 13. Chandramohan D, Maude GH, Rodrigues LC, Hayes RJ (1998) Verbal autopsies for adult deaths: Their development and validation in a multicentre study. Trop Med Int Health 3(6): 436–446.
- 14. Yang G, Rao C, Ma J, Wang L, Wan X, et al. (2006) Validation of verbal autopsy procedures for adult deaths in China. Int J Epidemiol 35(3): 741–748.
- 15. Morris SK, Bassani DG, Kumar R, Awasthi S, Paul VK, et al. (2010) Factors associated with physician agreement on verbal autopsy of over 27000 childhood deaths in India. PLoS One 5(3): e9583.
- 16.
World Health Organization, editor (2007) International statistical classification of diseases and related health problems, 10th revision.
- 17. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1): 159–174.
- 18. Reichenheim ME (2004) Confidence intervals for the kappa statistic. Stata Journal 4(4): 421–428.
- 19. Victora CG, Huttly SR, Fuchs SC, Olinto MT (1997) The role of conceptual frameworks in epidemiological analysis: A hierarchical approach. Int J Epidemiol 26(1): 224–227.
- 20.
StataCorp (2007) Stata statistical software: Release 10.