Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Does Consideration and Assessment of Effects on Health Equity Affect the Conclusions of Systematic Reviews? A Methodology Study

  • Vivian Welch ,

    Vivian.welch@uottawa.ca

    Affiliations Clinical Epidemiology Unit, Ottawa Hospital Research Institute, Ottawa Hospital, Ottawa, Ontario, Canada, Centre for Global Health, Institute of Population Health, University of Ottawa, Ottawa, Ontario, Canada

  • Mark Petticrew,

    Affiliation London School of Hygiene and Tropical Medicine, London, United Kingdom

  • Erin Ueffing,

    Affiliation Campbell and Cochrane Equity Methods Group, Centre for Global Health, Institute of Population Health, University of Ottawa, Ottawa, Ontario, Canada

  • Maria Benkhalti Jandu,

    Affiliation Centre for Global Health, Institute of Population Health, University of Ottawa, Ottawa, Ontario, Canada

  • Kevin Brand,

    Affiliation Telfer School of Management, University of Ottawa, Ottawa, Ontario, Canada

  • Bharbhoor Dhaliwal,

    Affiliation Centre for Global Health, Institute of Population Health, University of Ottawa, Ottawa, Ontario, Canada

  • Elizabeth Kristjansson,

    Affiliation School of Psychology, University of Ottawa, Ottawa, Ontario, Canada

  • Janet Smylie,

    Affiliations Centre for Research on Inner City Health, Keenan Research Centre of the Li Ka Shing Knowledge Institute, St. Michael's Hospital, Toronto, Ontario, Canada, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada

  • George Anthony Wells,

    Affiliations Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa, Ontario, Canada, University of Ottawa Heart Institute, Ottawa, Ontario, Canada

  • Peter Tugwell

    Affiliations Clinical Epidemiology Unit, Ottawa Hospital Research Institute, Ottawa Hospital, Ottawa, Ontario, Canada, Centre for Global Health, Institute of Population Health, University of Ottawa, Ottawa, Ontario, Canada, Department of Medicine, University of Ottawa, Ottawa, Ontario, Canada

Abstract

Introduction

Tackling health inequities both within and between countries remains high on the agenda of international organizations including the World Health Organization and local, regional and national governments. Systematic reviews can be a useful tool to assess effects on equity in health status because they include studies conducted in a variety of settings and populations. This study aims to describe the extent to which the impacts of health interventions on equity in health status are considered in systematic reviews, describe methods used, and assess the implications of their equity related findings for policy, practice and research.

Methods

We conducted a methodology study of equity assessment in systematic reviews. Two independent reviewers extracted information on the reporting and analysis of impacts of health interventions on equity in health status in a group of 300 systematic reviews collected from all systematic reviews indexed in one month of MEDLINE, using a pre-tested data collection form. Any differences in data extraction were resolved by discussion.

Results

Of the 300 systematic reviews, 224 assessed the effectiveness of interventions on health outcomes. Of these 224 reviews, 29 systematic reviews assessed effects on equity in health status using subgroup analysis or targeted analyses of vulnerable populations. Of these, seven conducted subgroup analyses related to health equity which were reported in insufficient detail to judge their credibility. Of these 29 reviews, 18 described implications for policy and practice based on assessment of effects on health equity.

Conclusion

The quality and completeness of reporting should be enhanced as a priority, because without this policymakers and practitioners will continue lack the evidence base they need to inform decision-making about health inequity. Furthermore, there is a need to develop methods to systematically consider impacts on equity in health status that is currently lacking in systematic reviews.

Introduction

Health inequities have been defined as unfair and avoidable inequalities in health across socioeconomic, demographic and geographic factors [1]. Specific population groups have been identified as “vulnerable” to health inequities. This vulnerability in turn is linked to social systems that differentially distribute resources across sociodemographic strata such as gender, socioeconomic position, and ethnicity. Health inequity exists both within and between countries. For example, under five childhood mortality is less than 6 per 1000 births in industrialized countries compared to 160 per 1000 in sub-Saharan Africa [2]. Within low and middle income countries (LMIC), under-five mortality is a median of two times higher in the poorest people compared to the highest wealth quintile. Within high income countries, health inequalities also exist. For example, Aboriginal people in Canada have 5 years lower life expectancy than non-Aboriginals [3].

We have shown with the Equity Effectiveness Loop framework that the effectiveness of clinical and public health interventions may be reduced by up to two thirds in vulnerable populations. This is due to a “staircase” effect due to lower coverage, worse screening, poor provider compliance and lower consumer adherence [4]. Because of this staircase effect, interventions may have the unintended adverse effect of increasing inequity in health status.

Decision-makers are increasingly under public pressure to consider the effects of programs and policies on health equity [5], [6]. There is increasing acceptance that systematic reviews can inform policy and practice since they reduce the chance of being misled, increase confidence in results, are an efficient use of time and are more easy to critically appraise and apply [7], [8]. Systematic reviews also represent an opportunity to identify what works to reduce health inequity [5]. However, decision-makers cite lack of evidence on impacts of health interventions on equity as a major barrier to using systematic reviews for evidence-informed decision-making [9], [10].

We have a moral obligation to ensure that clinical and public health interventions help the disadvantaged, but all too often interventions justified in the name of the poor benefit the most advantaged people, not the poorest [11]. NICE in the UK has taken a leadership role in insisting that equity evidence is needed to inform decisions about the likely impacts of clinical and public health interventions in different population groups. Specifically, the 2009 NICE Public Health methods guidance asks “How does effectiveness and cost-effectiveness vary according to the age, gender, class and ethnicity of the target audience? Is there any differential impact on inequalities in health within and between different population groups?” [2]. Furthermore, the World Health Organization Commission on Social Determinants of Health and Campbell and Cochrane Equity methods group have proposed that more attention on health equity in systematic reviews could increase relevance for policy and clinical practice [12].

Despite this need to assess the evidence on impacts of health interventions on equity in health status, systematic reviews rarely assess whether interventions have an impact on health equity[13]. Furthermore, our Cochrane review of methods found that none of the included studies had assessed the credibility of subgroup analyses nor the importance of equity assessment for the implications for practice and policy [14]. Lack of credibility and failure to discuss implications are a substantial barrier in using systematic reviews for policy and practice decisions.

This methodology study aimed to describe the extent to which impacts of health interventions on equity in health status are considered in systematic reviews, describe methods used, assess their credibility and assess the implications of their equity-related findings for policy, practice and research.

The specific objectives were to: 1) Evaluate definitions of health equity in systematic reviews; 2) Assess methods used by systematic reviews to assess impacts of health interventions on equity in health status; 3) Assess subgroup analyses according to seven credibility criteria; and 4) Assess implications of equity findings on conclusions.

Methods

Ethical approval was not required for this study.

Definition of health equity

As above, health equity is defined as unfair and avoidable health differences [1]. The acronym PROGRESS-Plus defines sociodemographic factors across which differences in effectiveness of interventions could be considered inequitable [15], [16]: Place of residence; Race/ethnicity/culture; Occupation; Gender/sex; Religion; Education; Socioeconomic status; Social capital. The “Plus” was proposed to promote the inclusion of additional factors across which disadvantage may exist due to discrimination and other reasons such as age, sexual orientation and disability [17].

Differences in health status outcomes across PROGRESS-Plus were considered differences in equity in health status if they were classified as inequitable or unfair by the authors of the systematic review, or they met the Whitehead criteria for inequity in health [1].

Study design

This study is designed as a methodology study, defined by the Cochrane Methodology Review group as: “a study of the methods used in randomized trials, other healthcare evaluations or systematic reviews. Data for methodology studies can come from clinical studies, such as randomized trials, epidemiological studies, from participants in a new study, or from systematic reviews of clinical studies” [18].

This study assessed the methods used in systematic reviews of effectiveness to consider effects on health equity.

Data source

We selected all systematic reviews indexed on MEDLINE in one month (November 2004) because the characteristics of these systematic reviews were already well described by Moher et al [19], and we wanted a group of greater than 100 systematic reviews with a diversity of health conditions and interventions since we expected less than 10% of systematic reviews would assess effects on health equity. We previously found that there was no increase in equity analyses over time in systematic reviews between 2004 and 2008 (unpublished data).

This group of systematic reviews was assembled using Montori's empirical search terms for high sensitivity (>98%) in retrieval of systematic reviews [20]. The search was run on February 18, 2005, with the terms: (1) 200411$.ed; (2) limit 1 to English (3) 2 and (cochrane database of systematic reviews.jn. or search.tw. or metaanalysis.pt. or medline.tw. or systematic review.tw. or ((metaanalysis.mp,pt. or review.pt. or search$.tw.) and methods.ab.)). Two reviewers screened the titles and abstracts, then a single reviewer screened the full-text to identify articles which met the following definition for a systematic review: “the authors stated objective was to summarize evidence from multiple studies and the article described explicit methods” [19]. A second reviewer independently screened a random sample of 10% of the full-text reviews. The search was limited to English-language due to resource implications of including non-English articles [19]. There is no evidence from two previous studies that non-English language papers are likely to assess impacts on equity in health status differently or more frequently [13], [21]. The search retrieved 300 systematic reviews.

For the purpose of this methodology study on health equity, systematic reviews were included if the stated purpose was to assess the effects of an intervention on health outcomes. Adhering to this criterion resulted in the exclusion of 76 of the 300 systematic reviews. Reasons for exclusion fell into two main categories. Either the study did not assess the effectiveness of an intervention on health outcomes or the study was not concerned with health outcomes (e.g. focused on literacy instead of health).

Data extraction for this study

Two reviewers (two of BD, EU, VW, MBJ) independently extracted data on reporting and analysis of differences in effectiveness across PROGRESS-Plus factors, using a pre-tested data extraction form. This data extraction form included 51 items on the characteristics of the population, intervention, comparison (setting), outcomes and study design (i.e. randomized controlled trials, observational studies or both), whether health inequalities or health inequities were described and how they were defined. Subgroup analysis across PROGRESS-Plus was assessed, using the definition in the Cochrane Handbook: “Subgroup analyses involve splitting all the participant data into subgroups, often so as to make comparisons between them. Subgroup analyses may be done for subsets of participants (such as males and females), or for subsets of studies (such as different geographical locations)” [22].

Extraction by two independent extractors was compared and differences were resolved by discussion. There were few discrepancies between data extractors.

Objective 1.

We assessed whether health equity or health inequalities were defined and described in the systematic reviews. We looked for any description of how health equity was defined such as whether differences were avoidable or unfair, whether proxy measures were used (such as receipt of health insurance for the poor) and how the judgment of fairness or avoidability was made. We expected such definitions only for those reviews where the review intended to assess effects on health equity across one or more PROGRESS-Plus factor.

Objective 2.

We assessed the methods used by systematic reviews to assess impacts of health interventions on equity in health status

Objective 3.

Subgroup analyses conducted across PROGRESS-Plus factors were assessed according to the seven ‘credibility’ criteria for subgroup analysis, proposed by Oxman and Guyatt [23], that are in the Cochrane Handbook of Systematic Reviews [22]. These credibility criteria are intended to minimize the over-interpretation of spurious differences: 1. Clinically important difference? 2. Statistically significant difference? 3. A priori hypothesis? 4. Subgroup analysis one of small number of hypotheses tested? 5. Difference suggested by comparisons within studies? 6. Difference consistent across studies? 7. Indirect evidence that supports hypothesized difference?

Objective 4.

We assessed whether differences in equity in health status across PROGRESS-PLUS factors were described in the discussion section and the implications for research and practice.

Results

Characteristics of included systematic reviews

Of the group of 300 systematic reviews, 224 were classified as assessing the effects of an intervention on health outcomes (web appendix S1). Of the 76 systematic reviews that were excluded, 16 assessed test characteristics of diagnostic methods, 21 conducted systematic reviews of research methods (e.g. quality assessment) and 39 assessed the association of patients' characteristics with outcomes.

Of the 224 included systematic reviews, 153 (68%) described the characteristics of the populations in the primary studies across one or more PROGRESS-Plus characteristics as follows: gender/sex (49%), age (47%), place of residence (22%), and LMIC setting (9%) were most frequently reported (Table 1), followed by race/ethnicity/culture (4%), socioeconomic status (3%), occupation (1%), education (1%) and social capital (1%).

thumbnail
Table 1. PROGRESS-Plus factors described in 224 systematic reviews.

https://doi.org/10.1371/journal.pone.0031360.t001

Objective 1: Definition of health inequalities

No reviews explicitly mentioned health equity. A small proportion (18/224) did describe in the background or discussion section that one or more PROGRESS-Plus factors were hypothesized to modify the impact of interventions. These 18 systematic reviews stated that health status differences across PROGRESS-Plus factors have been hypothesized elsewhere to affect the effectiveness of interventions, but none of these systematic reviews attached the term “health inequity” to these differences, nor hinted at this concept by describing these differences in health as unfair and avoidable.

Objective 2: Methods used to assess differences in effects across PROGRESS-Plus equity determining variables

Of 224 systematic reviews, 29 assessed differences in effects across PROGRESS-Plus variables, using explicit methods of subgroup analysis (n = 15) or targeted analysis of vulnerable populations (n = 14) (Web appendix S2). None of the reviews used a gradient approach, defined as assessing differences in health across the whole range of a measure of disadvantage such as across income levels or grades of socioeconomic position, thus recognizing a systematic pattern of worsening health outcomes with greater disadvantage [24] Nine of these 29 systematic reviews described health inequalities which were hypothesized to influence the effectiveness of interventions.

Of these 29 systematic reviews, 24 found differences in effects of interventions on health or social outcomes across PROGRESS-Plus variables.

Objective 3: Assess subgroup analyses according to seven credibility criteria for subgroup analyses

Subgroup analyses were classified into two types: 1) pooled, and 2) description of within-study differences. Seven systematic reviews compared pooled results from more than one study using statistical methods. Eight systematic reviews described differences in effects across one or more PROGRESS-Plus factor within individual studies, without combining data.

a) Pooled results (n = 7 systematic reviews).

Seven systematic reviews assessed impacts of health interventions on equity in health status using subgroup analysis. These subgroup analyses met a median of three out of the seven credibility criteria (min-max 2–5) (Table 2). These seven systematic reviews assessed differences in relative effects (n = 2) and absolute effects (n = 5). Only one of these seven systematic reviews found a statistically significant difference; between studies of educational interventions which reported the gender ratio and those that did not [25]. According to the authors of the systematic review, this result may be related to lack of reporting rather than true differences between boys and girls [25].

thumbnail
Table 2. Characteristics of subgroup analyses in systematic reviews which combined data from different studies across PROGRESS-Plus factors, according to Oxman and Guyatt credibility criteria for subgroup analyses.

https://doi.org/10.1371/journal.pone.0031360.t002

b) Description of within-study differences without pooling (n = 8).

Eight systematic reviews described differences in effects across PROGRESS-Plus characteristics observed within individual studies (Web Appendix S2). These systematic reviews described the effects within individual studies across gender/sex (n = 7), socioeconomic status (n = 3), education (n = 1), occupation (n = 1), religion (n = 1) and age (n = 2).

Objective 4: Influence of considering health equity on conclusions

Of those 29 reviews which evaluated effects on equity in health status, 18 (62%) described implications for policy, practice or research (Web Appendix S2).

Discussion

This study of 224 systematic reviews is the largest study of equity assessment in a random sample of systematic reviews, and it is the first study to assess the reporting of subgroup analyses related to vulnerable populations using explicit criteria and to assess the influence of equity analyses on the implications and conclusions of reviews. This study found that subgroup analyses were not reported in sufficient detail to judge their credibility. Secondly, this study found that equity results influenced the implications for two thirds of reviews.

We found that 13% of systematic reviews assessed effects of interventions in vulnerable populations using either subgroup analysis or targeted approaches, and 68% of systematic reviews described the population according to one or more PROGRESS-Plus factor. We propose that description of populations for some characteristics of PROGRESS-Plus, such as age and sex, is necessary to adhere to reporting standards and does not imply an equity perspective. However, some characteristics such as socioeconomic status and low and middle income country setting may indeed suggest an equity perspective is taken, and this would increase the proportion of systematic reviews with an equity perspective to higher than 13%.

None of the systematic reviews assessed whether effects were different across a gradient of disadvantage. None of the systematic reviews used the term “health equity” or described differences in effects as unfair. This is not necessarily undesirable and may reflect uncertainty about the normative judgment of fairness in different settings. For those systematic reviews that conducted subgroup analysis, the methods for reporting them met only three out of seven criteria for assigning credibility to a subgroup analysis [23].

This study has a number of strengths. We performed double extraction with verification and comparison of results and a pre-tested extraction form developed using published tools and guidance. This random sample of systematic reviews was assembled with transparent search criteria. The reporting characteristics of these systematic reviews are given elsewhere [19]. We identified that there was no time trend in planned or conducted subgroup analyses across PROGRESS-Plus factors from 2004 to 2008, thus justifying the use of this sample from November 2004.

The weaknesses of this study are that we did not assess the availability of data on effects in vulnerable populations from the primary studies. Three other studies have found that approximately 10% of primary studies report data disaggregated by one or more of the PROGRESS-Plus factors [13], [21], [26]. We addressed this weakness by explicitly assessing whether systematic reviews described availability of data in primary studies. Another limitation is that this methodology study was limited to systematic reviews indexed in MEDLINE. Since we aimed to assess effects of health interventions, this was an appropriate sample for our study. However, systematic reviews in non-medical databases such as Campbell Collaboration reviews may be more likely to assess differences in effects across socioeconomic status because they may focus on more upstream interventions [27], [28]. This is unlikely a serious limitation, and very unlikely to affect the conclusions of this study. Another limitation is that we excluded non-English systematic reviews. In two previous methods study, non-English studies were not qualitatively or quantitatively different from English studies in how they considered health equity or reported results [13], [21]. Thus, it is unlikely that this exclusion would affect the conclusions of this study.

We did not assess whether subgroup analyses that were planned were not reported or not conducted due to insufficient data or non-statistically significant results or other factors. Thus, there may be more equity subgroup analyses conducted that are not statistically significant and would not have an influence on the conclusions. In a prior study, we found that approximately half of planned subgroup analyses were not reported in the results (unpublished data).

Conclusions

Our findings suggest that there is inadequate consideration of effects on health equity in systematic reviews. We propose that the decision about whether to assess health equity needs to be informed by the theory underlying the intervention, and that this needs to be described in systematic reviews. Subgroup analyses are only one way that effects on health equity can be assessed and they are subject to spurious results and may be misleading. Applicability and targeted approaches are also useful for considering effects on health equity. The quality and completeness of reporting should be enhanced as a priority, because without this policymakers and practitioners will continue lack the evidence base they need to inform decision-making about inequity. However it is not only about quality of reporting, it is about the fundamental need to find a way to systematically consider impacts on equity in health status in a way that is currently missing from systematic review processes. The Campbell and Cochrane Equity Methods Group is developing data-driven guidance on these methods for systematic review authors and users.

Supporting Information

Web Appendix S2.

Systematic reviews (n = 29) which assessed differences across PROGRESS-Plus by: 1) subgroup analysis; 2) targeted; or 3) description of individual studies.

https://doi.org/10.1371/journal.pone.0031360.s002

(DOCX)

Acknowledgments

We are grateful to David Moher and Jennifer Tetzlaff for providing access to this group of systematic reviews. Colin McCann helped with developing the reference list of included studies. The first author is grateful for moral support from Jordi Pardo Pardo.

Author Contributions

Conceived and designed the experiments: VW PT MP EK JS KB GW. Performed the experiments: VW EU BD MBJ. Analyzed the data: VW EU BD MBJ. Wrote the paper: VW MP EK JS KB GW EU BD MBJ.

References

  1. 1. Whitehead M (1992) The Concepts and Principles of Equity and Health. International Journal of Health Services 22: 429–445.
  2. 2. UNICEF (2004) Progress for Children: a child survival report card.
  3. 3. Tjepkema M, Wilkins R, Senecal S, Guimond E, Penney C (2010) Mortality of urban Aboriginal adults in Canada, 1991–2001. Chronic Dis Can 31: 4–21.
  4. 4. Tugwell P, de SD, Hawker G, Robinson V (2006) Applying clinical epidemiological methods to health equity: the equity effectiveness loop. BMJ 332: 358–361.
  5. 5. Mackenbach JP (2003) Tackling inequalities in health: the need for building a systematic evidence base. Journal of Epidemiology & Community Health 57: 162.
  6. 6. Marmot M, Commission on Social Determinants of Health (2007) Achieving health equity: from root causes to fair outcomes. Lancet 370: 1153–1163.
  7. 7. World Health Assembly (2005) 5834 p. Resolution 5834 on the Ministerial Summit on Health Research.
  8. 8. Lavis JN, Davies HTO, Gruen RL (2006) Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthcare Policy 1: 21–33.
  9. 9. Petticrew M, Whitehead M, Macintye SJ, Graham H, Egan M (2004) Evidence for public health policy on inequalities: 1: the reality according to policymakers. J Epidemiol Community Health 58: 811–816.
  10. 10. Lavis JN, Davies HTO, Gruen RL (2006) Working within and beyond the Cochrane Collaboration to make systematic reviews more useful to healthcare managers and policy makers. Healthcare Policy 1: 21–33.
  11. 11. Gwatkin DR, Ergo A (2010) Universal health coverage: friend or foe of health equity? The Lancet epub ahead of print. DOI: 10.1016/S0140-6736(10)62058-2.
  12. 12. Tugwell P, Petticrew M, Kristjansson E, Welch V, Ueffing E, et al. (2010) Assessing equity in systematic reviews: realising the recommendations of the Commission on Social Determinants of Health. BMJ 341:
  13. 13. Tsikata S, Robinson V, Petticrew M, Kristjansson B, Moher D, et al. (2003) Do Cochrane systematic reviews contain useful information about health equity? [abstract]. XI Cochrane Colloquium: Evidence, Health Care and Culture; 2003 Oct 26 31; Barcelona, Spain 77.
  14. 14. Welch V, Tugwell P, Petticrew M, deMontigny J, Ueffing E, et al. (2010) How effects on health equity are assessed in systematic reviews of interventions. Cochrane Database of Systematic Reviews Issue 12:
  15. 15. Kavanagh J, Oliver S, Harden A, Gray J, Oliver K, et al. (2007) Health promotion, public health and the health of young people: a systematic map of inequalities research [abstract]. XV Cochrane Colloquium; 2007 Oct 23–27; Sao Paulo, Brazil 115–116.
  16. 16. Evans T, Brown H (2003) Road traffic crashes: operationalizing equity in the context of health sector reform. Injury Control & Safety Promotion 10: 11–12.
  17. 17. Kavanagh J, Oliver S, Lorenc T (2008) Reflections in developing and using PROGRESS-Plus. Equity Update 1–3.
  18. 18. The Editorial Team (2011) Cochrane Methodology Review Group.
  19. 19. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG (2007) Epidemiology and reporting characteristics of systematic reviews. PLoS Med 4: e78.
  20. 20. Montori VM, Wilczynski NL, Morgan D, Haynes RB, Hedges team (2005) Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. British Medical Journal 330: 1162–1163.
  21. 21. Tugwell P, Maxwell L, Welch V, Kristjansson E, Petticrew M, et al. (2008) Is health equity considered in systematic reviews of the Cochrane Musculoskeletal Group? Arthritis Care and Research 59: 1603–1610.
  22. 22. Higgins JPT, Green S, editors. (2009) Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.2 [updated September 2009]. The Cochrane Collaboration.
  23. 23. Oxman AD, Guyatt GH (1992) A consumer's guide to subgroup analyses. Ann Intern Med 116: 78–84.
  24. 24. Bonnefoy J, Morgan A, Kelly MP, Butt J, Bergman V (2007) pp. 1–337. Constructing the evidence base on the social determinants of health: A guide.
  25. 25. Swanson HL (1999) Reading research for students with LD: a meta-analysis of intervention outcomes. Journal of learning disabilities 32: 504–532.
  26. 26. Doull M, Runnels VE, Tudiver S, Boscoe M (2010) Appraising the Evidence: Applying Sex- and Gender-Based Analysis (SGBA) to Cochrane Systematic Reviews on Cardiovascular Diseases. Journal of Women's Health 19: 997–1003.
  27. 27. Shadish W, Myers D (2004) Campbell Collaboration Research Design Policy Brief. Campbell Collaboration. Available: http://www.campbellcollaboration.org/artman2/uploads/1/Research_Design_Policy_Brief.pdf.
  28. 28. Wilson SJ, Lipsey M (2006) The effects of school-based social information processing interventions on aggressive behavior, part I: Universal Programs. Campbell Systematic Reviews 5: DOI:https://doi.org/10.4073/csr.2006.5.
  29. 29. Wendel-Vos GC, Schuit AJ, Feskens EJ, Boshuizen HC, Verschuren WM, et al. (2004) Physical activity and stroke. A meta-analysis of observational data. Int J Epidemiol 33: 787–798.
  30. 30. Conaty S, Watson L, Dinnes J, Waugh N (2004) The effectiveness of pneumococcal polysaccharide vaccines in adults: a systematic review of observational studies and comparison with results from randomised controlled trials. Vaccine 22: 3214–3224.
  31. 31. Matthews SJ, McCoy C (2004) Peginterferon alfa-2a: a review of approved and investigational uses. [Review] [149 refs]. Clinical Therapeutics 26: 991–1025.
  32. 32. Twetman S, Petersson L, Axelsson S, Dahlgren H, Holm AK, et al. (2004) Caries-preventive effect of sodium fluoride mouthrinses: a systematic review of controlled clinical trials. [Review] [79 refs]. Acta Odontologica Scandinavica 62: 223–230.
  33. 33. Ramakrishnan U, Aburto N, McCabe G, Martorell R (2004) Multimicronutrient interventions but not vitamin a or iron interventions alone improve child growth: results of 3 meta-analyses. [Review] [80 refs]. Journal of Nutrition 134: 2592–2602.
  34. 34. Swanson HL, Sachse-Lee C (2000) A meta-analysis of single-subject-design intervention research for students with LD. Journal of learning disabilities 33: 114–136.