Figures
Abstract
Background
Health Technology Assessments (HTAs) are used to inform decision-making and their usefulness depends on the quality and relevance of research and specific studies for health-policy decisions. Little is known about the country of origin of studies used for HTAs.
Objective
To investigate which countries have made the largest contributions to inform health policy decisions through studies included in HTAs in Germany.
Methods
The country of origin was extracted from all studies included in HTAs of the German Institute for Quality and Efficiency in Health Care, (IQWiG), published from 2/2006 to 9/2010. Studies were ranked according to the total number of studies per country, adjusted for population size, gross domestic product (GDP), and total health expenditure.
Results
1087 studies were included in 54 HTA reports. Studies were assigned to 45 countries. Most of the studies (27%) originated from the United States (USA), 18% were multinational, followed by 7% from the United Kingdom (UK) and 5% from Germany. Nordic countries led the ranking when adjusting for population size/million (ranks 1-3,6,9/45 countries), GDP/billion US$ (1,2,5,9,14/45), or health expenditure/billion US$ (1,3,5,12,13/45). The relative contribution of the UK was stable in the analyses when adjusted for population size (7/45), GDP (7/45), and health expenditure (9/45), whereas the USA (13, 18, and 30/45) and Germany (17, 19, and 21/45) dropped in the ranking.
Conclusions
More than half of the studies relevant for evidence-informed decision-making in Germany originated from the USA, followed by multinational research and the UK. Only 5% of the studies originated from Germany. According to our findings, there appears to be some discrepancy between the use of globally generated evidence and the contribution to the knowledge pool by individual countries.
Citation: Herrmann KH, Wolff R, Scheibler F, Waffenschmidt S, Hemkens LG, Sauerland S, et al. (2013) All Nations Depend on the Global Knowledge Pool – Analysis of Country of Origin of Studies Used for Health Technology Assessments in Germany. PLoS ONE 8(3): e59213. https://doi.org/10.1371/journal.pone.0059213
Editor: Johanna I. Westbrook, University of New South Wales, Australia
Received: November 14, 2012; Accepted: February 12, 2013; Published: March 14, 2013
Copyright: © 2013 Herrmann et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This work was supported by the Institute for Quality and Efficiency in Health Care (www.iqwig.de). Neither the funder itself nor any individuals employed or contracted by the funders (other than the named authors) played any role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Gerd Antes is a member of the scientific board of the ISRCTN register. Robert Wolff is affiliated to Kleijnen Systematic Reviews Ltd. This does not alter the authors' adherence to all the PLOS ONE policies on sharing data and materials.
Introduction
Health Technology Assessment (HTA) is a multidisciplinary field that systematically investigates the clinical efficacy and effectiveness, safety, cost, cost-effectiveness of health care interventions, as well as organizational implications, social consequences, and legal and ethical considerations [1]. HTA plays a crucial role in health systems throughout the world, supporting decision-making on how to access, distribute and implement technologies and innovation. Health technologies include pharmaceuticals, devices, diagnostics and treatments, and other clinical, public health, and organizational interventions. HTA reports include a systematic review of the clinical evidence in a transparent, unbiased, and robust manner in order to quantify the potential benefits and risks of health technologies [2], [3].
The Federal Joint Committee (G-BA) and the Institute for Quality and Efficiency in Health Care (IQWiG) are the most relevant HTA institutions in Germany. G-BA is the central self-governing body within the German health care system, a committee comprising important stakeholders in the health care system, e.g. health care providers and statutory health insurance funds. ‘It issues directives for the benefit catalogue of the statutory health insurance funds for more than 70 million insured persons and thus specifies which services in medical care are reimbursed’ [4]. IQWiG is an independent scientific institute and investigates the benefits and harms of medical interventions, usually to inform the decisions made by the G-BA [5].
A variety of publications are available on the analysis of the research activities of different countries. These are mainly bibliometric analyses focusing on Cochrane reviews [6] or specific medical fields such as anesthesia [7], [8], dentistry [9], nuclear medicine [10], pharmacological trials [11], primary care [12], radiology [13] or surgery [14].
In addition to these scientific analyses, inclusion in an HTA report emphasizes and measures the impact of clinical research on evidence-informed decision-making and thus the value in research expenditure as well as the return on investment in patient-oriented research. Consideration of studies for HTA reports indicates the quality and relevance of research for health-policy decisions. However, little is known about the country of origin of studies used for HTAs. It is therefore of interest to analyze which countries conduct studies used for HTAs and hence provide research findings relevant to health-policy decisions.
The objective of our study is to investigate the country of origin of clinical studies included in HTAs in a specific country, using Germany as an example.
Methods
Search strategy
The IQWiG website www.iqwig.de, which provides an overview of IQWiG projects from 2004 onwards, was searched in September 2010 for completed HTA reports. IQWiG products comprise benefit assessments (full reports and rapid reports), working papers, appraisals of clinical practice guidelines (CPGs), as well as health information. Products pursuant to § 35a of the German Social Code Book V (assessments of dossiers submitted by pharmaceutical companies) and health economic evaluations were not considered in this analysis, as these types of documents were only published after our project had been completed. Both full reports and rapid reports are HTAs applying the same methods with regard to the actual content of the report. However, they differ in terms of procedures, e.g. in contrast to rapid reports, preliminary versions of full reports are discussed in a public hearing. Working papers provide information on relevant developments in health care or methodological issues. Health information (e.g. feature articles, fact sheets and research summaries) is produced to inform the general public. Appraisals of CPGs aim to describe current health care standards. The latter three types of products are not considered to be HTAs [5].
Inclusion criteria
Eligible HTA reports were benefit assessments (full reports and rapid reports) published on the IQWIG website between 2004 and September 2010.
To be included in an HTA report prepared by IQWiG, studies need to contain data relevant to the specific project, i.e. report patient-relevant outcomes such as mortality, morbidity, adverse events and quality of life related to the health technology under investigation [5]. In our analysis we considered all studies meeting the inclusion criteria of the HTA reports. No further criteria were applied.
Exclusion
We excluded IQWiG products not considered to be HTA reports, such as working papers, health information articles or appraisals of CPGs.
Data extraction
Clinical studies were ascribed to a country as obtained from data extraction tables in published HTA reports. The HTAs reported the country of origin of the studies included based on the location of the study center(s) of the specific study. In cases where this was unclear to the HTA authors, the origin of a study was classified as “unknown”. A study was classified as “multinational” in cases where multiple study centers in various countries (or even a continent) were reported. Neither multinational studies nor those of unclear geographic origin were considered for the adjustments of the subsequent analyses. The overall number of studies included in the HTA reports from one country was used in the analysis and calculated in Excel 2010 (Microsoft Corporation, USA). A ranked order was displayed for further comparisons and analyses.
Quality assessment
Inclusion in an HTA report was used as a quality indicator. These reports are conducted following a rigorous assessment of the risk of bias (e.g. assessment of study design, allocation concealment) and follow recognized standards of evidence-based medicine. They typically included randomized controlled trials (RCTs) for the evaluation of clinical effects, as well as other study designs (e.g. diagnostic accuracy studies) [5]. We did not conduct any additional quality assessments of the primary studies included in the HTA reports.
Analysis
We analyzed how the studies included were distributed across countries adjusted for size of population (studies per 1 million population), gross domestic product (GDP, studies per 1 billion US$ GDP) and national spending on health (studies per 1 billion US$ health expenditure).
Data from the International Monetary Fund (2010) [15] were extracted to adjust for GDP. To adjust for the size of population and total health expenditure, data from the World Health Organization (WHO) report ‘World Health Statistics 2010’ [16] were obtained. Data for the Republic of China (ROC) were not included in the WHO report and therefore obtained from the Central Intelligence Agency report ‘The World Factbook’ 2010 [17] and ‘Lists of countries by total health expenditure 2007’ [18].
For a more detailed analysis we divided the reports into those on drugs and those on non-drug interventions. As an example, the rankings of countries of origin were analyzed according to health expenditure.
Results
81 projects were finalized and published between 06/02/2006 and 11/09/2010. 54 completed HTA reports (full and rapid reports) were included. 26 of these were reports on drugs and 28 on non-drug interventions (table 1), e.g. surgical procedures and diagnostic devices. 27 projects were guideline appraisals, working papers and health information and therefore excluded from the analysis (table 2).
In total, 1087 clinical studies were included in these 54 reports (20.1 studies per report on average). Six reports included more than 50 studies. Of all studies, 843 were assigned to 45 countries while 193 studies (18%) were multinational. 51 studies (5%) were classified as unknown, which was mainly due to the fact that the IQWiG reports lacked information on the country of origin [19], [20].
Number of clinical studies by countries
Figure 1 displays the absolute number of studies included in IQWiG HTA reports per country of origin, which shows that the United States (USA) led the ranking (293 studies, 27%). Following in descending order were: 193 multinational clinical studies (18%), the United Kingdom (UK, 79 studies, 7%), and Germany (55 studies, 5%).
Clinical studies by countries adjusted by population, GDP, and health expenditure
After adjusting the absolute number of studies by population size, Finland (3.21 studies per 1 million population) ranked first, followed by other Nordic countries (figure 2). The UK (1.29) dropped to position 7, the USA (0.94) and Germany (0.67) to positions 13 and 17 (figure 2).
When using GDP to measure and adjust for a country's wealth, Finland (95.0 studies per billion US$ GDP) and Iceland (82.7) led the ranking. Other Nordic countries followed on ranks 5, 9 and 14 (62.8-27.80). The UK (37.2) was still at 7th position while the USA (20.8) and Germany (19.6) dropped to ranks 18 and 19. Bulgaria rose to 4th position (66.7) (figure 3).
Adjusted for health expenditure, Finland (1.2 studies per 1 billion US$ health expenditure) again ranked first, followed by Bulgaria (0.91). Other Nordic countries followed on ranks 3, 5, 12 and 13 (0.88–0.31). The Ivory Coast moved up to sixth position (0.67), the UK (0.44) dropped to position 9, Germany (0.19) to 21 and the USA (0.13) to position 30 (figure 4).
Influence of research topic
When dividing the reports by type of intervention (figure 5 – 12), most of the multinational studies and studies with unknown country of origin were on drugs (multinational 149, unknown 31, out of 410 studies in total); fewer were on non-drug interventions (multinational 44, unknown 20, out of 677 studies in total) (figure 5 and 9).
When adjusted for health expenditure, most countries remained stable in their ranking (figure 8 and 12). Only 6 of 28 countries with sufficient data showed differences of more than 20 places in the ranking of research on drugs and non-drug interventions. China, ROC, Ukraine and USA were more productive in drug research, whereas New Zealand and South Korea were more productive in non-drug research (table 3).
Discussion
HTA reports are important for translating research into policy-making. They aim to inform policy-making comprehensively and with a minimized risk of bias. In order to do so, the global pool of clinical studies, often described as the “body of evidence”, has to be exploited. The objective of our study is to investigate the country of origin of clinical studies included in HTAs in a specific country, using Germany as an example. The results reflect the national contributions with a particular focus on research relevant to supporting health policy, as HTA is defined as a policy support tool, and within this framework reports are produced to provide answers to relevant questions.
Summary of findings
The findings of our analysis confirm the leading role of the USA and UK as major contributors to the global pool of clinical studies providing relevant information for health-policy decisions. These are followed by a large proportion of multinational studies. Germany contributes only 5% of the research input included in IQWiG reports. When adjusted for population size or economic variables, Nordic countries dominate the ranking, while the relevance of the USA and Germany decreases noticeably. After adjustments, the position of the UK is more stable than that of the USA and Germany.
One limitation in the present analysis is that neither studies of unknown origin nor multinational studies were analyzed in depth, since data on the distribution of countries in the study reports were not reported in detail in the IQWIG reports. We stratified studies from drug and non-drug reports and found that most of the multinational studies were within the pool of drug studies. This might be explained by the fact that to be granted widespread approval for the same drug, a pharmaceutical company must submit approval studies to different regulatory authorities in different countries applying different legislation. It therefore makes sense to conduct large multinational studies in a variety of countries. Different and less stringent regulations apply for the approval of non-drug interventions.
Germany performs poorly compared with other countries of similar economic power. This finding is especially surprising as one would expect German HTA reports to include a higher proportion of German studies, as clinical research in a given country is more likely to address the same research questions of relevance as investigated in national HTA reports (e.g. for demographic or epidemiologic reasons). The underrepresentation of German studies might be caused by limited clinical research activity or by a lower output of studies relevant to health policy decision-making.
Overall, our data show vast differences between contributing countries. These differences are particularly striking when adjusted for country population, GDP or health care expenditure, i.e. showing national contributions per capita or per money unit. Rich countries such as Germany show a poor contribution to the global knowledge pool, which is in sharp contrast to the dependence of these countries on global knowledge for decision-making.
It should be noted that regardless of the size of a contribution, all countries are dependent on knowledge generated globally. In countries such as the USA, which contributes a large number of studies to the knowledge pool (in our analysis: 27%), users of information might be tempted to base their decision-making process on their ‘own’ trials. However, succumbing to this temptation is likely to cause serious problems. First, ignoring large parts of the available evidence is a waste of resources and would introduce bias, as decision-making in health care should be based on all of the available evidence. Second, stratification for medical specialties would change the country league tables considerably, in some fields even dramatically [6]. The obvious conclusion is that all countries should consider themselves as contributors to and beneficiaries from the global body of evidence.
Research results in context
Comparison of the present analysis to previous ones largely confirmed earlier findings of studies comparing national activities in patient-oriented research. There are many similarities, regardless of whether an analysis was based on studies cited by Cochrane reviews [6], [21], pharmacological trials [11], or high-ranking publications in primary care [12], surgery [14], anesthesia [7], [8], nuclear medicine [10], or dentistry [9]; Nordic and Anglo-American countries usually take the lead.
Gluud and Nikolova [21] described various factors that have to be taken into account to explain a country's scientific output. Population size, economic wealth and research expenditure are obvious and relatively simple factors to include, while historical and cultural aspects are more difficult to cover. Factors with a strong impact are the research expenditure of pharmaceutical companies, as well as collaboration between researchers and industry. National government budgets play a key role in the funding of clinical research, as well as in the regulation of RCTs on drugs, especially the time taken to obtain regulatory approval [11].
There are numerous other factors possibly explaining the lack of clinical studies; for example, the promotion policies of clinical research, the funding situation, and the specific requirements concerning the availability of specially trained and experienced medical and research staff might be very different from those in other countries [12], [22]–[24].
Numerous hypotheses for future testing can be derived from our analysis. It would be interesting to see whether our findings are supported by similar research in other countries. In addition, the cultural component should be investigated, as well as other questions related to decision science, for example, when to use national or multinational studies or foreign studies from similar or very different countries.
Strengths and limitations
Our analysis differs from existing approaches. We did not merely conduct a bibliometric analysis, but investigated which countries contributed the largest proportion of studies included in HTAs in Germany. Studies had thus undergone a rigorous quality assessment according to IQWiG methods [5]. At the same time our analysis considered the relevance of the research output. All 1087 studies assessed by IQWiG to inform health-policy decisions were included, reflecting the performance of different countries in producing research relevant to decision-making.
One limitation in the present analysis was that neither studies with unknown origin nor multinational studies were analyzed in depth, since data on the distribution of countries in the study reports were not reported in detail in the IQWIG reports.
Some results need to be interpreted with caution. As an example, when weighted by national health expenditures, the Ivory Coast reached sixth place. This is rather misleading, as only one study [25] was included in an HTA report on test accuracy in ultrasound screening in pregnancy [26]. However, due to very low national health expenditure, the Ivory Coast achieved this relatively high ranking.
There might potentially be other factors associated with the output of clinical trials than population size, GDP, or health expenditure, such as promotion policies within departments, the number of universities, their programs, and funding in this field [12], [24], which were not analyzed in our study.
Conclusion
According to our findings, there is a discrepancy between the use of globally generated evidence and the contribution to the knowledge pool by individual countries. In absolute numbers, by far the most studies relevant to evidence-informed decision-making in Germany were conducted in the USA, followed by multinational research and the UK.
From the perspective of contributing countries, absolute numbers are misleading as they imply contributions, which do not exist on a per capita level but are merely due to a “large country effect”. Our study confirms that some small countries have a remarkable input in relation to their population size, health expenditure, or GDP. In contrast, some larger rich countries profit from these imbalances. Germany belongs to this category, with only 5% of the studies in German HTA reports actually conducted in Germany. Even for countries with larger contributions, it would be unwise to ignore the globally available evidence, and even harmful in certain fields because of much richer information outside their own countries.
The often-noted and criticized lack of studies for many relevant clinical questions is a consequence of many countries not taking the responsibility to contribute to global knowledge on the same scale as they are benefitting from it. Limited resources are a crucial issue in all research fields. In the medical field the existing gaps in knowledge are not a pure research problem but have a serious impact on health care decisions on an individual and public health level. A better-balanced contribution of all countries to the generation of global knowledge and its translation into policy and practice are urgently required to eradicate these deficits [27].
Acknowledgments
The authors would like to thank Nadine Czyply, Mariella Franken and Christina Steffen for data management.
Author Contributions
Commented on the manuscript: FS LGH SW. Conceived and designed the experiments: GA RW. Performed the experiments: KHH RW FS. Analyzed the data: KHH FS LGH. Wrote the paper: KHH RW GS SS.
References
- 1.
Health Technology Assessment International (2011) Health Technology Assessment International (HTAi) website. What is HTAi? Available: http://www.htai.org/index.php?id=420. Accessed 2013 Feb 19.
- 2.
Velasco Garrido M, Kristensen FB, Palmhøj Nielsen C, Busse R (2008) Health technology assessment and health policy-making in Europe: current status, challenges and potential. Copenhagen: WHO Regional Office for Europe. 181 p.
- 3.
Kristensen FB, Sigmund H, editors (2008) Health Technology Assessment handbook 2007. Copenhagen: Danish Centre for Health Technology Assessment. 188 p.
- 4.
Gemeinsamer Bundesausschuss (2012) The Federal Joint Committee website. The Federal Joint Committee. Available: http://www.english.g-ba.de. Accessed 2013 Feb 19.
- 5.
Institute for Quality and Efficiency in Health Care (2011). IQWiG website. General methods: version 4.0. Available: https://www.iqwig.de/download/General_Methods_4-0.pdf. Accessed 2013 Feb 19.
- 6.
Wolff RF, Reinders S, Barth M, Antes G (2011) PLoS One website. Distribution of country of origin in studies used in Cochrane Reviews. PLoS One 6: e18798. Available: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0018798. Accessed 2013 Feb 19.
- 7. Figueredo E, Sánchez Perales G, Muñoz Blanco F (2003) International publishing in anaesthesia: how do different countries contribute? Acta Anaesthesiol Scand 47: 378–382.
- 8. Swaminathan M, Phillips-Bute BG, Grichnik KP (2007) A bibliometric analysis of global clinical research by anesthesia departments. Anesth Analg 105: 1741–1746.
- 9. Gil-Montoya JA, Navarrete-Cortes J, Pulgar R, Santa S, Moya-Anegón F (2006) World dental research production: an ISI database approach (1999-2003) Eur J Oral Sci 114: 102–108.
- 10. Signore A, Annovazzi A (2004) Scientific production and impact of nuclear medicine in Europe: how do we publish? Eur J Nucl Med Mol Imaging 31: 882–886.
- 11. Lambers Heerspink HJ, Knol MJ, Tijssen RJ, van Leeuwen TN, Grobbee DE, et al. (2008) Is the randomized controlled drug trial in Europe lagging behind the USA? Br J Clin Pharmacol 66: 774–780.
- 12. Glanville J, Kendrick T, McNally R, Campbell J, Hobbs FD (2011) Research output on primary care in Australia, Canada, Germany, the Netherlands, the United Kingdom, and the United States: bibliometric analysis. BMJ 342: d1028.
- 13. Özsunar Y, Unsal A, Akdilli A, Karaman C, Huisman TAGM, et al. (2001) Technology and archives in radiology research: a sampling analysis of articles published in the AJR and Radiology. AJR Am J Roentgenol 177: 1281–1284.
- 14. van Rossum M, Bosker BH, Pierik EG, Verheyen CC (2007) Geographic origin of publications in surgical journals. Br J Surg 94: 244–247.
- 15.
International Monetary Fund (2010) International Monetary Fund website. World economic and Financial Surveys. World Economic Outlook database by countries. Available: http://www.imf.org/external/pubs/ft/weo/2010/02/weodata/WEOOct2010all.xls. Accessed 2013 Feb 19.
- 16.
World Health Organization (2010) World Health Statistics 2010. Geneva: WHO. 177 p.
- 17.
Central Intelligence Agency (2010) Taiwan. Central Intelligence Agency website. In: The World Factbook. Available: https://www.cia.gov/library/publications/the-world-factbook/geos/tw.html. Accessed 2010 Oct 20.
- 18.
Wikipedia contributors (2007) Wikipedia website. List of countries by total health expenditure (PPP) per capita. Available: http://en.wikipedia.org/w/index.php?title=List_of_countries_by_total_health_expenditure_(PPP)_per_capita&oldid=500988430. Accessed 2012 Aug 02.
- 19.
Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (2009) IQWiG website. Selective serotonin and norepinephrine reuptake inhibitors (SNRI) for patients with depression: final report A05-20A [in German]. Available: https://www.iqwig.de/download/A05-20A_Abschlussbericht_SNRI_bei_Patienten_mit_Depressionen.pdf. Accessed 2013 Feb19.
- 20.
Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (2007) IQWiG website. Stem cell transplantation in adults with acute lymphoblastic leukaemia (ALL) or acute myeloid leukaemia (AML): final report N05-03A [in German]. Available: https://www.iqwig.de/download/N05-03A_Abschlussbericht_Stammzelltransplantation_be_ALL_und_AML.pdf. Accessed 2013 Feb 19.
- 21.
Gluud C, Nikolova D (2007) Trials website. Likely country of origin in publications on randomised controlled trials and controlled clinical trials during the last 60 years. Trials 8: 7. Available: http://www.trialsjournal.com/content/8/1/7. Accessed 2013 Feb 19.
- 22.
Bührlen B, Vollmar HC, Georgieff P (2010) The Office of Technology Assessment at the German Bundestag website. Clinical research in Germany with special focus on non-commercial studies [in German]. Available: http://www.tab-beim-bundestag.de/en/publications/reports/ab135.html. Accessed 2013 Feb 19.
- 23. Büchler MW, Diener MK, Weitz J (2011) Scientific evaluation of modern clinical research: we need a new currency! Langenbecks Arch Surg. 396: 937–939.
- 24.
Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, Department for Employment and Learning (2008) Research Assessment Exercise, rae2008 website. Research assessment exercise 2008: the outcome. Available: http://www.rae.ac.uk/results/outstore/RAEOutcomeAE.pdf. Accessed 2013 Feb 19.
- 25. N'Dri K, Delmotte C, Gbazi CG, Konan A, Burdin-Mensah GD, et al. (1997) Ultrasound diagnosis of fetal malformations in utero: apropos of 30 cases [in French]. Sante 7: 246–250.
- 26.
Institut für Qualität und Wirtschaftlichkeit im Gesundheitswesen (2008) IQWiG website. Ultrasound screening in pregnancy: test accuracy with regard to detection rates of foetal abnormalities; final report S05-03 [in German]. Available: https://www.iqwig.de/download/S05-03_Abschlussbericht_Ultraschallscreening_in_der_Schwangerschaft.pdf. Accessed 2013 Feb 19.
- 27. Antes G, Clarke M (2012) Knowledge as a key resource for health challenges. Lancet 379: 195–196.