Introduction

Health technology assessment (HTA) has received much attention by clinicians, researchers and policy-makers in various health-care systems [1,2]. It has emerged as an important tool for evidence-based policy and clinical decision-making that comprehensively assesses safety, efficacy, cost-effectiveness and ethical concerns in utilizing health technology. In recent years, many HTA research units have been established around the world and a large number of HTA studies have been carried out.

However, conducting HTA studies is one thing, using HTA evidence to influence policy- or decision-making is another. For instance, in China, one of the largest developing countries, although technology assessment was a responsibility of the Department of Science and Education of the Ministry of Health (MoH) and many other administrative entities, HTA has little or no impact on most health technology-related decisions, though there is some evidence that HTA findings have influenced policy decision-making in such areas as gamma knife technology and prenatal diagnosis technology [3,4]. For HTA to be more policy relevant and for health technology-related decision-making to be truly evidence-based, promoting knowledge translation (KT), namely, the “exchange, synthesis and ethically-sound application of knowledge—within a complex system of interactions among researchers and users” [5], is a necessary first step [6].

Some studies have been conducted in other countries to measure KT and explore its determinants. For example, Peltz has categorized how scientific knowledge is used in public policy into three types, namely, symbolic utilization, conceptual utilization, and instrumental utilization. Symbolic utilization refers to justifying existing policies or positions of organizations (e.g., government). Conceptual utilization refers to changing one’s thinking but not necessarily one’s particular actions. Instrumental utilization refers to research findings being utilized as a basis for action, such as shaping policies and guiding implementation [7]. Going one step further, Estabrooks has developed measurement scales to separately measure each type of use [8].

Additionally, there are studies on the determinants of KT in general. Some studies have focused on researchers’ characteristics, such as their academic ranks fields of study and attitudes toward KT [9–14]. Others have focused on the research project itself and have explored the influence of timeliness, practicality and policy-relevance of research on KT [15–17]. It has also been suggested that efforts to make reports more readable and easier to understand would enhance adoption of research-based knowledge [18]. Finally, the importance of relationships between researchers and knowledge users, which include communication and collaboration during the research process, has also been examined in previous studies [19–21].

There are also studies more specific to HTA KT. For instance, studies have shown that providing timely information that is in line with decision-makers’ priorities increases research knowledge utilization [22,23]. Using less technical language tends to facilitate research uptake [24]. Building partnerships with user groups also plays a beneficial role. Previous studies have also highlighted the importance of leadership and creating cooperative relationships between researchers and knowledge users [25–27].

These studies are useful in understanding HTA KT and identifying its potential influencing factors. However, most of these studies are about KT in clinical practice, mainly focusing on the knowledge users, such as clinical decision-makers and so on. There is a dearth of literature on HTA KT, especially from the perspective of HTA researchers. Researchers and decision-makers often differ in terms of values, language patterns, and time perspective. Furthermore, production of HTA knowledge is the starting point of the KT process and researchers are the ones responsible for undertaking the investigation. Thus, it is necessary to understand the researchers’ perspectives, their attitudes toward research, and what they think about researchers’ social obligations. This does not diminish the importance of the perspectives of HTA knowledge users. But as noted earlier, much of existing HTA KT research focuses on policy- and decision-makers, and not enough is known about HTA researchers. Although we have published an earlier article exploring the differences in HTA KT between policy-makers and HTA researchers based on the same study, it mainly included factors at the individual level and the effect of the factors at the organizational level still remained unclear [28]. Thus, this study is an attempt to address this knowledge gap.

Additionally, much of the research on HTA utilization and its KT determinants has been conducted in developed nations but how HTA is utilized and how HTA KT is conducted in developing countries and transitional societies remain largely unknown. Owing to differences between developed and developing nations in socioeconomic development and health system management, the determinants of KT from HTA to health policy-making may be very different. To avoid the pitfall of wholesale transfer of inapplicable experiences from developed nations, it is necessary to carry out relevant research in developing countries such as China, one of the largest developing countries. Findings about KT determinants in China will probably be applicable to other developing countries.

Therefore, to fill the knowledge gap, we have selected China as a case study, with a view to investigating KT determinants both at the individual and organizational level from HTA research to health policy-making from the perspective of researchers.

Material and methods

Results

Characteristics of respondents

We distributed 561 questionnaires and received 382 completed questionnaires, achieving a response rate of 68.1%. The 382 HTA researchers ranged in age from 24 to 76, with the majority in the 30–49 age group. Slightly less than half of the researchers were male. Most researchers had a doctoral (51.8%) or a master’s (38.8%) degree. Nearly 80% worked at universities. About 30% and 35% of them were associate professors and full professors, respectively. Regarding primary research funding sources, 55% of the researchers had funding support from the government. With respect to the nature of their HTA research, more than half focused on health systems assessment and approximately 30% focused on drug assessment (Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Demographic characteristics of the respondents.

https://doi.org/10.1371/journal.pone.0190732.t001

Regarding activity level by type of KT activity, our results indicated that 30%~40% of the HTA researchers usually or always published their research findings in academic journals or submitted HTA results to policy-makers. On the other hand, about 8.6% indicated that they never engaged in such knowledge dissemination activities. As one moves from academic translation to application translation, the proportion of respondents indicating that they never undertook that KT activity increased. Nonetheless, about 15% of the respondents reported that their HTA findings always or usually led to applications (Table 2).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Activity level by type of KT activity.

https://doi.org/10.1371/journal.pone.0190732.t002

A KT summary score was calculated for each researcher and the results of the Duncan’s test were reported. The means of the KT summary scores ranged from 51.06 for “others” to 61.72 for “systems”. The Duncan’s test appears to show that there were two homogeneous subsets of HTA research field. There was no statistically significant difference between “others” and “drugs” with respect to levels of KT activity. Likewise, there were no statistically significant differences between “drugs”, “procedures”, “equipment” and “systems” (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Means of knowledge translation for different HTA research fields in homogeneous subsets (Duncan’s test).

https://doi.org/10.1371/journal.pone.0190732.t003

The principal component analysis resulted in four factors with items of high loadings (0.50 or over). An examination of the variables associated with these four factors suggested they represented linkage mechanisms, attitudes, organizational support, and organizational linkages of HTA research units, which were similar to how our original questions were grouped. The factors, reliabilities, means and standard deviations of these variables were listed in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Factor analysis of scale on HTA knowledge translation and its items (n = 382).

https://doi.org/10.1371/journal.pone.0190732.t004

Multivariate linear regression analysis

The regression analysis results were summarized in Table 5. In the comprehensive model that included all fields of HTA research, academic ranks, government as the main source of funding support, relevance to policy-making, attitudes, “making HTA report easier to understand”, “individual linkage mechanism”, “organizational support” and “organizational linkage” were significantly and positively related to KT level. Scientific rigor was negatively associated with KT level (P < 0.05). Other variables, such as sex, education, and organizational affiliation, were found to be unrelated to KT level. The total amount of variance explained by this model, as shown by the adjusted R², was 0.50.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Multivariate linear regression models for predicting KT levels of HTA.

https://doi.org/10.1371/journal.pone.0190732.t005

The next ten columns in Table 5 reported the regression results of the different fields of HTA research. The factor of individual linkage mechanism explained KT level in all fields of HTA research. The factor of organizational linkage and academic rank were significantly and positively associated with KT level in “procedures”, “systems” and “other” research field, but not in “drugs” and “device and equipment”. “Making HTA reports easier to understand” explained KT level in the research fields of “drugs” and “systems”, but not in “procedures” and “device and equipment”. Attitude toward KT, practicality of HTA evidence and organizational support were significantly and positively related to KT level for HTA research field of “systems”, but not for the other fields. As for government as the main source of funding support, it was significantly and positively related to KT level in HTA research field of “drugs”, but not in the other fields. The other six variables, namely, sex, education, affiliation, scientific rigor, submitting HTA reports to policy-makers at the right time and HTA relevance to policy-making were not found to be associated with KT level in all fields of HTA research. The total amount of variance explained in the research fields of “drugs”, “device and equipment”, “procedures”, “systems” and “others”, as shown by the adjusted R², was 0.43, 0.38, 0.48, 0.53 and 0.49, respectively.

For linear regression models mentioned above, variable inflation factors (VIFs) were also computed for the independent variables to detect multi-colinearity. The maximum VIF among the independent variables was 5.18 (much smaller than 10), which indicated that multi-colinearity was not a significant problem [31].

Discussion

This paper derived its dependent and independent variables from prior research on KT. Based on the responses from 382 HTA researchers, the findings of this study showed that, on the whole, between 30% and 40% of the HTA researchers surveyed usually or always had their research findings published in academic journals or submitted to policy-makers; but between 20% and 30% of the researchers did not even do the more basic KT tasks. Additionally, between 40% and 50% of the HTA researchers said that they rarely or never had their research findings adopted by policy-makers. These findings suggest that the application of HTA findings to policy-making is not widespread in China at present. Furthermore, comparisons of the means of KT summary scores show that no significant differences exist between the various fields of HTA research, such as “drugs”, “procedures”, “device and equipment” and “systems”. These findings prompted us to further explore whether the influencing factors of KT level were the same in different fields of HTA research. These questions were examined by using regression models.

The importance of collaboration between HTA researchers and policy-makers [18–20] was also highlighted in this study. Frequent exchanges between researchers and policy-makers during the processes of topic selection, research implementation, and result dissemination will enhance mutual understanding and trust. They may result in more targeted research questions and more useful research results. This suggests that more attention be paid to linkages between HTA researchers and policy-makers. However, such linkages often require time and other resources. Thus, it may be advisable to devise some incentive schemes to compensate HTA organizations and researchers for investing resources in establishing and maintaining linkage mechanisms [30].

Researchers’ characteristics, such as attitudes, academic ranks and research funding sources, have been shown to influence knowledge utilization in previous studies [8–13]. Similarly, in this study, attitudes and academic rank were shown to be good predictors in the comprehensive model and the model for the research field of healthcare systems. As the producers of HTA knowledge, HTA researchers’ attitudes, such as willingness to transfer their knowledge to users, their expectations of the use of HTA evidence in practice, may facilitate or hinder KT. Likewise, HTA researchers with higher academic ranks are likely to be more productive in research and publication, which may lead to greater utilization [8,9]. Additionally, funding support mainly from government is a good predictor of KT. Greater policy-relevance of studies funded by government may be a plausible explanation.

In line with previous studies conducted in other countries [14–16,21,22], we have found that policy-relevance of HTA research, practicality of HTA evidence and making research outcomes easier to understand influence KT level. Therefore, to enhance the impact of HTA, it is important for researchers to take policy-makers’ needs into consideration and make recommendations more “actionable”. Especially for those studies focusing on policy-making, there is a requirement to directly engage with policy-makers throughout the entire research process, tailor research programs and outputs to policy-makers’ needs, and adopt an appropriate communication format to make it easier for non-researchers to understand. These efforts may entail some costs, which may not usually be covered by research funds. Therefore, research funders or research users may consider compensating or even rewarding researchers who engage in appropriate KT activities.

An interesting result is that scientific rigor of HTA research was negatively related to KT. This seemingly counter-intuitive finding is particularly worth-noting because scientific rigor is always considered a priority in academia. It is possible that policy-makers are not so much opposed to scientific rigor in research as “alienated” by impenetrable technicalities and academic jargons in some HTA reports and most academic publications. While it is necessary to maintain scientific rigor in research, it is equally important to ensure that research dissemination be done in a user-friendly manner. Translating research outcomes into policy-making is a difficult balancing act between ensuring research relevance and preserving scholarly integrity. This point is also supported by previous studies [17,23,32].

The importance of leadership and creating an environment favorable to KT has been noted in previous studies [24–26]. Our findings show that support from research organizations, such as providing guidance for HTA KT, having KT specialists on staff, establishing an incentive mechanism to reward KT activities and so forth will greatly facilitate HTA KT. This is particularly important because the current academic incentive system in China (as well as in many other countries) tends to focus more on SCI paper publication than on encouraging cooperation between researchers and policy-makers in KT or policy uptake. Changing this academic incentive system and research culture will be a challenging undertaking.

The contribution of this study is two-fold. On the one hand, it contributes to the advancement of knowledge. Most empirical studies on HTA KT have focused on developed countries and on clinical practice. Our study has attempted to better understand the factors at both the individual level and the organizational level (such as organizational support, organizational linkage of HTA) that influence the process from HTA research to health policy-making in China. On the other hand, by exploring the determinants of HTA KT activities, this study has suggested ways for both researchers and policy-makers in China, and possibly in other developing countries, to make the best use of HTA with a view to promoting evidence-based policy- and decision-making.

Some limitations of this study should be noted. Firstly, social desirability biases cannot be ruled out in this self-report study. Secondly, owing to the absence of a complete listing of HTA researchers in China, we resorted to a snowball sampling technique, which is commonly considered to be a non-probabilistic sampling method. We took several measures to control potential biases in the selection of the study participants, but since about 30% of the approached HTA researchers did not return their questionnaires, the existence of some biases cannot be ruled out. For those willing to participate in this study were more likely to be aware of the importance of KT and more likely to engage in KT activities, the results of KT level and other indicators reported in this study may be more positive than real situation Thirdly, this paper looks at HTA KT solely from the researchers’ perspective, but KT inevitably involves at least two sets of actors—researchers and users of research.

Conclusion

HTA is expanding rapidly in China, which is a positive development. But for HTA to be as effective as it should be, researchers need to pay equal attention to the transmission of research findings to potential users and the translation of research into policies, decisions, clinical practice, etc. Without this, HTA is just an academic exercise that is of little value to health care and has little impact on society.

Our study has shown that while many HTA researchers in China were aware of the importance of using research to support policy- and decision-making, HTA KT activities were not widespread. Also, although some HTA researchers were involved in KT to a greater or lesser extent, more effort is needed and such effort needs to be more systematic and targeted. On the other hand, successful KT relies not just on the endeavours of researchers, but also on the support of HTA organizations and research commissioning or funding agencies. Such support may take the forms of incentive mechanisms to encourage KT involvement, providing appropriate KT training to researchers, hiring KT specialists to work with researchers, including funding for KT activities in research grants, etc. As well, decision- and policy-makers need to reach out to researchers and be receptive to research evidence. This last aspect is not well understood in China at this time and more research is needed.

Our study is about the HTA KT situation in China, but our findings could have wider implications. Other developing nations, particularly those at developmental levels similar to that of China, could learn from China’s experience and design strategies that suit their own unique circumstances or meet their special needs.

Supporting information

Acknowledgments

We gratefully acknowledge the funding support of the China Medical Board (CMB 11–067), Shanghai Public Health Key Discipline Construction Project-Health Economics (No 12GWZX0601), Shanghai Public Health Senior Professional Oversea Fellowship (GWHW2012S10) and Shanghai Public Health Excellent Discipline Leadership Program (GWDTR201205). We also gratefully acknowledge the cooperation of HTA researchers taking part in the survey, whose participation has made this study possible. And many thanks to the research assistants for collecting and entering data.