Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

Engaging stakeholders in the co-development of programs or interventions using Intervention Mapping: A scoping review

  • Umair Majid ,

    Contributed equally to this work with: Umair Majid, Claire Kim, Albina Cako, Anna R. Gagliardi

    Roles Data curation, Formal analysis, Project administration, Writing – original draft, Writing – review & editing

    Affiliation Toronto General Hospital Research Institute, University Health Network, Toronto, Ontario, Canada

    ⨯
  • Claire Kim ,

    Contributed equally to this work with: Umair Majid, Claire Kim, Albina Cako, Anna R. Gagliardi

    Roles Data curation, Writing – original draft

    Affiliation Toronto General Hospital Research Institute, University Health Network, Toronto, Ontario, Canada

    ⨯
  • Albina Cako ,

    Contributed equally to this work with: Umair Majid, Claire Kim, Albina Cako, Anna R. Gagliardi

    Roles Data curation

    Affiliation Toronto General Hospital Research Institute, University Health Network, Toronto, Ontario, Canada

    ⨯
  • Anna R. Gagliardi

    Contributed equally to this work with: Umair Majid, Claire Kim, Albina Cako, Anna R. Gagliardi

    Roles Conceptualization, Methodology, Project administration, Resources, Software, Supervision, Writing – original draft, Writing – review & editing

    anna.gagliardi@uhnresearch.ca

    Affiliation Toronto General Hospital Research Institute, University Health Network, Toronto, Ontario, Canada

    ⨯

Abstract

Background

Health care innovations tailored to stakeholder context are more readily adopted. This study aimed to describe how Intervention Mapping (IM) was used to design health care innovations and how stakeholders were involved.

Methods

A scoping review was conducted. MEDLINE, EMBASE, Cochrane Library, Scopus and Science Citation Index were searched from 2008 to November 2017. English language studies that used or cited Intervention Mapping were eligible. Screening and data extraction were done in triplicate. Summary statistics were used to describe study characteristics, IM steps employed, and stakeholder involvement.

Results

A total of 852 studies were identified, 449 were unique, and 333 were excluded based on title and abstracts, 116 full-text articles were considered and 61 articles representing 60 studies from 13 countries for a variety of clinical issues were included. The number of studies published per year increased since 2008 and doubled in 2016 and 2017. The majority of studies employed multiple research methods (76.7%) and all 6 IM steps (73.3%). Resulting programs/interventions were single (55.4%) or multifaceted (46.4%), and 60.7% were pilot-tested. Programs or interventions were largely educational material or meetings, and were targeted to patients (70.2%), clinicians (14.0%) or both (15.8%). Studies provided few details about current or planned evaluation. Of the 4 (9.3%) studies that reported impact or outcomes, 3 achieved positive improvements in patient or professional behaviour or patient outcomes. Many studies (28.3%) did not involve stakeholders. Those that did (71.7%) often involved a combination of patients, clinicians, and community organizations. However, less than half (48.8%) described how they were engaged. Most often stakeholders were committee members and provide feedback on program or intervention content or format.

Conclusions

It is unclear if use of IM or stakeholder engagement in IM consistently results in effective programs or interventions. Those employing IM should report how stakeholders were involved in each IM step and how involvement influenced program or intervention design. They should also report the details or absence of planned evaluation. Future research should investigate how to optimize stakeholder engagement in IM, and whether use of IM itself or stakeholder engagement in IM are positively associated with effective programs or interventions.

Citation: Majid U, Kim C, Cako A, Gagliardi AR (2018) Engaging stakeholders in the co-development of programs or interventions using Intervention Mapping: A scoping review. PLoS ONE 13(12): e0209826. https://doi.org/10.1371/journal.pone.0209826

Editor: Holly Seale, University of New South Wales, AUSTRALIA

Received: September 5, 2018; Accepted: December 12, 2018; Published: December 26, 2018

Copyright: © 2018 Majid et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All relevant data are within the paper and its Supporting Information files.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Background

Health care innovations such as guidelines, procedures, treatments, technology or programs have limited impact on health service delivery and health outcomes unless they are actively implemented. A crucial step in implementation planning is identifying determinants of innovation use including barriers or facilitators [1]. Flottorp consolidated several frameworks to generate a checklist of 57 potential determinants of the use of innovations that were organized in seven domains: guideline factors, individual health professional factors, patient factors, professional interactions, incentives and resources, capacity for organizational change, and social, political, and legal factors [2]. Knowledge of determinants can be collected through interviews, focus groups, surveys or observation of behaviour among target users of the innovation [3]. This information can then be used to select and tailor interventions that are most likely to promote and support use of the innovation. Research shows that interventions selected and tailored to address identified determinants were more likely to improve professional practice compared with either no intervention or simple dissemination of information about the innovation [4]. Suitable interventions can be identified by mapping pre-identified determinants in existing taxonomies of behaviour change interventions such as the Expert Recommendations for Implementing Change compendium [5], or by using resources such as the Theoretical Domains Framework [6].

Despite the guidance provided by these useful processes and tools, there is no consistently reliable mechanism for matching determinants to interventions that guarantees the successful implementation and use of innovations [7,8]. One approach for optimizing the selection, tailoring, implementation and impact of interventions is to engage target users of the innovation in implementation planning. They can provide important insight on how the determinants influence practice, which interventions are the best fit, and the most practical way to implement them. This approach, sometimes referred to as participatory research, engaged scholarship or mode 2 research, and more recently as integrated knowledge translation (IKT), recognizes that intervention developer-user collaboration can enhance the relevance and use of innovations [9]. However, such collaborations are challenging to develop and maintain. Our scoping review of 13 IKT-based studies published between 2005 and 2014 identified numerous barriers including differing priorities among participants, lack of skill or experience with IKT, unclear roles and goals, lack of incentives to participate, little continuity due to infrequent participation or turnover, and lack of resources [9]. Even when well-funded, the impact of IKT collaborations on the use of innovations, and improvements in health service delivery and health outcomes has been modest. For example, large-scale investment to foster IKT through the Collaborations for Leadership in Applied Health Research and Care (CLAHRC) in England [10] and Academic Collaborative Centres (ACC) for Public Health in the Netherlands [11] did not appear to eliminate barriers or lead to greater output or use of innovations. Hence, further knowledge is needed on how to achieve and support IKT. In particular, ongoing research should identify processes that are conducive to interaction among the developers and users of innovations, and appropriate roles for innovation users in the implementation planning process.

Intervention Mapping (IM) is a process that is specifically meant to engage stakeholders in developing interventions that address pre-identified determinants through six steps: assess needs and barriers, establish objectives, select theory-informed interventions, design and pilot-test the intervention, implement and assess fidelity of the intervention, and evaluate the impact of the intervention [12,13]. First published in 1998, IM is not a new process. However it is unclear, given growing recognition of the need to engage stakeholders in implementation planning, if and how IM has been used to co-develop innovations. The purpose of this study was to review the health care literature to describe how stakeholders were involved in co-developing programs or interventions including their roles and processes employed to engage them. The findings could generate insight on how to optimize IM and IKT in IM.

Methods

Approach

Given the aim, which was to describe the characteristics of published research that employed IM, a scoping review was conducted, comprised of five steps: scoping, searching, screening, data extraction and data analysis [14]. Similar in rigour to a systematic review, the purpose of a scoping review is to examine the nature of research activity for a particular topic, the volume of which cannot be known without such an assessment. Such reviews do not synthesize outcomes reported across studies or assess their methodological quality, as is customary of systematic reviews, nor do they assume a theoretical stance. However, they can assess what is known about a specific topic, identify if sufficient research is available to conduct a future systematic review, and/or reveal knowledge gaps that warrant future research. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) criteria guided reporting of the methods and findings (S1 Table) [15]. Data were publicly available so institutional review board approval was not necessary. A protocol for this review was not registered.

Scoping

Relevant literature was first explored to become familiar with the IM literature and refine study methods. To do this, MEDLINE was searched using only the term “intervention mapping” as a keyword. AC and ARG independently screened titles and abstracts and discussed the findings. This knowledge was used to plan a more comprehensive search strategy and to generate eligibility criteria based on the PICO (participants, intervention, comparisons, outcomes) framework, as follows.

Populations referred to health care researchers, policy-makers or managers, clinicians, allied health staff, patients, care partners or consumers involved in the IM process that was used to plan an intervention relevant to screening, prevention, treatment (including drug/non-drug treatment, counseling, education, rehabilitation), follow-up, supportive care or palliative care offered in any health care setting. The intervention of interest was the IM process or any of its six steps (S2 Table). Relevant publications may have reported the entirety of the six-step IM process or several intervention planning steps (study type #1); often implementation and evaluation steps (study type #2) were reported in separate publications. Comparisons were based on study type. Type #1 studies tended to be qualitative studies, qualitative single or multiple case studies, or mixed methods studies that described the methods and outcomes of one or more IM steps. Type #2 studies tended to compare single or multiple interventions developed through the IM process either alone or compared with no intervention (sometimes described as “usual care” or “control”), or in comparison with another type of single or multiple intervention. The research design of type #2 studies included randomized or pragmatic controlled trials, observational studies (retrospective, prospective or before-after cohort studies), surveys, qualitative research (interviews, focus groups) or mixed methods research. Outcomes included but were not limited to IM steps used, type of innovation users involved and how, and with what outcome, meaning were interventions or programs developed, were they implemented, and did they influence health care delivery or patient health.

Studies were not eligible if they examined the effectiveness of clinical interventions (tests, procedures, treatment); used IM as a framework to extract data from research studies or documents such as practice guidelines but not to develop an intervention; used IM to develop policy or system-level interventions; used IM to develop health-related interventions applied in sport or school settings; or were publications in the form of editorials, letters, commentaries, protocols, meeting abstracts or proceedings, or conceptual analyses of IM. Systematic reviews were not eligible but were used to identify eligible primary studies.

Searching

A comprehensive literature search was conducted. MEDLINE, EMBASE, Cochrane Library and Scopus were searched on April 21, 2016 for studies published since 1998, the year in which the IM process was first published [12], for the phrase “intervention mapping” as a keyword. Science Citation Index was searched for all subsequently published studies that referenced the 1998 IM publication [12]. The search was subsequently updated. MEDLINE, EMBASE and Science Citation Index were searched on November 8, 2017 for studies published in 2016 and 2017. The references of all eligible studies were scanned to identify additional eligible articles.

Screening

AC and ARG independently screened all titles and abstracts of the initial search according to specified eligibility criteria. For the updated search, CK and ARG independently screened the title and abstract of the first 25 search results, then compared and discussed discrepancies, and how to interpret and apply the eligibility criteria. CK and ARG screened another 25 titles and abstracts and achieved congruent decisions regarding eligibility; henceforth, CK screened remaining titles and abstracts. At this stage we retained all studies published from 2008 and later so that findings and corresponding implications would be relevant to more recent IKT conduct and reporting practices. All items selected by at least one reviewer were retrieved. If more than one publication described a single study and reported different data, they were all included but counted as a single study. AC and ARG, and then CK and ARG met on multiple occasions to jointly review full text articles and resolve eligibility issues. Screening results from the initial search and the updated search were blended to create a single PRISMA diagram.

Data extraction

A data extraction form was developed to collect information on author, publication year, country, setting, health care issue requiring an intervention, study type (all or partial IM steps), stakeholder involvement, characteristics of the intervention that was designed using the IM process, and any outcomes reported by the study reflecting the impact of the intervention. Details about the intervention were based on the Workgroup for Intervention Development and Evaluation Research (WIDER) reporting checklist and included content, mode of delivery, duration and/or frequency, participants and personnel [16]. For the initial search results, AC and ARG independently pilot-tested the form on the same three articles and compared findings by discussion through four iterations at which time data extraction was congruent, after which AC extracted data from remaining articles. After the search was updated, CK and ARG pilot-tested data extraction on the same three articles and compared findings by discussion; this process was repeated two more times to achieve congruent data extraction, after which CK extracted data from remaining articles. All extracted data were independently checked by UM, who met with ARG on multiple occasions to discuss and resolve data extraction issues.

Data analysis

Summary statistics were used to describe the number of studies by country, year of publication, study design and health care issue, number of studies using the full or partial IM process, key processes employed in IM steps, intervention design (single or multifaceted), target population for the intervention (patient only, clinician only, patient and clinician), number of studies that employed theory, the most commonly used theories, number of studies involving different stakeholders in co-generation. The most commonly utilized theories, and intervention characteristics and outcomes. The common denominator used for all computations was based on 60 studies representing 61 articles.

Results

Search results

After duplicate titles were removed, a total of 449 unique titles and abstracts were screened, of which 346 were discarded, leaving 103 full-text articles to be screened. Among those, 42 were excluded: IM not undertaken (14), setting not eligible (12), published prior to 2008 (5), study protocol (4), IM mentioned but not employed (4), and duplicate study (2). Review of references for eligible items yielded no further studies. A total of 61 articles representing 60 studies were included in the review (Fig 1). The data extracted from each study is summarized in S3 Table [17–78].

thumbnail
Download:
Fig 1. PRISMA diagram.

https://doi.org/10.1371/journal.pone.0209826.g001

Study characteristics

Included articles were published from 2008 to 2017 inclusive. While not continuous, the number of studies published per year increased from a high of 8 in 2012 to 19 in 2017 (Fig 2). This translates to half of included studies published in the last two years (30/60, 50.0%). Studies were conducted in 13 countries, most often in the Netherlands (22, 36.7%) followed by the United States (16, 26.7%), Canada (8, 13.3%), United Kingdom (6, 10.0%) and Iran (2, 3.3%). One study (1.7%) was conducted in each of Australia, Brazil, Denmark, Estonia, Germany, Italy, South Africa and South Korea.

thumbnail
Download:
Fig 2. Number of IM studies published per year.

https://doi.org/10.1371/journal.pone.0209826.g002

Table 1 summarizes the disease or health service modalities addressed in each study. The most common disease or health service modality categories were health promotion (15, 24.2%), preventive medicine (11, 17.7%), cardiovascular disease (10, 16.1%), and mental health (8, 12.9%).

thumbnail
Download:
Table 1. Number of studies by disease or health service modality.

https://doi.org/10.1371/journal.pone.0209826.t001

With respect to research design, the majority of studies employed multiple methods including one or more approaches for Needs Assessment plus one or more approaches for Evaluation of a program or intervention generated through the IM process (46, 76.7%). For the 14 (23.3%) of studies employing a single research design, 4 (23.5%) were randomized controlled trials, 3 were survey studies, 2 studies used focus groups, 2 studies used interviews, 2 studies employed a prospective observational research design, and 1 study involved a literature review.

IM steps employed

Table 2 summarizes the IM steps used by each included study. Among included studies, 73.3% (44/60) employed all 6 IM steps. The remaining studies employed steps 1 to 4 (10, 16.7%), step 1 only (3, 5.0%), steps 2 to 6 (2, 3.3%) and steps 1 to 3 (1, 1.7%).

thumbnail
Download:
Table 2. Intervention Mapping steps used by included studies.

https://doi.org/10.1371/journal.pone.0209826.t002

Step 1: Needs assessment.

All but 2 studies conducted a Needs Assessment (58, 96.7%). Most of the studies that conducted a Needs Assessment employed multiple approaches to assess determinants (facilitators and barriers) of target performance objectives (50, 84.7%). Literature or systematic reviews (49, 98.0%), qualitative interviews or focus groups (43, 86.0%), and questionnaires (24, 48.0%) were commonly used. Other approaches included discussion among stakeholders such as town hall meetings (13, 26.0%), review of documents or medical records (2, 4.0%), and concept mapping (2, 4.0%). Among 8 (13.8%) studies that assessed determinants using a single approach, 4 (50.0%) employed literature reviews, 3 (37.5%) used questionnaires, and 2 (25.0%) used either qualitative interviews or focus groups.

Step 2: Program objectives.

All but 2 studies (58, 96.7%) explicitly stated Program Objectives including desired behavioural outcomes and change objectives among a specific population in either the form of a matrix or as stated objectives.

Step 3: Theoretical methods and practical strategies.

A total of 55 (91.7%) studies considered a Theoretical Framework to inform the selection and design of programs or interventions. Among these studies, 37 (67.3%) used more than one theory. Social Cognitive Theory (24, 43.6% of 55), the Health Belief Model (10, 18.2% of 55), and the Theory of Planned Behaviour (10, 18.2% of 55) were named most frequently by these studies.

Step 4: Program or intervention design.

Of the 56 (93.3%) studies that described Program Design, programs or interventions were pre-/pilot-tested by 34 (60.7%) studies to refine their design. For example, through qualitative interviews and satisfaction questionnaires with 23 patients and 12 caregivers, one study evaluated the acceptability of a heart failure self-help resource that aimed to engage patients and caregivers in exercise and symptom management [38]. This resource was reported acceptable and patients and caregivers were satisfied with the content and format of the resource. In another example, the authors evaluated an educational program that consisted of individual learning sessions for caregivers of patients with Alzheimer’s disease [75]. Using qualitative interviews with two caregivers, the authors found that both caregivers reported acquiring learning from the program, using reframing as a coping strategy, and improved communication with their affected relatives.

Among the 57 studies that described the program/intervention generated through use of the IM process, 40 (70.2%) targeted patients, 8 (14.0%) targeted clinicians, and 9 (15.8%) were aimed at both patients and clinicians. Of the 40 programs/interventions targeting patients, 24 (60.0%) described educational meetings or materials, 23 (57.5%) described social strategies such as clinicians supporting patients with head and neck cancer to determine their rehabilitation goals, actions, and coping plans [26], 8 (20.0%) were self-management apps or websites, and 3 (7.5%) were tools/guides, for example, a pre-chemotherapy online communication tool for patients to communicate with their nurses throughout treatment [35]. Among the 8 programs/interventions targeting clinicians, 6 (75.0%) involved educational materials or meetings and 3 (37.5%) involved point-of-care resources or tools. Of the 9 programs/interventions aimed at both patients and clinicians, 2 (22.2%) involved educational materials or meetings, 2 (22.2%) involved social strategies such as a resistance exercise coaching program for elderly individuals [34], and 2 (22.2%) employed point-of-care tools, for example, a tool to facilitate communication between patients and nurses [33].

Step 5: Adoption and implementation.

A total of 44 (73.3%) studies addressed Adoption/Implementation. Of these studies, 38 (86.4% of 44) mentioned a plan to implement the program or intervention. Of those, 24 (54.5%) studies offer non-specific or vague details. For example, expert groups were asked for feedback on program design and implementation [30], or an implementation guide was developed but no details were provided [57]. Remaining studies that mentioned an implementation plan (20, 45.5%) provided somewhat greater detail about who would implement the program or intervention and how. For example, details were provided about who generated an implementation plan for a 45-minute mental health counselling session and how the plan was devised, and details were provided on the implementation strategy, which comprised a 3-day training program for all clinicians, after which sessions were supervised by the first author with hands-on support as well as email and telephone support [71].

Step 6: Monitoring and evaluation.

Among 43 (71.2%) studies that addressed Monitoring/Evaluation, 20 (46.5%) provided rudimentary details about how the program or intervention would be evaluated in future, 19 (44.2%) described how the intervention was in the process of being evaluated but did not report the results, and 4 (9.3%) reported the impact or outcomes of program or intervention evaluation.

Of the 4 studies that reported the impact or outcomes of the program or intervention generated through use of the IM process, 3 reported positive findings [18,22,74] and 1 [65] reported mixed results. All 3 studies that reported positive findings reported all 6 IM steps [18,22,74]. The single study that achieved mixed results reported IM steps 2 to 6, but was published in 2009 before IM step 1, needs assessment, was added to the IM process [65]. The effectiveness of an educational intervention to increase colorectal cancer screening rates was evaluated using a randomized controlled trial in four groups of participants (men and women) from eight health centres [18]. Four months after the intervention, screening rates increased significantly to 87.1%, 61.3%, 54.8% and 1.6% for participants in the following intervention conditions: education and free Fetal Occult Blood Test (FOBT) screening, online education, free FOBT screening only, and control group, respectively (p<0.001). A randomized controlled trial was used to determine that an eating disorder prevention educational program and materials significantly improved knowledge about the causes and treatments of eating disorders among patients and health care providers (p<0.001) in 27 dental and dental hygiene training programs [22]. A pre- and post-questionnaire study found significantly improved expectations for positive patient outcomes and intention to prescribe diuretics among 147 health care educators following an educational program that comprised of in-person workshops that supported clinician educators to disseminate the results of a blood pressure treatment guideline [74]. In the fourth study, which reported conflicting results, a mixed methods design was used to evaluate an educational meeting that aimed to increase clinician adherence to a suicide prevention guideline [65]. Statistical modeling showed that the intervention did not have a statistically significant impact on adherence to the guideline (p>0.05, R2 = 0.27). A survey of health professionals showed that, apart from perceived behavioural control (p<0.027), there was no change in scores for attitude (p = 0.80), subjective norm (p = 0.76) or intention to use the guideline (p = 0.84).

Stakeholder involvement

S3 Table summarizes if and how stakeholders were engaged in the IM process. Among 60 included studies, 17 (28.3%) made no mention of involving stakeholders in any IM steps reported. Among the 43 (71.7%) studies that did mention stakeholder involvement, all but two (41, 95.3%) specified the types of stakeholders engaged. Most studies involved a combination of the following stakeholder groups: patients, clinicians, community organizations or representatives, and researchers; 12 (29.3% of 41) involved clinicians only, 5 (12.2% of 41) involved patients only, and 13 (31.7% of 41) involved both clinicians and patients. Community organizations and representatives were involved in 10 (24.4%) of the included studies.

Of the 43 studies that mentioned stakeholder involvement, less than half (21, 48.8%) described how they were engaged. In those 21 studies, stakeholders were members of committees or planning groups. The frequency of meetings was mentioned in 6 (28.6%) of the 21 studies and varied from quarterly, monthly or weekly, or a finite number of meetings held during the process. Four (19.0%) studies involved stakeholders on committees or groups in discussing the findings of Needs Assessment (Step 1). All 21 studies involved stakeholders in Program Design (Step 4) such as the review of program/intervention content and/or format (8, 38.1%), creation of program/intervention materials (7, 33.3%), recruitment of participants (6, 28.6%), and tailoring the program/intervention to the target population (3, 14.3%). Three (14.3% of 21) studies engaged stakeholders in developing program objectives and matrices of change outcomes. One (4.8%) study involved stakeholders in selecting the theory-based methods and strategies to guide the IM process. Two (9.5%) studies solicited stakeholder input and guidance on developing the intervention plan. Finally, 1 (4.8%) study involved stakeholders in reviewing the evaluation plan of intervention.

Discussion

The use of IM has steadily increased since 2008 and doubled from 2016 to 2017. IM was used in 13 countries to plan single or multi-faceted programs or interventions, which were largely based on educational materials or meetings aimed at patients, clinicians or both, to improve health care delivery or associated outcomes for a wide variety of clinical issues or conditions. Most studies reported having completed all 6 IM steps, and nearly all studies completed at least the first 4 IM steps, meaning that a program or intervention was developed. Activities undertaken for the first 4 steps, Needs Assessment, Program Objectives, Theoretical Framework and Program/Intervention Design, were well-reported by nearly all studies. While many studies reported how they planned to implement the program or intervention in future (step 5), fewer reported pilot-testing the program or intervention (part of step 4), or provided details about current or planned evaluation (step 6). Of only 4 studies that evaluated the program or intervention, 3 reported improvements in patient or clinician knowledge or behaviour. With respect to stakeholder engagement, 72% of studies said that patients, clinicians and/or representatives of community organizations were involved, but few studies reported the activities in which they were engaged or the impact of their engagement.

Two other studies reviewed the use of IM. Garba and Gadanya examined 22 IM studies published from 1999 to 2014 focused on designing disease prevention interventions but largely focused on whether interventions achieved the desired improvements when rigorously evaluated, rather than examining how the IM process was employed [79]. Their review found that IM resulted in significant uptake of disease prevention programs that were specifically evaluated using randomized controlled trials. Durks et al. examined how IM was used in 17 studies published up to May 2015 but focused only on the first 4 IM steps and largely described the health care practices that needed improvement and the proposed interventions [80]. Two additional reviews specific to particular disciplines examined the fidelity of the IM process and described interventions developed using IM. Similar to our review, both studies found that details of stakeholder engagement were sparse, and few interventions achieved the desired improvements. Fassier et al. conducted a systematic review of interventions developed with IM in work disability prevention from 2007 to August 2017 [81]. Among 8 studies included, stakeholder engagement was poorly described, and 2 interventions were reported as effective. Lamort-Bouché et al. conducted a systematic review of interventions developed with IM in the field of cancer up to August 2017 [82]. Stakeholder engagement was described in 6 of 16 studies, and 5 interventions were reported as effective. In contrast to these studies, our review is the most up-to-date, comprehensive of various health care issues or conditions, inclusive of all 6 IM steps, and descriptive of how stakeholders were engaged. We also included many more studies than prior reviews. This may in part be due to the fact that ours is the most recent and the number of IM studies doubled in the last two years of our review (2016, 2017). It may also be due to how we searched for studies; prior reviews employed search strategies, for which it is challenging to achieve an optimal balance of sensitivity and specificity, whereas we searched for any studies that included the phrase “intervention mapping” or cited the original publications that introduced the IM process [12,13], which may have been a more comprehensive approach.

The strengths of this study include use of rigorous scoping review methods [14] and compliance with standards for the conduct and reporting of reviews [15]. Several issues may limit the interpretation and application of the findings. Although scoping reviews often include consultation with stakeholders to interpret the findings, this was not done due to logistics and feasibility. Our search was limited to English language studies, and we did not search the grey literature given the methodological challenges of doing so that have been identified by others [83,84]; therefore, we may not have included all relevant studies. However, as noted earlier, our search approach resulted in a greater yield than previously published reviews of IM [79–82]. We did not search for other studies published by the same authors in an attempt to capture subsequent studies that may have evaluated the impact of the programs or intervention; however, few of the included studies reported evaluation results, and assessing the impact or outcomes of programs or interventions generated by IM was not our goal.

Several implications emerge in relation to the aims of this review, and inform issues that warrant ongoing research. The first aim was to gain insight on how to optimize IM. Overall, IM appears to be largely well-used and reported given the fidelity of details pertinent to its 6 steps. However, it is unclear if the use of IM consistently results in a successful program or intervention because few studies reported evaluation results. To be clear, step 6 of the IM process requires only that an evaluation plan be established, not actually operationalized. While some studies did describe ongoing or future evaluations, we included studies from 2008 to current and expected evaluations would have been published for earlier studies. This is confirmed by prior reviews, which also found that either interventions were not evaluated or they did not achieve improvements [81,82]. Reasons for the lack of evaluation or of effectiveness of programs or interventions generated using IM are unclear. They may have never undergone evaluation, or faced publication bias if the results of evaluation were negative. Given the sharp increase in IM use over the last two years, a future review is warranted to examine whether evaluations of programs or interventions developed more recently get published, and if those programs or innovations, informed by the engaged stakeholders, improve behaviour or outcomes. If effectiveness of programs or innovations developed using the IM process is still not established, then future research could seek to modify IM steps in an effort to optimize the process. That research should first assess whether lack of impact is due to attributes of the program or innovation or the way it was implemented, which could reflect limitations in how stakeholders were involved in the IM process, or barriers or challenges in the environment that prevent the program or innovation from being adopted, factors not related to the IM process that would necessitate other types of interventions that support program/innovation use.

The second aim of this research was to gain insight on how to optimize IKT for IM. Stakeholder engagement in IM is meant to increase the relevance, feasibility and ultimate success of programs or interventions. However, this review found that many studies did not specify if and how stakeholders were involved, or how involvement influenced program or intervention design, and those that did engage patients, clinicians and/or representatives of community agencies provided sparse details. It is unclear how stakeholder characteristics or involvement influenced the design, implementation or evaluation of the program or intervention. Thus, an understanding of how stakeholders should be chosen and involved in IM remains unknown. Although literature on IKT [9] and on patient/public engagement in health service planning and improvement is growing [85], optimal ways to choose and involve various stakeholders that are representative of the target users in research or in health service planning or improvement is limited. In our review, studies that described engagement were most likely to involve stakeholders via committees or planning groups in program design (step 4) by reviewing program/intervention content and/or format. This suggests that stakeholders provided input on already-established programs or interventions, rather than being involved from the outset, which may have resulted in a less-than-ideal program or intervention. If programs or interventions were not tailored based on stakeholder values and context, which has been shown to improve the likelihood of improved behaviour or outcomes [4], or if engagement was tokenistic, this could explain why programs or interventions developed using IM have not achieved the intended impact (previous refs). Future research should investigate the short-term impact of various ways to engage stakeholders on the design of programs or interventions, and the long-term impact of engagement options on the effectiveness of programs or interventions.

Conclusions

While IM is increasingly widely used many studies did not engage stakeholders, failed to describe how stakeholders were engaged, or involved stakeholders in a tokenistic manner, and many studies did not pilot-test or evaluate the impact or outcomes of the program or intervention. Therefore, the design of programs or interventions may be suboptimal because they were not informed by and tailored to address stakeholder context, and it is unclear if the use of IM consistently results in effective programs or interventions. Those employing IM should more thoroughly report how stakeholders were involved in each IM step and how involvement influenced program or intervention design. They should also report details of ongoing or planned evaluation, or specify if no such evaluation is planned and the rationale. Future research should investigate how to optimize stakeholder engagement in IM, and whether use of IM itself or stakeholder engagement in IM are positively associated with effective programs or interventions.

Supporting information

S1 Table. PRISMA checklist.

https://doi.org/10.1371/journal.pone.0209826.s001

(DOCX)

S2 Table. Intervention Mapping process.

https://doi.org/10.1371/journal.pone.0209826.s002

(DOCX)

S3 Table. Data extracted from included studies.

https://doi.org/10.1371/journal.pone.0209826.s003

(DOCX)

References

  1. 1. Field B, Booth A, Ilott I, Gerrish K. Using the Knowledge to Action Framework in practice: a citation analysis and systematic review. Implement Sci. 2014;9:172. pmid:25417046
  2. 2. Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko M, et al. A checklist for identifying determinants of practice: A systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:1–11.
  3. 3. Krause J, Van Lieshout J, Klomp R, Huntink E, Aakhus E, Flottorp S, et al. Identifying determinants for tailoring implementation in chronic diseases: an evaluation of different methods. Implement Sci. 2014;9:102. pmid:25112492
  4. 4. Baker R, Camosso-Stefinovic J, Gillies C, Shaw EJ, Cheater F, Flottorp S, et al. Tailored interventions to overcome identified barriers to change: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2015;4:CD005470.
  5. 5. Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci 2015, 10: 21. pmid:25889199
  6. 6. Michie S, Wood CE, Johnston M, Abraham C, Francis JJ, Hardeman W. Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data). Health Technol Assess 2015;19:1–188.
  7. 7. Aakhus E, Granlund I, Odgaard-Jensen J, Oxman AD, Flottorp SA. A tailored intervention to implement guideline recommendations for elderly patients with depression in primary care: a pragmatic cluster randomised trial. Implement Sci 2016;11:32. pmid:26956726
  8. 8. Van Lieshout J, Huntink E, Koetsenruijter J, Wensing M. Tailored implementation of cardiovascular risk management in general practice: a cluster randomized trial. Implement Sci 2016;11:115. pmid:27515970
  9. 9. Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci 2016;11:38. pmid:26988000
  10. 10. Currie G, Lockett A, El Enany N. From what we know to what we do: lessons learned from the translational CLAHRC initiative in England. J Health Serv Res Policy 2013;18(Suppl 3):27–39.
  11. 11. Molleman G, Fransen G. Academic collaborative centres for health promotion in the Netherlands: building bridges between research, policy and practice. Fam Pract 2012;29(Suppl 1):157–62.
  12. 12. Bartholomew LK, Parcel GS, Kok G. Intervention mapping: a process for developing theory- and evidence-based health education programs. Health Educ Behav 1998;25:545–63. pmid:9768376
  13. 13. Fernandez ME, Gonzales A, Tortolero-Luna G, Partida S, Bartholomew LK. Using intervention mapping to develop a breast and cervical cancer screening program for Hispanic farm workers. Health Promot Pract 2005;6:394–404. pmid:16210681
  14. 14. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005;8:19–32.
  15. 15. Moher D, Liberati A, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6:e1000097. pmid:19621072
  16. 16. Albrecht L, Archibald M, Arseneau D, Scott SD. Development of a checklist to assess the quality of reporting of knowledge translation interventions using the Workgroup for Intervention Development and Evaluation Research (WIDER) recommendations. Implement Sci 2013; 8: 52. pmid:23680355
  17. 17. Athilingam P, Clochesy J, Labrador M. Intervention Mapping Approach in the Design of an Interactive Mobile Health Application to Improve Self-care in Heart Failure. CIN: Computers, Informatics, Nursing. 2017;36(2):90–97.
  18. 18. Besharati F, Karimi-Shahanjarini A, Hazavehei S, Bashirian S, Bagheri F, Faradmal J. Development of a Colorectal Cancer Screening Intervention for Iranian Adults: Appling Intervention Mapping. Asian Pac J Cancer Prev. 2017;18(8):2193–2199. pmid:28843255
  19. 19. Caminiti C, Schulz P, Marcomini B, Iezzi E, Riva S, Scoditti U et al. Development of an education campaign to reduce delays in pre-hospital response to stroke. BMC Emergency Medicine. 2017;17(1).
  20. 20. Cho S, Lee H, Kim J, Lee M, Lee Y. Culturally Adaptive Walking Intervention for Korean-Chinese Female Migrant Workers. Research and Theory for Nursing Practice. 2017;31(2):179–196. pmid:28482996
  21. 21. Côté J, Cossette S, Ramirez-Garcia P, Rouleau G, Auger P, Boudreau F et al. Improving Health and Reducing Comorbidity Associated with HIV: The Development ofTAVIE en santé, a Web-Based Tailored Intervention to Support the Adoption of Health Promoting Behaviors among People Living with HIV. BioMed Research International. 2017;2017:1–10.
  22. 22. DeBate R, Bleck J, Raven J, Severson H. Using Intervention Mapping to Develop an Oral Health e-Curriculum for Secondary Prevention of Eating Disorders. Journal of Dental Education. 2017;81(6):716–725. pmid:28572418
  23. 23. Dumas A, Lemieux S, Lapointe A, Provencher V, Robitaille J, Desroches S. Development of an Evidence-Informed Blog to Promote Healthy Eating Among Mothers: Use of the Intervention Mapping Protocol. JMIR Research Protocols. 2017;6(5):e92. pmid:28526669
  24. 24. Golsteijn R, Bolman C, Volders E, Peels D, de Vries H, Lechner L. Development of a computer-tailored physical activity intervention for prostate and colorectal cancer patients and survivors: OncoActive. BMC Cancer. 2017;17(1).
  25. 25. Krops L, Hols D, Folkertsma N, Dijkstra P, Geertzen J, Dekker R. Requirements on a community-based intervention for stimulating physical activity in physically disabled people: a focus group study amongst experts. Disability and Rehabilitation. 2017;:1–8.
  26. 26. McEwen S, Dunphy C, Rios J, Davis A, Jones J, Lam A et al. Development and pre-testing of a rehabilitation planning consultation for head-and-neck cancer. Current Oncology. 2017;24(3):153. pmid:28680274
  27. 27. Merkx A, Ausems M, de Vries R, Nieuwenhuijze M. Come On! Using intervention mapping to help healthy pregnant women achieve healthy weight gain. Public Health Nutrition. 2017;20(09):1666–1680.
  28. 28. Miranda J, Côté J. The Use of Intervention Mapping to Develop a Tailored Web-Based Intervention, Condom-HIM. JMIR Public Health and Surveillance. 2017;3(2):e20. pmid:28428162
  29. 29. Muir S, Newell C, Griffiths J, Walker K, Hooper H, Thomas S et al. MotivATE: A Pretreatment Web-Based Program to Improve Attendance at UK Outpatient Services Among Adults With Eating Disorders. JMIR Research Protocols. 2017;6(7):e146. pmid:28747295
  30. 30. Puijk-Hekman S, van Gaal B, Bredie S, Nijhuis-van der Sanden M, van Dulmen S. Self-Management Support Program for Patients With Cardiovascular Diseases: User-Centered Development of the Tailored, Web-Based Program Vascular View. JMIR Research Protocols. 2017;6(2):e18. pmid:28179214
  31. 31. Sakakibara B, Lear S, Barr S, Benavente O, Goldsmith C, Silverberg N et al. Development of a Chronic Disease Management Program for Stroke Survivors Using Intervention Mapping. Archives of Physical Medicine and Rehabilitation. 2017;98(6):1195–1202. pmid:28219685
  32. 32. Shegog R, Begley C. Clinic-Based Mobile Health Decision Support to Enhance Adult Epilepsy Self-Management: An Intervention Mapping Approach. Frontiers in Public Health. 2017;5.
  33. 33. van Belle E, Zwakhalen S, Caris J, Van Hecke A, Huisman-de Waal G, Heinen M. Tailoring of the Tell-us Card communication tool for nurses to increase patient participation using Intervention Mapping. Journal of Clinical Nursing. 2017;27(3–4):62630.
  34. 34. van Dongen E, Leerlooijer J, Steijns J, Tieland M, de Groot L, Haveman-Nies A. Translation of a tailored nutrition and resistance exercise intervention for elderly people to a real-life setting: adaptation process and pilot study. BMC Geriatrics. 2017;17(1).
  35. 35. van Dulmen S, Driesenaar J, van Weert J, van Osch M, Noordman J. PatientVOICE: Development of a Preparatory, Pre-Chemotherapy Online Communication Tool for Older Patients With Cancer. JMIR Research Protocols. 2017;6(5):e85. pmid:28490421
  36. 36. Beentjes T, van Gaal B, Goossens P, Schoonhoven L. Development of an e-supported illness management and recovery programme for consumers with severe mental illness using intervention mapping, and design of an early cluster randomized controlled trial. BMC Health Services Research. 2016;16(1).
  37. 37. Dalum P, Brandt C, Skov-Ettrup L, Tolstrup J, Kok G. The Systematic Development of an Internet-Based Smoking Cessation Intervention for Adults. Health Promotion Practice. 2016;17(4):490–500. pmid:27101996
  38. 38. Greaves C, Wingham J, Deighan C, Doherty P, Elliott J, Armitage W et al. Optimising self-care support for people with heart failure and their caregivers: development of the Rehabilitation Enablement in Chronic Heart Failure (REACH-HF) intervention using intervention mapping. Pilot and Feasibility Studies. 2016;2(1).
  39. 39. Hall C, Hall A, Kok G, Mallya J, Courtright P. A needs assessment of people living with diabetes and diabetic retinopathy. BMC Research Notes. 2016;9(1).
  40. 40. Jones T, Dear B, Hush J, Titov N, Dean C. Application of Intervention Mapping to the Development of a Complex Physical Therapist Intervention. Physical Therapy. 2016;96(12):1994–2004. pmid:27256070
  41. 41. Kerstenetzky L, Birschbach M, Beach K, Hager D, Kennelty K. Improving medication information transfer between hospitals, skilled-nursing facilities, and long-term-care pharmacies for hospital discharge transitions of care: A targeted needs assessment using the Intervention Mapping framework. Research in Social and Administrative Pharmacy. 2016;14(2):138–145.
  42. 42. Norris S, Ho J, Rashed A, Vinding V, Skau J, Biesma R et al. Pre-pregnancy community-based intervention for couples in Malaysia: application of intervention mapping. BMC Public Health. 2016;16(1).
  43. 43. Romeike K, Lechner L, de Vries H, Oenema A. Development of a computer-tailored nutrition and physical activity intervention for lower-educated women of Dutch, Turkish and Moroccan origin using content matching and ethnic identity tailoring. BMC Public Health. 2016;16(1).
  44. 44. Shakibazadeh E, Bartholomew L, Rashidian A, Larijani B. Persian Diabetes Self-Management Education (PDSME) program: evaluation of effectiveness in Iran. Health Promotion International. 2016;31(3):623–634. pmid:25761821
  45. 45. Smith S, Whitehead M, Sheats J, Fontenot B, Alema-Mensah E, Ansa B. Formative research to develop a lifestyle application (app) for African American breast cancer survivors. Journal of the Georgia Public Health Association. 2016;6(1).
  46. 46. Steffen A, Epstein J, George N, MacDougall M. The Sandwich Generation Diner: Development of a Web-Based Health Intervention for Intergenerational Caregivers. JMIR Research Protocols. 2016;5(2):e91. pmid:27269632
  47. 47. Gray WN, Resmini AR, Baker KD, Holbrook E, Morgan PJ, Ryan J et al. Concerns, barriers, and recommendations to improve transition from pediatric to adult IBD care: Perspectives of patients, parents, and health professionals. Inflamm Bowel Dis. 2015;21:1641–51. pmid:25966837
  48. 48. Highfield L, Hartman MA, Mullen PD, Rodriguez SA, Fernandez ME, Bartholomew LK. Intervention Mapping to Adapt Evidence-Based Interventions for Use in Practice: Increasing Mammography among African American Women. BioMed research international. 2015.
  49. 49. Cabassa LJ, Gomes AP, Meyreles Q, Capitelli L, Younge R, Dragatsi D et al. Using the collaborative intervention planning framework to adapt a health-care manager intervention to a new population and provider group to improve the health of people with serious mental illness. Implement Sci. 2014;9:1.
  50. 50. Geidl W, Semrau J, Pfeifer K. Health behaviour change theories: contributions to an ICF-based behavioural exercise therapy for individuals with chronic diseases. Disabil Rehabil. 2014;36:2091–100. pmid:24564358
  51. 51. Hesselink G, Zegers M, Vernooij-Dassen M, Barach P, Kalkman C, Flink M et al. Improving patient discharge and reducing hospital readmissions by using Intervention Mapping. BMC Health Serv Res. 2014;14:1.
  52. 52. Pyne JM, Fischer EP, Gilmore L, McSweeney JC, Stewart KE, Mittal D et al. Development of a Patient-Centered Antipsychotic Medication Adherence Intervention. Health Educ Behav. 2014;41:315–324. pmid:24369177
  53. 53. De Brito‐Ashurst I, Perry L, Sanders TA, Thomas JE, Dobbie H, Yaqoob MM. Applying research in nutrition education planning: a dietary intervention for Bangladeshi chronic kidney disease patients. J Hum Nutr Diet. 2013;26:403–13. pmid:23240718
  54. 54. Laisaar KT, Uusküla A, Sharma A, DeHovitz JA, Amico KR. Developing an adherence support intervention for patients on antiretroviral therapy in the context of the recent IDU-driven HIV/AIDS epidemic in Estonia. AIDS Care. 2013;25:863–73. pmid:23391132
  55. 55. Munir F, Kalawsky K, Wallis DJ, Donaldson-Feilder E. Using intervention mapping to develop a work-related guidance tool for those affected by cancer. BMC Public Health. 2013;13:1.
  56. 56. Theunissen KA, Hoebe CJ, Crutzen R, Kara-Zaïtri C, de Vries NK, van Bergen JE et al. Using intervention mapping for the development of a targeted secure web-based outreach strategy named SafeFriend, for Chlamydia trachomatis testing in young people at risk. BMC Public Health. 2013;13:1.
  57. 57. Byrd TL, Wilson KM, Smith JL, Heckert A, Orians CE, Vernon SW et al. Using Intervention Mapping as a Participatory Strategy Development of a Cervical Cancer Screening Intervention for Hispanic Women. Health Educ Behav. 2012;39:603–11. pmid:22388451
  58. 58. Cherrington A. Intervention mapping as a guide for the development of a diabetes peer support intervention in rural Alabama. Preventing Chronic Disease. 2012;9.
  59. 59. Cornélio ME, Godin G, Rodrigues R, Agondi R, Spana T, Gallani MC. Development of the SALdavel programme to reduce salt intake among hypertensive Brazilian women: an intervention mapping approach. Eur J Cardiovasc Nur. 2012;12:385–92.
  60. 60. Gillison F, Greaves C, Stathi A, Ramsay R, Bennett P, Taylor G et al. ‘Waste the Waist’: the development of an intervention to promote changes in diet and physical activity for people with high cardiovascular risk. Brit J Health Psych. 2012;17:327–45.
  61. 61. Noordegraaf AV, Huirne JA, Pittens CA, van Mechelen W, Broerse JE, Brölmann HA et al. eHealth program to empower patients in returning to normal activities and work after gynecological surgery: intervention mapping as a useful method for development. J Med Internet Res. 2012;14:e124. pmid:23086834
  62. 62. Scarinci IC, Bandura L, Hidalgo B, Cherrington A. Development of a theory-based (PEN-3 and health belief model), culturally relevant intervention on cervical cancer prevention among Latina immigrants using intervention mapping. Health Promotion Practice. 2012;13:29–40. pmid:21422254
  63. 63. Suzuki R, Peterson JJ, Weatherby AV, Buckley DI, Walsh ES, Kailes JI et al. Using intervention mapping to promote the receipt of clinical preventive services among women with physical disabilities. Health promotion practice. 2012;13:106–15. pmid:21059870
  64. 64. Zwikker HE, van den Bemt B, van den Ende C, van Lankveld W, den Broeder A, van den Hoogen F et al. Development and content of a group-based intervention to improve medication adherence in non-adherent patients with rheumatoid arthritis. Patient Educ Couns. 2012;89:143–51. pmid:22878029
  65. 65. Hanbury A, Wallace LM, Clark M. Multiple outcome measures and mixed methods for evaluating the effectiveness of theory-based behaviour-change interventions: a case study targeting health professionals' adoption of a national suicide prevention guideline. Psychology, health & medicine. 2011;16:291–303.
  66. 66. Looijmans-van den Akker I, Hulscher ME, Verheij TJ, Riphagen-Dalhuisen J, van Delden JJ, Hak E. How to develop a program to increase influenza vaccine uptake among workers in health care settings?. Implement Sci. 2011;6:1.
  67. 67. Looijmans-Van Den Akker I, Van Delden JJ, Verheij TJ, van der Sande MA, Van Essen GA, Riphagen-Dalhuisen J et al. Effects of a multi-faceted program to increase influenza vaccine uptake among health care workers in nursing homes: A cluster randomised controlled trial. Vaccine. 2010;28:5086–92. pmid:20580740
  68. 68. Van Der Veen YJ, Van Empelen P, Richardus JH. Development of a culturally tailored Internet intervention promoting hepatitis B screening in the Turkish community in the Netherlands. Health Promot Int. 2011:dar044.
  69. 69. Van Rijssen HJ, Schellart AJ, Anema JR, de Boer WE, van der Beek AJ. Systematic development of a communication skills training course for physicians performing work disability assessments: from evidence to practice. BMC medical education. 2011;11:1.
  70. 70. Detaille SI, van der Gulden JW, Engels JA, Heerkens YF, van Dijk FJ. Using intervention mapping (IM) to develop a self-management programme for employees with a chronic disease in the Netherlands. BMC Public Health. 2010;10:1.
  71. 71. Koekkoek B, van Meijel B, Schene A, Hutschemaekers G. Development of an intervention program to increase effective behaviours by patients and clinicians in psychiatric services: Intervention Mapping study. BMC Health Serv Res. 2010;10:1.
  72. 72. Schmid AA, Andersen J, Kent T, Williams LS, Damush TM. Using intervention mapping to develop and adapt a secondary stroke prevention program in Veterans Health Administration medical centers. Implement Sci. 2010;5:1.
  73. 73. Albada A, Van Dulmen S, Otten R, Bensing JM, Ausems MG. Development of E-info geneca: A website providing computer-tailored information and question prompt prior to breast cancer genetic counseling. Genet Counsel. 2009;18:326–38.
  74. 74. Bartholomew LK, Cushman WC, Cutler JA, Davis BR, Dawson G, Einhorn PT et al. Getting clinical trial results into practice: design, implementation, and process evaluation of the ALLHAT Dissemination Project. Clin Trials. 2009;6:329–43.
  75. 75. Ducharme F, Beaudet L, Legault A, Kergoat MJ, Lévesque L, Caron C. Development of an intervention program for Alzheimer's family caregivers following diagnostic disclosure. Clinical Nursing Research. 2009;18:44–67. pmid:19208820
  76. 76. Ramirez-Garcia P, Côté J. Development of a nursing intervention to facilitate optimal antiretroviral-treatment taking among people living with HIV. BMC Health Serv Res. 2009;9:1.
  77. 77. Côté J, Godin G, Garcia PR, Gagnon M, Rouleau G. Program development for enhancing adherence to antiretroviral therapy among persons living with HIV. AIDS Patient Care ST. 2008;22:965–75.
  78. 78. Fransen GA, Hiddink GJ, Koelen MA, Van Dis SJ, Drenthen AJ, Van Binsbergen JJ et al. The development of a minimal intervention strategy to address overweight and obesity in adult primary care patients in The Netherlands. Fam Pract. 2008;25 Suppl 1:i112–5.
  79. 79. Garba RM, Gadanya MA. The role of intervention mapping in designing disease prevention interventions: A systematic review of the literature. PLOS One. 2017;12:e0174438. pmid:28358821
  80. 80. Durks D, Fernandez-Llimos F, Hossain LN, Franco-Trigo L, Benrimoi SI, Sabater-Hernandez D. Use of intervention mapping to enhance health care professional practices: A systematic review. Health Educ Behav. 2017;44:524–35. pmid:28580805
  81. 81. Fassier JB, Sarnin P, Rouat S, Peron J, Kok G, Letrilliart L, et al. Interventions developed with the intervention mapping protocol in work disability prevention: A systematic review of the literature. J Occup Rehabil. 2018; pmid:29948471
  82. 82. Lamort-Buoche M, Sarnin P, Kok G, Rouat S, Peron J, Letrilliart L, et al. Interventions developed with the Intervention Mapping protocol in the field of cancer: A systematic review. Psychooncology. 2018;27:1138–49. pmid:29247578
  83. 83. Benzies KM, Premji S, Hayden KA, Serrett K. State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs. 2006;3:55–61. pmid:17040510
  84. 84. Adams J, Hillier-Brown FC, Moore HJ, Lake AA, Araujo-Soares V, White M, et al. Searching and synthesising ‘grey literature’ and ‘grey information’ in public health: Critical reflections on three case studies. Syst Rev. 2016;5:164. pmid:27686611
  85. 85. Liang L, Cako A, Urquhart R, Straus SE, Wodchis WP, Baker GR, et al. Patient engagement in hospital health service planning and improvement: A scoping review. BMJ Open. 2018;8:e018263. pmid:29382676