Figures
Abstract
Background
With the evolution of treatments for neurological diseases, the contents of core neurological examinations (NEs) for medical students may need to be modified. We aimed to establish a consensus on the core NE items for neurology clerks and compare viewpoints between different groups of panelists.
Methods
First, a pilot group proposed the core contents of NEs for neurology clerks. The proposed core NE items were then subject to a modified web-based Delphi process using the online software “SurveyMonkey”. A total of 30 panelists from different backgrounds (tutors or learners, neurologists or non-neurologists, community hospitals or medical centers, and different academic positions) participated in the modified Delphi process. Each panelist was asked to agree or disagree on the inclusion of each item using a 9-point Likert scale and was encouraged to provide feedback. We also compared viewpoints between different groups of panelists using the Mann-Whitney U test.
Results
Eighty-three items were used for the first round of the Delphi process. Of them, 18 without consensus of being a core NE item for the neurology clerks in the first round and another 14 items suggested by the panelists were further discussed in the second round. Finally, 75 items with different grades were included in the recommended NE items for neurology clerks.
Citation: Liu C-H, Hsu L-L, Hsiao C-T, Hsieh S-I, Chang C-W, Huang ES, et al. (2018) Core neurological examination items for neurology clerks: A modified Delphi study with a grass-roots approach. PLoS ONE 13(5): e0197463. https://doi.org/10.1371/journal.pone.0197463
Editor: Jong-Ling Fuh, Taipei Veterans General Hospital, TAIWAN
Received: January 21, 2018; Accepted: May 2, 2018; Published: May 17, 2018
Copyright: © 2018 Liu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This work was provided by Chang Gung Memorial Hospital (grant numbers: CMRPG3G0261 and NMRPG3G6411-2) and Ministry of Science and Technology (grant number: 106-2511-S-182A-002 -MY2) to Chi-Hung Liu. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Neurology is regarded as a difficult component of the medical curriculum. Neurophobia is defined as a fear of neurological diseases, and it is a recognized problem among medical students which may prevent them using their basic neurological knowledge at the bedside [1–6]. In addition, this can impede a young doctor’s motivation and confidence of learning neurology and being a neurologist [2, 4, 7, 8]. Neurophobia may start early in medical school [9]. Previous studies have suggested effective strategies to cure neurophobia [9]. Of them, transformation of teaching methods in neurological examinations (NEs) has been shown to play an important role [9]. Recently, the concept of milestone development has been integrated into medical education [10]. Different levels of learners should have different entrustable professional activities (EPAs) and corresponding skills in NEs [11–13]. Well trained doctors should be able to efficiently perform task-specific NEs to target the chief complaints and main symptoms, however, performing comprehensive NEs is difficult for beginners. Inadequate preparedness has been associated with stress and anxiety in medical students [14]. It is reasonable that students should learn about NEs sequentially, starting with the basic but essential items and then advancing to comprehensive knowledge. The essential NE items have previously been established in the core curriculum of neurology clerkship [15]. With the evolution in epidemiological distribution and treatment of neurological diseases, investigations are needed to determine whether the contents of NE education should be modified in the current era.
Using a modified Delphi process [16–18], the aim of this study was to update the consensus on core NE items that should be taught during the NE milestones in neurology clerkships based on the concept of a grass-roots approach [19, 20]. Consensus was made by panelists with different backgrounds, and in particular learners and clinical tutors. We also compared viewpoints between different groups of panelists.
Materials and methods
Definition of neurology clerkship
In Taiwan, medical students enter medical school after graduating from senior high school and study there for 6 years. Thereafter, they need to receive 2 years of post-graduate-year training, after which they can apply for residency training. Neurology clerks are 5th grade medical students who have completed basic medical subjects and lectures on clinical neurology before starting clinical rotation. These students have very little experience in clinical neurology or performing NEs. We assumed that these students would be more likely to experience frustration when encountering the complicated and numerous items of comprehensive NEs.
The pilot group
We enrolled five attending physicians from the Neurology Department of Chang Gung Memorial Hospital in the pilot group (Fig 1). These participants had different subspecialties in neurology and had actively contributed to medical education for more than 3 years. The objective of the pilot group was to generate a list of proposed core NE items for neurology clerks using NE textbooks (e.g. DeJong’s The Neurologic Examination, 7th edition) as a reference using a LINE (an online chat application) discussion group. These items were then used in a Delphi process (S1 and S2 Tables).
This figure showing the evolution and decision making process of all of the NE items concluded from the pilot group and each round of the modified Delphi process. NE = neurological examinations.
The modified Delphi process
The panelists.
Representativeness and diversity are key factors for panelist selection in the modified Delphi process, and feedback on the elements of training may help to improve this problem [21, 22]. The needs of learners and clinical tutors are important in a grass-roots approach [19]. Therefore, professors, clinical tutors, and learners were the main targets for the panelists in this study [23]. A total of 30 panelists were enrolled for the modified Delphi process, who were selected from different learning (tutors and learners), expertise (neurologists and non-neurologists), levels of teaching hospital (community hospitals and medical centers), and academic positions. All of the selected learner panelists in this study had qualified academic performance in their neurology class (Table 1). As the neurology clerks may have had limited experience in neurology and therefore may have underestimated or overestimated the true significance of each NE item, we selected the learner panelists from different levels (medical students, interns, and residents). Six general neurologists came from community hospitals. We also enrolled 12 experts from medical centers with different subspecialties, including six from Chang-Gung Memorial Hospital and six from other medical centers. The Ethics Institutional Review Board of Chang Gung Memorial Hospital approved this study (104-9585B), and all of the panelists provided written inform consent.
The survey tool.
In the traditional Delphi process, meetings should be held with all of the participants. This is time consuming, expensive, and difficult to arrange the attendance of all the clinical physicians. Therefore, in this study, we designed a web-based online anonymous modified Delphi process. We used the online software SurveyMonkey (SM; https://zh.surveymonkey.com/) to overcome the problems of distance and time. Personal e-mails were used as the main connection between the principal investigator and the panelists. SM sent e-mails to each panel member with a URL linking to the survey. We first sent a greeting e-mail with educational background surveys before the first round of the modified Delphi process. This step helped to reveal the self-reported expertise, age, and years of neurological education of the panelists. We also sent follow-up reminder e-mails to the non-responders in each round. Each survey round was available for 1 week. We copied the list of responders from each round into new recipient lists for subsequent rounds.
The first round.
In the first round, we requested the panelists to evaluate whether an item should be included or excluded from the core NE items for the neurology clerks. We asked each panelist to agree or disagree with the inclusion of each item using a 9-point Likert scale, from 1 (strongly disagree) to 9 (strongly agree). We discarded the items if less than 10% of the panelists responded to them. We then discussed the items with response rates between 10% and 90% in the next round, and discussed the items with a response rate over 90% in this round. Of them, we defined the items with a score of 9 as strong agreement, and those with scores of 7 and 8 as agreement. The items with the scores between 4 and 6 were also discussed in the next round. We discarded the items with scores between 1 and 3. We asked the 30 panelists to provide feedback, and we also asked them to suggest NE items that were not initially included. We also confirmed the opinions of the panelists by phone to make sure that their suggestions were expressed correctly. The suggested NE items were further evaluated in the next round (Fig 1 and S1 Table).
The second round.
We revised the proposed core NE items according to the suggestions and ratings in the first round, and drafted a revised list for the second round. In the second round, we discarded the items if less than 80% of the panelists responded to them, and discussed the items with a response rate of over 80%. Of these items, we discarded the items with scores between 1 and 6. For the items with scores between 7 and 9, those with a response rate less than 90% were also discarded. Following this round, all items were either included or excluded and consensus was achieved (Fig 1 and S1 Table).
Data collection
We recorded the percentage of agreement of the NE items and used the 9-point Likert scale to register the importance of these items in each round of the modified Delphi process. Finally, we generated a recommended core list of NE items for neurology clerks.
Statistical analysis
All statistical analyses were performed using the Statistical Package for the Social Sciences (IBM SPSS Statistics version 22.0). The Likert scale rating for each NE item from all of the panelists was expressed as median (quartile 1, quartile 3). Items with median scores between 7 and 9 in the first or second round were regarded to be recommended NE items. In addition, we further grouped the panelists based on their backgrounds (tutors and learners, neurologists and non-neurologists, experts in medical center and others) to compare differences in perspective between them using the Mann-Whitney U test (nonparametric data). Statistical significance was set at p < 0.05.
Results
The proposed core NE items for neurology clerks from the pilot group
In total, 98 items were initially provided for open discussion in the pilot group meeting (S2 Table). Some of these items were discarded as they were considered to be too difficult for medical students or combined to make the topic more comprehensive. Finally, 83 items were included in the proposed core list of NE items for further discussion in the modified Delphi process (Fig 1 and S3 Table).
Background characteristics of the panelists
Among the panelists, 12 (40%) had experience of performing NEs for 0–10 years, and 18 (60%) had more than 11 years of experience. Fourteen of the panelists (46.7%) had experience of instructing NEs to students for more than 11 years, and 8 (26.7%) had less than 10 years of experience. The panelists included 12 learners (40%) and 18 tutors (60%). Among the 18 tutors, three (16.7%) were professors, four (22.2%) were associate professors, and five (27.8%) were assistant professors. Overall, 33.3% of the tutors were general neurologists in community hospitals. In addition, 10 (55.6%) of the 18 tutors had experience in curriculum design or had served as a program director. The most common expertise of the panelists was general neurology (55.6%), followed by headache (38.9%), cerebrovascular disease (33.3%), and neuro-critical care (33.3%). The panelists in this study covered most fields of subspecialties of clinical neuroscience (Table 2).
Results of the first round of the modified Delphi process
Twenty-nine (96.7%) of the 30 panelists completed the ratings. All 83 of the core proposed NE items were used for the first round of the modified Delphi process, of which 65 (78.3%) had agreement or strong agreement. Of these 65 items, 16 were strongly recommended as the core competence of neurology clerkship with a rating of 9 (S4 Table). However, 18 (21.7%) of the 83 items were put in the second round for further confirmation. Another 14 items were suggested to be added to the core list of NE items by the panelists in the first round (Fig 1). The panelists mentioned the main reasons why these items were considered to be important for neurology clerks in the “comment column” of the online questionnaires (S5 Table).
Results of the second round of the modified Delphi process
The 18 items that were not considered to be core competence for neurology clerks in the first round were discarded after thorough reconsideration in the second round of the modified Delphi process. The ratings of these items were consistent between the two rounds. Ten of the 14 (71.4%) added items suggested by the panelists were regarded to be core competence for neurology clerkship (S5 Table). The 22 items discarded during the Delphi process were summarized in the S6 Table. Finally, 75 items were included in the recommended core list of NE items for neurology clerks (Fig 1 and Table 3).
Differences in viewpoints between the panelists with diverse backgrounds
Tutors vs. learners.
Twelve learners (residents or students) and 17 tutors (attending physicians) completed the questionnaires. Between these two groups, the main differences in viewpoint were touching the pharyngeal wall with a cotton wool stick (tutors vs. learners, 7 vs. 9, p = 0.04), checking muscle strength innervated by different nerves (tutors vs. learners, 6 vs. 8, p = 0.03), checking breathing sound (tutors vs. learners, 7 vs. 9, p = 0.01), neck lymph node examination (tutors vs. learners, 6 vs. 9, p < 0.01), asking about erectile function (tutors vs. learners, 5 vs. 6, p = 0.04), starch test (tutors vs. learners, 2 vs. 3.5, p = 0.02), and assessing basic mood status (tutors vs. learners, 7 vs. 8.5, p = 0.02) (Table 4).
Neurologists vs. non-neurologists.
Comparing the 19 neurologists (tutors and residents of the neurology department) with the 10 non-neurologists, the non-neurologists strongly favored checking lacrimation/salivation (neurologists vs. non-neurologists, 4 vs. 6.5, p = 0.02), breathing sounds (neurologists vs. non-neurologists, 7 vs. 9, p = 0.02), neck lymph node examination (neurologists vs. non-neurologists, 6 vs. 9, p < 0.01), starch test (neurologists vs. non-neurologists, 2 vs. 3.5, p = 0.04), and assessing basic mood status (neurologists vs. non-neurologists, 7 vs. 8.5, p = 0.05). The neurologists thought that listening to the carotid pulse (neurologists vs. non-neurologists, 7 vs. 6, p = 0.03), and checking pronator drift (neurologists vs. non-neurologists, 9 vs. 7, p = 0.02) were the most important NE items for neurology clerks (Table 4).
Experts in medical centers vs. others.
The experts in medical centers considered that checking breathing sounds (experts in medical centers vs. others, 6 vs. 9, p < 0.01), checking pulse and heart rate (experts in medical centers vs. others, 7.5 vs. 9, p = 0.03), neck lymph node examination (experts in medical centers vs. others, 4 vs. 9, p < 0.01), understanding the definition of coma, semi-coma, stupor, confusion, delirium, and dementia (experts in medical centers vs. others, 8 vs. 9, p = 0.02), and assessing basic mood status (experts in medical centers vs. others, 5.5 vs. 8, p < 0.01) were less important NE items for neurology clerks (Table 4).
Discussion
The 16 items with strong agreement in this study were identical to those in previous guidelines for a comprehensive NE for neurology clerks [15, 24]. Our study further highlights the importance of performing physical examinations in neurological patients. In addition, our findings emphasized the importance of NE items on the autonomic system, meningeal irritation, acute ischemic stroke (AIS) screening and patient selection for thrombolytic therapy, abnormal gait and phenomenology of abnormal movements. All of these factors can be associated with the advancements in diagnosis and treatment of neurological emergencies and the increased epidemiological trend of degenerative diseases.
Previous recommendations from the American Academy of Neurology (AAN) stated that dealing with neurological emergencies is essential content that should be taught to neurology clerks (https://www.aan.com/siteassets/home-page/tools-and-resources/academic-neurologist--researchers/clerkship-and-course-director-resources/neurology-clerkship-core-curriculum-guidelines.new.pdf). Meningitis and subarachnoid hemorrhage are common life-threatening neurological emergencies, and a delayed diagnosis of these diseases may lead to serious consequences. Checking meningeal irritation is a simple method to screen these diseases, however it was not listed in the guidelines for a comprehensive NE. Our panelists generally agreed that all neurology clerks should be familiar with this skill. With improvements in intravenous thrombolysis and mechanical thrombectomy, the aim of AIS treatment now includes the early detection and early reperfusion [25]. It is therefore reasonable that all primary physicians, not only neurologists, should be aware of the major signs of AIS and should have the ability to screen the patients who might be candidates for reperfusion therapy. In the current study, the tutors and neurologists placed more emphasis on the items of carotid bruit auscultation, pronator drift, and the National Institute of Health Stroke Scale. This suggests that AIS screening and stroke severity evaluation should be highlighted in the core curriculum for neurology clerks [26]. In addition, the epidemiological trend in the aging society raises the importance of degenerative diseases including dementia and movement disorders in medical education. Mental status and involuntary movements are listed in the AAN NE guidelines for neurology clerks. However, our results further highlight that neurology clerks should be able to holistically assess basic cognitive function using the Mini-Mental State Examination. We also identified the involuntary movements that neurology clerks should learn.
We ranked the importance of the core NE items using median scores. Far more recommended NE items were identified in the current study compared to the AAN NE guidelines. However, the contents were generally similar between the AAN NE guidelines and our items with median scores of 8 and 9 (Table 3). We further separated each NE item into several parts (e.g. we separated the light reflex examination into direct light reflex, indirect light reflex, and relative afferent pupillary defect) and rated them individually. Simple examinations to assess brain stem function of patients with stupor and coma should be a core EPA for all clinical doctors and not only neurologists. Evaluating muscle power, understanding the Medical Research Council grading system, checking sensations, deep tendon reflexes, Babinski sign, Hoffmann’ reflex, cerebellar signs, and Romberg test are also important. Therefore, understanding the anatomy of the motor, sensory, and cerebellar systems is crucial before learning these NE skills. Checking spasticity, rigidity, and bradykinesia, and understanding an abnormal gait were also emphasized in our findings. The items with a median score of 7 included common contents that were frequently taught to our learners. Our results indicate that these items, such as the phenomenology of abnormal movements, may still be important for the students who have become skilled at the NE items with median scores of 9 and 8. Interestingly, our learners did not give as high ratings to the items for examining movement disorders as the tutors and experts. However movement disorders are not uncommon in clinical practice. This suggests that it would be worthwhile to develop curriculum to improve students’ basic concepts of these presentations [27, 28]. We believe that prioritizing the items in this order can help neurology clerks to learn NE more efficiently.
Traditionally, panelists in the Delphi process are experts and program directors, and it is possible that learner panelists may not be as well experienced in performing NEs or in different clinical scenarios. This may potentially confound the results. However, the core NE items for neurology clerks should not be too difficult for beginners. But these items should still meet the learners’ needs for their future clinical practice. To fulfill this demand, a bottom-up grassroots approach could be a practical method [20, 29, 30]. User panelists can still be included in the Delphi process [31]. In the current study, we included the opinions from learners and general neurologists. To reduce this bias, we assured the dominance of the tutors (60%) in the composition of the panelists, and we carefully selected the learners from different levels. The learners’ academic performance was within the top 50% in their same degree. We believe that this could reduce any confounding effect. Moreover, we further compared the viewpoints between the learners and tutors. This demonstrated a consensus with the tutors’ opinions only (S7 Table). The results showed small differences in the viewpoints between the learners and tutors, which is similar to previous studies [24]. However our learners and non-neurologists tended to give higher ratings to the NE items than the tutors, neurologists, and experts. It is possible that the learners and non-neurologists did not have enough clinical experience of patients with neurological diseases, and therefore they may have been more anxious of missing an important element of a NE that may lead to an incorrect diagnosis [14]. In addition, the learners and non-neurologists may have been less familiar with the neuroanatomy and clinical features of neurological diseases, and thus they felt the need to perform comprehensive NEs to reduce errors. In contrast, the neurologists tended to integrate the NE items instead of analyzing them separately, and thus they could better perform a hypothesis-driven rather than screening NE [32]. Based on these results, we suggest that medical students should focus on simplified core NE items before they have experience of neurological diseases and clinical care. In addition, it may be more important to establish a core NE structure than to comprehensively introduce a wide range of NE items for neurology clerks. It is essential to design an integrated curriculum which incorporates NE skills with regards to knowledge and clinical practice for neurological diseases, and this may facilitate medical students to use cognitive and meta-cognitive strategies [33]. The ability to integrate hypothesis-driven NEs into their practice may also help to improve motivation and self-regulated learning [34].
The strengths of this study include the web-based modified Delphi study, which could overcome the barriers of distance, time, and expense [35]. In addition, the heterogeneity of the members may have resulted in a better performance. We tried to expand the heterogeneity of the panelists, and we believe that this may have improved the results [31]. Adhesion is crucial to the results of the Delphi process, and we used the following methods to improve adhesion. First, we sent a letter explaining the methods to all of the panelists before the beginning of the study to help them understand the sequence of the study. Second, we used an online survey tool instead of e-mails to accomplish this study. The panelists could perform ratings on multiple devices including a computer, tablet, or smartphone. The ratings could thus be completed anywhere and at any time if their device could connect to the internet. The convenience of the online tool led to the high adhesion rate (96.7%) in this study. However, there are still several limitations to this study. First, the panelists who were not familiar with mobile devices may have had difficulty in completing the questionnaires. However, we supplied printed versions to these panelists if they complained about the inconvenience of using the online tools. Second, online panel discussions also have potential disadvantages including lower levels of interaction and engagement [36]. The web-based Delphi process may have limited free discussion among the panelists during the brainstorming process, and this may have particularly influenced the consensus gathering process with regards to the disputed items. To reduce this bias, we recorded the panelists’ opinions and provided their comments to all of the panelists separately by phone if requested. Finally, not all of the panelists were course directors, and we did not enroll international panelists. These factors may have influenced the generalizability of our findings to other countries.
Conclusions
Through our modified Delphi process, we provided a reference regarding the core NE items corresponding to the milestones of neurology clerkship. We hope our results can help to integrate clinical neurology and NE skills into a curriculum design and focus the content on the core issues. Future studies are needed to develop an assessment tool for NEs and to evaluate the reliability and validity of that tool.
Supporting information
S2 Table. Items of neurological examinations initially provided for the pilot group.
https://doi.org/10.1371/journal.pone.0197463.s002
(DOCX)
S3 Table. List of proposed core neurological examination items for the modified Delphi process.
https://doi.org/10.1371/journal.pone.0197463.s003
(DOCX)
S4 Table. Results of the first round modified Delphi process.
https://doi.org/10.1371/journal.pone.0197463.s004
(DOCX)
S5 Table. Results of the second round modified Delphi process.
https://doi.org/10.1371/journal.pone.0197463.s005
(DOCX)
S6 Table. Summary of the 22 items discarded during the Delphi process.
https://doi.org/10.1371/journal.pone.0197463.s006
(DOCX)
S7 Table. Results of the modified Delphi process from the tutors’ opinions.
https://doi.org/10.1371/journal.pone.0197463.s007
(DOCX)
Acknowledgments
We thank all the panelists of the modified Delphi process, and all the participants of the pilot group in this study for their help. We also thank (1) the Division of Medical Education, Graduate Institute of Clinical Medical Sciences, College of Medicine, Chang Gung University, (2) Chang Gung Medical Education Research Centre, Chang Gung Memorial Hospital, (3) Department of Neurology, Chang Gung Memorial Hospital, Linkou, and (4) Ministry of Science and Technology for their administrative and financial support.
References
- 1. Hudson JN. Linking neuroscience theory to practice to help overcome student fear of neurology. Med Teach 2006; 28:651–653. pmid:17594559
- 2. Matthias AT, Nagasingha P, Ranasinghe P, Gunatilake SB. Neurophobia among medical students and non-specialist doctors in Sri Lanka. BMC Med Educ 2013; 13:164. pmid:24321477
- 3. Youssef FF. Neurophobia and its implications: evidence from a Caribbean medical school. BMC Med Educ 2009; 9:39. pmid:19570231
- 4. Jozefowicz RF. Neurophobia: the fear of neurology among medical students. Arch Neurol 1994; 51:328–329. pmid:8155008
- 5. Flanagan E, Walsh C, Tubridy N. 'Neurophobia'—attitudes of medical students and doctors in Ireland to neurological teaching. Eur J Neurol 2007; 14:1109–1112. pmid:17880566
- 6. Ridsdale L, Massey R, Clark L. Preventing neurophobia in medical students, and so future doctors. Pract Neurol 2007; 7:116–123. pmid:17430877
- 7. McGee J, Maghzi AH, Minagar A. Neurophobia: a global and under-recognized phenomenon. Clin Neurol Neurosurg 2014; 122:iii–iv. pmid:24726892
- 8. Gupta NB, Khadilkar SV, Bangar SS, Patil TR, Chaudhari CR. Neurology as career option among postgraduate medical students. Ann Indian Acad Neurol 2013; 16:478–482. pmid:24339563
- 9. Abushouk AI, Duc NM. Curing neurophobia in medical schools: evidence-based strategies. Med Educ Online 2016; 21:32476.
- 10. Swing SR, Beeson MS, Carraccio C, Coburn M, Iobst W, Selden NR et al. Educational milestone development in the first 7 specialties to enter the next accreditation system. J Grad Med Educ 2013; 5:98–106. pmid:24404235
- 11. Norbury JW, Faulk CE, Harrell KM, Lawson LE, Moore DP. Impact of a Revised Curriculum Focusing on Clinical Neurology and Musculoskeletal Care on a Required Fourth-Year Medical Student Physical Medicine and Rehabilitation Clerkship. Rehabil Res Pract 2016; 2016:6197961. pmid:28025624
- 12. Sklar DP. Competencies, milestones, and entrustable professional activities: what they are, what they could be. Acad Med 2015; 90:395–397. pmid:25803411
- 13. Carraccio C, Englander R, Gilhooly J, Mink R, Hofkosh D, Barone MA et al. Building a Framework of Entrustable Professional Activities, Supported by Competencies and Milestones, to Bridge the Educational Continuum. Acad Med 2017; 92:324–330. pmid:26959225
- 14. Bosch J, Maaz A, Hitzblech T, Holzhausen Y, Peters H. Medical students' preparedness for professional activities in early clerkships. BMC Med Educ 2017; 17:140. pmid:28830418
- 15. Gelb DJ, Gunderson CH, Henry KA, Kirshner HS, Jozefowicz RF, Consortium of Neurology Clerkship D et al. The neurology clerkship core curriculum. Neurology 2002; 58:849–852. pmid:11914397
- 16. Esmaily HM, Savage C, Vahidi R, Amini A, Zarrintan MH, Wahlstrom R. Identifying outcome-based indicators and developing a curriculum for a continuing medical education programme on rational prescribing using a modified Delphi process. BMC Med Educ 2008; 8:33. pmid:18510774
- 17. McCarthy CJ, Rushton A, Billis V, Arnall F, Oldham JA. Development of a clinical examination in non-specific low back pain: a Delphi technique. J Rehabil Med 2006; 38:263–267. pmid:16801210
- 18. Kizawa Y, Tsuneto S, Tamba K, Takamiya Y, Morita T, Bito S et al. Development of a nationwide consensus syllabus of palliative medicine for undergraduate medical education in Japan: a modified Delphi method. Palliat Med 2012; 26:744–752. pmid:21920987
- 19. Rabbani F, Shaikh BT, Mahmood Q, Khan KS, Israr SM, Memon Y. Medical education and training: responding to community needs. Med Sci Monit 2005; 11:SR21–25. pmid:16192917
- 20. Crone RK. Global health interdependence. A grass-roots approach. JAMA 1992; 268:1462–1463. pmid:1512917
- 21. Goldacre MJ, Taylor K, Lambert TW. Views of junior doctors about whether their medical school prepared them well for work: questionnaire surveys. BMC Med Educ 2010; 10:78. pmid:21070622
- 22. Goldacre MJ, Lambert TW, Svirko E. Foundation doctors' views on whether their medical school prepared them well for work: UK graduates of 2008 and 2009. Postgrad Med J 2014; 90:63–68. pmid:23223778
- 23. Humphrey-Murto S, Varpio L, Gonsalves C, Wood TJ. Using consensus group methods such as Delphi and Nominal Group in medical education research. Med Teach 2017; 39:14–19. pmid:27841062
- 24. Moore FG, Chalk C. The essential neurologic examination: what should medical students be taught? Neurology 2009; 72:2020–2023. pmid:19506224
- 25. Powers WJ, Rabinstein AA, Ackerson T, Adeoye OM, Bambakidis NC, Becker K et al. 2018 Guidelines for the Early Management of Patients With Acute Ischemic Stroke: A Guideline for Healthcare Professionals From the American Heart Association/American Stroke Association. Stroke 2018.
- 26. Aroor S, Singh R, Goldstein LB. BE-FAST (Balance, Eyes, Face, Arm, Speech, Time): Reducing the Proportion of Strokes Missed Using the FAST Mnemonic. Stroke 2017; 48:479–481. pmid:28082668
- 27. Watson P, Stevenson M, Hawkins S. Neurology assessment by objective structured video examination. Clin Teach 2016; 13:348–351. pmid:26395465
- 28. Lim EC, Ong BK, Seet RC. Using videotaped vignettes to teach medical students to perform the neurologic examination. J Gen Intern Med 2006; 21:101.
- 29. Vaucher C, Bovet E, Bengough T, Pidoux V, Grossen M, Panese F et al. Meeting physicians' needs: a bottom-up approach for improving the implementation of medical knowledge into practice. Health Res Policy Syst 2016; 14:49. pmid:27431911
- 30. Schelvis RM, Oude Hengel KM, Wiezer NM, Blatter BM, van Genabeek JA, Bohlmeijer ET et al. Design of the Bottom-up Innovation project—a participatory, primary preventive, organizational level intervention on work-related stress and well-being for workers in Dutch vocational education. BMC Public Health 2013; 13:760. pmid:23947538
- 31. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One 2011; 6:e20476. pmid:21694759
- 32. Kamel H, Dhaliwal G, Navi BB, Pease AR, Shah M, Dhand A et al. A randomized trial of hypothesis-driven vs screening neurologic examination. Neurology 2011; 77:1395–1400. pmid:21900631
- 33. Aminoff MJ. The Future of the Neurologic Examination. JAMA Neurol 2017; 74:1291–1292. pmid:28973109
- 34. Kim KJ, Jang HW. Changes in medical students' motivation and self-regulated learning: a preliminary study. Int J Med Educ 2015; 6:213–215. pmid:26708325
- 35. Chami K, Gavazzi G, de Wazieres B, Lejeune B, Carrat F, Piette F et al. Guidelines for infection control in nursing homes: a Delphi consensus web-based survey. J Hosp Infect 2011; 79:75–89. pmid:21719149
- 36. Khodyakov D, Hempel S, Rubenstein L, Shekelle P, Foy R, Salem-Schatz S et al. Conducting online expert panels: a feasibility and experimental replicability study. BMC Med Res Methodol 2011; 11:174. pmid:22196011