https://orcid.org/0000-0003-1417-5082
Figures
Abstract
Due to limited space and resources, it can be difficult to train students on audiological procedures adequately. In the present study, we compared audiology training outcomes between a traditional approach and a recently developed immersive virtual reality (VR) approach in audiology students. Twenty-nine first-year audiology students participated in the study; 14 received traditional training (“TT group”), and 15 received the VR training (“VRT group”). Pre- and post-training evaluation included a 20-item test developed by an audiology educator. Post-training satisfaction and self-confidence were evaluated using Likert scales. Mean post-training test scores improved by 6.9±9.8 percentage points in the TT group and by 21.1±7.8 points in the VRT group; the improvement in scores was significant for both groups. After completing the traditional training, the TT group was subsequently trained with the VR system, after which mean scores further improved by 7.5 points; there was no significant difference in post-VR training scores between the TT and VRT groups. After training, the TT and VRT groups completed satisfaction and self-confidence questionnaires. Satisfaction and self-confidence ratings were significantly higher for the VR training group, compared to the traditional training group. Satisfaction ratings were “good” (4 on Likert scale) for 74% of the TT group and 100% of the VRT group. Self-confidence ratings were “good” for 71% of the TT group and 92% of the VRT group. These results suggest that a VR training approach may be an effective alternative or supplement to traditional training for audiology students.
Citation: Bakhos D, Galvin J, Aoustin J-M, Robier M, Kerneis S, Bechet G, et al. (2020) Training outcomes for audiology students using virtual reality or traditional training methods. PLoS ONE 15(12): e0243380. https://doi.org/10.1371/journal.pone.0243380
Editor: Rafael da Costa Monsanto, Universidade Federal de Sao Paulo/Escola Paulista de Medicina (Unifesp/epm), BRAZIL
Received: July 8, 2020; Accepted: November 19, 2020; Published: December 3, 2020
Copyright: © 2020 Bakhos et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: Audilab provided support in the form of salaries for authors JMA, MR, NM but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the author contributions section.
Competing interests: JMA, MR, NM are employees of Audilab. There are no patents, products in development or marketed products to declare. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Training in audiometric procedures and diagnosis is an essential part of educating audiology students. Audiometric procedures include mathematical, physiological, and psychophysical rules. For example, when measuring audiometric thresholds, it is important to know the appropriate levels of contralateral noise masking to avoid misdiagnosis or, more importantly, overmasking. In France, during the first year of audiology education, audiometry training involves theoretical course work (80 hours), lectures, and clinical training. Traditional clinical training is performed using tutorials (3 hours), where students train with each other or with their instructor. During the second and third year, the students practice under the direct and indirect supervision of a licensed practitioner during their internship. Given the number of students, as well as limited resources in terms of the training time and space available, this supervision is not always possible. In addition, such training, even when supervised, does not meet the recommendation by the French National Authority for Health that students be sufficiently trained before they are allowed to work with patients or volunteers [1].
Simulations have become an increasing part of medical education. Using simulators has been shown to improve patient safety and reduce costs and morbidity [2]. Various computer-assisted simulators have been developed in recent years to support health professional training. These models have been created to improve the experience before medical students interact with human patients or volunteers. In this context, simulators are strongly supported by supervisors in terms of students’ acquisition of both technical and non-technical skills [3]. Using simulators to train audiometry will better prepare students for future clinical practice. Over the last two decades, simulations have been increasingly integrated into medical education to facilitate the acquisition of knowledge and practice in a safe environment, allowing students to train and learn from their mistakes without risk to patients [4–7]. Simulators allow students to repeatedly learn and practice until they feel confident interacting with a patient. Currently, there are few simulators directed at developing students’ audiometry skills.
One area of recent development is the use of virtual reality (VR) simulators, which allow for a more immersive experience for the learner. Given advances in computer technology, VR simulators for the medical field have been greatly improved. VR simulators have been shown to improve learning outcomes for various surgery training steps. VR simulators have also been used to train clinical reasoning in the field of traumatology to enhance decision-making among students [8]. We recently developed a VR simulator for audiometric training [9] that simulated the clinical environment. Seven clinical cases were simulated, including otosclerosis, presbycusis, vestibular schwannoma, incus luxation, malingering presentation, sudden idiopathic deafness, and bilateral hearing loss due to cholesteatoma. For each clinical case, “beginner” and “expert” modes were created. The beginner mode provides feedback during the training session to inform students of errors and to provide correction and explanations; expert mode provides no feedback. At the end of the VR training session, a report is produced that summarizes the errors made for each clinical case. External experts (medical educators and licensed practitioners) and otolaryngology students reported satisfactory validity [9].
The initial evaluation of the VR simulator involved expert educators and otolaryngology students that had already experienced traditional audiometry training. However, it is still unclear how training outcomes may differ between a traditional or VR simulation approach. We hypothesized that, given the differences between traditional and VR training, VR training would lead to equal or better post-training evaluation scores, compared to traditional training. We also expected greater student satisfaction with the VR training, given the immersive experience. The objective of this study was to compare training outcomes between a group of audiology students that received traditional audiometry training compared to a single session of VR training. We also evaluated students’ satisfaction and self-confidence with the two training approaches.
Methods
Participants
We conducted a prospective single-center study with parallel groups at the Fougeres School of Audiology in France. Thirty-one first-year students were recruited for the study; two were excluded because they had repeated their first year. Accordingly, 29 students participated in the study (mean age = 21.1±4.1 years). The study was conducted in December 2019, after students had completed all theoretical course work for audiometry and audiometric findings in pathology. None of the participants had yet begun an internship at a hospital or audiology center. All data were anonymized prior to analysis. The ethics committee of Tours Hospital approved this project (2018–091). Students gave their verbal consent to participate to this study to a student in second year (BG) and their teacher (MN, LS).
Fourteen students (mean age = 20.4±1.7 years) were included in the Traditional Training (TT) group. The TT group received 3 hours of training supervised by a teacher in the audiology school. During the training, the teacher first reviewed basic audiometry principles; students were allowed to ask questions if they were not confident in their knowledge from the theorical lessons. Because the teacher had mild presbycusis, students were allowed to practice audiometry techniques for this sort of clinical case. The students were also allowed to train on each other. For example, a student would train with another student who simulated unilateral conductive hearing loss by plugging one ear with an ear plug.
Fifteen students (mean age = 21.7±5.6 years) were included in the Virtual Reality Training (VRT) group. The VR training was performed using the previously developed simulator [9]. During the training, a supervisor first explained how the VR system works and introduced the VR hardware, which included the headset, 2 captors, and handles (Oculus Rift ®). Next, each student was trained on 3 clinical cases (presbycusis, vestibular schwannoma, and sudden idiopathic deafness) using the beginner mode, which provided feedback during the session. Then, audiometric diagnosis and management were evaluated for each of the clinical cases, and a report was generated that summarized the errors during the evaluation. The duration of training for each case was approximately 30 minutes, and the total time of the VR training session was approximately 3 hours, including the 20 minutes of introducing the system and 20-minute breaks between clinical cases to avoid fatigue. Fig 1 shows screenshots of the VR system.
A: Captions for the handle functions; B: Otoscopy interpretation; C: Pure-tone audiometry and determination of auditory thresholds; D: Speech audiometry and determination of auditory thresholds; E: Example of feedback regarding masking for speech audiometry.
Outcome measures
The evaluation was developed by experts in audiology teaching (i.e., authors MN, LS in the present study), and included 20 questions regarding otoscopy, pure-tone and speech audiometry, acoumetry, masking and tympanometry. The questions were based on the audiometry theoretical coursework and lectures; as such, the questions did not directly address the “hands on” practical training in the TT or VRT group. English and French versions of the evaluation can be found in S1 and S2 Appendices, respectively. The TT and VR training supervisors did not know the evaluation questions. The scoring for each question was weighted depending on the number of errors: 1 point in case of no error, 0.5 points in case of 1 error, 0.3 points in case of 2 errors, and no points in case of 3 or more errors. The maximum possible score was 20, and scores were converted to percent correct. The post-test was administered immediately after the training session for both groups. After completing the post-test evaluation, students in the TT group performed the same VR training as for the VRT group and then were re-tested.
After completing the TT or VR training and the post-test, post-training satisfaction and self-confidence were evaluated using subscales with 5-point Likert response (1 = “Strongly disagree”, 2 = “Disagree”, 3 = “Indifferent”, 4 = “Agree”, 5 = “Strongly agree”). For satisfaction, the five items included: realism of training, feedback, support, comprehension, and degree of complement to the theoretical lessons. For self-confidence, the six items included: consultation, otoscopy interpretation, audiometry thresholds determination, speech audiometry procedures, masking, and overall confidence for clinical interaction with a patient. After training with the VR simulator, a four-item subscale with a 5-point Likert response was used to evaluate the immersive and realistic aspects of the VR simulator in relation to theoretical lessons using freely available online software (https://personalpages.manchester.ac.uk/staff/tim.wilding/PTA_Sim/index.html). Overall satisfaction or self-confidence ratings were calculated, and scores ≥ 4 were considered satisfactory.
Statistical analyses
Statistical analyses was performed using GraphPad Prism V6 software (2002; version 8.0.0 for Windows; GraphPad Software Inc., San Diego, California USA: www.graphpad.com). Mann-Whitney U tests were used to compare training outcomes, training satisfaction, and self-confidence between groups. A Wilcoxon matched-pairs signed rank test was used to compare outcomes in the TT group with the traditional training or the subsequent VR training. Statistical significance was set at p < 0.05.
Results
Fig 2 shows scores before and after training; mean scores are indicated by the horizontal lines. The mean pre-test score was 41.2±10.2 percent correct for the TT group and 39.5±6.5 percent correct for the VRT group. No significant differences in pre-test scores were observed between groups (U = 87; p = 0.444). The mean post-test score for the TT group significantly improved to 48.1±8.7 percent correct (W = 72; p = 0.021). Following the subsequent VR training, the mean TT post-test score further improved to 55.6±10.9 percent correct, significantly better than pre-test scores (W = 95; p = 0.0012) and significantly better than post-test scores after the traditional training (W = 89; p = 0.0029). The mean post-test score for the VRT group significantly improved to 60.5±7.1 percent correct (W = 120; p<0.001). The mean improvement relative to pre-test scores was significantly larger for VRT group (21.1±7.8 percentage points) than for the TT group (6.9 ±9.8 percentage points) (U = 22.5; p<0.0001).
The black horizontal lines show mean scores. The asterisks indicate significant differences (* = p < 0.05; ** = 0.001 < p < 0.05; *** = p < 0.001).
The post-training improvement in test scores was compared between the TT and VRT groups for individual questions. The post-training improvement was significantly larger for the VRT group than for the TT group for 4 questions: #6 (U = 44; p = 0.0046), #12 (U = 50.5; p = 0.0120), #13 (U = 54.5; p = 0.0117), and #17 (U = 49.5; p = 0.0117). These questions dealt with speech audiometry (#6), otoscopy (#12), tympanometry (#13) and audiometry interpretation of speech recognition thresholds (#17). For the remaining 16 questions, the post-training improvement remained significantly larger (U = 49.5; p = 0.014) for the VRT group (mean = 2.9±1.3 points) than for the TT group (mean = 1.29±1.47 points). After the subsequent VR training for the TT group, there was no significant difference in post-test scores between the TT group and the VRT group (U = 66; p = 0.0908).
Post-training satisfaction ratings were ≥ 4 (“Agree”) in 74% of the TT group and 100% of the VRT group. Post-training self-confidence ratings were ≥ 4 in 71% of the TT group and 92% of the VRT group. Fig 3 shows the results for each item of the surveys. Significantly higher ratings were observed in the VRT group than in the TT group for: realism of the audiological cases (U = 45.5; p = 0.0026), support during the training session (U = 0; p<0.0001), the degree of complement to the theoretical courses (U = 52.5; p = 0.0022), confidence for speech audiometry (U = 38.5; p = 0.0017), and confidence regarding the use of masking (U = 45; p = 0.0048).
For each item, data are shown for the VRT and TT groups. Each bar represents the percentage of respondents for each rating. The asterisks indicate significant differences between groups (*** = p<0.001).
Fig 4 shows the ratings for the VR system after the VRT and TT groups completed the VR training. Compared to theoretical courses, students agreed (a rating of 4) or strongly agreed (a rating of 5) that the VR training was more realistic (93%) and immersive (100%) than theoretical course work. They also agreed or strongly agreed that the VR training was more realistic (53%) and immersive (67%) than online software. All students (100%) agreed or strongly agreed that the VR training was complementary to theoretical courses and would recommend VR training to their colleagues.
Each bar represents the percentage of respondents for each rating.
Discussion
The present study compared traditional audiometry training versus VR training for first-year audiology students. While post-test scores improved after training for both groups, the improvement in the VRT group was significantly larger than in the TT group. These findings suggest that the tested VR training may be a useful tool for audiology education. VR training offers the possibility for the students to practice their skills in an extensive and repetitive manner with feedback but without any consequences to real patients or volunteers [10]. We also found great enthusiasm for the VR training among the students tested.
Some previous studies have investigated interest in VR training versus traditional training in other medical fields. In general, VR training offered an advantage for learning theoretical knowledge over traditional, lecture-based education for dentistry [11] and anatomy [12]. In surgery, the goal of training is quite different because evaluation is based on the acquisition of technical skills. Some meta-analyses were conducted for gastro-intestinal endoscopy training [4, 5] and for surgery of the ear, nose, and throat [6] using VR. They showed that VR training alone allowed for similar acquisition of technical skills as with traditional training, and thus considered VR simulators to be a useful learning tool. As in our study, most previous studies that compared traditional and VR training underscore the students’ enthusiasm for VR and the immersive environment. It would be ideal to simulate the same clinical cases for the traditional and VR training. However, this is more difficult for traditional training, where someone must “fake” some sort of hearing loss in the presence of their fellow students and instructor. The presentation of these simulated cases should also be consistent across students, which may be difficult. In the VR system, many more clinical cases can be easily and consistently implemented. This may partly explain the advantages of the VR training observed in the present study.
Feedback during training is essential to help students to improve performance, allowing students to learn from their mistakes quickly. In the present VR training system, concurrent feedback in beginner mode is cued by a sound alarm in case of error, followed by explanations of why the response was wrong. In this study, VR training was performed only using beginner mode. In our VR system (but not used in the present study), expert mode provides no concurrent feedback, but rather terminal feedback at the end of the evaluation. Some studies have suggested that terminal feedback is more effective than concurrent feedback, given the guidance hypothesis [13], which indicates that while concurrent feedback may help to learn a skill, learners are not dependent on concurrent feedback [14]. In the present study, students appreciated the concurrent feedback, with an overall satisfaction score of 4 for 80% of the TT group after the subsequent VR training, and for 100% of the VRT group. Only negative feedback was used in the VR training to indicate cases of error; positive feedback in cases of success has also demonstrated training benefits [15].
Traditional training for objective audiometry using normal-hearing volunteers, hearing-impaired volunteers, or manikin simulators has demonstrated improved self-confidence among audiology and speech language pathology students [16, 17]. This improved self-confidence has been also demonstrated after “boot-camps” [18], where students can practice with immediate feedback. Self-confidence as well as self-assessments play a major role in developing and improving clinical skills [19]. In the present study, the students similarly exhibited greater self-confidence after the training sessions, especially in the VRT group.
There are different types of simulators described in the literature, depending on their fidelity and resemblance to reality, divided into three main classes ranging from low- to high-fidelity [20]. Non-computerized manikins correspond to low-fidelity simulators (https://www.aheadsimulations.com/carl-for-training). They are used to train hearing aid manipulation, the removal of cerumen, real-ear unaided response measurements, and high-gain hearing aid fitting [21, 22]. Some computerized simulations allow training for specific tasks such as otoscopy, pure-tone air and bone conduction audiometry with online virtual patients [23]. Some of these online solutions are freely available (https://personalpages.manchester.ac.uk/staff/tim.wilding/PTA_Sim/index.html), while others require payments. One high-fidelity simulator exists in the field of audiology (Intelligent Hearing Systems, Miami, FL), and is a computerized manikin to train objective audiometry (e.g., auditory brainstem response, otoacoustic emissions). The present VR training software is somewhat different from these simulators, given that it is a 3D immersive experience, rather than some combination of hardware and software. Increasingly VR is considered to be a high-fidelity simulator [23, 24].
Some limitations to the present study should be discussed. First, our study was from a single center; a multi-center study is needed to confirm the present results. Second, we did not evaluate long-term retention of the training because students went on to internships at different audiology centers; it is possible that additional learning may have occurred at these centers, making it difficult to assess training retention. Previous studies have shown no difference in retention with traditional or VR training [25, 26]. However, greater satisfaction and engagement was reported by students who received VR training [26]. Similarly, greater satisfaction with VR training was reported in the present study, and students were very satisfied with the immersive and realistic environment. We note that some procedural learning may have been possible, because the same test was given before and after training (3 times for the TT group, as they were tested before training, after traditional training, and after the subsequent VR training). Even if some procedural learning may have occurred, the post-training improvement was more than double for VRT group, compared to the TT group. After traditional training, the mean improvement for the TT group was 6.9 percentage points; after completing the subsequent VR training, the mean score further improved by 7.5 percentage points. However, even after the VR training (and possible procedural learning), the mean score for the TT group remained 4.9 percentage points below that of the VRT group (although the difference at these endpoints was not significant). For future implementation, it would be advisable to create different questionnaires that cover different aspects of the clinical cases across pre- and post-training testing.
The present VR training system has also some limitations, especially for training situations in which precise hand movements are required (e.g., otoscopy, placement of air or bone transducers, using the audiometer, etc.). These gestures can be learned with a manikin or during a traditional training session with a teacher. Furthermore, the VR training does not presently permit evaluation of a student’s ability to explain the audiometric procedure to a patient. However, that is the role of the hospital and audiological center internships during second and third year. Thus, the goal of VR is not to replace, but to complement other modes of audiological training.
The present VR simulator satisfies the recommendations by the French National Authority for Health, which requires sufficient training before students can work with patients or volunteers. In the future, the benefit of VR training may be greatly improved by including a broader range of clinical cases and procedures. For example, behavioral measures of auditory thresholds in children are challenging; future VR systems could train students on age-appropriate audiometric techniques (e.g., evaluating infant responses). Further studies could also be directed at norming VR evaluation scores across different levels of medical education. For example, it would be useful to know the range of scores for first-year and third-year students. Once this range is identified, the VR evaluation could also determine whether students have sufficiently progressed in their audiological education, or to know if VR offers similar benefits for basic or advanced training. Furthermore, a complementary study could be designed to investigate the benefits of feedback for traditional or VR training. For traditional training, feedback could be provided by a supervisor for each student; such an approach is likely to be time-consuming and might require multiple supervisors to provide feedback to all students. While probably resource-intensive, it would be interesting to compare the benefits of such feedback between traditional and VR training approaches. VR training could also be used for continuing education and certification programs for practitioners. Further studies are needed to demonstrate the real-life benefit of VR training for practitioners and patients and to determine whether VR training can reduce the number of medical errors in clinical practice.
Conclusions
Simulations are widely used to train students in various fields of medicine; VR enables a safe training environment without risk to patients or volunteers. In the present study, our immersive VR training system provided audiology students with better learning outcomes and self-confidence than found with traditional training. Presently, the VR simulator can be used as a supplement to traditional audiology training; additional studies are needed to know whether it can replace traditional training. Further technological developments are also needed to expand the audiology training modules, such as behavioral and objective hearing tests for pediatric patients.
References
- 1.
Granry JC, Moll MC. Rapport de mission. État de l'art (national et international) en matière de pratiques de simulation dans le domaine de la santé: Dans le cadre du développement professionnel continu et de la prévention des risques associés aux soins. [Mission report. State of the art (national and international) in health simulation practices: As part of ongoing professional development and prevention of risks associated with care. French National Authority for Health. 2012].
- 2. Graafland M, Schraagen JM, Schijven MP. Systematic review of serious games for medical education and surgical skills training. Br J Surg. 2012; 99:1322–1330. pmid:22961509
- 3. Yiasemidou M, Glassman D, Tomlinson J, Song D, Gough MJ Perceptions about the present and future of surgical simulation: A national study of mixed qualitative and quantitative methodology. J Surg Educ. 2017; 74:108–116. pmid:27617919
- 4. Ziv A, Wolpe PR, Small SD, Glick S. Simulation-based medical education: an ethical imperative. Acad Med. 2003; 78:783–788. pmid:12915366
- 5. Khan R, Plahouras J, Johnston BC, Scaffidi MA, Grover SC, Walsh CM. Virtual reality simulation training for health professions trainees in gastro-intestinal endoscopy. Cochrane Database Syst Rev. 2018; 17:CD008237.
- 6. Walsh CM, Sherlock ME, Ling SC, Carnahan H. Virtual reality simulation training for health professions trainees in gastro-intestinal endoscopy. Cochrane Database Syst Rev. 2012; 13:CD008237.
- 7. Piromchai P, Avery A, Laopaiboon M, Kennedy G, O’Leary S. Virtual reality training for improving the skills needed for performing surgery of the ear, nose or throat. Cochrane Database Syst Rev. 2015; 9:CD010198. pmid:26352008
- 8. Harrington CM, Kavanagh DO, Quinlan JF et al. Development and evaluation of a trauma decision-making simulator in Oculus virtual reality. Am J Surg. 2018; 215:42–47. pmid:28262203
- 9. Aussedat C, Robier M, Aoustin JM et al. Using virtual reality in audiological training: Our experience in 22 otolaryngology residents. Clin Otolaryngol. 2020 Jul;45(4):643–648. pmid:32306528
- 10. Kron FW, Fetters MD, Scerbo MW, et al. Using a computer simulation for teaching communication skills: A blinded multisite mixed methods randomized controlled trial. Patient Educ Couns. 2017; 100:748–759. pmid:27939846
- 11. Soltanimehr E, Bahrampour E, Imani MM, Rahimi F, Almasi B, Moattari M. Effect of virtual versus traditional education on theoretical knowledge and reporting skills of dental students in radiographic interpretation of bony lesions of the jaw. BMC Med Educ. 2019; 25;19:233. pmid:31238927
- 12. Zhao J, Xu X, Jiang H, Ding Y. The effectiveness of virtual reality-based technology on anatomy teaching: a meta-analysis of randomized controlled studies. BMC Med Educ. 2020; 25;20:127. pmid:32334594
- 13. Walsh CM, Ling SC, Wang CS, Carnahan H. Concurrent versus terminal feedback: it may be better to wait. Acad Med. 2009; 84:S54–57. pmid:19907387
- 14. Salmoni AW, Schmidt RA, Walter CB. Knowledge of results and motor learning: A review and critical reappraisal. Psychol Bull. 1984; 95:355–386. pmid:6399752
- 15. Lim DH, Johnson SD. Trainee perceptions of factors that influence learning transfer. Int J Train Dev. 2002; 6:36–48
- 16. Alanazi AA, Nicholson N. Audiology and speech-pathology simulation training on the 1-3-6 early hearing detection and intervention timeline. Am J Audiol. 2019; 10;28:348–361. pmid:31095403
- 17. Alanazi AA, Nicholson N, Atcherson SR, et al. Use of baby ISAO simulator and standardized parents in hearing screening and parent counseling education. Am J Audiol. 2016; 1;25:211–223. pmid:27653494
- 18. Yeh DH, Fung K, Malekzadeh S. Boot camps: Preparing for residency. Otolaryngol Clin North Am. 2017; 50:1003–1013. pmid:28669460
- 19. Lambert T, Blessberger H, Fellner A, Steinwender C, Tekian A. Self-assessment of medical students and young physicians in training: Self-confidence of clinical competencies and working or learning environment. Wien Klin Wochenschr. 2020; 132:155–160. pmid:32130522
- 20. Maran NJ, Glavin RJ. Low- to high-fidelity simulation–a continuum of medical education? Med Educ. 2003; 37: 22–28. pmid:14641635
- 21. Koch RW, Moodie S, Folkeard P, et al. Face and content validity of a probe tube placement training simulator. J Am Acad Audiol. 2019; 30:227–234. pmid:30461412
- 22. Koch RW, Saleh H, Folkeard P, et al. Skill transference of a probe-tube placement training simulator. J Am Acad Audiol. 2020; 31:40–49. pmid:31274072
- 23. Dzulkarnain AA, Rahmat S, Mohd Puzi NA, Badzis M. Towards developing high-fidelity simulated learning environment training modules in audiology. Med J Malaysia. 2017; 72:37–45. pmid:28255138
- 24. Thai-Van H, Bakhos D, Bouccara D, Loundon N, Marx M, Mom T, et al. Telemedecine in Audiology. Best practice recommendations from the French Society of Audiology (SFA) and the French Society of Otorhinolaryngology-Head and Neck Surgery (SFORL). Eur Ann Otorhinolaryngol Head Neck Dis. 2020; 20:S1879-7296(20)30244-1.
- 25. Phelps A, Fritchle A, Hoffman H. Passive vs. active virtual reality learning: the effects on short- and long-term memory of anatomical structures. Stud Health Technol Inform. 2004; 98:298–300. pmid:15544293
- 26. Ryan E, Poole C. Impact of virtual learning environment on students' satisfaction, engagement, recall, and retention. J Med Imaging Radiat Sci. 2019; 50:408–415. pmid:31229502