http://orcid.org/0000-0002-2017-4492
Figures
Abstract
Gait analysis is a technique that is used to understand movement patterns and, in some cases, to inform the development of rehabilitation protocols. Traditional rehabilitation approaches have relied on expert guided feedback in clinical settings. Such efforts require the presence of an expert to inform the re-training (to evaluate any improvement) and the patient to travel to the clinic. Nowadays, potential opportunities exist to employ the use of digitized “feedback” modalities to help a user to “understand” improved gait technique. This is important as clear and concise feedback can enhance the quality of rehabilitation and recovery. A critical requirement emerges to consider the quality of feedback from the user perspective i.e. how they process, understand and react to the feedback. In this context, this paper reports the results of a Quality of Experience (QoE) evaluation of two feedback modalities: Augmented Reality (AR) and Haptic, employed as part of an overall gait analysis system. The aim of the feedback is to reduce varus/valgus misalignments, which can cause serious orthopedics problems. The QoE analysis considers objective (improvement in knee alignment) and subjective (questionnaire responses) user metrics in 26 participants, as part of a within subject design. Participants answered 12 questions on QoE aspects such as utility, usability, interaction and immersion of the feedback modalities via post-test reporting. In addition, objective metrics of participant performance (angles and alignment) were also considered as indicators of the utility of each feedback modality. The findings show statistically significant higher QoE ratings for AR feedback. Also, the number of knee misalignments was reduced after users experienced AR feedback (35% improvement with AR feedback relative to baseline when compared to haptic). Gender analysis showed significant differences in performance for number of misalignments and time to correct valgus misalignment (for males when they experienced AR feedback). The female group self-reported higher utility and QoE ratings for AR when compared to male group.
Citation: Braga Rodrigues T, Ó Catháin C, O’Connor NE, Murray N (2020) A Quality of Experience assessment of haptic and augmented reality feedback modalities in a gait analysis system. PLoS ONE 15(3): e0230570. https://doi.org/10.1371/journal.pone.0230570
Editor: Bijan Najafi, Baylor College of Medicine, UNITED STATES
Received: September 29, 2019; Accepted: March 3, 2020; Published: March 23, 2020
Copyright: © 2020 Braga Rodrigues et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The work presented in this paper has been supported by the Irish Research Council under grant GOIPG/2017/803 awarded to T.B.R. This publication has also been supported by the Science Foundation Ireland (SFI) under grant number SFI/12/RC/2289_P2 awarded to N.OC. and grant number SFI/13/RC/2106 awarded to N.M. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
1 Introduction
The assessment of human gait facilitates identification of movement deficiencies and abnormalities that are associated with the development of chronic injuries and disease. It provides objective data to support rehabilitation and retraining. Gait can be analysed and assessed using a variety of methods such as: clinical evaluation techniques; the use of high-speed cameras; force plates; and inertial sensors [1]. The hip and knee are weight bearing joints and play a key role in gait stability. The displacement of knee—called varus/valgus—is a misalignment of the tibiofemoral joint. The valgus knee (as per Fig 1a) is a condition whereby the knees turn outwards, whilst in the varus knee (Fig 1c) is a condition that causes the knees to turn inwards inwards [2]. This disorder occurs because the tibia is not aligned correctly with the femur, giving a different shape to the leg line.
Red arrows represent misalignment in the tibiofemoral joint. The blue arrows represent alignment of the tibiofemoral joint.
Excessive varus/valgus alignment can lead to serious orthopedics problems such as osteoarthritis [3]. Extreme cases of knee misalignment may need to be addressed surgically. If not properly treated, it can result in severe injuries from joint wear to diseases, e.g. knee arthrosis and osteoarthritis. However, in less severe cases, symptoms can be reduced with physiotherapy, corrective exercises, and through gait re-training [4]. There are some rehabilitation procedures to help with varus/valgus knee such as strengthening of hip and knee muscles [5]. Critical to all types of rehabilitation is appropriate feedback.
Feedback is a powerful tool for motor skill learning and helps with the sensory perceptual information as part of performing and learning a skill [6]. The accuracy of exercise performance with feedback in physiotherapy influences the healing process of the patient greatly. Crucial to successful rehabilitation is for the patient to understand the feedback, be it from a clinician or system. [7]. Some of the feedback systems include modalities such as: 2D screens; haptic; audio; expert guidance; and in more recent times Virtual Reality (VR) and Augmented Reality (AR) [8–10].
All of the different feedback approaches have advantages and disadvantages. For example, with 2D screen feedback, the user is limited in terms of the direction they can walk i.e. they are always required to walk towards the screen and must have their head up facing the screen. Audio guidance addresses this issue, but users need to clearly understand the guidance. With audio, this has shown to be an issue and a source of confusion [8]. The use of expert guidance has many benefits, but it requires the user to attend an expert clinic; the expert to be available; and is based on subjectivity of the clinician. Considering existing feedback modalities, 2D screens, audio and expert guidance, haptic has been shown to support the greatest user improvement for gait re-training [11, 12]. However, this requires the accurate placement of the haptic feedback display. AR has been very successful in education and considering the limitations of existing approaches, and the potential of AR as a portable, wearable and visual feedback modality is under researched and certainly worth investigating.
AR is an interactive experience in a real-world environment whereby real world objects are augmented with virtual information [13]. The future of AR points to a deeper use of technology augmenting human performance across a range of application domains [14]. Despite the countless possible applications and advances in the industrial sector, the understanding of user perception of AR technology is limited. Hence, there is a real need for user studies to determine the usability and utility of AR in different domains. This can be addressed through the Quality of Experience (QoE) framework. In this work we employed a questionnaire in order allow users to self-report on their perception of AR and Haptic feedback systems (in terms of utility, usability, interaction, and immersion). In terms of system utility (e.g. easiness to adjust to feedback), usability (e.g. feedback easy to understand), interaction (e.g. how users interact with feedback), immersion (e.g. awareness of body whilst moving.). The use of AR via wearable smart glasses in the field of gait rehabilitation is certainly an area under researched to-date. This study investigates if AR has the potential to be a lightweight and portable feedback alternative for rehabilitation protocols considering both objective (performance) and subjective (user QoE) evaluations.
Whist the previous discussion has justified the importance of understanding user perceptual quality of haptic and AR based gait feedback, the task of measuring user perceptual quality of multimedia experiences is complex. QoE is a user centric paradigm that allows us to evaluate the “degree of enjoyment or annoyance of an application, system, or service” of a multimedia experience [15]. It represents “the fulfillment of user’s expectation in respect to utility and enjoyment of that application or service” [16]. In order to evaluate any service and system from a QoE perspective, different Influencing Factors (IF) need to be considered. There are three main IF categories in QoE research: Human IF (e.g. gender, background), Context IF (e.g. Physical condition of varus/valgus in the case of this work, task), and System IF (e.g. AR versus Haptic, colour, screens).
In the recent years, with the advent of internet, advanced sensors, and internet of things (IoT), new proposals on evaluating QoE in a continuous manner have been proposed [17, 18], and models of assessing several multimedia systems were built [19]. Here, with the utility of the feedback as a key concern, the proposed work presents a novel QoE “system” level comparison of two feedback modalities (AR vs Haptic) within a gait analysis system. Our QoE comparison includes data analysis from post-test self-reported measures and also objective data comparison in terms of user responses (i.e. changes in gait if any) to each of the feedback modalities. In addition, we include analysis on the human factor (gender) and its effects on QoE and performance.
2 Related work
This section contains a critique of related research in terms of multimodal gait feedback systems and QoE assessments approaches for Haptic and AR (not all are specific to gait feedback). Each of these aspects are relevant to the scope of this work.
Haptic feedback has been studied in many works related to human activities [20], motor learning [21], and gait retraining [22]. Numerous works have compared haptic feedback with other modalities and have reported haptic: to be “less intrusive” than virtual reality feedback [23]; to be better in supporting task performance when compared to visual feedback for lower extremities [24] in gait; not to affect ecological validity of interaction compared with other modalities [25]. In addition, Haptic has been reported as easier to understand and follow when compared to auditory and visual stimuli [26]. Haptic feedback was also used to enhance the realism of a walking experience in multimodal environments [27]. Haptic feedback has also been used as an important tool in gait retraining for treatment of knee osteoarthritis [28]. In [11], closely aligned to the focus of this work, a gait re-training system employed haptic feedback to change gait parameters including varus/valgus misalignments. The system and results served as basis for this work by informing the use of haptic feedback to capture and improve gait parameters including knee alignment. They also highlighted issues whereby users were confused when receiving more than one feedback simultaneously (i.e. on different parts of the body). Such issues are again validation for why QoE assessments of such feedback mechanisms are required.
Some authors have applied Augmented Reality in gait analysis. In [29], a low-cost gait analysis system was developed using AR markers and a single video camera. The AR markers were used to track body segments and capture gait variables. Even though the authors achieved calibration and accurate tracking for gait angles, they highlighted the use of markers as a limitation (e.g. this system could not be used for treadmill walking). The use of different AR devices was also reported for guided walking in [30, 31]. These works indicated that novel AR technologies could be used in walking guidance with performance, body stability with positive impact in gaze and locomotor control [32, 33].
Considering these works, the use of AR for gait feedback has not been deeply explored. There are some exploratory works that suggest employing AR for gait retraining. The results reported in [34] results in significant improvement in gait over a 2D monitor. Other research has reported the use of AR in gait posture training [35] reported statistically significant improvement in posture, balance, and velocity. In [36], a gait retraining system was developed to modify footprint parameters. The authors concluded that AR could help to quickly modify user’s footprint parameters. Although these works make a valuable contribution, there was no qualitative metric employed that informs if users were satisfied or enjoyed the feedback experience. This is critical because it informs designers about how the users enjoy, engage, and experience such systems.
Several authors have used QoE assessment in multimedia systems as a paradigm to quantify how various factors of the system influence perceived quality levels from the user perspective. In [37], user QoE levels were compared in an immersive Virtual Reality and Augmented Reality applications. A sample size of twenty-one participants was divided randomly into two groups. Both objective and subjective metrics were gathered. The authors considered system, psychological, and user factor to evaluate quality. The QoE evaluation suggested that users felt safer and accustomed with the use of AR when compared to virtual reality. In [38], a QoE evaluation of a motor skills rehabilitation game was developed. The authors have assessed QoE through user engagement, task success, interaction, and socialization. This study reported that high QoE scores can be linked to high performance. These works demonstrate the need of a qualitative study for different applications. Several works have reported that valgus/varus incidence is different across gender groups [39, 40]. Anatomical differences between males and females lead to differences in knee alignment, and are a potential cause of anterior cruciate ligament injuries in females [41, 42]. Females, in general, have wider hips than males, influencing kinematic factors related to injury such as knee valgus/varus. Since the incidence of valgus/varus misalignments is different between males and females, we considered gender as an important factor to consider of this study.
Considering existing literature, the novelty of the work presented in this article lies in the evaluation and analysis of users’ QoE (self-reported measures and objective measures) of Haptic and AR feedbacks in our gait analysis system. The focus is on comparing subjective and objective metrics for correcting knee alignment with these two different feedback modalities (Haptic and AR).
3 System and feedback architecture
Our gait system is composed of a capturing module, a presentation module and a data processing module. The capturing module consists of 6 Inertial Measurement Units (IMUs). The Feedback module contains two components: Haptic and Augmented Reality modules. Finally, the data processing system is a quadcore Intel Core i7 laptop, 16GB DDR4 RAM, 3.2Ghz, GTX 1060-6GB was used to integrate all modules and is also the Wi-Fi WebSocket server for all modules as per Fig 2.
The figure shows sensor placement and coordinate systems from different views.
3.1 Capturing module—IMU
The capturing module contains 6 X-Sens IMU’s [43] and placed on the body as per Fig 2. A real-time Wi-Fi synchronization and streaming protocol for multiple IMUs was developed in C#. This streaming protocol is important for it ensures that no data is lost, that feedback is presented without delay, and all modules can work independently. In terms of internal configuration of each IMU, 10 streams of data were captured: 3D acceleration from triaxial accelerometer (Accxyz), 3D angular velocity from triaxial gyroscope (Gyroxyz), 3D magnetic field from a triaxial magnetometer (Magxyz), and UNIX timestamp. As discussed later in this section, the Accxyz, Gyroxyz, and Magxyz were fused to provide quaternion representation. The developed protocol fuses, in real-time, accelerometer, gyroscope, and magnetometer data and generates the quaternion orientation. The datasets from the IMU’s were synchronized with the computer CPU clock ensuring no packet loss. This module, processes in real time, the quaternion and Euler angles of each sensor and generates angles for knees, hips, tibia, and trunk lean. Data from the sensors was sampled at 40Hz on all three axes and sent through a Wi-Fi interface to the server computer. Further details on the multi-IMU streaming protocol is available in [44, 45] for the interested reader.
To represent the orientation of a rigid body or frame coordinates in 3D space, a quaternion representation was employed. This complex number representation defines any spatial rotation around a fixed point or coordinate system. A quaternion q = [q0q1q2q3] was used to calculate an angle θ about a fixed Euler axis [46, 47]. To get the angle between two joints with IMU, quaternion matrices were obtained by fusion of the 3 internal modules (Accxyz, Gyroxyz, Magxyz) using a Madgwick-based orientation filter [48]. The quaternion generated by the orientation filter represent s the spatial rotation of each IMU and can generate any joint angle (knee angle in this case) for each axis. Having each Euler angle, it is then possible to reference one IMU to another and determine the angle between two sensors. This angle between the two IMU’s was used as part of the walking evaluation during experiments. At the start of each test, while the user was stand, sensor calibration was obtained using the IMU quaternion in Euler angles θx, θy, θz in North-East-Down (NED) Z-Y-X sequence as in Eq (1).
(1)
To find the tibia projection angle in the frontal, lateral, and sagittal planes, we need to calculate unit vectors on each quaternion coordinate system. This calculation converts the current quaternion of each IMU to direction cosine matrices. We take then the calibrated θx, θy, θz and convert them into a unit vector in the ZYX order as in Eq (2). We then applied this to the calibrated Euler angles.
(2)
To get any IMU joint angle (tibia angle in our case), we convert each IMU quaternion into Direction Cosine Matrices (DCMxyz) (Eq (3)) and multiply the direction vector IMUxyz as in Eq (4). Finally, we apply trigonometry of right-angle triangle of IMUxyz of the directional vector vxyz on the desired plane (Eq (5)). The angle θ between two IMU will be as in Eq (6).
(3)
(4)
(5)
(6)
3.2 Feedback modules
In this section, Haptic and Augmented Reality feedback modules are presented.
3.2.1 Haptic module.
A bespoke wearable haptic module was designed for gait feedback purposes as illustrated in Fig 3. No off-the-shelf haptic modules satisfied our requirements of being lightweight, wearable, and provide a haptic sensation. The Haptic module was developed to provide the correct feedback to the user according to his/her movements [49]. The two haptic modules had an ESP8266 Wi-Fi micro-controller board with a WebSocket client. Each module was composed of a leg mounted strap; two vibration units (Fig 3a); and communication and micro-controller with battery unit (Fig 3b).
It contains haptic motors (a) and the Wi-Fi microcontroller responsible for the web-socket client (b). All the units are sheltered within ABS plastic cases (30x30x10mm) for the haptic module and (40x30x10mm) for the Wi-Fi micro-controller.
The leg mounted bracelet is attached to the users’ skin as per Fig 2. The vibration units are enclosed within the plastic casing. The design of the circuit contains MOSFET transistors operating as switches. There was also a pulse width modulation control to allow precise change of the intensity of the vibration unit if required. When the signal is received by the communication unit, the vibrating unit provides a high level TTL output signal to the transistor’s gate. This signal leads the transistor to operate in the “saturation region” and permitting the current to reach the motor. A freewheel diode was installed across each motor of the vibration units to remove voltage spikes due inductive nature of the load when switched off [50]. This prevents malfunction of the hardware, protecting the I/O ports of the microcontroller inside the communication unit from electromotive force (EMF).
3.2.2 Augmented reality module.
Our AR module consisted of an Epson Moverio Bt-300 Smart Glasses [51] connected with a WebSocket protocol. A WebSocket client in the AR module was employed as it allowed the web server to establish a connection with the feedback application and communicate directly with it without any delay (typically web communication consists of a series of requests and responses between the client and the web server, where, for real-time applications, this technique is not well suited [52]). With the use of WebSockets, we established a connection only once, and the communication between the server and the feedback application could follow without problems related to delay and synchronization.
3.2.3 Activation of feedback modules.
The feedback state diagram is shown in Fig 4. The user input is compared with the kinematic model which controls the feedback mechanism according to the activation threshold. The kinematic model was defined as per Fig 4, with activation thresholds for each feedback defined at +7o for valgus, and -7o for varus i.e. if valgus/varus angle extended beyond the defined threshold, feedback was provided to the user. These values represent normal angle limits of knee alignment [53]. The model constantly evaluates the current tibia angle in order to compare with threshold values. Each person has their own walking style and for this reason it is difficult for a participant to have perfect alignment throughout every single part of the gait cycle while walking naturally. Because of this, every small change between baseline (no feedback) and test (both feedback) was observed during testing.
These diagrams represent the feedback control system. User knee angle is used as input, which will be compared constantly with kinematic model. The user then receives haptic or AR stimuli to correct knee alignment.
The feedback in the Haptic module was presented as vibrations on each leg whenever the participant’s tibial angle was above or below the activation thresholds for valgus and varus. The correct alignment of each leg resulted in “no vibration” (i.e. no feedback provided) on the Haptic bracelet. During the training phase (see section IV), participants were told that no feedback from haptic means they are in correct alignment. The objective given to the participant was to receive the least amount of vibration as possible. The feedback in the AR module was presented as circle visualizations on the AR glasses (see Figs 4 and 5). The user sees a projection of 6 circles in their field of view (3 of each leg as per Fig 5). Again, whenever the tibial angle was above or below the activation thresholds for valgus and varus. For each leg, three circles control the states of the knee according to valgus and varus angles. The correct alignment of each leg is achieved when the blue circle in the middle is lit. The objective given to the participant is to keep the circles blue during trial.
AR feedback iscontrolled by colored circles: redfor misalignments and blue for alignment. Haptic controls are vibrations oneach leg: 1 and 4 for Valgus, 2 and 3 for Varus.
4 Experimental protocol
This research was approved by the Athlone Institute of Technology Research Ethics Committee on the 23rd of January of 2019. Participants consent was obtained in written format and stored in a secure location. Data were anonymized for all trials and participants. After ethics approval, a test with healthy participants was conducted. A convenience sampling approach was employed to recruit twenty-six participants (13 males, 13 females) with an average age of 27.54 (± 6.57) years. Due to previous knee or walk abnormalities, data of two participants was omitted. The gender balance guidelines have been applied as per ITU-P913 standards for objective and subjective quality assessment [54]. A within group experimental design was employed; hence each participant experienced both the haptic and AR feedback modalities. The ordering of how the participants experienced the feedback was randomised. Participants were tested on two different days and the protocol adhered to the approach taken in numerous related works in the literature [17, 37, 55] and included the steps outlined in Fig 6.
This protocol was consisted during all trials for all participants. The full protocol is available in S1 File.
During the information phase, each participant was greeted and thanked for their participation. After a brief explanation, written consent was obtained. Participants were brought to the waiting room and were provided with an information sheet that fully described the experiment. The screening phase assessed a participant’s visual acuity color perception, and ability to perceive the haptic stimuli [56–58]. The screening process for participants for visual acuity, color perception, and haptic sensation required participants to achieve a threshold score to be eligible for the actual testing. For the Snellen test, a score of 20/20 was required. For the Ishihara test, thirty-eight color plates were used and only 4 errors were allowed during examination. For the haptic screening, participants were required to differentiate 4 vibration patterns and location [58]. Upon completion, baseline metrics of gait angles: left and right hip, left and right knee, left and right tibia (for varus/valgus assessment), and trunk lean were captured over a two-minute period using the devices outlined in Section III. For this experiment, we only analyzed tibia angle to evaluate feedback. Full gait analysis considering all angles will be evaluated as part of a future work study.
For training and testing phases, participants were randomly assigned into two groups (Haptic/AR, and AR/Haptic) depending on which feedback the participant experienced first. Each participant experienced one of the feedback modalities and had a week break before they were presented with the alternative modality feedback. As part of the training, participants were introduced to the AR and the haptic modules as appropriate for the given test day. The devices were fitted to the participant by the principal investigator and an opportunity for adjustment was provided to ensure there was no discomfort. After sensor placement, participants were securely guided to a treadmill where they were asked to select a walking speed with which they felt comfortable (the range selected by users was between 2.5 and 4 miles per hour). Following this, in the test, the speed each participant selected was maintained for training and testing of both feedback modalities. Instructions for each feedback were explained with 3 feedback sheets (available in S1 File) showing the difference of the three different knee states (valgus, normal, varus). Participants were aware that each leg was independent so that even though one leg was on valgus state, the other one could be aligned for example. Participants walked 2 minutes for base-line capture (no feedback), 30 seconds for feedback training, and 2 minutes (with feedback).
4.1 QoE questionnaire
As per [59], twelve questions asked were asked of all participants on the experience of both feedback modalities. For the subjective analysis, QoE factors were evaluated in form of questionnaires after the gait assessment phase as per Fig 6. QoE takes into consideration how system, human and contextual factors contributes to a user’s perceived quality of a system [19]. The literature suggests that the accepted approach to measuring a user’s perceived quality of his or her experience is based on self-reported measures via post-experience questionnaires. The developed questionnaire was used to determine an overall mean opinion score (MOS) based on feedback from users [60].
The twelve questions were developed to evaluate system utility (questions 1-3), usability (questions 4-6), interaction (questions 7-9), and immersion (questions 10-12). For each of those 4 assessment variables, 4 standard questionnaires were used as guidelines: The System Usability Scale (SUS), ITU-T methods for subjective assessment of quality, Igroup presence questionnaire (IPQ), and Computer System Utility Questionnaire (CSUQ) [54, 61–63]. The rating system used was a seven-point Likert scale to determine whether or not the participant agreed with the statement. The full questionnaire is available in [59] and per Table 3 in the results section. The ordering of the questions was randomized for the different participants to negate any ordering effects.
4.2 Data processing and statistics
As outlined in the methodology section, QoE and objective metrics were captured for each trial. Participants were categorized into AR and haptic. Subgroups of males (N = 13) and females (N = 13) were also randomly defined for gender analysis purposes. In order to compare differences across groups, a Shapiro-Wilk normality test [64] was conducted. All variables were with a normal distribution (p>0.05). A dependent samples t-test was performed on the data with 95% confidence level. For the objective analysis, we have reported differences between AR and haptic groups for number of alignments after receiving feedback, and the amount of time participants were not aligned. We have also reported the same analysis considering gender. These comparisons were done by dependent samples t-test at 95% confidence level. The QoE model (QoEMF) for each feedback for a number p of participants was designed to be average of the four-assessment metrics: Utility (UtF), Usability (UsF), Interaction (InF), and Immersion (ImF) as in Eq (7).
(7)
5 Results
In this section we present analysis and discussion of the data captured during the experiment: objective measures of performance (i.e. number of misalignments for each feedback modality); and subjective evaluation from post-test QoE questionnaires for each of the feedback modalities. In addition, we include analysis by gender.
5.1 Objective results
For the objective data, we analysed how the participant reacted to each of the types of feedback i.e. if or how did they change their walking style based on each feedback modality. For each leg, 3 distinct states were defined: varus, correct position, and valgus. We report, for each state, the time the participants remained in misalignment during the experiment, and the number of times the participant needed feedback (feedback cue) during the experiment (2 minutes). We also provide detail on the number of complete alignments (both legs in correct position) and misalignments for each leg.
Table 1 contains performance report of varus and valgus alignment of all participants after experiencing AR and Haptic feedback. It also includes a further categorization by gender. The results show statistically significant differences between the AR and Haptic feedback in terms of the number of varus, valgus, and total misalignments for baseline and test. Participants performed better with AR feedback, with a reduction of 31% for varus, 13% for valgus. All reported results considered 95% and 90% confidence interval. Statistically significant differences in performance is reported for the AR feedback in reducing varus and total misalignments with a two-tailed p < 0.1 and p < 0.05. For gender analysis, the male improved for varus (45% p = 0.034) and valgus (18% p = 0.073) while females did not have statistically significant improvement. The ordering of feedback did not influence performance (p > 0.1).
Table 2 contains performance data in terms of how long users were in the varus and valgus positions during the 2 minutes trials. We have confirmed that only AR feedback could reduce varus time with statistically significant difference for baseline and testing. Participants had better performance in time with AR feedback in reducing varus in 11%, valgus 64% and Total misalignments 37%. Males had significant improvement in valgus time (63% p = 0.047). The performance for the Haptic feedback increased the number of misalignments with the male group (-49% p = 0.06). This suggests that the users were somewhat confused by the haptic feedback. Statistically significant difference in performance was only reported for the AR feedback in reducing varus and total misalignments with a two-tailed p<0.05. The ordering of feedback did not influence performance (p > 0.1).
5.2 Self-reported questionnaire results
Table 3 present results of the MOS self-reported measures via post-test questionnaires. Table IV presents the results considering the gender variable. Since the AR and Haptic groups were randomized repeated measures, a dependent samples t-test was performed on the data with 95% confidence level using the IBM statistical analysis software package SPSS [65].
As per Table 3, out of the 12 questions asked, only Question 1, which was asked if whenever the participant received feedback, he or she adjusted easily and quickly, reported a statistically significant difference between AR and Haptic feedback with a two-tailed p value of 0.015, p<0.05. The AR group reported a MOS rating of 4.458 whereas the Haptic feedback 3.5. This result is confirmed that even not knowing performance, participants felt the AR feedback was more effective in reducing misalignments. Considering the discussion in section V.A about how participants responded to the haptic feedback (i.e. increase in misalignments), this results raises an interesting questions about the ease of understanding of haptic feedback for participants. For all other questions, excluding Question 2, the AR feedback had greater MOS than Haptic feedback (although not statistically significant).
Table 4 presents results of the MOS Questionnaire by gender. The female group reported a statistically significant difference between AR and Haptic for Question 1. Male group also reported a statistically significant difference for Question 2 (“My walking style changed during experiment.”) and Question 12 (“I was engaged with the system.”).
Utility, Usability Interaction, Immersion, and QoEM scores of AR and Haptic feedback by gender are shown in Fig 7. AR feedback showed significant Utility (p < 0.05) for female group, which indicated that females found AR feedback more useful than Haptic feedback for this experiment. This QoE factor is related to adjustment to feedback, changes in walking styles and system support.
6 Discussion
In this section we discuss the results of the comparison between AR and haptic feedback. Due to the fact that haptic feedback has been reported as a viable feedback modality across many fields such as rehabilitation and gait re-education, our assumption was that haptic feedback would report better results in terms of user performance (and also possibly QoE).
Haptic information is given directly at the joint that the user needs to change whilst AR feedback the participant needed to process visual information and change the leg related to that change. Surprisingly as seen in the results, AR feedback not only reduced the number of misalignments, but from the subjective questionnaire analysis, users reported that AR feedback helped to reduce the number of misalignments better than haptic. Although the results indicate that both feedback modalities reduce the occurrence of varus and valgus misalignments, AR feedback significantly reduced the number of varus misalignment (by 31%) when compared to baseline readings. Whilst the reductions for valgus (for AR) and neither varus nor valgus are significant for haptic, approximate reductions of between 9%-15% are positive.
Looking deeper at the analysis considering gender influence on the results, for the male AR group, the level of reduction for varus was 45% (and 18% for valgus misalignments). Consistent with the male group, although to a lesser extent, AR feedback reduced the number of varus misalignments by 35% for the female group (not significant when compared to baseline). These results demonstrate the utility of employing both feedbacks, but in particular AR feedback. It also raises an interesting question to understand why females’ knee did not have a significant change after receiving feedback. Feedback and users’ responses to same is an important topic to understand. In our use case, it can have a significant impact on a person’s Quality of Life. Reducing misalignments can also reduce the injury incidence more. These results are important for the research community and was also a good indicator for future work, where we will extend the research for understanding physiological measures and what happens in a clinical setup for males and females.
For the QoE analysis, subjective evaluation of questionnaires for feedback utility, usability, interaction, and immersion was performed. Table 3 reported results of the MOS questionnaire for all participants. When participants were asked about adjustment after feedback in Question 1 (“When I received feedback, I adjusted easily and quickly.”), they felt that AR was more effective in changing varus and valgus misalignments. This correlates with the objective analysis in Table 1. For the MOS questionnaire considering gender, the male group reported that they believed their walking style changed based on the AR feedback. They also reported higher engagement when using the AR glasses than haptic devices. The female group reported higher utility of AR feedback. These difference between gender groups highlight the importance of considering human factors and employing QoE analysis in these types of novel feedback studies. Considering that many researches were conducted using current feedback tools such as 2D screen and haptic, this study can be a new paradigm in using immersive technologies in gait re-training and promotion of rehabilitation protocols.
7 Conclusion
This paper presented a comparison of Haptic and Augmented Reality as feedback modalities in a gait analysis system. It compared, in terms of objective and subjective ratings, how users perceived and responded to Haptic and Augmented Reality feedback. Based on the results, the novel AR approach has significant potential as a method of gait rehabilitation. The objective evaluation tells us that AR significantly reduces the number of knee misalignment. In addition, subjective questionnaire assessment provides interesting results in terms of how users feel their walk changed positively with AR feedback. The agreement of objective and subjective evaluations serves as basis of using AR as part of a rehabilitation protocol. Both gender groups considered reported that AR had greater utility than haptic feedback. The male group showed statistically significant improvement in varus, valgus, total Misalignment, and valgus time. Future work will also assess the validity that AR feedback not only provides higher QoE scores but also promotes less cognitive workload in comparison with haptic as well as instantiation of the QoE model proposed above. Physiologic measures and pupillary response will also be evaluated and their inference to QoE will be analysed.
Acknowledgments
The authors would like to acknowledge Dr. Paul Archbold and Mr. Eoin Woodlock for the use of the laboratory space for data collection.
References
- 1. Harris GF, Wertsch JJ. Procedures for gait analysis. Archives of Physical Medicine and Rehabilitation. 1994;75:216–225. pmid:8311681
- 2. Sharma L, Song J, Dunlop D, Felson D, Lewis CE, Segal N. Varus and valgus alignment and incident and progressive knee osteoarthritis. Annals of the rheumatic diseases. 2010;69:1940–1945. pmid:20511608
- 3. Freisinger GM, Hutter EE, Lewis J, Granger JF, Glassman AH, Beal MD. Relationships between varus-valgus laxity of the severely osteoarthritic knee and gait, instability, clinical performance, and function. Journal of orthopaedic research: official publication of the Orthopaedic Research Society. 2017;35:1644–1652.
- 4. Baudet A, Morisset C, Athis P, Maillefert JF, Casillas JM, Ornetti P. Cross-Talk Correction Method for Knee Kinematics in Gait Analysis Using Principal Component Analysis (PCA): A New Proposal. PLOS ONE. 2014;9:102098–102098.
- 5. Jafarnezhadgero AA, Majlesi M, Etemadi H, Robertson DGE. Rehabilitation improves walking kinematics in children with a knee varus: Randomized controlled trial. Annals of Physical and Rehabilitation Medicine. 2018;61:125–134. pmid:29476933
- 6. Figueiredo LS, Ugrinowitsch H, Freire AB, Shea JB, Benda RN. External Control of Knowledge of Results: Learner Involvement Enhances Motor Skill Transfer. Perceptual and Motor Skills. 2018;125:400–416. pmid:29350078
- 7. Carr K, Zachariah N, Weir P, Mcnevin N. An Examination of Feedback Use in Rehabilitation Settings. Critical Reviews in Physical and Rehabilitation Medicine. 2011;23(1-4):147–160.
- 8. Ahrens A, Lund KD, Marschall M, Dau T. Sound source localization with varying amount of visual information in virtual reality. PLOS ONE. 2019;14:214603–214603.
- 9. Hartanto D, Kampmann IL, Morina N, Emmelkamp PGM, Neerincx MA, Brinkman WP. Controlling Social Stress in Virtual Reality Environments. PLOS ONE. 2014;9:92804–92804.
- 10. Stepp CE, An Q, Matsuoka Y. Repeated Training with Augmentative Vibrotactile Feedback Increases Object Manipulation Performance. PLOS ONE. 2012;7:32743–32743.
- 11.
Lurie KL, Shull PB, Nesbitt KF, Cutkosky MR. Informing haptic feedback design for gait retraining. 2011 IEEE World Haptics Conference. 2011; p. 19–24.
- 12.
Shull PB, Jirattigalachote W, Zhu X. An Overview of Wearable Sensing and Wearable Feedback for Gait Retraining. In: ICIRA; 2013.
- 13. Vu TM, Katushin N, Pumwa J. Motion tracking glove for augmented reality and virtual reality. Paladyn. 2019;10:160–160.
- 14. Cipresso P, Giglioli IAC, Raya MA, Riva G. The Past, Present, and Future of Virtual and Augmented Reality Research: A Network and Cluster Analysis of the Literature. In: Front. Psychol.; 2018.
- 15.
Brunnstrm K, Moor KD, Dooms A, Egger-Lampl S, Garcia MN, Hossfeld T. Qualinet White Paper on Definitions of Quality of Experience; 2013.
- 16. Robitza W, Ahmad A, Kara PA, Atzori L, Martini MG, Raake A. Challenges of future multimedia QoE monitoring for internet service providers. Multimedia Tools and Applications. 2017;76: 22243–22266.
- 17. Murray N, Lee B, Qiao Y, Miro-Muntean G. The Impact of Scent Type on Olfaction-Enhanced Multimedia Quality of Experience. IEEE Transactions on Systems, Man, and Cybernetics: Systems. 2017;47:2503–2515.
- 18.
Floris A, Atzori L. Quality of Experience in the Multimedia Internet of Things: Definition and practical use-cases. 2015 IEEE International Conference on Communication Workshop (ICCW). 2015; p. 1747–1752.
- 19.
Martini M, Chen CW, Chen Z, Dagiuklas T, Sun L, Zhu X. QoE-Aware Wireless Multimedia Systems; 2012.
- 20.
Linden JV, Schoonderwaldt E, Bird J. Towards a real-time system for teaching novices correct violin bowing technique. 2009 IEEE International Workshop on Haptic Audio visual Environments and Games. 2009; p. 81–86.
- 21. Lieberman J, Breazeal C. TIKL: Development of a Wearable Vibrotactile Feedback Suit for Improved Human Motor Learning. IEEE Transactions on Robotics. 2007;23:919–926.
- 22.
Xu J, Lee UH, Bao T, Huang Y, Sienko KH, Shull PB. Wearable sensing and haptic feedback research platform for gait retraining. 2017 IEEE 14th International Conference on Wearable and Implantable Body Sensor Networks. 2017; p. 125–128.
- 23. Gaffary Y, Gouis BL, Marchal M, Argelaguet F, Arnaldi B, Lcuyer A. AR Feels Softer than VR: Haptic Perception of Stiffness in Augmented versus Virtual Reality. IEEE Transactions on Visualization and Computer Graphics. 2017;23:2372–2377. pmid:28809699
- 24. Koritnik T, Koenig A, Bajd T, Riener R, Munih M. Comparison of visual and haptic feedback during training of lower extremities. Gait Posture. 2010;32(4):540–546. pmid:20727763
- 25.
Kauhanen L, Palomki T, Jylnki P, Aloise F, Nuttin M, Millan JDR. Haptic Feedback Compared with Visual Feedback for BCI. Infoscience EPFL scientific publications. 2006;.
- 26. Sigrist R, Rauter G, Riener R, Wolf P. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychonomic Bulletin & Review. 2013;20:21–53.
- 27. Turchet L, Burelli P, Serafin S. Haptic Feedback for Enhancing Realism of Walking Simulations. IEEE Transactions on Haptics. 2013;6:35–45. pmid:24808266
- 28.
Shull P, Lurie K, Shin M, Besier T, Cutkosky M. Haptic gait retraining for knee osteoarthritis treatment. 2010 IEEE Haptics Symposium. 2010; p. 409–416.
- 29. Nagymt G, Kiss RM. Affordable gait analysis using augmented reality markers. PLOS ONE. 2019;14:212319–212319.
- 30. Diaz GJ, Parade MS, Barton SL, Fajen BR. The pickup of visual information about size and location during approach to an obstacle. PLOS ONE. 2018;13:192044–192044.
- 31. Matthis J, Barton S, Fajen B. The critical phase for visual control of human walking over complex terrain. Proceedings of the National Academy of Sciences;114:7–24.
- 32. Binaee K, Diaz GJ. Assessment of an augmented reality apparatus for the study of visually guided walking and obstacle crossing. Behavior Research Methods. 2019;51:523–531. pmid:30132240
- 33.
Kothari R, Binaee K, Matthis JS, Bailey R, Diaz GJ. Novel apparatus for investigation of eye movements when walking in the presence of 3D projected obstacles. In: presented at the Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications; 2016.
- 34. Sekhavat YA, Namani MS. Projection-Based AR: Effective Visual Feedback in Gait Rehabilitation. IEEE Transactions on Human-Machine Systems. 2018;48:626–636.
- 35. Lee CH, Kim Y, Lee BH. Augmented reality-based postural control training improves gait function in patients with stroke: Randomized controlled trial. Hong Kong Physiotherapy Journal. 2014;32:51–57.
- 36. Bennour S, Ulrich B, Legrand T, Jolles B, Favre J. A gait retraining system using augmented-reality to modify footprint parameters: Effects on lower-limb sagittal-plane kinematics. Journal of Biomechanics. 2017;66(26-35). pmid:29137725
- 37.
Keighrey C, Flynn R, Murray S, Murray N. A QoE evaluation of immersive augmented and virtual reality speech & language assessment applications. 2017 Ninth International Conference on Quality of Multimedia Experience (QoMEX. 2017; p. 1–6.
- 38.
Omelina L, Bonnechre B, Jan SVS, Jansen B. Analyzing the quality of experience of computer games in rehabilitation: the therapist’s perspective. In: and others, editor. REHAB’16 Proceedings of the 4th Workshop on ICTs for improving Patients Rehabilitation Research Techniques, 13-14, Lisbon. ACM; 2016.
- 39. Haines T, Mcbride J, Triplett N, Skinner J, Fairbrother K, Kirby TJ. A comparison of men’s and women’s strength to body mass ratio and varus/valgus knee angle during jump landings. Journal of sports sciences. 2011;29:1435–1477. pmid:21916796
- 40. Schmitz RJ, Ficklin TK, Shimokochi Y, Nguyen AD, Beynnon BD, Perrin DH. Varus/valgus and internal/external torsional knee joint stiffness differs between sexes. The American journal of sports medicine. 2008;36:1380–1388. pmid:18535251
- 41. Russell KA, Palmieri RM, Zinder SM, Ingersoll CD. Sex differences in valgus knee angle during a single-leg drop jump. Journal of athletic training. 2006;41:166–171. pmid:16791301
- 42. Quatman CE, Hewett TE. The anterior cruciate ligament injury controversy: is “valgus collapse” a sex-specific mechanism? British journal of sports medicine. 2009;43:328–335. pmid:19372087
- 43.
Technology XDMT. MTw Awinda—Wireless Motion Tracker; 2019. Available from: https://www.xsens.com/products/mtw-awinda/.
- 44.
Rodrigues T, Cathain C, Devine D, Moran K, O’Connor NE, Murray N. An Evaluation of a 3D Multimodal Marker-less Motion Analysis System. In: ACM Multimedia Systems Conference; 2019.
- 45.
Rodrigues TB, Salgado DP, Catháin CO, Connor N, Murray N. Human gait assessment using a 3D marker-less multimodal motion capture system. Multimedia Tools and Applications. 2019;.
- 46.
Hu V, Charry E, Umer M, Ronchi A, Taylor S. An inertial sensor system for measurements of tibia angle with applications to knee valgus/varus detection. In: and others, editor. 2014 IEEE Ninth International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP), 21-24, Singapore. IEEE; 2014.
- 47. Vargas-Valencia LS, Elias A, Rocon E, Bastos-Filho T, Frizera A. An IMU-to-Body Alignment Method Applied to Human Gait Analysis. Sensors. 2016;16:2090–2090.
- 48.
Sebastian M, OH, Andrew H, JL, Ravi V. Estimation of IMU and MARG orientation using a gradient descent algorithm. In: 2011 IEEE International Conference on Rehabilitation Robotics; 2011.
- 49.
Mcdaniel T, Krishna S, Villanueva D, Panchanathan S. A haptic belt for vibrotactile communication. 2010 IEEE International Symposium on Haptic Audio Visual Environments and Games. 2010; p. 1–2.
- 50. Rahman MF, Patterson D, Cheok A, Betz R. 30—Motor Drives. Power Electronics Handbook. 2018; p. 945–1021.
- 51.
Corp ©SE. MOVERIO BT-30C—Smart Glasses; 2019. Available from: https://moverio.epson.com/.
- 52. Bhatia V, Joshi S, Chapaneri R. Websocket-Evented Real-Time Online Coding Collaborator. In: Smart Intelligent Computing and Applications; 2019. p. 325–334.
- 53. Ilahi O, Kadakia N, Huo M. Inter- and intraobserver variability of radiographic measurements of knee alignment. The American journal of knee surgery. 2001;14:238–280. pmid:11703037
- 54.
Itu-T. Methods for the subjective assessment of video quality, audio quality and audiovisual quality of Internet video and distribution quality television in any environment. TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU2016;.
- 55.
Egan D, Brennan S, Barrett J, Qiao Y, Timmerer C, Murray N. An evaluation of Heart Rate and ElectroDermal Activity as an objective QoE evaluation method for immersive virtual reality environments. 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX). 2016; p. 1–6.
- 56. Gregori N, Feuer W, Rosenfeld P. Novel method for analyzing snellen visual acuity measurements. Retina, The Journal of Retinal and Vitreous Diseases. 2010;30(7):1046–1050.
- 57. Pickford RW. The Ishihara Test for Colour Blindness. Nature. 1944;153:656–657.
- 58. Bell-Krotoski JA, Fess EE, Figarola JH, Hiltz D. Threshold Detection and Semmes-Weinstein Monofilaments. Journal of Hand Therapy;8:155–162. pmid:7550627
- 59.
Rodrigues T. QoE Questionnaire of Gait Feedback System; 2019. Available from: http://bit.ly/QoEGaitFeedback.
- 60.
Laghari AA, Hui H, Shafiq M, Khan A. Assessing effect of Cloud distance on end user’s Quality of Experience (QoE). 2016 2nd IEEE International Conference on Computer and Communications (ICCC). 2016; p. 500–505.
- 61. Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction. 2008;24:7–7.
- 62. Schubert T, Friedmann F, Regenbrecht H. The Experience of Presence: Factor Analytic Insights. Presence: Teleoperators and Virtual Environments. 2001;10:266–281.
- 63. Lewis J, R J. IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction. 1993;7(1):57–78.
- 64. Mishra P, Pandey CM, Singh U, Gupta A, Sahu C, Keshri A. Descriptive statistics and normality tests for statistical data. Ann Card Anaesth. 2019;22(1):67–72. pmid:30648682
- 65.
Field A; 2013. Discovering Statistics using IBM SPSS Statistics (4th. ed.). Sage Publications Ltd.