https://orcid.org/0000-0003-0545-7627
Figures
Abstract
Background
The applicability of mobile digital technology to promote clinical care of people with multiple sclerosis (pwMS) is gaining increased interest as part of the implementation of patient-centered approaches. We aimed at assessing adherence to a smartphone-based e-diary, which was designed to collect patient-reported outcomes (PROs). Secondary objectives were to evaluate the construct and predictive validity of e-diary derived PROs and to explore the various factors that were associated with changes in PROs over time.
Materials and methods
In this observational cohort study patients downloaded an MS tailored e-diary into their personal smartphones. Report of PROs was enquired once monthly for a period of one year through a smartphone-based application, using previously validated tools. An e-diary derived bodily function summary score (eBF) was defined as the sum of scores depicting vision, limbs function, pain, bowl/ bladder dysfunction, pseudobulbar affect and spasticity. Multiple linear regression and analysis of covariance were used to determine the association between PROs, clinician-reported outcomes (ClinROs) of disease activity and quality of life (QoL). Regression coefficient analysis was used to compare the slope of change in eBF before and after a relapse.
Results
97 pwMS downloaded the e-diary [Female: 64 (66%), EDSS 3.4±2.1]. 76 patients (78%) completed the 12-month study period. 53 patients (55%) submitted ≥75% of requested surveys. Anxiety was negatively associated with adherence to periodic PROs assessments by the e-diary. E-diary derived PROs were significantly correlated with corresponding functional system scores (0.38< r <0.8, P<0.001). eBF score significantly predicted QoL (β = -0.36, P = 0.001) while EDSS did not. Change in eBF score over time was independently associated with the occurrence of an MS relapse (F = 4.4, P = 0.04), anxiety (F = 6.4, P = 0.01) and depression (F = 5.1, P = 0.03). Individual regression slopes of eBF scores were significantly higher pre-relapse than post-relapse (3.0±3.3 vs. -0.8±2.0, P = 0.007).
Citation: Golan D, Sagiv S, Glass-Marmor L, Miller A (2021) Mobile-phone-based e-diary derived patient reported outcomes: Association with clinical disease activity, psychological status and quality of life of patients with multiple sclerosis. PLoS ONE 16(5): e0250647. https://doi.org/10.1371/journal.pone.0250647
Editor: Marcello Moccia, Universita degli Studi di Napoli Federico II, ITALY
Received: December 10, 2020; Accepted: April 12, 2021; Published: May 5, 2021
Copyright: © 2021 Golan et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: Funding for this research was provided by Teva Pharmaceutical Industries Ltd as part of the Israeli National Network of Excellence in Neuroscience (NNE) and by the Merry Sherman Saifer Fund. The funders were not involved in the study design, data collection and analysis, decision to publish, or preparation of the manuscript".
Competing interests: Funding for this research was provided in part by Teva Pharmaceutical Industries Ltd. This does not alter our adherence to PLOS ONE policies on sharing data and materials. All authors declare no other competing interests relevant to the submitted research.
Introduction
The advent of mobile electronic platforms opened up the possibility for interactive diaries that remind patients to enter data, prospectively collect information about patients’ symptoms while at their natural environment and present a comprehensive reflection of disease status from the patients’ perspective as part of the implementation of patient centric approach and participatory medicine [1–4]. As the everydayness of our lives becomes increasingly digitized, data generated outside of healthcare encounters holds promise to fill recognized gaps in real-world evidence [5].
Collection of patient-reported outcomes (PROs) by electronic diaries (e-diaries) may be suitable to supplement clinician-reported outcomes (ClinROs) for people with MS (pwMS) [6]. PROs have many potential uses, such as promoting communication between patients and physicians, assessing change in disease status, screening for unidentified symptoms as well as monitoring safety and efficacy of medications [7, 8]. Traditional physician derived endpoints, like the neurological examination and the expanded disability status scale (EDSS), do not capture the gamut of MS symptoms, such as pain, fatigue, depression and sleep disturbance. These are better depicted by PROs, which despite being subjective, have been shown to correlate with objective assessments [9–11], predict objective disease worsening [12] and have an important association with quality of life (QoL) [13]. Traditionally, PROs have been assessed using paper questionnaires but on-line administration of PROs gains popularity and has been shown to be equivalent [14]. In most assessments of currently available electronic tools related to medical use, the most prevalent functionalities were providing users with information/education, assisting users with their therapy adherence and helping users monitor the effect and possible side effects of their medications [15]. Despite the abundance of healthcare-related applications very little research has been undertaken to investigate the validity and efficacy of these tools as well as their potential risks [16]. A randomized clinical trial demonstrated that a mobile device application enhances reporting of adverse drug reactions by pwMS compared to traditional reporting by phone or e-mail [17]. A small study of an MS custom application designed to gather both PROs and objective performances (motor, cognitive and visual) reported high attrition among patients with visual as well as with subjective cognitive difficulties [18]. Still, other researchers have shown that the majority of pwMS use mobile phones regularly and are competent to use electronic patient relationship management systems [19]. It has also been reported that pwMS frequently use internet-based social media to gain knowledge about their disease and its management and are capable to distinguish reliable users, from those who post inaccurate information [20].
We developed an electronic diary (e-diary) smartphone application for longitudinal collection of PROs from pwMS. The potential added values of the smartphone application are: (1) to improve clinical practice by conveying summary scores of various disease related symptoms to clinicians. These scores and their change over time may facilitate discussion between patients and their caregivers regarding those symptoms and their management. (2) valid PRO summary scores can be used as efficacy endpoints in clinical trials of novel therapies. (3) electronic PRO measures can promote Patient-Powered Research Networks (PPRN) by collecting patient data with greater frequency than clinic visits, as already implemented by the Patient-Centered Outcomes Research Institute (PCORI) [21]. In this study we aimed at evaluating adherence to an electronic diary (e-diary) smartphone application for longitudinal collection of PROs from pwMS over 12 months and to describe predictors for adherence. Secondary objectives were to evaluate the construct and predictive validity of the e-diary derived PROs and to explore the various factors, disease related and psychological, that were associated with changes in PROs over time.
We premised good adherence to the tool and significant correlations between e-diary derived PROs and related clinician-reported outcomes. We hypothesized that changes in e-diary derived PROs over time would be associated with anxiety and depression but would nevertheless predict clinical MS relapses. We also hypothesized that e-diary derived PROs will be predictive of participants’ quality of life.
Materials and methods
This was an observational cohort study. All participants were recruited at the multiple sclerosis clinic of Carmel medical center, Haifa, Israel. Patients were enrolled between February 2016 and January 2017. Data collection was continued until the last patient completed 12 months follow-up.
Participants
97 pwMS according to the revised McDonald criteria were recruited for this study. Inclusion criteria were: age 18 to 70 years, able to browse the internet and use a smartphone, willing and able to give informed consent, EDSS ≤ 7. Patients with neurological conditions involving the central nervous system other than MS were excluded. Patients were offered to take part in the study consecutively, during their routine clinic visits. Of the patients approached, only five declined to participate. The study was approved by the institutional review board of the Lady Davis Carmel Medical Center, Haifa, Israel. IRB approval number: CMC-0065-13. All participants gave written informed consent.
The e-diary application
An e-diary, tailored for pwMS, was developed. The e-diary is an internet smartphone application. Previously validated PROs questionnaires from multiple sources were computed into the e-diary application in order to cover the spectrum of MS related symptoms. The PROs that were used and data regarding their internal consistency, reliability and validity are elaborated in the S1 Table. Our intention in selecting the composition of PROs to be computed into the e-diary was to create a tool that could specifically grade MS symptoms in a comprehensive manner, with emphasis on symptoms frequency and intensity. PROs selection was in line with the intended purposes of the e-diary to enable patients to convey a comprehensive report of their symptoms in a timely and efficient manner, as well as a potential tool to collect PROs as efficacy end-point in therapeutic clinical trials.
PROs surveys were divided into two components, bodily function and mental. The bodily function survey included the impact of visual impairment scale, pain effects scale and bowel & bladder control scales from the MS quality of life inventory (MSQLI) [22]. Additionally included in the bodily function survey were the upper and lower extremity function short forms from the Neuro-QOL inventory [23], the numeric rating scale to measure spasticity [24] and the Center for Neurologic Study- Lability Scale (CNS-LS) to measure pseudo-bulbar affective lability [25].
The mental survey consisted of the mental health inventory, which measures anxiety, depression, positive affect and behavioral control, as well as the abbreviated modified fatigue impact scale, both from the MSQLI [22]. Also included in the mental survey were the abbreviated perceived deficits questionnaire [22], which measures subjective cognitive complaints, and the sleep disturbance short form [23].
Plots could be produced by the application, which presented PROs summary scores of each domain by time from study entry. These plots enabled physician and patients to gain insight into the various symptoms scores and their trajectory over time. The application also sent reminders to take disease modifying drugs (DMDs) and asked users to report their DMDs intake. Monitoring of adherence to DMDs by this e-diary was reported elsewhere [26].
Study measures and procedures
Patients were clinically evaluated at baseline, after 6 months and by the end of a 12 month follow up period. On each visit relapses were recorded, and a detailed neurological examination was performed and quantified according to the neurostaus definitions. EDSS score as well as pyramidal, cerebellar, sensory, brainstem bowel & bladder and visual system scores were obtained [27].
At baseline, QoL was assessed using the abbreviated version of the generic world health organization quality of life questionnaire (WHOQOL-BREF) [28]. This QoL measure has a broad scope, including ‘physical health and autonomy’, ‘psychological health’, ‘social relationships’ and ‘environmental aspects’. The total score is between 0 (worst health) to 20 (best health). The WHOQOL-BREF showed satisfactory levels of internal consistency with Cronbach’s alpha between 0.63 and 0.81among pwMS [29].
Anxiety and depression were evaluated at baseline with the Hospital Anxiety and Depression Scales (HADS). A cut-point of ≥11 indicates clinically meaningful symptoms of anxiety and depression [30]. Cognitive function at baseline was screened using the Symbol Digit Modalities Test (SDMT) [31]. Raw scores were converted to age adjusted z-scores, using normative data [32]. Baseline evaluations of QoL, anxiety, depression and cognition were assessed using traditional paper and pencil questionnaires.
Participants were asked to submit a comprehensive monthly PROs report by means of the e-diary application. The e-diary reminded participants to submit the monthly report. In addition, patients could submit additional reports any time, at their discretion. A summary e-diary derived bodily function score (eBF) was computed as the sum scores of the ’bodily function’ components, namely the sum of visual impairment, lower and upper extremity, pain, bowel & bladder, pseudobulbar and spasticity sub scores (S1 Table).
Construct validity of e-diary derived PROs was assessed by looking for correlations between subjective symptom-specific measures and physician determined systems scores of the corresponding domains [27]. Predictive validity of e-diary derived PROs was estimated by looking for independent association with QoL and by looking for an association between change in e-diary derived PROs over time and MS relapse activity.
Statistical analysis
Data analysis was performed using SPSS, version 23.0 (IBM Corp., Armonk, N.Y., USA). Data distribution was inspected for normality. Continuous variables were analyzed using between-groups t-test. Categorical variables were analyzed with chi-square or Fisher’s exact tests. Correlation coefficients are Pearson’s r or Spearman’s rho, according to data distribution. Linear regression modeling was used to assess the independent contribution of variables that were significantly associated with QoL in the univariate correlation analyses. Maximal change in bodily function PROs was computed for each participant as the difference between the maximal and minimal e-diary derived bodily function score (maximal eBF score minus minimal eBF score). Analysis of covariance (ANCOVA) was used to assess the independent contribution of anxiety, depression and clinical MS relapses to maximal change in bodily function PROs over time. Regression coefficient analysis and individual regression slopes were used to assess the trend of change in individual eBF scores over time [33]. For participants with an on-study relapse, pre relapse regression slopes were determined between baseline and relapse date. Post relapse regression slopes were determined between relapse date and 6 months post relapse or end of follow up, whichever obtained first (Fig 1). Pre relapse regression slopes were compared to post relapse regression slopes using paired t-test. Pre relapse regression slopes were also compared to whole study regression slopes of patients who did not relapse during the study (baseline to end) using independent groups t-test. Descriptive statistics are reported as mean ± standard deviation or frequency (%). Sample size considerations are detailed in the (S1 File).
The sum scores of e-diary derived bodily function PROs by time since study entry from a single participant are shown. This patient had an on-study relapse about 5 months after enrolment to the study (marked by the letter ’R’). The trend of change in bodily function PROs score is depicted by the dashed line, which represents the individual regression slope [33]. The pre-relapse regression slope (left panel) is positive, indicating worsening, while the post-relapse regression slope (right panel) is negative, indicating improvement.
Results
Participants and adherence to the e-diary
Demographic and clinical characteristics of the 97 patients who downloaded the e-diary application are given in Table 1. Study flow chart is shown in Fig 2. Drop out was defined as complete cessation of e-diary activity or non-show to clinical follow up. Patients who remained partially cooperative in the study were not regarded as dropouts. 17 patients dropped out before 6 months of follow up and additional 4 dropped out before 12 months of intended follow up. Reasons for discontinuation were: anxiety provoked by the PROs surveys (n = 5), internet or smart-phone malfunction (n = 4), did not attend clinic visits (n = 4), could not learn how to use the application (n = 3), time consuming / not interested (n = 3), pregnancy (n = 1), language difficulties (n = 1) and loss to follow up (n = 1).
21 patients dropped out before 12 months of intended follow up. EDSS = Expanded Disability Status Scale. SDMT = Symbol Digit Modalities Test. HADS = Hospital Anxiety and Depression Scale. WHOQOL-BREF = World health organization quality of life questionnaire. pwMS = people with multiple sclerosis.
Adherence to PROs collection by means of the e-diary is presented in Fig 3. Patients were asked to submit a bodily function report and a mental function report at least once a month. 80% of participants submitted at least 50% of the required PROs reports, while about 60% of participants submitted at least 75% of the required surveys. The adherence to bodily function reporting was slightly higher compared to the adherence to the mental function survey.
Patients were asked to submit a bodily function report and a mental function report at least once a month. 80% of participants submitted at least 50% of the required PROs reports, while about 60% of participants submitted at least 75% of the required surveys. The adherence to bodily function reporting was slightly higher compared to the adherence to the mental function survey.
Predictors of adherence to PROs surveys are presented in Table 2. Female patients were more adherent than male patients. Anxiety was associated with poor adherence to PROs surveys.
Construct validity of e-diary derived PROs
Correlations between e-diary derived PROs and quantified neurological examination scores at baseline are given in Table 3. Strong correlation was found between the sum of bodily function scores (eBF) and EDSS. Strong correlations were also found between the pyramidal system score and PROs of spasticity, lower and upper limb functions. Moderate correlations were found between sensory and visual system scores and corresponding PROs of pain and visual functions. Subjective PROs of cognitive difficulties and limb functions were weakly correlated with SDMT and cerebellar system scores, respectively. E-diary derived depression and anxiety scores were strongly associated with corresponding HADS-D and HADS-A scores (Pearson r = 0.68 for depression and 0.72 for anxiety, P value <0.001).
Predictive validity of e-diary derived PROs: Association with QoL
Correlations between various disease characteristics and QoL at baseline, as measured by the generic world health organization quality of life questionnaire (WHOQOL-BREF) are given in Table 4. On univariate analysis, the sum of bodily function scores (eBF), fatigue, anxiety, subjective cognitive difficulties and sleep were strongly correlated with QoL. EDSS, depression and SDMT were moderately correlated with QoL at baseline. Multivariate analysis showed that only the sum of bodily function PRO scores (eBF), depression and anxiety were independently associated with QoL. Objective/ physician derived measurements, such as EDSS and SDMT were not independently associated with QoL in this study.
Predictive validity of e-diary derived PROs: Association with change in disease status
The distribution of individual regression slopes of the sum of bodily function scores (eBF) are presented in Fig 4. 13 patients had a relapse during the study. The average pre relapse slopes were positive (indicating worsening), while post relapse slopes were negative on average (indicating improvement). Individual regression slopes of patients who did not relapse during the study were close to zero. Pre-relapse regression slopes were significantly higher than both post-relapse regression slopes, and regression slopes of patients who did not experience an MS exacerbation during the study.
13 patients had a relapse during the study. The average pre-relapse slopes were positive (indicating worsening), while post-relapse slopes were negative on average (indicating improvement). Individual regression slopes of patients who did not relapse during the study were close to zero. P value for the comparison between pre-relapse and post-relapse regression slopes is from a paired t-test. P value for the comparison between pre-relapse regression slopes and regression slopes of patients without clinical MS relapses is from an independent group t-test.
The independent effect of predictors of maximal change over time in e-diary derived bodily function PROs are given in Table 5. Depression and anxiety were independently associated with change in bodily function PROs over time. On study MS relapse was also an independent predictor of change in e-diary derived bodily function PROs. There were insignificant interactions between depression and on-study relapse as well as between anxiety and on-study relapse in association with maximal change in e-diary derived bodily function patient reported outcomes.
Discussion
In this study we show that pwMS can adhere to continuous collection of PROs by a smartphone-based e-diary. These PROs represent disease aspects with a significant and independent association with patients’ QoL. E-diary derived bodily function PROs were associated, tough not identical, with their corresponding domains from the quantified neurological examination, in support of their construct validity. Change in bodily function PROs over time on the group level was independently associated with clinical MS relapses, anxiety and depression.
The overall adherence to the e-diary as a tool for longitudinal PROs evaluation was acceptable. About 80% of participants submitted at least 50% of the required surveys, meaning that for 80% of patients at least one PROs report was available every 2 months, which is more often than the usual frequency of clinic visits. Thus, the e-diary carries the potential of tapping patients’ status in between clinic visits. This finding is in line with previous reports of higher adherence to e-diaries, when compared to paper diaries, partly due to compliance-enhancing features as alarms and reminders [35–37]. While it has been reported that male and younger patients had more favorable perceptions of PROs collection [7], we found better adherence to PROs surveys among female pwMS. Age was not significantly influencing adherence to PROs collection in this study, however most of our patients were young (Mean age 40.4). E-diary application questionnaires were in Hebrew. Although one patient dropped out due to language difficulties, primary language other than Hebrew was not associated with adherence to PROs collection by means of the e-diary.
Anxiety turned out to be a major cause of both complete drop-out and decreased adherence to PROs surveys. Patients reported that the PROs questionnaires exposed them to anxiety provoking awareness of MS related symptoms that they did not yet experience. Interestingly, the same observation was made with an e-diary for patients with headache. Endorsing higher daily anxiety was associated with lower odds of complete reporting [38]. Future e-diaries for pwMS should be personalized. A general question about the relevance of a symptom cluster to the respondent should be introduced first and detailed specific questions should follow only if necessary.
The bodily function component of the e-diary is based upon subjective measures, which were originally designed to capture the impact of neurological symptoms upon QoL, such as the MS quality of life inventory (MSQLI) [22] and the Neuro-QOL inventory [23]. The correlations between patients and physicians scores varied for different symptom domains (Table 3). Strong correlations were noted for pyramidal system dysfunction (muscle power, spasticity), in agreement with previously reported strong correlations between the Neuro-QOL lower extremity score and the timed 25 foot walk [23]. Only weak to moderate correlations were noted for visual, sensory, cerebellar and cognitive dysfunction, consistent with previous studies with similar analyses [39, 40]. Discrepancy between patients and physicians has many potential reasons. Pain is an inherently subjective experience, which is only partially captured by deficits in the sensory examination. Subjective cognitive complaints are known to be inflated among patients with depression and fatigue [41, 42], while ignored by those with frank dementia, who lose insight into their cognitive status [43]. The visual system score relies mainly upon visual acuity, while the patient may refer to visual impairment as a result of nystagmus, visual field defects and diplopia. Nevertheless, significant correlations between bodily function PROs and quantified neurological examination add credence to the concept of disease monitoring by using PROs.
The lack of perfect agreement between PROs and objective assessments highlight the importance of PROs to get a full picture of the impact of symptoms on QoL [44] as well as for comprehensive evaluation of response to therapy [45]. Indeed, bodily function PROs were independently associated with QoL, while EDSS did not (Table 4), confirming previous reports [46]. Furthermore, it seems that the exact composition of the PROs tool determines the degree of its correlation with objective assessments. In this study, the sum of bodily function PROs was strongly correlated with EDSS (r = 0.77), while the generic QoL questionnaire (WHOQOL-BREF) was only moderately associated with EDSS (r = -0.48). This is not surprising given that our intention in selecting the composition of PROs to be computed into the e-diary was to create a tool that could specifically grade MS symptoms in a comprehensive manner, with emphasis on symptoms frequency and intensity.
An important impediment in using PROs for individual patient clinical decision making is the large fluctuations in their scores, despite apparent clinical stability. For example, it has been shown that Multiple Sclerosis Walking Scale-12 (MSWS-12), a PRO of walking, remained stable over 2 years on the group level, although fluctuations on the individual level were found in up to 80% of the patients, above the minimal clinically important difference for that scale, which takes into account measurement standard errors and test-retest reliability. These fluctuations in walking PROs on the individual level, but not on the group level, were noted despite being free from clinical relapses [47]. In this study, the e-diary derived bodily function PRO (eBF score) was predictive of clinical disease activity on the group level. This summary score increased pre-relapse, decreased post-relapse and was close zero for participants without a relapse (Fig 4). Individual regression slopes moderate random fluctuations in PROs and may be more reliably indicative of change than comparing the last PRO value to the previous one. The small number of relapses in this study (13 events) precluded determination of sensitivity and specificity of individual regression slopes as relapse predictors on the individual level. Mental PROs: depression, anxiety, physical and mental fatigue–are not part of routine relapse determination in our medical center. Therefore, summary score of mental functions was deemed less suitable for the predictive validity analysis.
The maximal change in eBF score was independently associated with both clinical disease activity and the psychological status of patients (Table 5). It follows that change in bodily function PROs was driven by MS relapses, but the degree of change was also influenced by depression and anxiety. This finding is in agreement with previous reports [48]. Both depression [49] and anxiety [50] have been identified as independent predictors of self-rated disability and QoL among patients with MS. Further work is needed to determine thresholds for change in eBF scores or eBF individual regression slopes that could reliably predict clinical disease activity on the individual level. Higher thresholds are expected for patients with clinically significant depression or anxiety.
This study has several limitations. MRI was not part of this research protocol. Only 15 patients had a scan at baseline and by the end of the 1year clinical follow-up, which precluded the integration of MRI into the definition of disease activity in this study. Since the e-diary was considered experimental, patients knew that the information was not consistently used for providing medical care. It is reasonable to premise that if patients knew that the e-diary derived information was affecting their care, adherence to using the e-diary could have been even higher. Due to convenience sample of small size, our findings should be regarded as preliminary. Despite statistically significant difference between pre-relapse slopes and whole-study slopes of change in eBF among participants who did not relapse, the possibility of non-representative sample exists. Therefore, further studies with larger samples are needed to confirm validity of this e-diary, in terms of relapse prediction. Indeed, fully remote mobile health studies have enrolled in the tens of thousands from across distributed geographic regions, providing the opportunity for broad sampling across diverse populations in the real-world setting [51]. It is also unknown if e-diary derived PROs could predict the more subtle and gradual progression, which is independent of relapse activity.
Conclusion
PwMS seem to adhere to longitudinal PROs collection by using a smartphone-based e-diary application. The construct validity and responsiveness of e-diary derived PROs to change in objective clinical status on the group level, carry the potential for using smartphone-based PRO diaries in clinical research, to monitor therapeutic effects of medications from the patients’ perspective. Further work with a larger samples is needed to determine thresholds for significant change in PROs scores on the individual level, which may alert clinicians about possible disease worsening in between clinic visits. Integration of such PROs with the treating clinician’s perspective may provide a basis for informed and shared therapeutic decisions and may represent a further step in the implementation of digital health in MS care.
Supporting information
S1 Table. Patient reported outcome measures that were periodically collected by the e-diary.
https://doi.org/10.1371/journal.pone.0250647.s001
(DOCX)
Acknowledgments
The smartphone e-diary application, cloud data aggregation and report generation were provided by Irody, Inc.
References
- 1. Lejbkowicz I, Caspi O, Miller A. Participatory medicine and patient empowerment towards personalized healthcare in multiple sclerosis. Expert Rev Neurother. 2012;12: 343–352. pmid:22364333
- 2. Zissman K, Lejbkowicz I, Miller A. Telemedicine for multiple sclerosis patients: assessment using Health Value Compass. Mult Scler. 2012;18: 472–480. pmid:21965420
- 3. Lejbkowicz I, Paperna T, Stein N, Dishon S, Miller A. Internet usage by patients with multiple sclerosis: implications to participatory medicine and personalized healthcare. Mult Scler Int. 2010;2010: 640749. pmid:22096625
- 4. Rocca E, Anjum RL. Erice Call for Change: Utilising Patient Experiences to Enhance the Quality and Safety of Healthcare. Drug Saf. 2020;43: 513–515. pmid:32297160
- 5. Okun S. The Missing Reality of Real Life in Real-World Evidence. Clin Pharmacol Ther. 2019;106: 136–138. pmid:31002396
- 6. van Munster CEP, Uitdehaag BMJ. Outcome Measures in Clinical Trials for Multiple Sclerosis. CNS Drugs. 2017;31: 217–236. pmid:28185158
- 7. Lapin B, Udeh B, Bautista JF, Katzan IL. Patient experience with patient-reported outcome measures in neurologic practice. Neurology. 2018;91: e1135–e1151. pmid:30135254
- 8. Katzan IL, Thompson NR, Dunphy C, Urchek J, Lapin B. Neurologic provider views on patient-reported outcomes including depression screening. Neurol Clin Pract. 2018;8: 86–92. pmid:29708224
- 9.
Report AC. IgG4-Related Neuropathy. https://doi.org/10.1001/jamaneurol.2013.658.Text
- 10. Gromisch ES, Zemon V, Holtzer R, Chiaravalloti ND, DeLuca J, Beier M, et al. Assessing the criterion validity of four highly abbreviated measures from the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS). Clin Neuropsychol. 2016;30: 1032–1049. pmid:27279357
- 11. Lavorgna L, Sparaco M, Esposito S, Motl RW, Gallo A, Bisecco A, et al. Validity and reproducibility of the Italian version of the patient determined disease steps scale in people with multiple sclerosis. Mult Scler Relat Disord. 2017;18: 173–176. pmid:29141804
- 12. Cavallari M, Palotai M, Glanz BI, Egorova S, Prieto JC, Healy BC, et al. Fatigue Predicts Disease Worsening in Relapsing-Remitting Multiple Sclerosis Patients. Mult Scler J. 2016; 1–9. pmid:26920374
- 13. Riazi A. Patient-reported Outcome. Int MS J. 2006;13: 92–99. pmid:17101077
- 14. Lavorgna L, Miele G, Petruzzo M, Lanzillo R, Bonavita S. Online validation of the Italian version of the patient determined disease steps scale (PDDS) in people with multiple sclerosis. Mult Scler Relat Disord. 2018;21: 108–109. pmid:29547779
- 15. van Kerkhof LWM, van der Laar CWE, de Jong C, Weda M, Hegger I. Characterization of Apps and Other e-Tools for Medication Use: Insights Into Possible Benefits and Risks. JMIR mHealth uHealth. 2016;4: e34. pmid:27052946
- 16. Boulos MNK, Brewer AC, Karimkhani C, Buller DB, Dellavalle RP. Mobile medical and health apps: state of the art, concerns, regulatory control and certification. Online J Public Health Inform. 2014;5: 229. pmid:24683442
- 17. Defer G, Fedrizzi S, Chevanne D, Montastruc F, Briant AR, Parienti JJ, et al. Adverse Drug Reaction Reporting Using a Mobile Device Application by Persons with Multiple Sclerosis: A Cluster Randomized Controlled Trial. Drug Saf. 2021;44: 223–233. pmid:33048319
- 18. Bove R, White CC, Giovannoni G, Glanz B, Golubchikov V, Hujol J, et al. Evaluating more naturalistic outcome measures, A 1-year smartphone study in multiple sclerosis. Neurol Neuroimmunol Neuroinflamm. 2015;2. pmid:26516627
- 19. Haase R Schultheiss, Kempcke T, Thomas R, K Ziemssen T. Modern communication technology skills of patients with multiple sclerosis. Mult Scler. 2013;19: 1240–1. pmid:23388164
- 20. Lavorgna L, De Stefano M, Sparaco M, Moccia M, Abbadessa G, Montella P, et al. Fake news, influencers and health-related professional participation on the Web: A pilot study on a social-network of people with Multiple Sclerosis. Mult Scler Relat Disord. 2018;25: 175–178. pmid:30096683
- 21. Nowell WB. Information Patients Can Provide Will Strengthen the Real-World Evidence That Matters to Them. Clin Pharmacol Ther. 2019;106: 49–51. pmid:31112287
- 22. Ritvo PG, Fischer JS, Miller DM, Andrews H, Paty DW, LaRocca NG. Multiple Sclerosis Quality of Life Inventory: A user’s manual. Mult Scler. 1997; 1–65. Available: https://www.nationalmssociety.org/NationalMSSociety/media/MSNationalFiles/Brochures/MSQLI_-A-User-s-Manual.pdf pmid:9160341
- 23. Miller DM, Bethoux F, Victorson D, Nowinski CJ, Buono S, Lai JS, et al. Validating Neuro-QoL short forms and targeted scales with people who have multiple sclerosis. Mult Scler J. 2016;22: 830–841. pmid:26238464
- 24. Farrar JT, Troxel AB, Stott C, Duncombe P, Jensen MP. Validity, reliability, and clinical importance of change in a 0–10 numeric rating scale measure of spasticity: a post hoc analysis of a randomized, double-blind, placebo-controlled trial. Clin Ther. 2008;30: 974–985. pmid:18555944
- 25. Moore SR, Gresham LS, Bromberg MB, Kasarkis EJ, Smith RA. A self report measure of affective lability. J Neurol Neurosurg Psychiatry. 1997;63: 89–93. pmid:9221973
- 26. Golan D, Sagiv S, Glass-Marmor L, Miller A. Mobile phone-based e-diary for assessment and enhancement of medications adherence among patients with multiple sclerosis. Mult Scler J—Exp Transl Clin. 2020;6: 205521732093930. pmid:32864155
- 27.
Kappos L. Neurostatus–Neurostatus training and documentation DVD for a standardized neurological examination and assessment of Kurtzke׳s functional systems and expanded disability status scale for MS patients. 2002 [cited 23 Nov 2016]. Available: http://neurostatus.net/
- 28. Skevington SM, Lotfy M, O’Connell KA, WHOQOL Group. The World Health Organization’s WHOQOL-BREF quality of life assessment: Psychometric properties and results of the international field trial. A Report from the WHOQOL Group. Qual Life Res. 2004;13: 299–310. pmid:15085902
- 29. Wynia K, Middel B, van Dijk J, de Ruiter H, De Keyser J, Reijneveld SA. The Multiple Sclerosis Impact Profile (MSIP). Development and testing psychometric properties of an ICF-based health measure. Disabil Rehabil. 2008;30: 261–274. pmid:17852244
- 30. Zigmond AS, Snaith RP. The hospital anxiety and depression scale. Acta Psychiatr Scand. 1983;67: 361–70. pmid:6880820
- 31. Strober L, DeLuca J, Benedict RH, Jacobs A, Cohen JA, Chiaravalloti N, et al. Symbol Digit Modalities Test: A valid clinical trial endpoint for measuring cognition in multiple sclerosis. Mult Scler J. 2018; 135245851880820. pmid:30334474
- 32. Sheridan LK, Fitzgerald HE, Adams KM, Nigg JT, Martel MM, Puttler LI, et al. Normative Symbol Digit Modalities Test performance in a community-based sample. Arch Clin Neuropsychol. 2006;21: 23–28. pmid:16139470
- 33. Pfister R, Schwarz K, Carson R, Jancyzk M. Easy methods for extracting individual regression slopes: Comparing SPSS, R, and Excel. Tutor Quant Methods Psychol. 2013;9: 72–78.
- 34. Sumowski JF, Benedict R, Enzinger C, Filippi M, Geurts JJ, Hamalainen P, et al. Cognition in multiple sclerosis. Neurology. 2018;0: 10.1212/WNL.0000000000004977. pmid:29343470
- 35. Bandarian-Balooch S, Martin PR, McNally B, Brunelli A, Mackenzie S. Electronic-Diary for Recording Headaches, Triggers, and Medication Use: Development and Evaluation. Headache J Head Face Pain. 2017;57: 1551–1569. pmid:28925498
- 36. Stone AA, Shiffman S, Schwartz JE, Broderick JE, Hufford MR. Patient compliance with paper and electronic diaries. Control Clin Trials. 2003;24: 182–99. Available: http://www.ncbi.nlm.nih.gov/pubmed/12689739 pmid:12689739
- 37. Midaglia L, Mulero P, Montalban X, Graves J, Hauser SL, Julian L, et al. Adherence and satisfaction of smartphone- And smartwatch-based remote active testing and passive monitoring in people with multiple sclerosis: Nonrandomized interventional feasibility study. J Med Internet Res. 2019;21: 1–15. pmid:31471961
- 38. Seng EK, Prieto P, Boucher G, Vives-Mestres M. Anxiety, Incentives, and Adherence to Self-Monitoring on a Mobile Health Platform: A Naturalistic Longitudinal Cohort Study in People With Headache. Headache. 2018;58: 1541–1555. pmid:30334248
- 39. Fischer JS, LaRocca NG, Miller DM, Ritvo PG, Andrews H, Paty D. Recent developments in the assessment of quality of life in Multiple Sclerosis (MS). Mult Scler J. 1999;5: 251–259. pmid:10467384
- 40. Marrie R a, Goldman M. Validity of performance scales for disability assessment in multiple sclerosis. Mult Scler. 2007;13: 1176–1182. pmid:17623733
- 41. Bruce JM, Bruce AS, Hancock L, Lynch S. Self-reported Memory Problems in Multiple Sclerosis: Influence of Psychiatric Status and Normative Dissociative Experiences. Arch Clin Neuropsychol. 2010;25: 39–48. pmid:19959565
- 42. Kinsinger SW, Lattie E, Mohr DC. Relationship between depression, fatigue, subjective cognitive impairment, and objective neuropsychological functioning in patients with multiple sclerosis. Neuropsychology. 2010;24: 573–80. pmid:20804245
- 43. Marrie RA, Chelune GJ, Miller DM, Cohen JA. Subjective cognitive complaints relate to mild impairment of cognition in multiple sclerosis. Mult Scler. 2005;11: 69–75. pmid:15732269
- 44. Benito-León J, Manuel Morales J, Rivera-Navarro J, Mitchell AJ. A review about the impact of multiple sclerosis on health-related quality of life. Disabil Rehabil. 2003;25: 1291–1303. pmid:14617435
- 45. Klineova S, Farber R, Friedman J, Farrell C, Lublin FD, Krieger S. Objective and subjective measures of dalfampridine efficacy in clinical practice. Mult Scler J—Exp Transl Clin. 2018;4: 205521731878674. pmid:30090641
- 46. Miller A, Dishon S. Health-related Quality of Life in Multiple Sclerosis: The Impact of Disability, Gender and Employment Status. Qual Life Res. 2006;15: 259–271. pmid:16468081
- 47. Motl RW, Putzki N, Pilutti LA, Cadavid D. Longitudinal changes in self-reported walking ability in multiple sclerosis. PLoS One. 2015;10: 1–12. pmid:25932911
- 48. Mitchell AJ, Benito-León J, González J-MM, Rivera-Navarro J. Quality of life and its assessment in multiple sclerosis: integrating physical and psychological components of wellbeing. Lancet Neurol. 2005;4: 556–566. pmid:16109362
- 49. Magistrale G, Medori R, Cadavid D, Argento O, Incerti C, Pisani V, et al. Assessing clinical correlates of self-rated disability in patients with multiple sclerosis. Mult Scler J–Exp Transl Clin. 2015;1: 205521731559242. pmid:28607696
- 50. Pham T, Jetté N, Bulloch AGM, Burton JM, Wiebe S, Patten SB. The prevalence of anxiety and associated factors in persons with multiple sclerosis. Mult Scler Relat Disord. 2018;19: 35–39. pmid:29125968
- 51. Omberg L, Chaibub Neto E, Mangravite LM. Data Science Approaches for Effective Use of Mobile Device–Based Collection of Real-World Data. Clin Pharmacol Ther. 2020;107: 719–721. pmid:32036612