https://orcid.org/0000-0002-4050-4196
Figures
Abstract
Objectives
To assess the psychometric characteristics of the Spanish Post-COVID-19 Functional Status (PCFS) scale (web-based and paper-based forms) and the test-retest reliability of the web-based form.
Study design and setting
Cross-sectional study of 125 COVID-19 survivors. The test-retest reliability of the web-based form was assessed at 7 days after the first evaluation. We collected symptoms, functional status (PCFS scale), health-related quality of life (EuroQol-5D questionnaire, EQ-5D-5L), activities of daily living limitations [Barthel Index and Global Activity Limitation Index, GALI] and psychological state (Hospital Anxiety and Depression Scale, HADS).
Results
The paper- and web-based forms of the Spanish PCFS scale showed adequate construct validity, and the web-based form provided substantial test-retest reliability (kappa = 0.63). The percentage of agreement between the web-based and paper-based forms was high (88%). Functional status showed a high correlation with EQ-5D-5L (inverse) and GALI (direct) (both; Rho ≥ .743), a moderate correlation with HADS (Rho ≥ .409) and a low correlation with the Barthel Index (Rho < .30). The Kruskal–Wallis test showed statistically significant differences in EQ-5D-5L, GALI and HADS according to the degree of functional status.
Citation: Sacristán-Galisteo C, del Corral T, Ríos-León M, Martín-Casas P, Plaza-Manzano G, López-de-Uralde-Villanueva I (2022) Construct validity of the Spanish version of the Post-COVID-19 Functional Status scale and validation of the web-based form in COVID-19 survivors. PLoS ONE 17(6): e0269274. https://doi.org/10.1371/journal.pone.0269274
Editor: Dong Keon Yon, Seoul National University College of Medicine, REPUBLIC OF KOREA
Received: November 22, 2021; Accepted: May 17, 2022; Published: June 1, 2022
Copyright: © 2022 Sacristán-Galisteo et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript. The data that support the findings of this study are available on request from the corresponding author (TDC).
Funding: Funded by the Foundation for Biosanitary Research and Innovation in Primary Care (FIIBAP) and the Regional Ministry of Health of the Community of Madrid through non-refundable grants from the credits awarded to the Community of Madrid by the Spanish Government Fund COVID-19, included in Order HAC/667/2020.
Competing interests: The authors have declared that no competing interests exist.
Introduction
On March 11, 2020, the World Health Organization declared the outbreak of a global pandemic of novel severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) causing coronavirus disease 2019 (COVID-19). Substantial numbers of SARS-CoV-2-convalescent patients experience symptoms months after acute infection and have not returned to their initial health state prior to infection, raising the suspicion of post-COVID-19 syndrome [1, 2]. These long-term health consequences have a significant impact on the physical, cognitive, mental and social health status, even in patients with mild disease presentation [3–5], suggesting a long-term functional decline in COVID-19 survivors.
The continued assessment of patients with post-COVID-19 syndrome has become a major task, with the aim of defining and mitigating the socioeconomic and medical long-term effects of COVID-19. In the absence of a specific instrument to assess the degree of impairment in the functional status of individuals who have had COVID-19, various physical and psychological dimensions have been initially examined. Among the tests used for this purpose were the 6-minute walk test, the 1-minute sit-to-stand test and the Barthel Index, as well as scales used to measure dyspnoea, anxiety, depression and quality of life [6–10]. An optimal instrument is needed to assess the functional status/degree of disability post-COVID-19 and to determine the impact of the disease on daily activities and lifestyle changes. For this reason, Klok et al. [11] developed the Post-COVID-19 Functional Status (PCFS) scale, an ordinal 6-grade scale ranging from 0 (“no functional limitations”) to 5 (“death”), which is used as a patient-reported outcome measure to evaluate the consequences of COVID-19 on functional status. The PCFS scale has been validated for Dutch and Belgian populations, showing adequate construct validity [12]. The linguistic validation and cross-cultural adaptation of the Spanish version was performed previously in the Chilean population, showing conceptual and linguistic equivalence to the original instrument [13]; however, a construct validity analysis has not been performed.
The COVID-19 pandemic has forced healthcare providers to be adaptive and innovative in their patient care, to maintain adequate social distancing and to reduce the risk of disease transmission. Among the measures taken, hospital visits have been limited to a minimum, and telematic consultations have increased, becoming alternatives to conventional appointments [14]. It is therefore relevant to evaluate the metric properties of a self-administered web-based form of the Spanish version of the PCFS scale to assess functional status and to monitor a patient’s progression, without requiring their attendance.
The aim of this study was to assess the psychometric characteristics of the Spanish version of the PCFS scale (web-based and paper-based forms) and evaluate the test-retest reliability of the web-based form.
Materials and methods
Study design, settings and participants
This cross-sectional survey study was conducted in Madrid (Spain) between February 16 and April 16, 2021. The study was approved by the local ethics committee of Clínico San Carlos Hospital (Madrid) (21/039-E) and was conducted in accordance with the Declaration of Helsinki. The fundamental rights of the participants were guaranteed, ensuring their well-being at all times, as well as being fully informed of their rights (e.g., the right to self-determination and the right to make informed decisions about their participation in the study, both at the beginning and during the course of the study). Written informed consent was obtained from all the participants. The sample was recruited from 2 healthcare centres in the Community of Madrid (Spain) area using a consecutive sampling by convenience method. The sample size was established based on the criteria of several authors, who have proposed 100 observations as the minimum required, as well as according to the proposed rule of a ratio of 20 observations per measured variable [15–17]. Thus, a necessary minimum of at least 100 participants were determined to analyze the construct validity of the scale. To capture the real-life scenario of COVID-19 survivors, the only selection criteria were as follows: confirmed SARS-CoV-2 infection (positive reverse-transcription–polymerase chain reaction [RT-PCR] test from a nasopharyngeal or oropharyngeal swab or serological tests positive for SARS-CoV-2 antibodies), persistence of symptoms for at least 30 days from the date of symptom onset, at least 18 years of age, able to speak Spanish and having internet access at home. Participants with pre-existing disability were excluded from the study. Written informed consent was obtained from all participants.
Study procedures
The first evaluation required the attendance of the participants at the centre where the research was carried out. All the self-reported questionnaires in paper-based form were fulfilled during the interview (face-to-face), whereas the PCFS scale in web-based form was completed online. All participants answered both the paper- and web-based forms of the Spanish version of the PCFS scale on the same day with a minimum of 2 hours and with distracting activities between the administration of the 2 scales. The web-based form of the PCFS scale was an exact copy of the paper version and was placed on a secured Internet page. Participants could answer the questions by clicking the appropriate box; after finishing the scale, the results were submitted. The rest of the self-reported questionnaires were randomised and were completed during the interview. To assess the test-retest reliability of the web-based form, an invitation to complete the scale a second time was sent to the participants by e-mail 7 days after the first evaluation.
Outcomes
A physician collected data by interviewing the participants, including sociodemographic characteristics (sex, age, height, weight, educational level, occupation, marital and smoking status, date of symptoms onset and type of COVID-19 diagnosis), comorbidities and clinical characteristics (dyspnoea, fatigue/muscle weakness).
Functional status.
The consequences of COVID-19 on functional status were evaluated with the PCFS [11], an ordinal 6-grade scale: grade 0 (no functional limitations); grade 1 (negligible functional limitations); grade 2 (slight functional limitations); grade 3 (moderate functional limitations); grade 4 (severe functional limitations); and grade 5 (death). The scale was applied through a structured interview by a physician previously trained to validate the paper-based form and was self-administered by the participants to validate the web-based form. The final rating on the scale is the poorest functional status indicated by the participants’ answers.
Health-related quality of life.
To assess the participant’s quality of life, we employed the European Quality of Life-5 Dimensions questionnaire (EQ-5D-5L) [18], which consists of 5 dimensions (1, mobility; 2, self-care; 3, usual activities; 4, pain/discomfort; 5, anxiety/depression) with 5 response options based on severity level, ranging from 1 to 5. Based on these 5-dimension codes, an index score is provided, ranging from 0 (death) to 1 (full health). Additionally, the participants had to rate their current overall health on a visual analogue scale ranging from 0 (worst imaginable health) to 100 (best imaginable health).
Activities of daily living limitations.
Limitations in activities of daily living were evaluated by the Barthel Index [19] and Global Activity Limitation Index (GALI) [20]. The Barthel Index is an ordinal scale comprising 10 activities of daily living (feeding, grooming, bathing, dressing, bowel and bladder care, toilet use, ambulation, transfers and stair climbing) and is scored in steps of 5 points, resulting in a maximum total score of 100, with higher scores indicating more independence. The GALI is self-reported and asks “For the past 6 months at least, to what extent have you been limited because of a health problem in activities people usually do?”, to which there are 3 possible responses: not limited at all, limited but not severely and severely limited.
Anxiety and depression levels.
The Spanish version of the Hospital Anxiety and Depression Scale (HADS) [21], which consists of 14 items divided into 2 subscales for anxiety and depression, was used to assess the patient’s level of anxiety and depression. The subscales include 7 items each, with scores ranging from 0 to 21 for each item.
Data analysis
The data were analysed using SPSS v.25 software (SPSS Inc., Chicago, IL, USA). The significance level was set at P < 0.05. Continuous variables are presented as mean ± standard deviation or median and interquartile range, as appropriate. Categorical variables are presented as absolute and relative frequencies.
The percent agreement and kappa coefficients [22] were employed to estimate the concordance between the web-based and paper-based forms of the Spanish version of the PCFS scale, as well as the test-retest concordance after 7 days for the web-based form. Kappa values were interpreted according to Landis and Koch’s criteria [23]: poor agreement (<0.2), fair agreement (0.21–0.40), moderate agreement (0.41–0.60), substantial agreement (0.61–0.80), and almost perfect agreement (>0.80).
Construct validity was assessed by analysing the correlations between the scores of both PCFS scale forms with those obtained for quality of life (EQ-5D-5L), limitations in activities of daily living (GALI and Barthel Index) and psychological status (HADS). Spearman’s Rho correlation coefficient was used to determine the correlations due to the categorical nature of certain variables and the fact that the continuous variables did not have a normal distribution. A low correlation was considered <0.30, a moderate correlation 0.30–0.60, and a strong correlation >0.60 [24]. In addition, the discriminant ability of the Spanish version of the PCFS scale was assessed by comparing the scores obtained for quality of life, limitations of activities of daily living and psychological status between the various functional status grades. The functional status factor was analysed by employing the Kruskal–Wallis test, and multiple comparisons were performed with the Mann–Whitney U test.
Results
The total sample consisted of 125 participants (71 women and 54 men; 46.46 ± 14.18 years; 25.50 ± 4.5 kg/m2) who had a confirmed diagnosis of COVID-19 by RT-PCR. In addition, 16 (12.8%) participants required hospitalisation during the COVID-19 infection (5 of whom required Intensive Care Unit admission), and only 5 (4%) had a comorbidity. Most participants had a grade 3 functional status according to the PCFS scale, regardless of the form used (web-based form, 34.4%; paper-based form, 35.2%). Table 1 shows the participants’ sociodemographic and clinical characteristics. It is important to note that 4 participants did not complete the web-based form of the Spanish PCFS scale within 7 days of the first assessment. Hence, the test-retest agreement was analysed with a sample of 121 participants.
Agreement between the two forms and test-retest concordance for the web-based form
The percentage of agreement for the Spanish version of the PCFS scale rating between the web-based and paper-based forms was almost perfect when the scale was completed on the same day (both completions spaced approximately 2 hours apart), showing 88% agreement and a Kappa coefficient of 0.84 (0.76–0.92). Therefore, the two forms showed almost identical results and could apparently be applied interchangeably.
Regarding the stability of the score obtained again in the web-based form at 7 days (test-retest), the percentage of substantial agreement for the functional status grade was 72% (Kappa = 0.63 [0.52 to 0.74]). Thus, the stability over time of the web-based form of the PCFS scale was adequate.
Construct validity
The convergent validity of the PCFS scale was adequate, given that the correlational analysis showed a statistically significant relationship between the participant’s functional status and their quality of life, limitations in activities of daily living and psychological status (Table 2). Specifically, an increase in functional status was associated with an improvement in quality of life and psychological status, as well as less limitation in activities of daily living. All correlations between the Spanish PCFS scale, regardless of the form used, were of moderate-large magnitude, with the exception of those observed with the Barthel Index, which were low (Rho < .30). Functional status showed a particularly high correlation with quality of life (EQ-5D-5L; inverse correlation) and with limitation of usual activities of daily living (GALI; direct correlation) (Fig 1), with an absolute Rho value ≥ .743 in all cases.
Abbreviatures: GALI, Global Activity Limitation Indicator; PCFS, Post-COVID-19 Functional Status.
The Kruskal–Wallis test showed statistically significant differences in quality of life, limitation in activities of daily living and psychological status according to the functional status (P < .001). Table 3 shows the multiple comparisons between participants with differing degrees of functional impairment according to the PCFS scale. Regardless of the form of the PCFS scale used (web-based or paper-based versions), statistically significant differences in the EQ-5D-5L and GALI were observed between almost all grades of functional status, except in the comparison between grades 0 and 1. These differences showed a deterioration in quality of life and an increase in limitations in activities of daily living as the degree of functional impairment increased. Only those participants classified as grade 4 on the PCFS scale showed a statistically significant deterioration with respect to the other participants, with lower functional status grades for performing of activities of daily living as assessed by the Barthel Index. In addition, the participants with a higher degree of functional impairment were more psychologically distressed, with statistically significant differences in the levels of anxiety and depression between functional status grades 3–4 and the other grades.
Discussion
The present study assessed the psychometric characteristics of the Spanish version of the PCFS scale (web-based and paper-based forms), and the test-retest reliability of the web-based form. The results show for the first time the psychometric characteristics of the Spanish version of the PCFS scale, for both the web-based and paper-based forms, and the agreement between these 2 forms. Both forms showed adequate construct validity, and the web-based form provided substantial test-retest reliability. Scores derived through the 2 forms of the scale showed a high level of agreement between them, which suggests there was minimal within-subject variance between the 2 modes of administration. Thus, the paper and electronic format of the Spanish version of the PCFS scale can be used interchangeably in clinical practice for determining the functional status after COVID-19 in the Spanish-speaking population.
The percentage of agreement for the PCFS scale rating between the web-based and paper-based forms was high (88%). Prior published studies have validated an electronic or web-based form of a paper-based questionnaire, showing uniform psychometric properties between administration modes (pain assessments [25, 26] and respiratory symptoms [27, 28]). The COVID-19 pandemic has led to an unprecedented situation in which there has been a major push to implement telematic consultations, emphasising the need for assessment tools such as this web-based form, which offers the possibility to assess patients’ functional status without needing outpatient visits. The use of electronic data capture for health assessment offers data management capabilities, improved adaptability, enhanced ergonomics and appeal [29], as well as reducing the number of spoiled responses without altering the results [30].
Test-retest analyses for the online administration of the Spanish version of the PCFS scale demonstrated that data gathered with this method are stable over time with no notable changes in functional status. There was high concordance between the responses to the 2 testing modes, with a slightly lower decrease in concordance for the first day, as has been reported in other tests, such as the 6-minute walk test [31]. This result means that a longer time span increased the risk of a change in functional status control; thus, the time factor can determine the progression of functional status. This outcome highlights the scale’s sensitivity in detecting changes over time, as can be observed in the discriminant validity results.
The construct validity of the Spanish version of the PCFS scale, for both the web-based and paper-based forms, was adequate and similar to the original version [12]. All correlations between the scale and other validated tools, regardless of the form used (web-based or paper-based), were of moderate-large magnitude. Notably, the strongest associations were with the EQ-5D-5L and GALI scores, which is in line with a previous study by Machado et al. [12], who showed strong correlations with the “usual activities” domain of the EQ‐5D-5L questionnaire. A key element in explaining these results is that the PCFS scale [11] evaluates the same dimensions of functional limitations, including changes in lifestyle, sports and social activities, which are included in GALI [20], as well as frequent symptoms after COVID-19, such as pain, fatigue, anxiety and depression, which can reduce quality of life [32]. In fact, these associations were to be expected due to the fact that the course of the infection is mild or asymptomatic in approximately 80%–90% of cases [33]. Therefore, the survivors will perceive earlier functional limitations in their usual activities, such as employment and social activities, making this scale the most specific and comprehensive assessment for this condition.
We also found a moderate correlation between the PCFS scale and the psychological status listed in the HADS, which could partially be explained by the fact that it has been reported that psychological status is associated with functional impairment in patients with COVID-19 [34]. Moreover, the PCFS scale specifically inquiries about psychological factors; however, there are many other factors that can affect the functional status that could explain this moderate correlation. In contrast, the Barthel Index showed a significant but low correlation. Only those participants classified as grade 4 on the PCFS scale showed limitations in performing activities of daily living assessed by the Barthel Index. In fact, only 1 participant showed a total score <100 for the Barthel Index. These results are in line with those of other studies that showed low Barthel indices related to poorer functional ability after COVID-19 [6, 35, 36]. However, the low correlation observed suggests that the PCFS scale covers the entire range of severity for the functional status, allowing a precise discrimination of the dimensions affected in these patients, whereas the Barthel Index only detects functional limitations in activities of daily living in severely impaired individuals (ceiling effect). This conclusion disagrees with Pizarro-Pennaroll et al. [37] who recommend the use of the Barthel Index for assessing activities of daily living in patients post-COVID-19. One reason that could explain these contradictory results was that our sample consisted of nonhospitalised COVID-19 survivors with mild disease presentation.
The discriminant ability of the Spanish version of the PCFS scale has been demonstrated, given that the participants classified by the scale with differing grades of functionality also showed differences in quality of life, performance of activities of daily living and psychological status. These results are similar to those reported for the original version [11, 12]. According to our findings, the discriminant validity was particularly reinforced by the differences observed between the various grades of functionality established by the PCFS scale and the quality of life and activities of daily living evaluated by the EQ-5D-5L and GALI, respectively. These results are supported by a recent review that suggested that the EQ-5D-5L is an optimal tool for detecting differences related to COVID-19 severity [37]. In addition, the distress grade was higher in those participants with grade ≥ 3 measured by the PCFS scale. A possible explanation is that severe limitations in performing activities of daily living, such as employment, leisure activities and/or personal hygiene, logically affect the individual’s emotional state.
The first limitation of the current study is the limited external validity of the Spanish version of the PCFS scale given that most of the participants were nonhospitalised COVID-19 survivors. However, this version might be a reliable representation of the current population of nonhospitalised COVID-19 survivors with moderate long-term functional limitations. Future studies are needed to investigate the distribution of the scale grades in selected cohorts of individuals, such as hospitalised or post-discharge patients with COVID-19, to demonstrate associations between functional status and response to therapy and linking functional status measures to patient preferences and utilities. Another limitation lies in the potential difficulties with web-based monitoring in less literate or technologically sophisticated populations, which warrants further research in these populations.
The Spanish version of the PCFS scale is a validated tool to assess the degree of impairment in functional status in the Spanish population who have had COVID-19, and to generate new therapeutic approaches in the rehabilitation programs, guide post-COVID-19 care, monitor the recovery process and assess functional sequelae, all of which could help in making treatment decisions on the multidisciplinary interventions aimed at improving the psychological state, which will presumably lead to an increase in quality of life. With an increasing number of primary care practitioners and specialists using electronic resources due to the COVID-19 pandemic, the need to have validated electronic forms of scales is becoming paramount. This electronic form of the scale provided results comparable to the paper form; thus, its utilisation can aid healthcare practitioners collecting and analysing the consequences of COVID-19 on functional status for long-term management without the need for outpatient visits.
Conclusion
This study provides evidence of the psychometric characteristics of the Spanish version of the PCFS scale, both the web-based and paper-based forms, in COVID-19 survivors. Both versions showed adequate construct validity, and the web-based form provided substantial test-retest reliability. The comparability of a web-based form with the existing paper-based form allows the use of the web-based form in clinical practice. The Spanish version of the PCFS scale is an adequate tool for determining the degree of functional status after COVID-19 in the Spanish-speaking population.
References
- 1. Funke-Chambour M, Feldmeyer L, Hoepner R, Huynh-Do U, Maurer B, Rexhaj E, et al. The Long-COVID Syndrome—a New Clinical Picture after COVID-19 Infection. Praxis (Bern 1994). 2021;110: 377–382. pmid:34019446
- 2. Salamanna F, Veronesi F, Martini L, Landini MP, Fini M. Post-COVID-19 Syndrome: The Persistent Symptoms at the Post-viral Stage of the Disease. A Systematic Review of the Current Data. Front Med. 2021;8: 653516. pmid:34017846
- 3. Augustin M, Schommers P, Stecher M, Dewald F, Gieselmann L, Gruell H, et al. Post-COVID syndrome in non-hospitalised patients with COVID-19: a longitudinal prospective cohort study. Lancet Reg Heal—Eur. 2021;6: 100122. pmid:34027514
- 4. Huang C, Huang L, Wang Y, Li X, Ren L, Gu X, et al. 6-month consequences of COVID-19 in patients discharged from hospital: a cohort study. Lancet. 2021;397: 220–232. pmid:33428867
- 5. Walle-Hansen MM, Ranhoff AH, Mellingsæter M, Wang-Hansen MS, Myrstad M. Health-related quality of life, functional decline, and long-term mortality in older patients following hospitalisation due to COVID-19. BMC Geriatr. 2021;21. pmid:33752614
- 6. Curci C, Pisano F, Bonacci E, Camozzi DM, Ceravolo C, Bergonzi R, et al. Early rehabilitation in post-acute COVID-19 patients: Data from an Italian COVID-19 Rehabilitation Unit and proposal of a treatment protocol. Eur J Phys Rehabil Med. 2020;56: 633–641. pmid:32667150
- 7. Qu G, Zhen Q, Wang W, Fan S, Wu Q, Zhang C, et al. Health-related quality of life of COVID-19 patients after discharge: A multicenter follow-up study. J Clin Nurs. 2021. pmid:33656210
- 8. Raman B, Cassar MP, Tunnicliffe EM, Filippini N, Griffanti L, Alfaro-Almagro F, et al. Medium-term effects of SARS-CoV-2 infection on multiple vital organs, exercise capacity, cognition, quality of life and mental health, post-hospital discharge. EClinicalMedicine. 2021;31. pmid:33490928
- 9. Martin I, Braem F, Baudet L, Poncin W, Fizaine S, Aboubakar F, et al. Follow-up of functional exercise capacity in patients with COVID-19: It is improved by telerehabilitation. Respir Med. 2021;183. pmid:33964817
- 10. Frontera JA, Yang D, Lewis A, Patel P, Medicherla C, Arena V, et al. A prospective study of long-term outcomes among hospitalized COVID-19 patients with and without neurological complications. J Neurol Sci. 2021;426. pmid:34000678
- 11. Klok FA, Boon GJAM, Barco S, Endres M, Miranda Geelhoed JJ, Knauss S, et al. The post-COVID-19 functional status scale: A tool to measure functional status over time after COVID-19. Eur Respir J. 2020;56. pmid:32398306
- 12. Machado FVC, Meys R, Delbressine JM, Vaes AW, Goërtz YMJ, van Herck M, et al. Construct validity of the Post-COVID-19 Functional Status Scale in adult subjects with COVID-19. Health Qual Life Outcomes. 2021;19. pmid:33536042
- 13. Lorca LA, Torres-Castro R, Ribeiro IL, Benavente P, Pizarro M, San Cristobal B, et al. Linguistic Validation and Cross-Cultural Adaptation of the Post-COVID-19 Functional Status Scale for the Chilean Population. Am J Phys Med Rehabil. 2021;100: 313–320. pmid:33496442
- 14. Martínez-García M, Bal-Alvarado M, Santos Guerra F, Ares-Rico R, Suárez-Gil R, Rodríguez-Álvarez A, et al. Monitoring of COVID-19 patients by telemedicine with telemonitoring. Rev Clin Esp. 2020;220: 472–479. pmid:33994572
- 15. de Winter JCF, Dodou D, Wieringa PA. Exploratory Factor Analysis With Small Sample Sizes. Multivariate Behav Res. 2009;44: 147–181. pmid:26754265
- 16.
Hair JF, Anderson RE, Tatham RL, Black WC. Análisis Multivariante. 5th ed. Madrid, Spain: Pearson; 2004.
- 17. Kyriazos TA. Applied psychometrics: sample size and sample power considerations in factor analysis (EFA, CFA) and SEM in General. Psychology. 2018;9: 2207–2230.
- 18. Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D, et al. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res. 2011;20: 1727–1736. pmid:21479777
- 19. González N, Bilbao A, Forjaz MJ, Ayala A, Orive M, Garcia-Gutierrez S, et al. Psychometric characteristics of the Spanish version of the Barthel Index. Aging Clin Exp Res. 2018;30: 489–497. pmid:28770477
- 20. Van Oyen H, Bogaert P, Yokota RTC, Berger N. Measuring disability: A systematic review of the validity and reliability of the Global Activity Limitations Indicator (GALI). Archives of Public Health. BioMed Central Ltd.; 2018. pmid:29881544
- 21. Herrero MJ, Blanch J, Peri JM, De Pablo J, Pintor L, Bulbena A. A validation study of the hospital anxiety and depression scale (HADS) in a Spanish population. Gen Hosp Psychiatry. 2003;25: 277–283. pmid:12850660
- 22. Cohen J. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull. 1968;70: 213–220. pmid:19673146
- 23. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33: 159–74. pmid:843571
- 24.
Carter R, Lubinsky J, Domholdt E. Rehabilitation research: principles and applications. 4th Ed. St. Louis, Missouri: Elsevier Saunders; 2011.
- 25. Cook AJ, Roberts DA, Henderson MD, Van Winkle LC, Chastain DC, Hamill-Ruth RJ. Electronic pain questionnaires: A randomized, crossover comparison with paper questionnaires for chronic pain assessment. Pain. 2004;110: 310–317. pmid:15275781
- 26. Junker U, Freynhagen R, Längler K, Gockel U, Schmidt U, Tölle TR, et al. Paper versus electronic rating scales for pain assessment: A prospective, randomised, cross-over validation study with 200 chronic pain patients. Curr Med Res Opin. 2008;24: 1797–1806. pmid:18485269
- 27. Raat H, Mangunkusumo RT, Mohangoo AD, Juniper EF, Van Der Lei J. Internet and written respiratory questionnaires yield equivalent results for adolescents. Pediatr Pulmonol. 2007;42: 357–361. pmid:17335011
- 28. Juniper EF, Langlands JM, Juniper BA. Patients may respond differently to paper and electronic versions of the same questionnaires. Respir Med. 2009;103: 932–934. pmid:19028085
- 29. Vanhaelen Q. Web-based Tools for Drug Repurposing: Successful Examples of Collaborative Research. Curr Med Chem. 2020;28: 181–195. pmid:32003659
- 30. Caro JJ, Caro I, Caro J, Wouters F, Juniper EF. Does electronic implementation of questionnaires used in asthma alter responses compared to paper implementation? Qual Life Res. 2001;10: 683–691. pmid:11871589
- 31. Hansen H, Beyer N, Frølich A, Godtfredsen N, Bieler T. Intra-and inter-rater reproducibility of the 6-minute walk test and the 30-second sit-to-stand test in patients with severe and very severe COPD. Int J COPD. 2018;13: 3447–3457. pmid:30425474
- 32. Shanbehzadeh S, Tavahomi M, Zanjari N, Ebrahimi-Takamjani I, Amiri-arimi S. Physical and mental health complications post-COVID-19: Scoping review. J Psychosom Res. 2021;147: 110525. pmid:34051516
- 33. Pascarella G, Strumia A, Piliego C, Bruno F, Del Buono R, Costa F, et al. COVID-19 diagnosis and management: a comprehensive review. J Intern Med. 2020;288: 192–206. pmid:32348588
- 34. Mcloughlin BC, Miles A, Webb TE, Knopp P, Eyres C, Fabbri A, et al. Functional and cognitive outcomes after COVID-19 delirium. Eur Geriatr Med. 2020;11: 857–862. pmid:32666303
- 35. Belli S, Balbi B, Prince I, Cattaneo D, Masocco F, Zaccaria S, et al. Low physical functioning and impaired performance of activities of daily life in COVID-19 patients who survived hospitalisation. Eur Respir J. 2020;56. pmid:32764112
- 36. Sakai T, Hoshino C, Yamaguchi R, Hirao M, Nakahara R, Okawa A. Remote rehabilitation for patients with covid-19. J Rehabil Med. 2020;52. pmid:32871014
- 37. Pizarro-Pennarolli C, Sánchez-Rojas C, Torres-Castro R, Vera-Uribe R, Sanchez-Ramirez DC, Vasconcello-Castillo L, et al. Assessment of activities of daily living in patients post COVID-19: A systematic review. PeerJ. 2021;9: e11026. pmid:33868804