https://orcid.org/0000-0003-0887-7627
Figures
Abstract
Background
Patient-reported outcome measures (PROMs) are commonly used to evaluate surgical outcome in patients undergoing joint replacement surgery, however routine collection from the target population is often incomplete. Representative samples are required to allow inference from the sample to the population. Although higher capture rates are desired, the extent to which this improves the representativeness of the sample is not known. We aimed to measure the representativeness of data collected using an electronic PROMs capture system with or without telephone call follow up, and any differences in PROMS reporting between electronic and telephone call follow up.
Methods
Data from a pilot PROMs program within a large national joint replacement registry were examined. Telephone call follow up was used for people that failed to respond electronically. Data were collected pre-operatively and at 6 months post-operatively. Responding groups (either electronic only or electronic plus telephone call follow up) were compared to non-responders based on patient characteristics (joint replaced, bilaterality, age, sex, American Society of Anesthesiologist (ASA) score and Body Mass Index (BMI)) using chi squared test or ANOVA, and PROMs for the two responder groups were compared using generalised linear models adjusted for age and sex. The analysis was restricted to those undergoing primary elective hip, knee or shoulder replacement for osteoarthritis.
Results
Pre-operatively, 73.2% of patients responded electronically and telephone follow-up of non-responders increased this to 91.4%. Pre-operatively, patients responding electronically, compared to all others, were on average younger, more likely to be female, and healthier (lower ASA score). Similar differences were found when telephone follow up was included in the responding group. There were little (if any) differences in the post-operative comparisons, where electronic responders were on average one year younger and were more likely to have a lower ASA score compared to those not responding electronically, but there was no significant difference in sex or BMI. PROMs were similar between those reporting electronically and those reporting by telephone.
Conclusion
Patients undergoing total joint replacement who provide direct electronic PROMs data are younger, healthier and more likely to be female than non-responders, but these differences are small, particularly for post-operative data collection. The addition of telephone call follow up to electronic contact does not provide a more representative sample. Electronic-only follow up of patients undergoing joint replacement provides a satisfactory representation of the population invited to participate.
Citation: Harris IA, Cashman K, Lorimer M, Peng Y, Ackerman I, Heath E, et al. (2021) Are responders to patient health surveys representative of those invited to participate? An analysis of the Patient-Reported Outcome Measures Pilot from the Australian Orthopaedic Association National Joint Replacement Registry. PLoS ONE 16(7): e0254196. https://doi.org/10.1371/journal.pone.0254196
Editor: Mathieu F. Janssen, Erasmus MC Desiderius School, NETHERLANDS
Received: October 26, 2020; Accepted: June 22, 2021; Published: July 2, 2021
Copyright: © 2021 Harris et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because of legislative restrictions on the use of registry data from the Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR). Data are available from the AOANJRR for those who meet the criteria for access. External access to and use of de-identified AOANJRR data is permitted but must be in accordance with AOANJRR policies (Ref No POL.S3.3, S3.4, S3.5) available on the registry website: https://aoanjrr.sahmri.com/policies. Requests for data use can be made by contacting the AOANJRR Manager: Cindy Turner Manager AOANJRR Telephone: +618 8128 4284 Email: cturner@aoanjrr.org.au.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Patient reported outcome measures (PROMs) of health status are commonly recorded pre- and post-operatively in people undergoing joint replacement surgery as a measure of surgical thresholds and treatment effects for these common and resource intensive procedures. However, unlike registries that commonly have near-complete coverage of all procedures, PROMs collection is rarely complete, being limited by resources and patient responsiveness. Registry-based PROMs collection in joint replacement surgery has coverage rates rarely higher than 80%, and often less than 50% [1,2]. A 60% threshold has been suggested for completeness in PROMs collection [3], but with any threshold, it is important to know the representativeness of the sample so that conclusions based on the sample can be applied to the population. We consider it more important to understand the representativeness of a sample than the size of the sample or completeness. For example, data from a 50% sample may be considered meaningful if the differences between the sample and the population are understood, whereas an 80% sample may provide misleading information if the sample is unrepresentative.
The Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR) piloted a PROMs program, targeting all patients undergoing elective hip, knee or shoulder replacement from participating institutions. Due to the inefficiencies associated with using paper forms (either directly or by mail) within a national registry, the AOANJRR PROMs program uses direct electronic data capture. Data capture by telephone call using an interviewer who directly entered data electronically was also used for the pilot stage of the program, but only for patients who did not respond to direct electronic data entry following text message and email prompts for completion.
This study aims to answer the following questions: 1) are patients for whom PROMs data were captured directly (electronically) different to those who did not provide the data electronically (i.e. all others); 2) does adding telephone call follow up to those responding electronically improve representativeness; and 3) are patient-reported outcomes different between those who provide direct electronic data entry and those who provide information via telephone call after not responding electronically?
Methods
Between 30 July 2018 and 28 January 2020, the AOANJRR conducted the first stage of a PROMs pilot study, collecting PROMs data from patients pre-operatively and at six months post-surgery from 43 institutions across Australia, including metropolitan and regional, and private and public hospitals from all states and one territory. The analysis includes procedures registered (pre-op analysis) or performed (post-op analysis) between 30 July 2018 and 29 May 2019, to allow 8 months for follow up. The study was nested within the AOANJRR, a national registry that validates more than 97.8% of all joint replacement procedures for all hospitals (approximately 320) performing joint replacement surgery in Australia [4].
The following Australian ethics committees approved the pilot program from which these data were drawn: University of South Australia HREC (200890), Sydney Local Health District Ethics Review Committee (RPAH Zone, HREC/18/RPAH/90), Calvary Health Care Adelaide HREC (18-CHREC-F004), Mater Misericordiae Ltd HREC (HREC/18/MHS/45), St Vincent’s Health and Aged Care HREC (HREC 18/14), University of Tasmania HREC (H0017292), Calvary Health Care Tasmania HREC (010418), St John of God HREC (1408), Calvary Health Care (ACT)(25–2018). Consent was not obtained for the analyses used in this report as data were analyzed anonymously. Researchers accessed data on 18 May 2020, after the data were anonymized.
Data collection involved initial (pre-operative) patient data capture at hospital pre-admission clinics or private surgeon clinics using electronic-only methods which had the capacity to be conducted on multiple devices including smart phone, tablet or computer. Patients unable to complete data entry at initial contact were registered in the system and contacted electronically via email or text message two and five days after initial registration to allow direct data entry at their convenience. Post-operative collection involved direct electronic contact via email or text message links directing the patient to the online survey, which were sent 166 and 180 days after the patients’ procedure date. Non-responders or those without any electronic contacts including those with only home telephone numbers (both pre-operative and post-operative) were contacted by telephone and asked to complete the survey by telephone.
The analysis was restricted to primary elective procedures undertaken for osteoarthritis. The denominator used for the main analyses (the reference population) was all patients who were registered for the PROMs pilot program that were matched to routine AOANJRR data, because many patients who were registered may not have proceeded to surgery within the study period. Using known procedures and registered patients tests the ‘within-system’ representativeness by restricting the analysis to patients who were given the opportunity to respond. The comparison of telephone to electronic follow up was further restricted to those who had an email or mobile telephone number, again to compare the responsiveness in similar patient populations.
Information on the population included data routinely collected by the AOANJRR; age, gender, Body Mass Index (BMI), American Society of Anesthesiologists (ASA) physical status classification [5], unilateral/bilateral and approach (for hip replacement). Other demographic information (e.g., education and ethnicity) was not available for analysis. PROMs data include the Oxford Hip Score [6] (OHS), Oxford Knee Score [7] (OKS), EQ-5D-5L [8] Utility Index (using Australian preference weights) and Visual Analogue Scale (VAS), low back pain, affected joint pain, expected (post-operative) pain and function, the Hip injury and Osteoarthritis Outcome Score, 12 item [9] (HOOS-12) and the Knee injury and Osteoarthritis Outcome Score, 12 item [10] (KOOS-12), the latter two scores providing a summary score and domain scores for Pain, Function and Quality of Life.
Categorical data (proportions) were compared using chi squared tests. Continuous demographic data were compared using analysis of variance. Differences in pre-and post-operative PROMs responses between groups were compared using general linear models adjusting for age and sex. The critical value chosen to reject the alternative hypothesis was 0.05.
Results
Over the study period, 6,224 patients at 43 participating institutions having electronic contact information and primary THA, TKA or TSA for osteoarthritis were registered into the PROMs pilot study.
Of the 6,224 patients registered into the PROMs system, 417 patients had their pre-operative PROMs data collected directly by separate hospital systems, external to the electronic data capture system, and were excluded from pre-op analyses. A further 48 patients were excluded from pre-op analyses due to pre-op follow up phone calls ceasing. For post-op analyses, of the 6,224 patients registered, 24 died and 55 opted out prior to completing their post-op PROMs and were excluded from post-op analyses. A further 573 patients had their procedure after 29 May 2019 and were unable to be followed up by telephone due to post-op follow up phone calls ceasing and are excluded from post-op analyses.
For pre-op analyses, a total of 5,759 patients with a mobile phone and/or email address listed were matched to 6,095 primary hip, knee or shoulder replacement procedures performed for osteoarthritis over the same period and are included in the pre-op analyses. For post-op analyses, a total of 5,572 patients were matched to 5,892 primary hip, knee or shoulder replacement procedures performed for osteoarthritis and are included in the post-op analyses. For patients with multiple procedures, only the first procedure was included in the demographic analyses.
1. Electronic responders versus electronic non-responders
Of the 5,759 registered patients, 4,213 (73.2%) completed pre-operative PROMs data electronically, the remainder (“electronic non-responders”) were either followed up by telephone or not followed.
A comparison of those responding to pre-operative electronic data collection to electronic non-responders is provided in Table 1. There was no difference in the type of joint replacement between the groups, however, patients having bilateral procedures were more likely to respond. On average, responders were one year younger and more likely to be female. The average difference in BMI between responders and non-responders (0.3 kg/m2) was small.
The post-operative comparisons are provided in Table 1 and show that responders were, on average, one year younger and more likely to be healthy (lower ASA class) but there was no significant difference for BMI, sex or joint type.
2. All responders (electronic plus telephone) versus non-responders
The addition of telephone call follow up for patients not responding (or not able to respond) to electronic data entry increased pre-operative data completeness from 73.2% to 91.4%. The differences between all responders (electronic and telephone) and non-responders are provided in Table 2. There were no significant between group differences in the pre-operative or post-operative characteristics of all responders compared to non-responders.
Characteristics of electronic-only responders, all responders (electronic and telephone) and all patients are provided in Table 3 to quantify the difference in representativeness when telephone responders are added to electronic-only responders (compared to the total group). Statistical tests are not provided as nearly all differences were statistically significant due to the large sample size. There is no more than 1% or one unit (for age and BMI) difference between electronic-only and all responders (electronic plus telephone), except for post-operative ASA class where the addition of telephone follow up increased the response of ASA class 3 patients.
3. Patient reported outcomes in those responding electronically versus by telephone
Pre-operative and post-operative patient-reported outcomes for total hip arthroplasty (THA) patients responding electronically versus telephone follow up are provided in Table 4A and 4B. No clinically meaningful or statistically significant differences were identified for the pre-operative PROMs, and only a small difference in post-operative EQ-5D VAS scores was evident.
b: Post-operative patient-reported outcomes comparing patients responding directly (“electronic”) and those responding by telephone (“telephone”) for patients undergoing THA.
Comparisons between electronic and telephone responders for total knee arthroplasty (TKA) patients pre- and post-operatively are provided in Table 5A and 5B.
b: Post-operative patient-reported outcomes comparing patients responding directly (“electronic”) and those responding by telephone (“telephone”) for patients undergoing TKA.
Results for total shoulder arthroplasty (TSA) patients are not shown. For this group, there was one significant difference, whereby patients responding by telephone reported less post-operative joint pain than electronic responders (1.0 versus 3.1, mean difference 2.0, 95%CI 0.9–3.1).
Discussion
Our findings show that patients undergoing elective joint replacement surgery who are included in a PROMs program using direct, electronic data entry are, on average, younger, more likely to be female and have a lower ASA score than those that do not take part. However, the differences are small, particularly in the post-operative comparisons where there was a 1-year difference in age and a 1% difference in the distribution of sex. When telephone call follow up was added to the responder group, there were very little or no differences in the pre- and post-operative comparison to non-responders. The small differences seen may not be clinically important and statistical significance, where present, likely reflects the large sample size and corresponding statistical power. The representativeness of the samples did not change by more than 1% or one unit (year of age or unit of BMI) when telephone follow up was added to electronic follow up, except that telephone follow up detected more patients in a higher ASA class.
A previous study in elective surgery patients in England (including hip and knee arthroplasty) showed that responding patients were more likely to be female and older (which concurs with our findings) [11]. A recent study of patients included in a hip arthroscopy registry also reported that responding patients were more likely to be female and older, but they included a broader age spectrum (more younger people) [12]. Similar to our study, the differences between responders and non-responders in both these studies were small. A study of shoulder arthroplasty patients, however, showed that female patients were less likely to respond [13].
Comparing PROMs between electronic and telephone responders showed no significant difference for most outcomes. Pain outcomes, however, were often different between these groups, with patients responding electronically reporting higher expected pain than those responding by telephone. Post-operatively, however, while TKR patients responding electronically reported higher pain levels than those responding by telephone, there was no significant difference for THR patients. A similar study recently reported that patients undergoing hip surgery who responded electronically reported significantly but marginally more pain than in non-electronic responders, but no difference in knee or shoulder patients [14]. It should be noted that the differences in PROMs between electronic and telephone responders were small (all were less than clinically important thresholds) and any statistical significance likely reflects the large sample size that permitted the detection of small differences.
The differences in age and ASA score are likely due to lack of resources and individual capacity to respond electronically in older and less healthy patients. Interestingly, when younger orthopaedic patients have been included, they have been reported as having lower electronic response rates [14].
Given that the PROMs tools used in the current study were designed for direct patient entry, it is likely that any discrepancy between direct patient entry and telephone follow up is due to bias in the group responding by telephone, possibly due to features of the interviewer-patient interaction. Previous research has shown no difference between scores reported by telephone and those recorded by direct patient entry (paper based) for the EQ-5D survey or the Oxford hip or knee scores on patients undergoing joint replacement or other orthopaedic procedures [14–17]. Better EQ-5D scores have been reported for telephone administration compared to direct patient entry in other populations [18], but was not found in our study.
The addition of telephone call follow up for patients who did not respond electronically, increased data completeness but did not substantially alter the representativeness of the sample. Although higher completion rates have been considered desirable and minimum proportions have been suggested [3], the representativeness of PROMs samples is rarely reported in research publications. This has important practical and cost implications. Given the high per-patient cost of telephone follow up and annual volume of joint replacement surgery, this approach is unlikely to be cost-effective on a large scale to improve representativeness but may be used where high rates of follow up are required, e.g., in nested clinical trials. Since the AOANJRR PROMs pilot program, telephone follow up for patients not responding electronically has ceased as part of routine practice. It should be noted that these findings do not allow a direct comparison of primary telephone and electronic follow up, as the telephone follow up was only used for patients who did not or could not respond electronically. Rather, it compares electronic follow up to telephone follow up in people who did not respond electronically.
The strengths of this study are the large sample size and the broad range of institution sizes and locations. There may be differences in representativeness for variables not included in this analysis, for example, socioeconomic factors, ethnicity, education and language proficiency. Another limitation is that the study measured representativeness in patients registered in the PROMs system (i.e., it measured “within-system” representativeness) and did not test the representativeness of the system by looking at patients who were not registered. Therefore, the current study does not address the overall representativeness of the PROMs system and cannot make conclusions about the need to increase the overall coverage of such systems. The findings reported in this paper may not generalise to other clinical registries or jurisdictions.
Conclusion
Patients undergoing total joint replacement who provide direct electronic PROMs data are younger, healthier and more likely to be female than non-responders, but these differences are small, particularly for post-operative data collection. The addition of telephone call follow up for patients who do not respond electronically increases the response rate but only marginally improves the representativeness of the sample.
References
- 1. Tariq MB, Vega JF, Westermann R, Jones M, Spindler KP. Arthroplasty studies with greater than 1000 participants: analysis of follow-up methods. Arthroplast Today. 2019;5:243–250. pmid:31286051
- 2. Wilson I, Bohm E, Lübbeke A, Lyman S, Overgaard S, Rolfson O, et al. Orthopaedic registries with patient-reported outcome measures. EFORT Open Rev. 2019;4:357–367. pmid:31210973
- 3. Rolfson O, Bohm E, Franklin P, Lyman S, Denissen G, Dawson J, et al. Patient-reported outcome measures in arthroplasty registries Report of the Patient-Reported Outcome Measures Working Group of the International Society of Arthroplasty Registries Part II. Recommendations for selection, administration, and analysis. Acta Orthop. 2016;87 Suppl 1(Suppl 1):9–23.
- 4. Australian Orthopaedic Association National Joint Replacement Registry (AOANJRR). Hip, Knee & Shoulder Arthroplasty: 2019 Annual Report. Adelaide; 2019.
- 5.
ASA Physical Status Classification System: American Society of Anesthesiologists; Available from: https://www.asahq.org/standards-and-guidelines/asa-physical-status-classification-system. Accessed 30 September 2020.
- 6. Dawson J, Fitzpatrick R, Carr A, Murray D. Questionnaire on the perceptions of patients about total hip replacement. J Bone Joint Surg Br. 1996;78-B:185–190. pmid:8666621
- 7. Dawson J, Fitzpatrick R, Murray D, Carr A. Questionnaire on the perceptions of patients about total knee replacement. J Bone Joint Surg Br. 1998;80-B:63–69. pmid:9460955
- 8. Herdman M, Gudex C, Lloyd A, Janssen M, Kind P, Parkin D, et al. Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res. 2011;20:1727–1736. pmid:21479777
- 9. Gandek B, Roos EM, Franklin PD, Ware JE Jr. A 12-item short form of the Hip disability and Osteoarthritis Outcome Score (HOOS-12): tests of reliability, validity and responsiveness. Osteoarthritis Cartilage. 2019;27:754–761. pmid:30419279
- 10. Gandek B, Roos EM, Franklin PD, Ware JE Jr. A 12-item short form of the Knee injury and Osteoarthritis Outcome Score (KOOS-12): tests of reliability, validity and responsiveness. Osteoarthr Cartil. 2019;27(5):762–70. pmid:30716536
- 11. Hutchings A, Neuburger J, Grosse Frie K, Black N, van der Meulen J. Factors associated with non-response in routine use of patient reported outcome measures after elective surgery in England. Health Qual Life Outcomes. 2012;10:34. pmid:22462512
- 12. Poulsen E, Lund B, Roos E. The Danish Hip Arthroscopy Registry: Registration Completeness and Patient Characteristics Between Responders and Non-Responders. Clin Epidemiol. 2020;12:825–833. pmid:32801920
- 13. Ling DI, Finocchiaro A, Schneider B, Lai E, Dines J, Gulotta L. What Factors Are Associated with Patient-reported Outcome Measure Questionnaire Completion for an Electronic Shoulder Arthroplasty Registry? Clin Orthop Relat Res. 2021;479:142–147. pmid:32740479
- 14. OME Cleveland Clinic Orthopaedics. Value in Research: Achieving validated outcome measurements while mitigating follow-up cost. J Bone Joint Surg. 2020;102:419–427. pmid:31868824
- 15. Chatterji R, Naylor JM, Harris IA, Armstrong E, Davidson E, Ekmejian R, et al. An equivalence study: Are patient-completed and telephone interview equivalent modes of administration for the EuroQol survey? Health Qual Life Outcomes. 2017;15:18. pmid:28114993
- 16. Abdel Messih M, Naylor JM, Descallar J, Manickam A, Mittal R, Harris IA. Mail versus telephone administration of the oxford knee and hip scores. J Arthroplasty. 2014;29:491–494. pmid:24290739
- 17. Carpenter CVE, Blackburn J, Jackson J, Blom AW, Sayers A, Whitehouse MR. Validated repeatability of patient-reported outcome measures following primary total hip replacement: a mode of delivery comparison study with randomized sequencing. Acta Orthop. 2018;89:628–633. pmid:30451047
- 18. Hays RD, Kim S, Spritzer KL, Kaplan RM, Tally S, Feeny D, et al. Effects of mode and order of administration on generic health-related quality of life scores. Value Health. 2009;12:1035–1039. pmid:19473334