http://orcid.org/0000-0003-3150-6154
Figures
Abstract
Background
Disinvestment (removal, reduction, or reallocation) of routinely provided health services can be difficult when there is little published evidence examining whether the services are effective or not. Evidence is required to understand if removing these services produces outcomes that are inferior to keeping such services in place. However, organisational imperatives, such as budget cuts, may force healthcare providers to disinvest from these services before the required evidence becomes available. There are presently no experimental studies examining the effectiveness of allied health services (e.g., physical therapy, occupational therapy, and social work) provided on weekends across acute medical and surgical hospital wards, despite these services being routinely provided internationally. The aim of this study was to understand the impact of removing weekend allied health services from acute medical and surgical wards using a disinvestment-specific non-inferiority research design.
Methods and findings
We conducted 2 stepped-wedge cluster randomised controlled trials between 1 February 2014 and 30 April 2015 among patients on 12 acute medical or surgical hospital wards spread across 2 hospitals. The hospitals involved were 2 metropolitan teaching hospitals in Melbourne, Australia. Data from n = 14,834 patients were collected for inclusion in Trial 1, and n = 12,674 in Trial 2. Trial 1 was a disinvestment-specific non-inferiority stepped-wedge trial where the ‘current’ weekend allied health service was incrementally removed from participating wards each calendar month, in a random order, while Trial 2 used a conventional non-inferiority stepped-wedge design, where a ‘newly developed’ service was incrementally reinstated on the same wards as in Trial 1. Primary outcome measures were patient length of stay (proportion staying longer than expected and mean length of stay), the proportion of patients experiencing any adverse event, and the proportion with an unplanned readmission within 28 days of discharge. The ‘no weekend allied health service’ condition was considered to be not inferior if the 95% CIs of the differences between this condition and the condition with weekend allied health service delivery were below a 2% increase in the proportion of patients who stayed in hospital longer than expected, a 2% increase in the proportion who had an unplanned readmission within 28 days, a 2% increase in the proportion who had any adverse event, and a 1-day increase in the mean length of stay. The current weekend allied health service included physical therapy, occupational therapy, speech therapy, dietetics, social work, and allied health assistant services in line with usual care at the participating sites. The newly developed weekend allied health service allowed managers at each site to reprioritise tasks being performed and the balance of hours provided by each professional group and on which days they were provided. Analyses conducted on an intention-to-treat basis demonstrated that there was no estimated effect size difference between groups in the proportion of patients staying longer than expected (weekend versus no weekend; estimated effect size difference [95% CI], p-value) in Trial 1 (0.40 versus 0.38; estimated effect size difference 0.01 [−0.01 to 0.04], p = 0.31, CI was both above and below non-inferiority margin), but the proportion staying longer than expected was greater with the newly developed service compared to its no weekend service control condition (0.39 versus 0.40; estimated effect size difference 0.02 [0.01 to 0.04], p = 0.04, CI was completely below non-inferiority margin) in Trial 2. Trial 1 and 2 findings were discordant for the mean length of stay outcome (Trial 1: 5.5 versus 6.3 days; estimated effect size difference 1.3 days [0.9 to 1.8], p < 0.001, CI was both above and below non-inferiority margin; Trial 2: 5.9 versus 5.0 days; estimated effect size difference −1.6 days [−2.0 to −1.1], p < 0.001, CI was completely below non-inferiority margin). There was no difference between conditions for the proportion who had an unplanned readmission within 28 days in either trial (Trial 1: 0.01 [−0.01 to 0.03], p = 0.18, CI was both above and below non-inferiority margin; Trial 2: −0.01 [−0.02 to 0.01], p = 0.62, CI completely below non-inferiority margin). There was no difference between conditions in the proportion of patients who experienced any adverse event in Trial 1 (0.01 [−0.01 to 0.03], p = 0.33, CI was both above and below non-inferiority margin), but a lower proportion of patients had an adverse event in Trial 2 when exposed to the no weekend allied health condition (−0.03 [−0.05 to −0.004], p = 0.02, CI completely below non-inferiority margin). Limitations of this research were that 1 of the trial wards was closed by the healthcare provider after Trial 1 and could not be included in Trial 2, and that both withdrawing the current weekend allied health service model and installing a new one may have led to an accommodation period for staff to adapt to the new service settings. Stepped-wedge trials are potentially susceptible to bias from naturally occurring change over time at the service level; however, this was adjusted for in our analyses.
Conclusions
In Trial 1, criteria to say that the no weekend allied health condition was non-inferior to current weekend allied health condition were not met, while neither the no weekend nor current weekend allied health condition demonstrated superiority. In Trial 2, the no weekend allied health condition was non-inferior to the newly developed weekend allied health condition across all primary outcomes, and superior for the outcomes proportion of patients staying longer than expected, proportion experiencing any adverse event, and mean length of stay.
Trial registration
Australian New Zealand Clinical Trials Registry ACTRN12613001231730 and ACTRN12613001361796
Author summary
Why was this study done?
- This study was done to find out whether a health service that is routinely provided has the effect that it is intended to have.
- It considers whether it is worth continuing to provide this service when the resources we use to support this service could potentially be better used elsewhere in the health system.
What did the researchers do and find?
- We conducted 2 large trials across 6 acute medical and surgical wards in each of 2 hospitals.
- We removed the weekend allied health (e.g., physical therapy, occupational therapy, and social work) service in the first trial, redesigned it, and reintroduced it in the second trial.
- We found that removing the weekend allied health service had an uncertain impact on patient flow (the change in the percentage of patients who stayed longer than expected was +1% [95% CI: −1% to 4%]) and safety (the change in the percentage with any adverse event was +1% [95% CI: −1% to 3%]), but that having no weekend allied health service on these wards was no worse than having the newly developed service that was brought back in, and indeed was superior for some outcomes (the percentage who stayed longer than expected was 2% better [95% CI: 1% to 5%], and the percentage with any adverse event was 3% better [95% CI: 0% to 5%] with no weekend service).
What do these findings mean?
- These findings indicate that resources spent on providing allied health services on weekends across acute medical and surgical wards could possibly be better spent elsewhere in the healthcare system.
- This study also demonstrated that it is possible to undertake research of routinely provided services where it is uncertain whether they provide the benefits they are intended to.
Citation: Haines TP, Bowles K-A, Mitchell D, O’Brien L, Markham D, Plumb S, et al. (2017) Impact of disinvestment from weekend allied health services across acute medical and surgical wards: 2 stepped-wedge cluster randomised controlled trials. PLoS Med 14(10): e1002412. https://doi.org/10.1371/journal.pmed.1002412
Academic Editor: Aziz Sheikh, Edinburgh University, UNITED KINGDOM
Received: March 28, 2017; Accepted: September 21, 2017; Published: October 31, 2017
Copyright: © 2017 Haines et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This research was funded by the National Health and Medical Research Council (AU) Award Number: 1060696 (TH, ES, FM, LO, DMa, DMi), the National Health and Medical Research Council (AU) Award Number: 1069758 (TH), the Department of Health, State Government of Victoria (TH), and the Australian Government Postgraduate Award (RH and MS). KP worked for the Victorian Department of Health and Human Services and contributed to study design and manuscript preparation as an author. The funders had no other role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: I have read the journal's policy and the authors of this manuscript have the following competing interests: RH has been funded by an Australian Postgraduate Awards Scholarship. KM is a board member (volunteer/unpaid) for Wounds Australia and a member of Victorian Allied Health Leaders Council Executive (unpaid).
Introduction
Disinvestment is the process of (partially or completely) withdrawing health resources from any existing healthcare practices, procedures, technologies, or pharmaceuticals that are deemed to deliver little or no health gain for their cost and thus are not efficient health resource allocations [1]. Governments and professional bodies around the world are introducing processes to eliminate or limit access to services and procedures known to deliver little or no health gain for their cost. Examples from the United States include the Choosing Wisely campaign driven by the American Board of Internal Medicine Foundation [2], and the Patients Before Paperwork initiative led by the American College of Physicians [3].
There is some controversy surrounding what the key motivation for these disinvestments should be. Commentators have expressed concern that the Choosing Wisely campaign is seen by some as a cost-cutting exercise, and feel that it would be more acceptable if the focus were placed on helping to avoid patient harm that may arise from unnecessary tests and ineffective interventions [2]. We would argue that disinvesting from ineffective services could be seen as being more virtuous and acceptable if entwined with an argument based on opportunity costs: that resources spent delivering ineffective services to one patient are the same resources that can no longer be used to deliver an effective service to improve health outcomes for another. This position is best seen from the perspective of healthcare administrators with finite budgets who must choose between competing services they can offer with their available funds.
Much of the debate thus far has centred on disinvestment from individual services that are known to be ineffective or unnecessary, particularly for certain patient subgroups. But what is to be done with classes of services that are routinely provided, but have unknown effectiveness or cost-effectiveness? Continued provision of these services, if ineffective, creates an opportunity cost that is wasteful and stops other patients from potentially benefitting from those same resources. If services are effective, cessation of provision creates a situation for potential patient harm and poorer health outcomes than what otherwise would have been achieved. Thus, decision makers are left in an unenviable position—by virtue of the absence of evidence relevant to guide their decision—of choosing between continued provision, cessation, or reduction, which each carry an element of risk for their patients. This situation is further complicated in circumstances where a service is provided by clinicians who believe it is effective, despite the absence of evidence to this effect. Planning to remove such a service may create fear of negative patient and/or provider outcomes in staff members, which can act as a barrier to practice change [4,5].
One such area of uncertainty is the delivery of allied health services on weekends to patients in acute medical and surgical units. Allied health services (such as physical therapy or social work) are now commonly provided on weekends in hospitals internationally, though not to the same extent as during the week [6–10]. These models appear to have increased in popularity following observational studies identifying a ‘weekend effect’—poorer patient and service outcomes associated with admissions or procedures taking place on weekends [11–15]. However, the efficacy of these services has not been established. A recent, methodologically inclusive systematic review of these services amongst acute joint arthroplasty patients found some benefit in improving length of stay and function [16]. However, this finding was largely driven by findings of observational studies, and no randomised trials examining the efficacy of weekend allied health services in acute settings were identified. Links between allied health staffing levels and the weekend effect have not been established, and some have questioned the very existence of the weekend effect [17,18].
The aim of this study was to establish the impact of disinvesting from provision of allied health services on weekends across acute medical and surgical hospital wards.
Methods
Human Research Ethics Committee approval for this project was provided by the Monash Health (approval ref 13327B) and Melbourne Health (approval number: 2013.283) Human Research Ethics Committees.
Design
We have previously proposed an approach for use in the context of disinvestment from a service that has unknown effectiveness [4]. This approach is centred on simultaneous disinvestment from the health service in question while also generating the evidence examining the effectiveness of this service. In this research, we conducted 2 stepped-wedge cluster randomised controlled trials across each of 2 tertiary, metropolitan teaching hospitals, Dandenong Hospital and Footscray Hospital, in Victoria, Australia. Stepped-wedge trials are a form of cluster randomised trial with unidirectional crossover that are increasingly being used for health service evaluations [19]. Trial 1 was a disinvestment randomised trial where the ‘current’ weekend allied health service was removed from 1 ward per calendar month in an order determined at random. It contained examination of both non-inferiority and superiority hypotheses for each primary outcome examined, and superiority hypotheses for each secondary outcome examined. Testing non-inferiority hypotheses does not preclude testing of superiority hypotheses and can be done without statistical penalty [20]. Trial 2 was a conventional stepped-wedge randomised trial where a ‘newly developed’ weekend allied health service was reintroduced in a 180-degree rotated image of Trial 1 (Fig 1) [5]. This reintroduction of weekend allied health services in Trial 2 was considered necessary in the present context given the incremental and evolving development of the current weekend service delivery models that had taken place at the participating sites combined with heterogeneity in delivery models of these services internationally. The newly developed model that was reintroduced was intended to be a complex intervention that used a reproducible procedure for development. Its development involved gathering feedback from medical, nursing, and allied health staff working on the targeted wards. This feedback was reviewed by and discussed with local allied health managers in charge, who made the final decision as to the make-up of the newly developed model.
Numbers in boxes indicate occasions of service provided by allied health team on weekdays/on weekends that month. These data include approved clinical exceptions during no weekend service periods.
There were 3 deviations from the analysis plan and 1 change to the outcome measures described in our study protocol (S1 Text). First, a pre-planned meta-analysis that combined data from both sites and trials was not performed due to discordance in findings between Trial 1 and Trial 2 [21]. Second, a sensitivity analysis for Trial 1 was performed where a 1-month analysis washout period was introduced immediately following introduction of the ‘no weekend allied health service’ condition: the September 2014 block at Dandenong Hospital and the November 2014 block at Footscray Hospital served as the last block for this sensitivity analysis of Trial 1. Third, analyses without adjustment for monthly outcome data for the previous 2 years were undertaken. These analysis changes were inspired by peer-reviewers. The secondary outcome of staff absenteeism could not be extracted in clinical units relevant for this trial, so was abandoned. A CONSORT checklist for this study is provided as S2 Text.
Participants and setting
This research took place across 6 acute medical or surgical wards at Dandenong Hospital and 6 at Footscray Hospital, Victoria, Australia. Most of these wards were specialised in different areas of medical/surgical medicine and admitted different types of patients. At Dandenong Hospital, the ward specialties were orthopaedic surgery, stroke, thoracic/vascular/general surgery & medical, general medicine, head/neck/plastics, and surgical. At Footscray Hospital, the ward specialties were medical (2 wards), infectious diseases/respiratory, plastics/ENT/head/neck surgery, general surgery/colorectal/breast/endocrine/urology, and general surgery/vascular/thoracic/upper gastrointestinal. Patient allocation to wards was driven by patient specialty requirements and bed availability.
Patients who were exposed to the no weekend allied health service condition as well as either the current or newly developed weekend allied health service conditions were excluded to avoid research-design-induced contamination. The requirement to collect individual patient-level consent for researchers to access the primary and secondary outcomes reported in this paper was waived during the ethical approval process.
Randomisation and masking
Random allocation of wards at each site to starting position in the trial design was undertaken at public meetings at each site. Investigators (D. Mitchell and EHS) developed a list of pseudonyms to represent each ward and then provided these to another investigator (TPH) who was blinded to the meaning of each pseudonym. This investigator then used a random number generator command in Microsoft Excel to allocate wards to starting positions in the stepped-wedge trial design. From this point, staff and patients were not blinded to group allocations due to the practical need to notify staff when the weekend allied health service on their ward would cease.
Interventions
A detailed description of weekend allied health service models investigated in both of these trials (including TIDiER checklist [22]) and the process used to develop the newly designed model in Trial 2 was provided in our protocol [5]. Briefly, we undertook extensive consultation with relevant stakeholders (medical, nursing, and allied health staff and managers) on participating wards. These staff were interviewed (group interviews at the ward level and individual/key informant interviews) by investigators experienced in conducting qualitative and participatory action research (LO, FM). These staff were not asked to say which professional discipline they wanted to be employed on the weekends, rather, to identify and prioritise the tasks that they believe to be most important for allied health to perform on weekends in terms of improving patient health outcomes, improving patient flow, and reducing readmissions. They were also asked to reflect on the strengths and limitations of the current model of care, suggest areas for improvement, and examine patient incident and clinical exception data gathered during Trial 1 to inform their decisions. Allied health managers were provided with this list of tasks and other feedback gathered, so they could propose the new stakeholder-driven model of weekend allied health service they felt would work best. A Delphi meeting was used to facilitate this process separately at each hospital site [23]. This information was forwarded to each hospital’s allied health director, who made the final decision.
Services delivered under the original (Trial 1) and newly developed models (Trial 2) at each site are presented in S3 and S4 Texts, which present the amount of service provided by each discipline group when the service was fully operational (first month of Trial 1 and last month of Trial 2).
The no weekend service condition entailed delivery of no allied health services on weekends to wards affected, unless criteria for a clinical exception had been met. A safety mechanism specified in our trial protocol was that a process be established whereby the trial protocol could be violated for an individual patient if specific, pre-planned criteria set by local clinicians and approved by local administrators were met (S5 Text).
Outcomes
The primary outcomes were indicators of the domains of patient flow through the hospital (length of stay measured in days, proportion of patients staying longer than their diagnosis-related group average inlier length of stay), failures in discharge planning (proportion of patients with unplanned readmissions within 28 days of discharge), and failures in patient care (proportion of patients experiencing any of the following adverse events: in-hospital fall, Code Blue call, Medical Emergency Team call, pulmonary embolus, deep vein thrombosis, death, hospital-acquired pressure area, or intensive care unit admission from the ward). We used 2 indicators of patient flow due to the inherent limitations of using the intuitively attractive indicator (length of stay measured in days) in the context of our stepped-wedge design. The results of a stepped-wedge trial can be biased if certain hospital wards, but not others in the same trial, change the types of patients (particularly those with greater or lesser lengths of stay) they tend to admit over time to cope with seasonal demands. Statistically adjusting for patient diagnosis and procedure type is a possible approach to dealing with this problem, but only if the number of meaningful diagnostic/procedural groupings is small enough to enable statistical models to be validly calculated (which is not the case in this context). Our alternate approach was to use the dichotomous outcome of whether each patient stayed longer than expected for their diagnosis/procedural-related grouping. In Australia, the Australian Refined Diagnosis Related Groups approach is employed. Hospital data coders classify patients into these groupings that are based on similarity of conditions and usage of hospital resources, using information in the hospital morbidity record such as the diagnoses, procedures, and demographic characteristics of the patient [24]. The primary outcomes were collected through hospital administrative data systems and checked daily by research assistants interviewing ward representatives and checking handover documentation.
Secondary outcomes collected across all participants included the proportion of patients discharged to residential aged care facilities, the cost (in Australian dollars) per patient to the healthcare system per admission, the proportion of patients discharged on a Saturday or Sunday, and the number of compliments and complaints (total and allied health specific). These outcomes were extracted from hospital administrative data systems at completion of the study. The cost of inpatient treatment per patient was extracted from hospital-based clinical costing systems in August 2016 to allow finalisation of hospital costing processes. It should be noted here that clinical costing data are largely driven by length of stay, and hence this outcome has the same limitations as described above for the length of stay outcome.
Process measures collected included the occasions of weekend allied health service delivery (i.e., the number of times an allied health professional went and saw a patient on the weekend) and reason for clinical exceptions taking place. Occasions of allied health service delivery were recorded by allied health staff, collected through routine hospital administrative data systems, and extracted at the end of the study. The frequency and reason for clinical exceptions taking place were recorded by site investigators (KM, TC, and MS) who had local responsibility for approving these at the time of the exception being granted. Non-patient-related outcomes and qualitative, economic, subsample, and meta-regression analyses described in our protocol were reserved for further publications.
Procedure
An audit was conducted prior to the trial to establish the average amount of weekend service per ward and the financial costs of the overall service. During Trial 1, weekend services were titrated down according to pre-trial audit amounts. If a ward received 4 hours of physical therapy over a weekend prior to trial commencement, it is this amount that was removed when this service was ceased on this ward. For Trial 2, the service was reintroduced on a pro rata basis. So, if a hospital provided A$2,400 in allied health staffing costs per weekend across 6 wards overall prior to the trial, then A$400 worth of resources per ward was allowed to be reintroduced when the newly developed service was commenced. At the Dandenong Hospital site, managers reallocated their funds to increase the overall number of allied health hours provided by moving some hours to a Friday afternoon and Monday morning. This was intended to facilitate discharge planning prior to the weekend and immediately following and to maximise the total number of allied health hours. At the Footscray Hospital site, managers changed the personnel who provided the weekend allied health service from casual staff to allied health staff working in the intensive care unit and the Immediate Response Service, whose usual responsibility was to screen and assess presentations to the emergency department to prevent inappropriate admissions and facilitate discharge planning.
A CONSORT flow chart (Fig 2) provides study recruitment, data collection, and analysis details. Nursing staff managers on each ward were engaged prior to study commencement to identify which tasks usually performed by allied health staff on weekends could be modified, accelerated (to a Friday), delayed (to a Monday), or transferred to other staff who were present on weekends. They were also provided with the criteria for clinical exceptions on their wards. Study stopping rules and non-inferiority margins were also developed prior to trial commencement by hospital administrators and allied health managers at participating sites [5].
Project research assistant data collectors were present 7 days per week during the study period. They collected data through medical record review and interviews with hospital staff, enabling crosschecking of data collected between routine hospital administrative data systems and direct data collection approaches [25,26].
These trials were registered with the Australian New Zealand Clinical Trials Registry. Trial 1 was prospectively registered on 8 November 2013 (ACTRN12613001231730), and Trial 2 on 12 December 2013 (ACTRN12613001361796).
Statistical analysis
Multilevel, mixed-effects generalised linear model analyses were used to generate effect size estimates and 2-tailed 95% CIs and superiority hypothesis p-values. The 95% CIs were compared to the non-inferiority margins for each primary outcome measure to determine if they were completely below (indicating non-inferiority), completely above (indicating inferiority), or both above and below (indicating uncertain inferiority) the non-inferiority margin. The non-inferiority margins were an absolute difference of 2% in proportion outcomes (proportion of patients staying longer than the average inlier diagnosis-group-related length of stay, proportion with an unplanned readmission within 28 days, proportion experiencing 1 or more of the adverse events previously listed) and an absolute difference of 1 day for the mean length of stay outcome.
These analyses were conducted by a statistician (SM) independent to the research locations and blinded to ward allocation status through use of 6 mock codes representing different ward allocation patterns. An intention-to-treat analysis approach was employed. Models were initially constructed using patient-level data nested within ward nested within hospital site. Patient admissions were coded against the first study ward of their admission and calendar month of their first admission to a study ward. Log natural transformation of length of stay data was planned due to the anticipated skewed distribution [5]. Effect estimates were generated using ward-month-level data if effect estimates could not be calculated using patient-level data (on account of inability to ascertain starting values for models with binary outcomes). All analyses were adjusted for study month and ward as categorical fixed factors in line with recommendations for analysis of stepped-wedge designs [19]. Adjustment was also made for monthly outcome data from the previous 2 years for primary outcomes to account for potential seasonal fluctuations that are consistent from year to year. The main effect of having no weekend service was calculated, followed by pre-planned ‘no weekend service by site’ interaction effects to determine if site-level analyses were required [27]. Analyses were undertaken using Stata MP v.14 (StataCorp, College Station, Texas).
Three post hoc exploratory analyses were undertaken. The first investigated potential differences in outcomes between the 2 weekend allied health service delivery models (current versus newly developed) directly. This analysis was considered important as there was no guarantee that the newly developed model would be superior to the current model despite this being the intent of redesigning the weekend allied health service. The second examined whether the results under the no weekend service condition were comparable between the 2 trials. Previous studies have reported substantial changes over time within health services in outcomes examined in this trial (e.g., falls [28]), making it important to consider whether the no weekend allied health service conditions in the 2 trials were comparable. Adjustment for study month was not used in these exploratory analyses as this would have led to collinearity with the independent variables of interest. The third post hoc analysis was a sensitivity analysis for Trail 1 whereby a 1-month washout period was applied to each ward following their transition to the no weekend allied health service condition. This meant that the month of data immediately following the transition in model of care was excluded from the analysis for each ward and that the final month included in the analysis of Trial 1 became September 2014 for the Dandenong Hospital site and November 2014 for the Footscray Hospital site. This analysis was important as changes in practice can potentially affect hospital practices and processes in a way that is different to what happens once the change has had more time to become imbedded in practice.
Data were monitored by a committee drawn from senior clinical staff at participating sites who were not members of the study investigative team. Trial dates are provided (Fig 1), along with deviations from the registered trial protocol (Fig 2; inability to commence at 1 site and closure of 1 ward during Trial 2). Study data used in these analyses are provided (S1 and S2 Data). Study power analysis calculations are provided (S6 Text) and were described in our protocol paper [5].
Results
Trial dates of commencement and completion along with occasions of allied health service are presented (Fig 1). Demographic characteristics of trial participants (Trial 1, n = 14,834; Trial 2, n = 12,674) are presented (Table 1). There were 28 approved clinical exceptions in total (14 in each trial) during the no weekend allied health periods, of which the most common justification was a post-fall mobility review by a physiotherapist (16). The remaining occasions of service provision were not approved clinical exceptions during this period.
Data for primary and secondary outcomes for the full trial sample are presented (Table 2). Effect size estimates for each trial are presented (Table 3). The p-values presented in Table 3 reflect the probability of the findings observed when the superiority null hypothesis is true, and were derived from multilevel, mixed-effects generalised linear model analyses. Length of stay was greater for the no weekend service condition in Trial 1, but shorter in Trial 2. The no weekend allied health condition was of uncertain inferiority in Trial 1 for this outcome (95% CI: 0.85 to 1.77), but non-inferior in Trial 2 (95% CI: −2.03 to −1.13) relative to the +1 day non-inferiority margin. The proportion of patients staying longer than their diagnosis-related group average inlier length of stay was lower for the no weekend service condition in Trial 2 compared to the newly developed weekend allied health service condition. The no weekend allied health service condition was of uncertain inferiority for the proportion of patients who stayed longer than expected (95% CI: −0.01 to 0.04), the proportion who had an unplanned readmission within 28 days (95% CI: −0.01 to 0.03), and the proportion who had 1 or more adverse events (95% CI: −0.01 to 0.03) in Trial 1 relative to a +2% absolute increase non-inferiority margin. However, the no weekend allied health service condition was non-inferior to the newly developed weekend allied health service condition for the proportion of patients who stayed longer than expected (95% CI: −0.05 to −0.01), the proportion who had an unplanned readmission within 28 days (95% CI: −0.02 to 0.01), and the proportion who had 1 or more adverse events (95% CI: −0.05 to −0.004). It was also superior for the proportion of patients staying longer than expected and the proportion who had 1 or more adverse event outcomes.
Examination of site-by-intervention interaction effects identified an interaction for the proportion of patients with an unplanned readmission in Trial 1. The subsequent subgroup analyses indicated that at the Dandenong Hospital site the no weekend service condition had a higher proportion of unplanned readmissions (coefficient [95% CI]: 0.03 [0.01 to 0.05], p = 0.01), but this was not the case at the Footscray Hospital site (coefficient [95% CI]: −0.0002 [−0.03 to 0.02], p = 0.86).
Examination of superiority hypotheses for secondary outcomes demonstrated that there was difference between conditions in the proportion of patients discharged to residential aged care facilities. However, the cost to the healthcare system per admission was greater for the no weekend service condition in Trial 1, but less in Trial 2.
Examination of superiority hypotheses for the secondary outcome of the proportion of patients discharged on a Saturday or Sunday demonstrated that there was no difference between conditions in Trial 1, but proportion discharged on a Saturday or Sunday was marginally (2%) higher for the newly developed weekend service in Trial 2. This finding should be viewed in the context that length of stay was longer when a weekend allied health service was present in that trial.
Exploratory analyses (Table 4) indicated that patients in the no weekend service condition in Trial 1 had better outcomes across multiple domains than those in the no weekend service condition in Trial 2. Patients exposed to the original weekend allied health service delivery model did not experience different outcomes than those exposed to the newly developed model for the primary outcomes of the proportion staying in hospital longer than expected, the proportion who had an unplanned readmission within 28 days, and the proportion experiencing any adverse event when these patient groups were compared directly. There was a difference for mean length of stay (log transformed) and total cost favouring the original weekend allied health service delivery model, though these outcomes did not account for differences in patient diagnosis categories between phases. The sensitivity analyses of Trial 1, with the washout period added, demonstrated that the no weekend allied health condition was non-inferior to the current weekend allied health condition (95% CI: 0.01 to 0.60) for the length of stay outcome, whereas there was uncertain inferiority for this outcome in the primary analysis. The other outcomes were largely unaffected in this sensitivity analysis.
Analyses without adjustment for monthly outcome data for the previous 2 years had some inconsistencies with main analyses, where the adjustment was made (S7 Text). Differences in the proportion of patients staying longer than expected and with any adverse event were no longer significant between the no weekend service and newly developed weekend allied health service conditions in Trial 2. However, the 95% CIs were still completely below the non-inferiority margin, meaning that the interpretation of these results from this perspective remains unaffected. There were also site-by-intervention interaction effects for the log-transformed length of stay (Trial 1) and proportion of patients with any adverse event (Trial 2) outcomes. Here, the log-transformed mean length of stay was longer if exposed to the no weekend allied health service condition at the Dandenong Hospital (coefficient [95% CI]: −0.15 [0.07 to 0.23], p < 0.001), and the proportion of patients who experienced any adverse event was lower in the no weekend allied health service condition at Dandenong Hospital only (coefficient [95% CI]: −0.03 [−0.05 to −0.002], p = 0.03).
Discussion
In Trial 1, criteria to say that the no weekend allied health condition was non-inferior to current weekend allied health condition were not met, while neither the no weekend nor current weekend allied health condition demonstrated superiority. The result for the mean length of stay outcome from Trial 1 was sensitive to whether a 1-month washout period was applied in the analysis. When the washout period was applied, the no weekend allied health service condition was found to be non-inferior to the current weekend allied health service model. In Trial 2, the no weekend allied health condition was non-inferior to the newly developed weekend allied health condition across all primary outcomes, and superior for the proportion of patients staying longer than expected, proportion experiencing any adverse event, and mean length of stay.
The findings of this study were somewhat discordant with the only previously published randomised trial of the effect of weekend allied health services on patient and health service outcomes. This earlier trial reported no significant difference in length of stay or patient adverse events (p > 0.05) in those provided with a Saturday physical therapy service, in addition to a Monday to Friday service, on rehabilitation wards [29]. However, this trial did identify small benefits of the service in terms of improved functional independence and health-related quality of life attributable to the intervention, though these benefits were arguably below clinically meaningful thresholds [30–32]. It is difficult to directly compare these results given the differences in ward types and patient populations involved, the fact that the earlier study focused only on a physical therapy service, and the differences in the activities undertaken by allied health staff between acute and rehabilitation settings. Our study was the first to our knowledge to use this particular disinvestment research design to simultaneously disinvest from a routinely provided service with uncertain effectiveness while also developing evidence that had previously been missing as to the effectiveness of the intervention.
It should be noted that the models of care we examined in Trial 1 were limited to those in place at the outset of the trial at the study locations. However, the model of care in Trial 2 was a complex intervention, where the process used could be reproduced in other settings to develop models that are similarly tailored to local conditions. A strength of these two trials in addressing the research context area was their size, which led to narrow confidence intervals in our analyses and a high degree of certainty in the results. Our choice to investigate both the current weekend allied health service and a newly developed service also enhanced the generalisability of our findings to real world settings. However, these trials did not investigate the breadth of all possible permutations and combinations of weekend allied health service delivery models. Rather, we focused on the pragmatic scenario of what was currently being allocated and the budgetary envelope of this. Our study was limited in that we were unable to proceed with this research on 1 ward at 1 of our sites in Trial 2 due to closure of that clinical unit. We were also unable to proceed at a third site due to local opposition to participation in the study. This highlights some of the difficulties that can be encountered in conducting disinvestment research of this nature over an extended period of time. This study could have been further strengthened by considering the healthcare costs consumed by patients after they were discharged from these acute wards. It is possible that allied health services not provided on weekends in the acute setting may still have been provided in another setting later in a patient’s journey.
The scope of the clinical implications of our findings should be clearly defined. The study locations did not have all types of specialty wards (e.g., spinal, burns) that might be considered to be acute medical or surgical wards. The finding of no effect of weekend allied health services overall should not be extrapolated to weekday services. It is possible that variation in staffing profile and the lack of availability of community-based services on weekends that support patients upon discharge are an important difference. We also did not withhold weekend allied health services from those who met our clinical exception criteria. The low frequency of these exceptions may indicate that using staff from other areas that do have a weekend allied health service (e.g., intensive care unit) or using an on-call staffing model may be preferable to employing weekend allied health staff in fixed shifts to meet these patients’ needs.
This study reported discordant findings for the mean length of stay and total cost outcomes between Trials 1 and 2. This can be explained by 5 potential mechanisms. First, these changes could be attributed to background variation in patient case mix not accounted for by these outcomes. Winter months in Australia, which largely coincided with the no weekend services condition in Trial 1, are associated with a 20% increase in demand for medical admissions primarily due to infectious diseases [33]. Second, it may be that the current weekend allied health service delivery model was superior to the newly developed model. Our exploratory analyses directly comparing these models did not support this explanation as the log-transformed length of stay outcome favoured the newly developed service when these conditions were directly compared. Third, the current service model could be argued to have had an unfair advantage in a direct comparison with the newly developed service model. The current service model had several years of refinement locally and integration into usual care before being subjected to this evaluation, whereas the newly developed service model was evaluated as soon as it was introduced and did not have the same opportunity to be refined and integrated. Fourth, the patient cohort or standard of usual care provided at study sites could have changed within the no weekend service periods across Trials 1 and 2. Our exploratory analyses identified multiple outcomes where differences favoured those exposed to the no weekend service condition during Trial 2 compared to Trial 1, which would support this hypothesis. The fifth potentially concurrent explanation is that both withdrawing the weekend allied health service model and installing a new one required an accommodation period for staff to adapt to the new service settings. This hypothesis would also be supported by the differences observed between Trials 1 and 2 within the no weekend service periods. Models of organisational lag have been formally investigated since the 1980s [34] and may indicate that our original research design should have included a washout period following the transition to the new model of care. Our sensitivity analyses, where we introduced a 1-time-period washout to Trial 1, identified that the length of stay in the no weekend service condition was non-inferior to the current weekend service condition (though other results were not substantially changed).
There is potential that naturally occurring change over time could confound the results of these stepped-wedge trials. Naturally occurring change over time (maturation) is a potential confounder in every longitudinal interventional research design. The important concern is whether this may have biased the intervention effect size estimates calculated from each trial. In a stepped-wedge design, this problem is prominent given the unidirectional crossover employed. This means that a stepped-wedge trial conducted within a system/organisation that naturally improves over time will favour the condition that is tested second, and vice versa. This problem was first described in detail by Hussey and Hughes [35]. Their solution to this problem was to explicitly model the effect of each time period to eliminate this bias from the intervention effect estimate. We used this approach in every analysis presented in Table 3. Thus, our estimated effect sizes calculated from within Trial 1 and Trial 2 are statistically independent of the potentially confounding effects of change over time that occurred across the organisations involved, and can be considered to be free of bias from this source. The analyses that were not independent of these effects are the comparisons between Trial 1 and Trial 2 that are reported in Table 4. We could not use this approach in the comparisons between the 2 control periods, and in the direct comparisons of the current and the newly developed weekend services in Table 4 as these were not comparisons based on a stepped-wedge design. Rather, these were pre- versus post-intervention design comparisons with no overlap in time periods between the conditions being compared (thus a time covariate would directly confound the covariate of interest, which in this case was Trial 1 versus Trial 2). The washout sensitivity analysis model in Table 4 did use the approach described by Hussey and Hughes as this was a stepped-wedge design.
The key implication of this research is that resources being used to support weekend allied health service delivery to acute medical and surgical wards similar to those involved in this study could potentially be put to better use elsewhere in the healthcare system. Future research in the field of weekend allied health service delivery is warranted, particularly in other ward types and when examining higher dosage levels of service delivery. If higher dosage levels can demonstrate an effect on clinical or patient flow outcomes, they will still need to be justified economically. Future research using this disinvestment trial approach is also warranted. The value of a trial that finds a model of care that no longer includes a particular service to be non-inferior to one that includes it can be conceptualised as the future opportunity costs saved of no longer providing that service. Hence, commonly provided services that have a relative absence of evidence supporting their use and a high opportunity cost of delivery should be targeted.
Supporting information
S1 Text. Trial protocol.
Study protocol for 2 randomised controlled trials examining the effectiveness and safety of current weekend allied health services and a new stakeholder-driven model of weekend allied health services for acute medical/surgical patients versus no weekend allied health services.
https://doi.org/10.1371/journal.pmed.1002412.s004
(PDF)
S4 Text. Hours of service by discipline.
Hours of allied health service provision offered under current and newly developed weekend allied health service delivery models at each site.
https://doi.org/10.1371/journal.pmed.1002412.s007
(DOCX)
S7 Text. Analyses unadjusted for monthly ward data from previous 2 years.
Effect size estimates of main and trial-by-site interaction effects from each trial for primary and secondary outcomes. Main effects are interpreted as the impact of being exposed to the no weekend allied health condition compared to the current or newly developed weekend allied health conditions.
https://doi.org/10.1371/journal.pmed.1002412.s010
(DOCX)
References
- 1. Elshaug AG, Hiller JE, Tunis SR, Moss JR. Challenges in Australian policy processes for disinvestment from existing, ineffective health care practices. Aust New Zealand Health Policy. 2007;4:23. pmid:17973993
- 2. Hurley R. Can doctors reduce harmful medical overuse worldwide? BMJ. 2014;349:g4289. pmid:24993249
- 3. Erickson S, Rockwern B, Koltov M, McLean R, Medical Practice and Quality Committee of the American College of Physicians. Putting patients first by reducing administrative tasks in health care: a position paper of the American College of Physicians. Ann Intern Med. 2017;166:659–61. pmid:28346948
- 4. Haines T, O’Brien L, McDermott F, Markham D, Mitchell D, Watterson D, et al. A novel research design can aid disinvestment from existing health technologies with uncertain effectiveness, cost-effectiveness, and/or safety. J Clin Epidemiol. 2014;67:144–51. pmid:24275500
- 5. Haines TP , O’Brien L, Mitchell D, Bowles K, Haas R, Markham D, et al. Study protocol for two randomized controlled trials examining the effectiveness and safety of current weekend allied health services and a new stakeholder-driven model for acute medical/surgical patients versus no weekend allied health services. Trials. 2015;16:133. pmid:25873250
- 6. Campbell L, Bunston R, Colangelo S, Kim D, Nargi J, Hill K, et al. The provision of weekend physiotherapy services in tertiary-care hospitals in Canada. Physiother Can. 2010;62:347–54. pmid:21886374
- 7. Cusick A, Johnson L, Bissett M. Occupational therapy in emergency departments: Australian practice. J Eval Clin Pract. 2009;15:257–65. pmid:19335482
- 8. Shaw KD, Taylor NF, Brusco NK. Physiotherapy services provided outside of business hours in Australian hospitals: a national survey. Physiother Res Int. 2013;18(2):115–23. pmid:23038214
- 9. Ottensmeyer CA, Chattha S, Jayawardena S, McBoyle K, Wrong C, Ellerton C, et al. Weekend physiotherapy practice in community hospitals in Canada. Physiother Can. 2012;64:178–87. pmid:23449882
- 10. Ponto JM, Berg W. Social work services in the emergency department: a cost-benefit analysis of an extended coverage program. Health Soc Work. 1992;17:66–73. pmid:1537582
- 11. Aylin P, Alexandrescu R, Jen M, Mayer E, Bottle A. Day of week of procedure and 30 day mortality for elective surgery: retrospective analysis of hospital episode statistics. BMJ. 2013;346:f2424. pmid:23716356
- 12. Ruiz M, Bottle A, Aylin PP. The Global Comparators project: international comparison of 30-day in-hospital mortality by day of the week. BMJ Qual Saf. 2015;24:492–504. pmid:26150550
- 13. McIsaac DI, Bryson GL, van Walraven C. Elective, major noncardiac surgery on the weekend: a population-based cohort study of 30-day mortality. Med Care. 2014;52:557–64. pmid:24783992
- 14. Thomas C, Smith R, Uzoigwe C, Braybrooke J. The weekend effect. Bone Joint J. 2014;96:373–8. pmid:24589794
- 15. Freemantle N, Ray D, McNulty D, Rosser D, Bennett S, Keogh BE, et al. Increased mortality associated with weekend hospital admission: a case for expanded seven day services? BMJ. 2015;351:h4596. pmid:26342923
- 16. Haas R, Sarkies M, Bowles KA, O’Brien L, Haines T. Early commencement of physical therapy in the acute phase following elective lower limb arthroplasty produces favorable outcomes: a systematic review and meta-analysis examining allied health service models. Osteoarthritis Cartilage. 2016;24:1167–81.
- 17. Wise J. The weekend effect—how strong is the evidence? BMJ. 2016;353:i2781. pmid:27199275
- 18. Wise J. Higher weekend death rate is flawed, study finds. BMJ. 2016;353:i2598. pmid:27154791
- 19. Hemming K, Haines T, Chilton P, Girling A, Lilford R. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ. 2015;350:h391. pmid:25662947
- 20. Snapinn SM. Noninferiority trials. Curr Control Trials Cardiovasc Med. 2000;1:19–21. pmid:11714400
- 21. Greco T, Zangrillo A, Biondi-Zoccai G, Landoni G. Meta-analysis: pitfalls and hints. Heart Lung Vessel. 2013;5:219–25. pmid:24364016
- 22. Hoffmann TC, Glasziou PP, Boutron I, Milne R, Perera R, Moher D, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ. 2014;348:g1687. pmid:24609605
- 23. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15. pmid:11095242
- 24.
Australian Institute of Health and Welfare. Australian refined diagnosis-related groups (AR-DRG) data cubes. Canberra: Australian Institute of Health and Welfare; 2017 May 17 [cited 2017 Sep 12]. Available from: https://www.aihw.gov.au/reports/hospitals/australian-refined-diagnosis-related-groups-ar-dr/contents/data-cubes.
- 25. Sarkies M, Bowles K-A, Skinner E, Mitchell D, Haas R, Ho M, et al. Data collection methods in health services research: hospital length of stay and discharge destination. Appl Clin Inform. 2015;6:96–109. pmid:25848416
- 26. Sarkies MN, Bowles KA, Skinner EH, Haas R, Mitchell D, O’Brien L, et al. Do daily ward interviews improve measurement of hospital quality and safety indicators? A prospective observational study. J Eval Clin Pract. 2016;22:792–98. pmid:27291891
- 27. Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet. 2000;355:1064–9. pmid:10744093
- 28. Haines T, Bell R, Varghese P. Pragmatic, cluster randomized trial of a policy to introduce low-low beds to hospital wards for the prevention of falls and fall injuries. J Am Geriatr Soc. 2010;58:435–41. pmid:20398112
- 29. Peiris CL, Shields N, Brusco NK, Watts JJ, Taylor NF. Additional Saturday rehabilitation improves functional independence and quality of life and reduces length of stay: a randomized controlled trial. BMC Med. 2013;11:198. pmid:24228854
- 30. Pickard AS, Neary MP, Cella D. Estimation of minimally important differences in EQ-5D utility and VAS scores in cancer. Health Qual Life Outcomes. 2007;5:70. pmid:18154669
- 31. Coretti S, Ruggeri M, McNamee P. The minimum clinically important difference for EQ-5D index: a critical review. Expert Rev Pharmacoecon Outcomes Res. 2014;14:221–33. pmid:24625040
- 32. Beninato M, Gill-Body K, Salles S, Stark P, Black-Schaffer R, Stein J. Determination of the minimal clinical important difference in the FIM instrument in patients with stroke. Arch Phys Med Rehabil. 2006;87:32–9. pmid:16401435
- 33. Jones R. Myths of ideal hospital size. Med J Aust. 2010;193:298–300. pmid:20819051
- 34. Damanpour F, Evan WM. Organizational innovation and performance: the problem of “organizational lag”. Adm Sci Q. 1984;29:392–409.
- 35. Hussey M, Hughes J. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials. 2007;28:182–91. pmid:16829207