PBS trial

Twenty-three community ID services (total N = 246 people with ID and challenging behaviour) in England, UK, were randomly allocated to either the intervention (N = 11) condition or treatment as usual (TAU; N = 12) condition. Twenty-six volunteer health staff in the intervention condition were trained in PBS approaches (see main PBS trial paper for further methodological detail [28]. The staff who trained in PBS approaches, hence referred to as therapists, came from a number of professional backgrounds (11 nurses, 5 occupational therapists, 5 psychiatrists, 2 speech and language therapists, 1 clinical psychologist, 1 assistant psychologist and 1 social worker). The PBS trial received ethical approval by the National Research Ethics Service Committee London-Harrow (reference 12/LO/1378). All participants who had capacity provided written consent before participation in the trial; family or nominated consultees gave assent for those participants without capacity.

Participants in the intervention group received PBS as well as treatment as usual (TAU). The latter was defined as routine care that is available within the community ID services including multidisciplinary input from a variety of health and social care professionals, e.g. psychiatrists, psychologists, nursing.

Process evaluation–Implementation study

Stakeholders taking part in the qualitative interviews were derived from the main trial [28]. This included participants with mild to moderate ID and verbal ability, family and paid carers, service managers and therapists. We also used data regarding reach and dose of therapy collected as part of the clinical effectiveness study.

In order to illustrate how the PBS-based training might work and produce the anticipated outcomes, we developed a logic model adapted from the Kirkpatrick’s Four-Level Training Evaluation Model (1959) [29]. The model was also used as a reference framework to gather participants’ views about the quality and effectiveness of the intervention that they received and this has been reported elsewhere [30].

The logic model levels are:

1. Inputs- this domain describes what has to be in place in order for the PBS-based staff training and associated interventions to take place.
2. Processes- this domain describes the delivery of the training and the implementation of the intervention.
3. Actions- the domain establishes the key elements of the PBS-based training intervention which need to be implemented in order to produce the study outcomes.
4. Results- the domain outlines the anticipated outcomes of the intervention.

The outcomes of the process evaluation were assessed using qualitative semi-structured interviews with stakeholders who took part in the study. In addition, quantitative data from the study database was used as well as an external independent evaluation of the quality of PBS plans. The PBS plans (a central output of the PBS-based training) are developed from what has been learnt from the functional assessment or analysis of the behaviour. They detail the approaches that family or paid carers should take to adapt the environment and support participants to develop alternative skills and behaviours to replace challenging behaviour.

Independent quality assessment

An independent assessor (Board Certified Behaviour Analyst and Associate Fellow of the British Psychological Society with 35 years of experience) assessed the quality of PBS plans developed by the therapists in the study. The Behaviour Intervention Plan Quality Evaluation Scoring Guide II (BIP-QE II) was applied to examine whether the 12 key domains of behaviour plan were present in the plans (Table 1) [31]. Each of the 12 key domains was given a score of 0 (no information), 1 (some information) or 2 (all necessary information). Higher scores on the BIP-QE II indicate an increased likelihood that a behaviour plan and the interventions or approaches contained within it would be implemented with fidelity [31].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Behaviour intervention plan quality evaluation.

https://doi.org/10.1371/journal.pone.0221507.t001

Mechanisms of impact and contextual factors study

Data for the mechanisms of impact and contextual factors study derived from individual semi-structured interviews with a purposive sample of 13 participants with intellectual disability who had sufficient verbal ability, 11 family carers, 10 paid carers, 12 service managers, 12 therapists and a focus group with the 4 PBS trainers. These interviews were conducted to assess the context in which the intervention was delivered and stakeholders’ views of both the intervention and the training components. We obtained informed consent from all participants; easy read versions were used for the participants with mild to moderate intellectual disability. Researchers and the service user reference group (Camden Advocacy Project SURGE) developed topic guides triangulating available literature and the aims and objectives of the study which were piloted and changes were made as needed. The interviews were audio recorded and transcribed verbatim by the lead author (AB) who was not directly involved in either the delivery of the intervention nor the data collection. All text from study interviews were managed using the qualitative research software NVivo 10 [32].

The interview transcripts were coded and analysed according to the inductive method of thematic analysis described by Braun and Clark (2006) during which, themes and subthemes are data driven and emerge from the text and are not predefined by researchers [33]. By doing so, the analytic process involved the reading and re-reading of data for each broad theme or ‘topic’ related to the experience of participants of the PBS intervention, without being influenced by any previous research on behavioural interventions in ID population. Topics were given the status of ‘theme’ when it emerged more than twice from the transcript. Any topics which appeared less than twice from the text but were still relevant for the present work were discussed prior to a decision being made whether to include them in the analysis. Themes were processed at the ‘latent level’ with each theme being analysed beyond the pure descriptive and sematic level of narration [33]. Once the themes and sub-themes were generated, the researcher created a codebook for co-raters to test inter-rater reliability measured using Cohen’s Kappa coefficient and based on the parameters proposed by Landis and Koch, 1977. Co-raters were a Clinical Psychologist, three Consultants in the Psychiatry of ID and a family carer of a person with ID. Inter-rater reliability was in the range of 0.8–1, indicating an almost perfect agreement [34]. The co-raters undertook three rounds of codebook revisions to determine themes and subthemes. The service user advisory group was consulted on five different occasions in this process to cover the preparation of questions for the qualitative interviews; the exploration of the preliminary findings of the interviews with stakeholders and to gather final comment on the interpretation of the findings.

Implementation (resource and training)

Manual-assisted face-to-face training was delivered by expert PBS trainers in three two-day workshops over a period of 15 weeks. Out of the 26 therapists who volunteered to receive the training, 5 therapists dropped out at the beginning of the training (3 due to illness, 1 left their position and 1 for unknown reason) and were not further involved in the study. Twenty-one started the intervention work, but 7 therapists did not remain in post until the end of the study (1 due to maternity leave, 4 left their positions or were seconded to another position, 1 took sabbatical leave and 1 due to illness). None of these 7 therapists were replaced as it was not possible to carry out more training events.

Prior to inclusion in the PBS study each service agreed to reduce therapists’ routine caseload by around 20% (1 day a week) to allow for the addition of PBS work with a maximum of 7 participants. However, although each therapist recorded the time spent on delivering the intervention to his/her allocated number of participants with ID, no information was recorded on whether and to what extent their service related caseload was reduced sufficiently to allow them to work more intensively with the trial participants. It is clear from the interviews with the therapists, that the agreed reduction in caseload did not happen in the majority of cases. That is, the therapists were expected to work as usual and in addition carry out the study tasks.

Post-training mentoring for the time that a participant was in the study was available to the therapists who were responsible for utilising this support as required, but also were prompted by regular emails or phone calls by the study administrator who also liaised with the trainer mentors. The trainers provided feedback on plans, and problem solving regarding the delivery of PBS. Several meetings took place with service managers and therapists to promote buy-in and ongoing support for the therapists both before and after the study started. The research team offered assistance with collection of study related paperwork. Clinical supervision and case management remained with the clinical teams.

Fidelity, dose and reach

For each participant the therapists were expected to complete a Brief Behavioural Assessment Tool (BBAT); conduct observations to inform the functional analysis/assessment, to formulate a PBS Plan; complete the Goodness of Fit in discussion with the carers to ensure implementation and support problem solving. Finally, therapists completed the Fidelity Checklist (designed to assess how many of the anticipated elements of the PBS intervention they had completed).

Out of the 108 possible datasets that should have contained all of the above elements, 24 were fully completed, 47 were partially completed and for 37 participants there were no therapist datasets. The reasons for the missing or incomplete data are presented in Table 2. Among the reasons for missing intervention data, was a mild or absent level of challenging behaviour in participants with ID (n = 16) at the time that the therapist approached the participant and his/her carers to begin the intervention. The 33 “Goodness to Fit” forms which rated the effectiveness of the PBS plan received from 9 clusters, indicated that the family/paid carers found the plans helpful and confirmed that they were able to implement them in the long term (“in next 12 months”). We did not, however, perform any observations of the plan implementation by carers.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Number of full and partial datasets and reasons for missing data.

https://doi.org/10.1371/journal.pone.0221507.t002

Each therapist also completed a Log of engagement detailing the tasks and time spent in relation to the delivery of PBS. At the time of receiving training, the therapists were advised by the PBS trainers that an estimated 12.5 hours would be required per case. Available Log of Engagement data for 17 therapists revealed that they spent a median of 11.50 hours (IQR 8–32) per participant relating to the intervention broken down into the following tasks (median given): 0.5 hours for direct contact or observations of a participant; 3.5 hours for contact with family, paid carers and/or service at the assessment stage; 2.3 hours for PBS plan writing and study related paperwork; 1.5 hours for contact with family, paid carers and/or service at the intervention stage; 2 hours for further indirect work not accounted in other areas (e.g. travel).

Independent quality assessment

All complete PBS plans and a selection of incomplete work (N = 61) were sent to the external assessor to assess the quality of the work involved in the delivery of PBS using the Behaviour Intervention Plan Quality Evaluation Scoring Guide II (BIP-QE II). The assessment revealed that the 33 completed plans (representing 30.6% out of a total of 108 datasets) were all rated as low quality (weak plans) and unlikely to have an impact on challenging behaviour. Three main reasons were identified: 1) lack of evidence that functional analyses had been conducted; 2) lack of evidence of specific interventions (such as skills teaching or differential reinforcement) being carried out; or 3) information on factors that maintained the challenging behaviour not being recorded.

Mechanisms of impact and contextual factors study: Stakeholder experience of PBS-based staff training

Six main themes emerged from the interviews with stakeholders.

PBS Trainers overview.

The PBS trainers (N = 4), external to the study, had many years of experience in delivering PBS training to a variety of health and social care professionals and had developed the manual for the intervention. They were each allocated 4–5 therapists and maintained contact with them via emails and telephone calls. The trainers also took part in monthly teleconferences which ran for 12 months (2014–2015).

Absence of service support to therapists being involved in the study was seen as the main challenge which also had an impact on the development of the PBS plans.

‘If you were the participant who is being trained by us but you are working in a service context that isn’t supportive of what we are trying to do then… your power to affect change is hugely reduced’. (Trainer 4, male)

‘There is practice leadership issue. She [therapist] was trying to get things implemented in a staff team and there was no manager or the manager was not ever there…we provided a lot of training …but ultimately is the management with all staff that will implement PBS plan without our support’. (Trainer 1, male)

Further, the trainers raised the issue of paid carers leaving their jobs and the impact that had on the delivery of the intervention.

‘The issue was a very high staff turnover in one site in particular so she [therapist] talked to one staff one day who the week later no longer worked and she did some training with some staff, you know, never seen again really, and these concerns were so serious that escalated locally’. (Trainer 1, male)

The trainers argued that the challenging behaviour displayed by the study participants may have been less severe and therefore, not the major focus for concern or in need for intervention and that the diverse professional background of the therapists may have contributed to the challenges in delivering certain aspects of the intervention, e.g. functional assessments/analysis of the behaviour.

‘There is no behaviour at present and it is such a shame the way some teams are supposed to have PBS and not allocated suitable people for the study’. (Trainer 1, male)

‘One of the issues we all thought was that people with very different roles attended the training…not many had some prior knowledge of PBS and struggled to see the role of using the BBAT to achieve functional analysis and struggled to fit the PBS plan within their role’. (Trainer 3, male)

A summary of findings from the process evaluation is presented in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Summary of methods and findings from the process evaluation of the PBS-based staff training based on the MRC guidance [21].

https://doi.org/10.1371/journal.pone.0221507.t003

Lessons learnt and future research

The process evaluation provides useful insights into the barriers that might have influenced the delivery of PBS as it is clear that despite the fact that substantial support was provided to therapists the intervention was delivered with low fidelity. In the light of our process evaluation findings, we argue that there is urgent need to tackle the barriers to the implementation of PBS in order to optimise its delivery, which as discussed in the paper were:

• Service workloads/pressures
• Duration of intervention delivery and competing clinical responsibilities
• Paid carer turnover rates
• Insufficient managerial support
• Skill development of therapists in formulating PBS plans
• Role and timing of direct observation of the participant
• Mentoring and supervisory support on site

It is also important to define PBS core components (what can PBS change or influence?) and the impact of other mental health comorbidities which may require specific interventions within its framework in order to improve patient outcomes. For example, a significant proportion of participants in the trial were assessed with such comorbidities; therefore, timely diagnosis and treatment of those conditions is essential to avoid further impairment in adaptive functioning.

Other important issues for consideration include the high paid carer turnover, the resources required for training for the delivery of the intervention if staff undertake this work in addition to core duties, and the supervisory arrangements. Without a comprehensive approach to further investigating the mechanisms of the onset, course and prognosis of challenging behaviour [40–42] training in PBS may only address the needs of a proportion of all people with ID and of their families impacted by challenging behaviour.

It is recognised that often interventions that rely on therapist training and complex systems of delivery can be difficult to sustain long term [43]. This may help explain why many complex interventions that are found to have efficacy in single site phase II trials fail to demonstrate clinical effectiveness when tested at scale [43]. Hence, training staff members in PBS, or train a sole agent to deliver the intervention across services could be associated with long term implementation. This is an important point and although outside the scope of the present study it would need to be considered as a question to be answered in future research.

Future studies examining implementation of PBS should focus on long term sustainability in its delivery, particularly in social care facilities or in family homes. In addition, researchers may need to consider adaptive trial designs that allow for more flexibility in the conduct of a clinical trial [44–46]. Therefore, every effort must be made to ensure that any such schemes translate into high fidelity implementation in routine care. This concern is further borne out by the fact that whilst the stakeholders thought highly of the PBS training and that participants and provider services found the engagement with the therapists and the PBS strategies acceptable and useful, nevertheless, those elements did not improve fidelity. Current work in developing guidance on how to support effectively people with complex needs may also be an important contributor to the questions as to how best to implement such multicomponent frameworks such as PBS which depend on an indirect approach to working with individuals. Practice leadership, which is thought to be essential in good provider support, should be promoted as an enabling factor in ensuring services that are fit for purpose [47]. Whilst there is an important argument to be made about the need to censure high fidelity and improve training standards, there remains to be decided as to whether a less formal PBS approach as is likely to be taking place currently in community ID services, confers enough gains at least for the majority of adults who display challenging behaviour [48].