http://orcid.org/0000-0002-1202-9172
Correction
22 Jul 2020: PLOS ONE Staff (2020) Correction: School-based intervention to address self-regulation and executive functioning in children attending primary schools in remote Australian Aboriginal communities. PLOS ONE 15(7): e0236485. https://doi.org/10.1371/journal.pone.0236485 View correction
Figures
Abstract
Executive functioning and self-regulation influence a range of outcomes across the life course including physical and mental health, educational success, and employment. Children prenatally exposed to alcohol or early life trauma (ELT) are at higher risk of impairment of these skills and may require intervention to address self-regulation deficits. Researchers partnered with the local Aboriginal health organization and schools to develop and pilot a manualized version of the Alert Program® in the Fitzroy Valley, north Western Australia, a region with documented high rates of fetal alcohol spectrum disorder and ELT. This self-controlled cluster randomized trial evaluated the effect of an 8-week Alert Program® intervention on children’s executive functioning and self-regulation skills. Following parent or caregiver consent (referred to hereafter as parent), 271 students were enrolled in the study. This reflects a 75% participation rate and indicates the strong community support that exists for the study. Teachers from 26 primary school classrooms across eight Fitzroy Valley schools received training to deliver eight, one-hour Alert Program® lessons over eight-weeks to students. Student outcomes were measured by parent and teacher ratings of children’s behavioral, emotional, and cognitive regulation. The mean number of lessons attended by children was 4.2. Although no significant improvements to children’s executive functioning skills or behavior were detected via the teacher-rated measures as hypothesized, statistically significant improvements were noted on parent-rated measures of executive functioning and behavior. The effectiveness of future self-regulation programs may be enhanced through multimodal delivery through home, school and community based settings to maximize children’s exposure to the intervention. Despite mixed findings of effect, this study was an important first step in adapting and evaluating the Alert Program® for use in remote Australian Aboriginal community schools, where access to self-regulation interventions is limited.
Citation: Wagner B, Latimer J, Adams E, Carmichael Olson H, Symons M, Mazzucchelli TG, et al. (2020) School-based intervention to address self-regulation and executive functioning in children attending primary schools in remote Australian Aboriginal communities. PLoS ONE 15(6): e0234895. https://doi.org/10.1371/journal.pone.0234895
Editor: Amanda A. Webster, University of Wollongong, AUSTRALIA
Received: May 5, 2019; Accepted: June 4, 2020; Published: June 24, 2020
Copyright: © 2020 Wagner et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: "Data cannot be shared publicly to protect participant privacy due to small population sizes in individual clusters of participants, approved ethical protocols and feedback from local communities involved in this research. The Western Australian Aboriginal Health Council Ethics Committee does not provide approval for the public sharing data. The contact person is Erica Lewin, Aboriginal Health Council of Western Australia, ethics@achwa.org (Research project ref: 601) or Ali Hollingsworth, Research Data Manager, Biometrics, Research Governance and Platforms, Telethon Kids Institute: ali.hollingsworth@telethonkids.org.au.
Funding: This project was funded by the National Health and Medical Research Council project grant (1086145) https://nhmrc.gov.au/. BW is supported by an Australian Government Research Training Program (RTP) Scholarship https://www.education.gov.au/research-training-program. MS received support through the National Health and Medical Research Council FASD Research Australia Centre of Research Excellence (1110341). DC is supported by a National Health and Medical Research Council Research Fellowship (1119339). Community researcher training was supported by a philanthropic grant from 100 Women https://100women.org.au/. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The ability to regulate emotion, behaviour, and thought relies on a set of integrated brain processes known as the executive functions. These top-down (deliberate) cognitive functions are commonly believed to include inhibition, working memory, and cognitive flexibility [1, 2]. Additional higher-order executive functions are thought to include planning and reasoning [1]. Working together, these cognitive processes support individuals to purposely solve problems and function adaptively throughout life [3]. Self-regulation is a related concept [4, 5], referring to the volitional ability of a person to modify or maintain arousal states [6], emotions, thoughts, or behaviors appropriate to a situation [7]. Although research to define the exact association between executive functioning and self-regulation continues [7], a bidirectional relationship is thought to exist [8]. Firstly, the executive functions underlie self-regulation [9], meaning an individual employs top-down cognitive processes to gain control over their emotions, behaviors, and thoughts [7, 10]. Secondly, bottom-up self-regulatory processes enable top-down executive functioning. That is, the body receives and processes sensory information to modulate arousal, emotion, and attention, which impacts executive functioning [7, 8]. Ultimately, executive functioning and self-regulation influence a range of outcomes across the life course, including mental and physical health, educational success, and employment [11–13].
The nature of schooling demands constant use of children’s executive functioning and self-regulatory abilities [14]. For example, students must follow instructions, apply classroom rules, solve maths problems, comprehend texts, get along with peers and teachers, remember equipment, and transfer knowledge between situations. Without well-developed executive functioning and self-regulation skills, students can struggle with these basic school tasks. In these instances, teachers may view students as having problematic or disruptive behavior [15, 16]. Additionally, impairment in executive functioning and self-regulation amongst students can lead to disengagement from school, student-teacher conflict, poor academic achievement, and disrupted learning for other students [15, 17]. Deficits in executive functioning and self-regulation are widely reported as ‘hallmarks’ of fetal alcohol spectrum disorder (FASD) [18–20], a pervasive neurodevelopmental disorder arising from prenatal alcohol exposure [21]. Children’s exposure to adverse childhood experiences (ACEs), including stress and poverty, can also impair executive functioning and self-regulation development [22]. ACEs are sometimes referred to as early life trauma (ELT) [23]. Given evidence of the malleability of the executive functions and self-regulation [9], a key opportunity exists to develop these skills amongst children with or at risk of dysfunction.
Aboriginal leaders from the remote Fitzroy Valley region of Western Australia (WA) have been concerned about the potential of FASD and exposure to ELT to impact the behaviours of children in their local community. In 2009, local leaders developed a multifaceted strategy called the Marulu FASD/ELT Strategy (Marulu means ‘precious, worth nurturing’ in the Bunuba language) [24]. This strategy prioritized three aims: FASD prevention and diagnosis, and appropriate support for children and families affected by FASD and ELT. As part of this strategy, Fitzroy Valley leaders partnered with researchers to conduct Australia’s first FASD prevalence study otherwise known as the Lililwan Study (Lililwan means ‘all the little ones’ in Kimberley Kriol) [25]. In this study, 108 Fitzroy Valley children born between 2002 and 2003 were assessed. This study found a FASD prevalence of 194 per 1,000 school-aged children [26], this rate is considered extremely high. In contrast, global FASD prevalence has been reported as 7.7 per 1000 children [27]. Interestingly, of all the children assessed in this study, 30% were identified as having executive functioning impairment (albeit through brief assessment) and 26% as having behavioral problems [26, 28]. Preliminary Lililwan Study data also indicated 90% of children had been exposed to ELT with over 50% of children experiencing greater than three different ELTs [26]. There are significant stressors for Fitzroy Valley children and families arising from poverty, family violence, limited access to fresh food and transport, overcrowded and insecure housing, chronic illness, and separation from parents [25]. The 2005 Western Australian Aboriginal Child Health Survey also reported that 22% of carers of children living in the region had experienced seven or greater ‘life stress events’ within their family [29]. Therefore, in addition to the high prevalence of FASD, exposure to ELT may have contributed to the high rates of executive functioning and behavioral problems identified amongst children in the Lililwan Study [26].
As a result, community members, including school staff, requested training and support to address disruptive student behavior as a means to improve children’s learning [25]. However, the geographical isolation of Fitzroy Valley communities meant that available pediatric therapies were limited. To overcome service barriers, researchers, Fitzroy Valley communities, and schools together identified the Alert Program® as a feasible and appropriate program to improve students’ executive functioning and self-regulation skills [30].
The Alert Program®, developed by occupational therapists, teaches children about self-regulation through the analogy of a car engine. Children are taught that ‘just like a car engine, our bodies can be in a high, low or just right speed’ [6]. Through this analogy, children learn to recognize their own arousal states before learning to use strategies and tools from five sensorimotor categories (mouth, body, touch, look, listen) to help change or maintain an optimal level of arousal (engine/alertness levels) for different situations [6]. Upskilling of adults in key Alert Program® concepts is also a core aspect of the program [31].
The Alert Program® has previously been adapted and trialled in schools [32–34] and FASD clinic settings [35–38], with some reported improvements to children’s executive functioning and self-regulation abilities on a range of standardized and non-standardized measures [32, 34, 35, 37–39]. However, until this research project, no Australian Alert Program® studies, or studies with predominantly Indigenous participants, have been reported [40]. Therefore, this study aimed to assess the efficacy of an 8-week Alert Program® intervention, delivered by classroom teachers to children attending primary school in remote Australian Aboriginal settings, on students’ executive functioning and self-regulation skills. Student outcomes were measured by teacher and parent ratings of their behavioral, emotional, and cognitive regulation.
Methods and materials
A detailed protocol for this study was prospectively published and registered with the Australian New Zealand Clinical Trials Registry (ACTRN12615000733572) [41]. A detailed description of the development and content of the intervention training and delivery has also been reported elsewhere [30, 41, 42].
Ethics approvals
Ethics approvals were provided by The University of WA (RA/4/1/7234), the WA Aboriginal Health Ethics Committee (601), and the WA Country Health Service (2015:04). The study was also approved by the WA Department of Education and the Kimberley Aboriginal Health Planning Forum Research Sub-Committee.
Community collaboration
This study addressed a community identified need to support the development of adaptive student behaviors in the Fitzroy Valley schools. Prior to this study, researchers undertook a year-long formative process in partnership with local leaders (MC, EW), stakeholders, families, schools, and Aboriginal community researchers to develop culturally secure and contextually feasible research processes and protocols [30]. During a subsequent pilot study, researchers gathered feedback from school staff on the suitability of training, lessons, and resources for the remote school context [30, 42]. Guided by this feedback, researchers refined the intervention in consultation with the original Alert Program® authors. Results of the formative stage of this project and results of the pilot study have been published elsewhere [30, 42].
Community researchers
Recognising the cultural diversity of the region, 18 Aboriginal community members from the four main Fitzroy Valley language groups were employed to provide essential expertise in relationship brokerage, cultural protocols, language translation, and general research assistance as community researchers [43, 44]. The two-way partnerships fostered between Aboriginal and non-Aboriginal research staff were vital for ensuring cultural safety for participants and researchers alike [45].
Setting
The study took place at four government and four independent community schools in the remote Fitzroy Valley. This culturally diverse region is home to approximately 3,500 mainly Australian Aboriginal people, from the Bunuba, Walmajarri, Gooniyandi, and Wangkatjungka language groups [46]. Kimberley Kriol and local Aboriginal languages are commonly spoken, with English an additional language for many. Residents live across 45 distinct Aboriginal communities spread over an area of 40,000 square kilometres [47]. The remoteness of schools and communities means access to facilities such as grocery stores, health services, recreation facilities, and social services is limited. Each day, children may travel up to 100 kilometres one way from their home community to attend their closest school.
Study design
This study was a self-controlled cluster randomized trial in which primary school classrooms in each cluster of schools received the intervention at one of four time-points (school terms two or three, 2016 or 2017).
Randomization and cluster assignment
Schools were assigned by researchers to one of four clusters based on geographic proximity and school size. This responded to the challenge researchers experienced travelling vast distances to collect data from participants in different schools within the same cluster at varying calendar times. Pre-assigning schools to clusters also ensured a relatively even distribution of participants across each cluster, given that school size varied from six to 110 students. Clusters included one to three schools. Randomization to the intervention schedule occurred at the cluster level as determined by a computer-generated randomization pattern.
Inclusion criteria
All students enrolled in grades (years) one to six (5.5–12.5 years of age) at any of the eight study schools at the time of recruitment in their cluster were eligible to participate in the study with written parent consent. All teaching staff with responsibility for delivering the Alert Program® were also eligible to participate in the study with informed written consent.
Sample size
The available study population was 363 students [48, 49]. Community leaders requested that all eligible Fitzroy Valley families be invited into the study. Given the non-standard study design, sample size estimates were calculated using standard methods for a cross-over randomized control trial with 60 participants per cluster across time-points with 80% power to detect an effect at the 0.05 significance level [41].
Recruitment and consent
Initial approval to conduct the study was provided by community Elders and school principals following a series of engagement visits to the Fitzroy Valley [30]. Research assistants and Aboriginal community researchers then visited families together to explain the purpose of the study and to seek written consent for their child(ren)’s involvement. Community researchers provided verbal translation of study materials into Kimberley Kriol or other Aboriginal languages when required. Separate consent items included: (1) collection of student outcome data; (2) access to students’ medical records and students’ school attendance records; and (3) permission to take photos and videos of students during Alert Program® lessons. Researchers obtained informed written consent from teachers following a school staff information session. For this study, teachers consented to completing teacher-rated questionnaires for participating students and to record lesson information and student attendance data.
Alert Program® intervention
While a detailed description of the intervention has been described elsewhere, briefly, all school staff, including Aboriginal and Torres Strait Islander education officers (AIEOs), were invited to attend two one-day training sessions facilitated by the study coordinator [30]. It was important that AIEOs attended the training as they work in two-way teams with classrooms teachers to support teaching and learning activities including the translation of instructions or concepts from English to Kimberley Kriol for students when required. Training session one occurred before teachers delivered the first Alert Program® lesson, and training session two occurred between weeks three and four of program delivery. The training covered the history and purpose of study, and general information about FASD and executive functioning [30]. Staff also viewed the Alert Program® Online Course modules one to five, and had the opportunity to ask questions during the two sessions. The Alert Program® Leader’s Guide, lesson manual, equipment boxes, and other resources (such as picture cards and posters) required for lesson delivery were provided to teachers, who were encouraged to contact the study coordinator if they had further queries pertaining to lesson delivery or use of resources.
Classroom teachers were asked to deliver a one-hour manualized Alert Program® lesson to all students in their class, once a week, for eight consecutive weeks (8 hours in total). The intervention was designed by adapting Stage One and Two of the original three stage Alert Program®. Stage Three relates to independent and ongoing use of sensorimotor strategies by participants [6]. The resource intensive nature of this study, given participating schools were widely spread, meant that the longer-term follow-up necessary for evaluating the effectiveness of Stage Three was not possible. Therefore it was decided to deliver this adapted version of the Alert Program®. The intervention dose used in this study reflects the dose delivered in other studies evaluating the effectiveness of the Alert Program® in schools, and therefore was considered a justifiable starting point [32, 34]. During lessons one to three, students developed an awareness and feel for arousal levels in the body and practiced using the engine language. During lessons four to eight, students experimented with strategies from the five sensorimotor categories to change or maintain their arousal levels (engine/alertness levels) [6]. Teachers were also encouraged to embed program language and strategies throughout the school day [30].
Outcome measures
Sutter-Eyberg Student Behavior Inventory-Revised (SESBI) and Eyberg Child Behavior Inventory (ECBI).
The 38-item SESBI is a teacher-rated measure of student disruptive behavior at school, while the 36-item ECBI is a similar measure of child behavior at home [50]. Both questionnaires were used as proxy measures of problems in self-regulation. The SESBI and ECBI include two scales: the Intensity Scale, which measures frequency of disruptive behaviors on a scale from 1 ‘never’ to 7 ‘always’; and the Problem Scale, in which respondents rate each behavior as being a problem, 0 ‘no’ or 1 ‘yes’. A reduction in scores on a scale indicates improvement. Both the SESBI Intensity and Problem scales have high internal consistency (α = .98, α = .96) and high test-retest coefficients (r = .87, r = .93) [50]. Internal consistency and test-retest coefficients for the ECBI Intensity and Problem scales are also high (α = .95, α = .93; r = .80, r = .85) [50]. The SESBI Intensity Scale was chosen as the primary outcome for the current study.
Behavior Rating Inventory of Executive Function 2 Teacher and Parent Screening Forms (BRIEF2-SF).
The 12-item teacher-rated and parent-rated BRIEF2-SF comprise overall measures of global executive function that include cognitive, behavioral, and emotional regulation items [51]. Items are rated as occurring 1 ‘never’, 2 ‘sometimes’, or 3 ‘often’ and yield a score indexing the overall level of executive functioning. Lower scores indicate fewer executive functioning difficulties. Both the teacher and parent versions of the BRIEF2-SF have high internal consistency (α = .91, .89), test-retest reliability (r = .87, .79), and correlate strongly with the full BRIEF2 Global Executive Composite Score (r = .95, .96) [51]. Parents were not asked two of the cognitive regulation items from the BRIEF2-SF (Q6 and Q7), as pilot testing deemed these questions irrelevant to parents’ experiences in these remote communities [30]. Instead, excluded questions were assigned a score determined by pro-rating the remaining three cognitive regulation items, as suggested by the BRIEF2-SF developers (P. Isquith, personal communication, October 14, 2015).
In the context of the current study, the BRIEF2-SF was used as a secondary outcome measure. The screening form was chosen because of the battery of measures being used, and the need to reduce the burden for respondents by using a screening form measure. This was considered acceptable given that while the level of detail about specific profiles of executive function is limited in the screening form, it still provides a sound estimate of global executive function and is strongly correlated with the full BRIEF2 Global Executive Composite Score [51].
Data collection
Data were collected over two-week periods per the timeline in Table 1 [41]. For the control condition, data were collected for clusters two, three and four, and matched the timing of the preceding intervention cluster. Cluster one did not have a control period for several reasons: data collection was not feasible in school term one (February–March) due to cultural and school business, and the likelihood of seasonal flooding causing road closures. Also, given this was the beginning of the school year, teachers may have had insufficient knowledge of the student prior to be able to accurately complete questionnaires. For the intervention condition, data were collected immediately before (pre) and after the intervention (post), and at eight to nine-weeks follow-up (follow-up).
Teachers completed questionnaires independently. Parents completed questionnaires during a series of home visits by an Aboriginal community researcher and a research assistant. As many parents speak English as a second, third or fourth language, the Aboriginal community researchers translated questionnaires into written Kimberley Kriol and provided verbal translation into the local Aboriginal language when necessary.
Data collection using the questionnaires unavoidably overlapped with the first and last weeks of the intervention for some students. This occurred because some teachers and parents required additional time to complete the questionnaires and the unavailability of teachers during the school holidays meant that data collection could not be extended into the school vacation. Given education stakeholders had recommended the intervention be completed over one school term (ten-week period), data collection for some students needed to occur in the first and last week of the term when the intervention had just started or was finishing. All attempts were made to obtain data from a student’s original school if they moved between clusters during the study.
Demographics
Children’s demographic information including date of birth, sex, languages spoken at home, and Aboriginal and/or Torres Strait Islander status was collected from parents.
Identification of children with a diagnosis of FASD
Screening for and diagnosis of FASD were not part of this study. Review of participants’ electronic Department of Health medical records with parent consent identified children with a previous diagnosis of FASD. Due to the extremely limited FASD diagnostic services available in the region, it is possible that there were more affected children in the study than just those with a previously confirmed diagnosis. Children who performed poorly on study measures were subsequently referred to the local occupational therapy team for further follow-up.
Lesson implementation and student attendance
Lesson implementation and student attendance were recorded by teachers in the study lesson manual. Lessons were coded as implemented if teachers recorded the date next to the relevant lesson with missing lesson implementation data verified with individual teachers. Students were marked as present or absent for individual Alert Program® lessons. If these data were missing, individuals’ daily school attendance was requested from the school.
Blinding
Blinding of teachers, parents, and researchers was not possible because of the nature of the intervention and data collection methods.
Data analysis
All paper records were entered into Qualtrics, an online survey platform. Ten percent of all paper records and data extracted from medical records were checked for data entry errors. The mean error rate across outcomes was low (0.25%). Data were cleaned and coded in IBM SPSS version 25. Descriptive statistics were produced in SPSS to summarise participant demographics per cluster, lesson implementation and attendance information, and to inspect outcome data.
Outcome data were analyzed per the study protocol [41]. Participants were eligible for inclusion in analyses if their data were available for at least one pre- and post-intervention timepoint. No participants were excluded from analyses based on intervention exposure in accordance with intent-to-treat principles of study design. The primary outcome was change in students’ SESBI Intensity Scale scores, which was analyzed using a generalized linear mixed-effects model (GLMM) and assessed at the 5% significance level. Fixed effects included intervention, sex, age, and calendar time. To detect a possible intervention lag effect, intervention was coded as 0 (pre), 1 (post), and 2 (follow-up). Calendar time was coded 1–10 (Table 1) and was included to adjust for possible secular trends [52]. Random effects for individuals and individuals nested within clusters were also included in addition to an exchangeable correlation structure. All secondary outcomes were analyzed similarly with adjustment for multiple testing using the false discovery rate as described by Benjamini and Hochberg [53]. GLMM analyses were performed in Stata version 15.1.
SESBI and ECBI Intensity Scale scores were analyzed using the ‘meglm’ command, specifying a gamma error distribution, to account for an anticipated skewed error distribution [41]. Where mixed-effects negative binomial models were found to be non-significant, SESBI and ECBI Problem Scale and BRIEF2-SF scores were analyzed using the ‘mepoisson’ command. Over dispersion was evaluated in these analyses using mixed-effects models with negative binomial error distributions (‘menbreg’ command). Predicted means and proportions were estimated directly from the fixed and random effects.
Sensitivity analyses
Sensitivity analyses were performed for students who attended five or more Alert Program® lessons to determine the impact of intervention exposure on all student outcomes. To determine the difference in outcomes for children with and without FASD, separate sensitivity analyses was performed by including FASD diagnosis as a covariate in the models. However, an interaction between the intervention and FASD could not be modelled given the small number of children with a confirmed FASD diagnosis. Both sensitivity analyses were assessed at the 5% significance level and included the same random and fixed effects as previously described.
Results
Participants
Of the 363 students identified from 2015 school enrollment data as potentially eligible for participation in the study, 271 students (75%) participated in the research. This reduction in number was due to the fact that the actual age of children sometimes varied from school enrolment data making them ineligible (n = 33; Fig 1). Following initial entry of their child into the study, a further 21 parents withdrew consent as they no longer wanted to complete the battery of questionnaires. The number of participants lost to follow-up varied across data collection time-points. For the primary outcome measure, reasons included the teacher being uncontactable or not returning the questionnaires. Participants were eligible for inclusion in analyses if their data were available for at least one pre- and post-intervention time-point. Therefore, the final number of participants included in the primary data analysis (SESBI Intensity Scale) was 230. For reporting purposes, clusters were randomly assigned a new identifying code to preserve anonymity of the school clusters.
Participants’ mean age at the time of consent was 8.7 (range 5.8–12.1 years). English and Kimberley Kriol were spoken by 77% and 71% of students at home respectively while 60% of students spoke two or more languages at home. Additional participant characteristics by cluster are presented in Table 2. Following parent consent, researchers accessed lesson attendance data for 264/271 students (97%) and medical records for 239/271 students (88%). The medical record review identified twelve students (5%) with a documented FASD diagnosis, a lower than expected rate. The mean number of Alert Program® lessons attended by students was 4.2 of eight possible lessons (SD 2.6). Fifty-one percent of students (135/264) attended five or more lessons (Fig 2). Lesson implementation per classroom ranged between 81 and 100%.
Primary outcome.
SESBI intensity Scale score. Children’s SESBI Intensity Scale scores (n = 230), did not change significantly from pre- to post-intervention or pre-intervention to follow-up (Table 3).
Secondary outcomes–teacher questionnaires.
Children’s SESBI Problem Scale scores (n = 220), significantly decreased by a factor of 0.59 from pre- to post-intervention (Table 3). However, SESBI Problem Scale scores significantly increased from pre-intervention to follow-up by a factor of 1.98. This indicates that teachers rated some students’ behavior as less of a problem post-intervention, but as more of a problem at follow-up. Both results remained significant following adjustments for multiple testing. There were no significant differences for the teacher-rated BRIEF2-SF scores (n = 231) from pre- to post-intervention, or pre-intervention to follow-up.
Secondary outcomes–parent questionnaires.
Children’s ECBI Intensity (n = 209) and Problem Scale (n = 206) scores, increased significantly from pre- to post-intervention by factors of 1.07 and 1.31, respectively (Table 3). However, the Intensity scale result was no longer significant following adjustment for multiple testing. There were no significant changes in parent-rated BRIEF2-SF scores (n = 209) at post-intervention compared to pre-intervention. Compared to pre-intervention, at follow-up the parent-rated ECBI Intensity and Problem Scales, and parent-rated BRIEF2-SF scores, were significantly lower by factors of 0.80, 0.32 and 0.89, respectively. This demonstrates that some parents observed improvements in their children’s behavior and executive functioning at follow-up compared to before the Alert Program® intervention commenced. All three results remained significant following adjustment for multiple testing.
Sensitivity analyses
Five or more Alert Program® lessons.
A sensitivity analysis comparing the effect of the intervention in children who received the full intervention dose compared to those who did not could not be performed due to the small number of children who received the full intervention dose. However, a sensitivity analysis was performed comparing the effect of the intervention in children receiving five or more Alert Program® lessons compared to those who did not. These results are presented in Table 4. The results are similar to results of the primary and secondary outcome analyses (Table 3). Interestingly the one exception was parent-rated ECBI Intensity Scale scores which increased significantly from pre- to post-intervention by a factor of 1.12.
FASD diagnosis
Results of sensitivity analyses performed by including FASD diagnoses as a covariate for both teacher and parent-rated outcomes were consistent with results from models that excluded FASD as a covariate (Table 3).
Discussion
This study evaluated the effectiveness of an 8-hour teacher-delivered Alert Program®, manualized and adapted to the local context, that aimed to improve self-regulation and executive functioning skills of primary school aged students in the remote Fitzroy Valley. Effect was measured using teacher-rated and parent-rated outcomes. Importantly, the study resulted from a locally identified need for a practical, contextually relevant intervention to support children’s behavioral, cognitive, and emotional regulation. High reported rates of self-regulation impairment, exposure to ELT and alcohol prenatally, had been identified in a previous study reporting FASD prevalence.
While the results did not fully support the study hypotheses, the Alert Program® reduced the frequency of children’s disruptive behavior (as a proxy for improved self- regulation) and improved executive functioning skills, as measured by parent-rated standardized outcome measures. It is promising, that teachers rated student behaviors as significantly less of a problem immediately after the Alert Program® had finished despite reporting that the frequency of school behavior problems remained unchanged. This may reflect increased self-efficacy of teachers to manage problem behaviors, despite behaviors continuing among children. Providing information to teachers about high local FASD prevalence and rates of executive functioning impairment, coupled with training in self-regulation concepts, may have altered teacher perceptions of child behavior and executive functioning during treatment [54, 55]. Teachers may have been more understanding or tolerant of certain behaviors or felt better equipped to manage problem behaviors throughout the intervention period. However, these results were not maintained at follow-up. This could indicate that to sustain improvement, teachers require additional training or coaching to continue embedding Alert Program® language and strategies throughout the school day after completion of formalized lessons.
The use of ecological measures (SEBSI, ECBI and BRIEF2-SF) may partly explain the variability of results between teacher and parent ratings of student behavior and executive functioning. While ecological measures can assess the everyday meaning and presentation of behavioral problems specific to real-world contexts such as the classroom and home [56, 57], the objective measurement of these can be influenced by the raters’ own ideals, beliefs, and attitudes [57]. Many of these beliefs are influenced by differences between Aboriginal and Western cultural norms [58, 59] and because of this teachers and parents may over- or underestimate poor child behavior compared to each other. Factors such as self-efficacy of caregiving skills or the perception of ones’ teaching or parenting abilities being scrutinized, may have influenced ratings. This is especially pertinent given the absence of ecological executive functioning and self-regulation assessments normed for use with Australian Aboriginal children [58, 60].
Varying student attendance may have impacted results given the strong association between behavioral problems and low school attendance reported in the literature [61, 62]. Only 50% of student study participants received more than half of the Alert Program® lessons; a much lower rate than other Alert Program® studies that have reported intervention exposure [33, 37]. The small sample of participants who received exposure to the full intervention dose meant sensitivity analyses could not be performed for this group. However, sensitivity analyses performed for children who received five or more Alert Program® lessons revealed similar findings for teacher and parent-rated outcomes as those emerging from the primary and secondary outcome analyses.
It is possible therefore that children who stood to benefit the most from the program were the students who received the least exposure given their absence from school. This is one possible explanation for why significant changes were not detected via teacher-rated outcome measures. School attendance in the Kimberley region, which encompasses the Fitzroy Valley, is typically the lowest of any Western Australian school district [63]. Reasons for this include family travel to surrounding communities to attend funerals, medical appointments, and cultural events. The multiple biopsychosocial influences on child behavior and development in a remote community setting may require multilayered interventions. It is possible that a single intervention while necessary, was not sufficient to demonstrate a measurable effect. Any future efforts to identify or design contextually appropriate multi-model interventions must be done so in partnership with local communities and their stakeholders.
One real-world, positive outcome from this study was the provision of extra teacher support and training as requested by communities and schools during the Lililwan FASD prevalence study. School staff received support to manage difficult classroom behaviors [25] through training in self-regulation and Alert Program® concepts. To ensure cultural congruence the training, intervention, and resources were designed in partnership with local occupational therapists, then piloted at a local school, and further refined based on local teacher feedback prior to this study [30]. The whole-of-school and region approach to Alert Program® teacher training and classroom delivery meant children could receive immediate access to strengths-based classroom strategies and accommodations which was identified as a community priority [64].
Another positive outcome of study implementation was community acceptability of the study, and responsiveness to expressed community needs. Study acceptability was made possible through the existing relationships many research team members had established with, or as members of, the Fitzroy Valley Marulu FASD/ELT Strategy leadership group. Several researchers also had enduring relationships with Aboriginal community leaders, health, and education stakeholders through their previous or current roles working for local Aboriginal organisations, schools, and health agencies. These relationships enabled researchers to partner with key stakeholders, including the local Aboriginal cultural health service, allied health staff, and schools, to develop and refine the intervention, outcome measures, and other study protocols [30].
Although findings were mixed, a highly valuable project outcome was the employment and training of 18 local Aboriginal community researchers from various Fitzroy Valley communities to provide invaluable expertise in cultural protocols, language translation, relationship facilitation between researchers and families, and data collection. A separate capacity building grant also enabled several community researchers to complete a nationally recognised Certificate II in Community Services to support employment opportunities following the study. Non-Aboriginal researchers also received vital cultural guidance and training from community researchers, Aboriginal organisations, and Elders to better understand Indigenous ways of knowing and doing, and to help them work in a culturally secure way. Without the local knowledge and cultural guidance of the community researchers or local stakeholders, it is unlikely the project would have been feasible given the challenges of conducting multi-site research in remote community schools.
Limitations
The challenges of intervention research in remote communities are many and varied, even when an experienced research team that includes many Aboriginal community researchers puts measures in place to address these potential limitations. The findings of the current study therefore, must be interpreted in the context of several important limitations. As described elsewhere [41], the stepped-wedge cluster randomized control trial design originally proposed was not feasible in the remote community context. This necessitated a non-standard study design and meant that sample size calculations could only be estimated conservatively. While it is possible the study was underpowered for some outcomes, the largest possible number of participants were recruited from the entire available study population. A repeated measures design was employed to increase the likelihood of detecting any significant effects of the intervention on student outcomes. Surprisingly, despite the very high reported rates of FASD in the Fitzroy Valley, and substantial efforts to obtain information and refer children for diagnosis, only 12 students were identified with a confirmed FASD diagnosis during the Department of Health medical record review. This impacted the study’s ability to detect an interaction between diagnosis of FASD and impact of the intervention. Screening for prenatal alcohol exposure and subsequent FASD diagnosis as part of this study was not feasible due to budget constraints and the limited local diagnostic capacity. Given FASD prevalence in the Fitzroy Valley was established as 194 per 1000 children [26], it is likely that many children in the current study had undiagnosed FASD. Should this be the case, future efforts to identify effective interventions for children in regions such as the Fitzroy Valley may be limited.
Blinding of teachers, parents, and researchers to the intervention was not possible given the nature of the intervention and how the data needed to be collected. It is recognised that outcomes of non-blinded studies may be biased with a risk of inflating the likelihood of a type I error. Unblinded teachers and parents may have taken greater note of target behaviors at the post and follow-up data collection time-points, which could have affected the results.
In some instances, different teachers and parents completed questionnaires for the same student across time-points for reasons that included teacher absences or resignations, and parent availability. This may have impacted the results. The frequency of respondents changing was not recorded and is a limitation of this study.
There was an unavoidable overlap between the first and last Alert Program® lessons and pre- and post-intervention data collection, to enable sufficient time for teacher and parent questionnaires to be administered within the school term. Therefore, some students may have received exposure to the Alert Program® prior to completing the pre-intervention outcome measures, or may not have received the full intervention dose prior to completing the post-intervention outcome measures. Implementation fidelity of lesson activities is unknown because some parents declined consent for researchers to film lessons. A random audit using film was not possible.
Conclusions
While this study demonstrated no effect of the 8-week Alert Program® intervention on children’s disruptive behaviours as measured by the primary outcome, it provides important information for future research evaluating urgently needed self-regulation interventions for Aboriginal Australian children living in remote communities. The lack of effect demonstrated in this study must be viewed cautiously and may be due to many factors. These include that the program as delivered is ineffective, but also that the limitations of the study meant that we failed to find an effect when one existed. This may be due to the complex setting in which this work was conducted including the lack of validated measures available for use in Aboriginal populations.
Future research is urgently needed to identify, in consultation with local communities, effective self-regulation programs for supporting teachers, students, and their families to manage problematic behaviours that prevent students from achieving success at school. Despite mixed findings, this study was an important first step in adapting and evaluating one available self-regulation program for use in remote Australian Aboriginal community schools where access to evidence based interventions is extremely limited.
Acknowledgments
The authors gratefully acknowledge the Fitzroy Valley students, families, school staff and communities who were involved in this study. Thank you to the study’s chief investigators and project team members who contributed to this work; Professor Karen Edmond, Glenn Pearson, Professor Branko Celler, Professor Jane Valentine, Gayle Segar, Jacinta Freeman, Emma Douglas, and Kaashifah Bruce. We thank Sue Cherel (lead community researcher) and the team of Fitzroy Valley Aboriginal community researchers–Tamika Shaw, Maria Cox, Cynthia Cox, Lurlene Dawson, Kerry-anne Hobbs, Francine Jack, Rebecca Smith, Danielle Munday, Jennifer Jackie, Chantelle Berringal, Jeanelle Smith, Janice Moora, Samantha Ahfat, Irene Passenger, and Marissa Williams—for generously sharing their knowledge and expertize with the overall project team. Thank you to Dr Julie Marsh and Dr Therese Shaw who provided statistical advice. Thank you to the large team of volunteers, students, and Alcohol and Pregnancy & FASD Research Team colleagues who supported collection and entry of data. The authors also acknowledge WA Health, WA Department of Education, Nindilingarri Cultural Health Service and Marninwarntikura Fitzroy Women’s Resource Centre for their overall study support.
References
- 1.
Diamond A. Why improving and assessing executive functions in early life is critical. 2016 [cited 2018 March 5]. In: Executive function in pre-school aged children: integrating measurement, neurodevelopment, and translational research [Internet]. Washington DC: American Psychological Association, [cited 2018 March 5]; [11–43]. Available from: https://ovidsp-tx-ovid-com.ezproxy.library.uwa.edu.au/sp-3.32.2a/ovidweb.cgi.
- 2. Miyake A. Individual differences in executive functions: Unity/diversity framework and recent research findings. International Journal Of Psychology. 2016;51(s1):1032–.
- 3. Zhou Q, Chen SH, Main A. Commonalities and differences in the research on children’s effortful control and executive function: a call for an integrated model of self-regulation. Child Development Perspectives. 2012;6(2):112–21.
- 4. Bridgett DJ, Oddi KB, Laake LM, Murdock KW, Bachmann MN. Integrating and differentiating aspects of self-regulation: effortful control, executive functioning, and links to negative affectivity. Emotion (Washington, DC). 2013;13(1):47–63.
- 5. Hofmann W, Schmeichel BJ, Baddeley AD. Executive functions and self-regulation. Trends Cogn Sci. 2012;16(3):174–80. pmid:22336729
- 6.
Williams MS, Shellenberger S. "How does your engine run?" A leader's guide to the Alert Program for self-regulation. Albuquerque, NM: TherapyWorks, Inc.; 1996.
- 7. Nigg JT. Annual research review: on the relations among self-regulation, self-control, executive functioning, effortful control, cognitive control, impulsivity, risk-taking, and inhibition for developmental psychopathology. J Child Psychol Psychiatry. 2017;58(4):361–83. pmid:28035675
- 8. Ursache A, Blair C, Raver CC. The promotion of self‐regulation as a means of enhancing school readiness and early achievement in children at risk for school failure. Child Development Perspectives. 2011;6(2):122–8.
- 9.
McClelland MM, Geldhof J, Morrison F, Gestsdóttir S, Cameron C, Bowers E, et al. Self-Regulation. 2018 [cited 2018 February 15]. In: Handbook of life course health development [Internet]. Cham: Springer International Publishing, [cited 2018 February 15]; [275–98]. Available from: https://link-springer-com.ezproxy.library.uwa.edu.au/content/pdf/10.1007%2F978-3-319-47143-3_12.pdf.
- 10.
Barkley R. Executive functions: What they are, how they work, and why they evolved. New York: Guilford Press; 2012.
- 11. Moffitt TE, Arseneault L, Belsky D, Dickson N, Hancox RJ, Harrington H, et al. A gradient of childhood self-control predicts health, wealth, and public safety. Proceedings of the National Academy of Sciences. 2011;108(7):2693.
- 12. Blair C, Razza R. Relating effortful control, executive function, and false belief understanding to emerging math and literacy ability in kindergarten. Child Dev. 2007;78:647–63. pmid:17381795
- 13. Snyder HR, Miyake A, Hankin BL. Advancing understanding of executive function impairments and psychopathology: bridging the gap between clinical and cognitive approaches. Front Psychol. 2015;6:328–. pmid:25859234
- 14. McClelland MM, Cameron CE. Self‐regulation in early childhood: improving conceptual clarity and developing ecologically valid measures. Child Development Perspectives. 2012;6(2):136–42.
- 15. Portilla XA, Ballard PJ, Adler NE, Boyce WT, Obradović J. An integrative view of school functioning: transactions between self-regulation, school engagement, and teacher-child relationship quality. Child Dev. 2014;85(5):1915–31. pmid:24916608
- 16. Sulik MJ, Blair C, Mills‐Koonce R, Berry D, Greenberg M. Early parenting and the development of externalizing behavior problems: longitudinal mediation through children's executive function. Child Dev. 2015;86(5):1588–603. pmid:26082032
- 17. Blair C, Diamond A. Biological processes in prevention and intervention: the promotion of self-regulation as a means of preventing school failure. Dev Psychopathol. 2008;20(3):899–911. pmid:18606037
- 18. Mattson SN, Crocker N, Nguyen TT. Fetal alcohol spectrum disorders: neuropsychological and behavioral features. Neuropsychol Rev. 2011;21(2):81–101. pmid:21503685
- 19. Kodituwakku PW, Handmaker NS, Cutler SK, Weathersby EK, Handmaker SD. Specific impairments in self-regulation in children exposed to alcohol prenatally. Alcoholism: Clinical and Experimental Research. 1995;19(6):1558–64.
- 20. Rasmussen C, Horne K, Witol A. Neurobehavioral functioning in children with fetal alcohol spectrum disorder. Child Neuropsychol. 2006;12(6):453–68. pmid:16952890
- 21. Bower C, Elliott EJ, Zimmet M, Doorey J, Wilkins A, Russell V, et al. Australian guide to the diagnosis of foetal alcohol spectrum disorder: a summary. J Paediatr Child Health. 2017;53(10):1021–3. pmid:28975754
- 22. Blair C. Stress and the development of self‐regulation in context. Child Development Perspectives. 2010;4(3):181–8. pmid:21779305
- 23. Westfall NC, Nemeroff C. The preeminence of early life trauma as a risk factor for worsened long-term health outcomes in women. Current Psychiatry Reports. 2015;17(11).
- 24. Fitzpatrick JP, Oscar J, Carter M, Elliott EJ, Latimer J, Wright E, et al. The Marulu Strategy 2008–2012: overcoming fetal alcohol spectrum disorder (FASD) in the Fitzroy Valley. Aust N Z J Public Health. 2017;41(5):467–73. pmid:28749590
- 25.
Latimer J, Elliott EJ, Carter M, Oscar J, Fitzpatrick JP, Ferreira M, et al. Marulu, the Lililwan project, fetal alcohol spectrum disorders (FASD) prevalence study in the Fitzroy Valley: a community consultation. Sydney: The George Institute for International Health; 2010.
- 26. Fitzpatrick JP, Latimer J, Olson HC, Carter M, Oscar J, Lucas BR, et al. Prevalence and profile of neurodevelopment and fetal alcohol spectrum disorder (FASD) amongst Australian Aboriginal children living in remote communities. Res Dev Disabil. 2017;65:114–26. pmid:28499185
- 27. Lange S, Probst C, Gmel G, Rehm J, Burd L, Popova S. Global prevalence of fetal alcohol spectrum disorder among children and youth: a systematic review and meta-analysis. JAMA pediatrics. 2017;171(10):948–56. pmid:28828483
- 28. Dossetor PJ, Martiniuk ALC, Fitzpatrick JP, Oscar J, Carter M, Watkins R, et al. Pediatric hospital admissions in Indigenous children: a population-based study in remote Australia. BMC Pediatr. 2017;17(195).
- 29.
Zubrick S, Silburn S, Lawrence D, Mitrou F, Dalby R, Blair E, et al. The social and emotional wellbeing of children and young people: Malarabah (Derby) ATSIC region. Summary of findings from volume two of the Western Australian Aboriginal Child Health Survey Perth Telethon Institute for Child Health Research and Curtin University of Technology; 2005.
- 30. Wagner B, Fitzpatrick J, Symons M, Jirikowic T, Cross D, Latimer J. The development of a culturally appropriate school based intervention for Australian Aboriginal children living in remote communities: a formative evaluation of the Alert Program® intervention. Aust Occup Ther J. 2017;64(3):243(10). pmid:27966224
- 31.
Williams MS, Shellenberger S. An introduction to "how does your engine run?": the Alert Program for self-regulation Albuquerque, NM: Therapy Works Inc.; 1996.
- 32. Barnes KJ, Vogel KA, Beck AJ, Schoenfeld HB, Owen SV. Self-regulation strategies of children with emotional disturbance. Phys Occup Ther Pediatr. 2008;28(4):369–87. pmid:19042478
- 33. Blackwell AL, Yeager DC, Mische-Lawson L, Bird RJ, Cook DM. Teaching children self-regulation skills within the early childhood education environment: a feasibility study. Journal of Occupational Therapy, Schools & Early Intervention. 2014;7(3/4):204–24.
- 34. Mac Cobb S, Fitzgerald B, Lanigan-O’Keeffe C. The Alert Program for self-management of behaviour in second level schools: results of phase 1 of a pilot study. Emotional & Behavioural Difficulties. 2014;19(4):410–25.
- 35. Wells AM, Chasnoff IJ, Schmidt CA, Telford E, Schwartz LD. Neurocognitive habilitation therapy for children with fetal alcohol spectrum disorders: an adaptation of the Alert Program®. Am J Occup Ther. 2012;66(1):24–34. pmid:22251828
- 36. Soh DW, Skocic J, Nash K, Stevens S, Turner GR, Rovet J. Self-regulation therapy increases frontal gray matter in children with fetal alcohol spectrum disorder: evaluation by voxel-based morphometry. Front Hum Neurosci. 2015;9. pmid:25667571
- 37. Nash K, Stevens S, Greenbaum R, Weiner J, Koren G, Rovet J. Improving executive functioning in children with fetal alcohol spectrum disorders. Child Neuropsychol. 2014;21(2):191–209. pmid:25010354
- 38. Nash K, Stevens S, Clairman H, Rovet J. Preliminary findings that a targeted intervention leads to altered brain function in children with fetal alcohol spectrum disorder. Brain Sciences. 2018;8(1):7.
- 39. Soh DW, Skocic J, Nash K, Stevens S, Turner GR, Rovet J. Self-regulation therapy increases frontal gray matter in children with fetal alcohol spectrum disorder: evaluation by voxel-based morphometry. Front Hum Neurosci. 2015;9:108. pmid:25788884
- 40. Gill K, Thompson-Hodgetts S, Rasmussen C. A critical review of research on the Alert Program®. Journal of Occupational Therapy, Schools, and Early Intervention. 2018;11(2):212–28.
- 41. Wagner B, Fitzpatrick JP, Mazzucchelli TG, Symons M, Carmichael Olson H, Jirikowic T, et al. Study protocol for a self-controlled cluster randomised trial of the Alert Program to improve self-regulation and executive function in Australian Aboriginal children with fetal alcohol spectrum disorder. BMJ Open. 2018;8(3).
- 42. Wagner B, Carmichael Olson H, Symons M, Mazzucchelli TG, Jirikowic T, Latimer J, et al. Improving self-regulation and executive functioning skills in primary school children in a remote Australian Aboriginal community: a pilot study of the Alert Program®. Australian Journal of Education. 2019;63(1):98–115.
- 43. Coffin J. Rising to the challenge in Aboriginal health by creating cultural security. Aboriginal and Islander Health Worker Journal. 2007;31(3):22–4.
- 44.
Coffin J, Green C. Decolonising Australian community development tools. In: Kickett-Tucker C, Bessarab D, Coffin J, Wright M, editors. Mia Mia Aboriginal community development: fostering cultural security. Cambridge: Cambridge University Press; 2017.
- 45. Mc Loughlin F, Hadgraft NT, Atkinson D, Marley JV. Aboriginal health research in the remote Kimberley: an exploration of perceptions, attitudes and concerns of stakeholders. BMC Health Serv Res. 2014;14:517. pmid:25343849
- 46.
Morphy F. Population, people and place: the Fitzroy Valley population project Canberra: The Australian National University; 2010 [Available from: http://caepr.cass.anu.edu.au/research/publications/population-people-and-place-fitzroy-valley-population-project.
- 47.
Gooda M. 2010 Aboriginal and Torres Strait Islander Social Justice Commissioner social justice report Sydney: Australian Human Rights Commission; 2010 [158]. Available from: https://www.humanrights.gov.au/publications/social-justice-report-2010.
- 48.
Australian Curriculum: Assessment and Reporting Authority. My School Website [Website]. [updated 2015. Available from: http://www.myschool.edu.au/.
- 49.
Western Australian Department of Education. Schools on-line: a directory of public schools [Website]. Department of Education; [updated 2015. Available from: http://www.det.wa.edu.au/schoolsonline/home.do.
- 50.
Eyberg S, Pincus D. Eyberg child behavior inventory and Sutter-Eyberg student behavior inventory revised: professional manual. Lutz, FL: Psychological Assessment Resources Inc.; 1999.
- 51.
Gioia GA, Isquith PK, Guy SC, Kenworthy L. BRIEF2: Behavior Rating Inventory of Executive Function professional manual. Lutz, FL: Psychological Assessment Resources Inc; 2015.
- 52. Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ: British Medical Journal. 2015;350.
- 53. Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B (Methodological). 1995;57(1):289–300.
- 54. McIntosh K, Mercer SH, Hume AE, Frank JL, Turri MG, Mathews S. Factors related to sustained implementation of schoolwide positive behavior support. Except Child. 2013;79(3):293–311.
- 55.
Malbin D. Trying differently rather than harder: fetal alcohol spectrum disorder. 3rd ed: D. Malbin; 2002.
- 56. Rai JK, Abecassis M, Casey JE, Flaro L, Erdodi LA, Roth RM. Parent rating of executive function in fetal alcohol spectrum disorder: a review of the literature and new data on Aboriginal Canadian children. Child Neuropsychol. 2017;23(6):713–32. pmid:27282764
- 57. Mason BA, Gunersel AB, Ney EA. Cultural and ethnic bias in teacher ratings of behavior: a criterion-focused review. Psychol Sch. 2014;51(10):1017–30.
- 58. Dingwall KM, Cairney S. Psychological and cognitive assessment of Indigenous Australians. Aust N Z J Psychiatry. 2010;44(1):20–30. pmid:20073564
- 59. Ardila A. Towards a cross-cultural neuropsychology. Journal of Social and Evolutionary Systems. 1996;19(3):237–48.
- 60. Townsend C, White P, Cullen J, Wright C, Zeeman H. Making every Australian count: challenges for the National Disability Insurance Scheme (NDIS) and the equal inclusion of homeless Aboriginal and Torres Strait Islander Peoples with neurocognitive disability. Aust Health Rev. 2018;42(2):227–9. pmid:28355528
- 61. Kearney CA. An interdisciplinary model of school absenteeism in youth to inform professional practice and public policy. Educ Psychol Rev. 2008;20(3):257–82.
- 62. McShane G, Walter G, Rey JM. Characteristics of adolescents with school refusal. Aust N Z J Psychiatry. 2001;35(6):822–6. pmid:11990893
- 63.
Kimberley Workforce Development Alliance. Kimberley workforce development plan 2014–2017: Executive summary. Government of Western Australia: Department of Training and Workforce Development; 2013.
- 64. Latimer J, Elliott EJ, Carter M, Oscar J, Fitzpatrick JP, Ferreira M, et al. Marulu, the Lililwan project, fetal alcohol spectrum disorders (FASD) prevalence study in the Fitzroy Valley: a community consultation. 2010.