Figures
Abstract
Introduction
Executive functions (EFs) training interventions aimed at ADHD-symptom reduction have yielded mixed results. Generally, these interventions focus on training a single cognitive domain (e.g., working memory [WM], inhibition, or cognitive-flexibility). However, evidence suggests that most children with ADHD show deficits on multiple EFs, and that these EFs are largely related to different brain regions. Therefore, training multiple EFs might be a potentially more effective strategy to reduce EF-related ADHD symptoms.
Methods
Eighty-nine children with a clinical diagnosis of ADHD (aged 8–12) were randomized to either a full-active-condition where visuospatial WM, inhibition and cognitive-flexibility were trained, a partially-active-condition where inhibition and cognitive-flexibility were trained and the WM-training task was presented in placebo-mode, or to a full placebo-condition. Short-term and long-term (3-months) effects of this gamified, 25-session, home-based computer-training were evaluated on multiple outcome domains.
Results
During training compliance was high (only 3% failed to meet compliance criteria). After training, only children in the full-active condition showed improvement on measures of visuospatial short-term-memory (STM) and WM. Inhibitory performance and interference control only improved in the full-active- and the partially-active condition. No Treatment-condition x Time interactions were found for cognitive-flexibility, verbal WM, complex-reasoning, nor for any parent-, teacher-, or child-rated ADHD behaviors, EF-behaviors, motivational behaviors, or general problem behaviors. Nonetheless, almost all measures showed main Time-effects, including the teacher-ratings.
Conclusions
Improvements on inhibition and visuospatial STM and WM were specifically related to the type of treatment received. However, transfer to untrained EFs and behaviors was mostly nonspecific (i.e., only interference control improved exclusively in the two EF training conditions). As such, in this multiple EF-training, mainly nonspecific treatment factors – as opposed to the specific effects of training EFs—seem related to far transfer effects found on EF and behavior.
Trial Registration
trialregister.nl NTR2728. Registry name: improving executive functioning in children with ADHD: training executive functions within the context of a computer game; registry number: NTR2728.
Citation: Dovis S, Van der Oord S, Wiers RW, Prins PJM (2015) Improving Executive Functioning in Children with ADHD: Training Multiple Executive Functions within the Context of a Computer Game. A Randomized Double-Blind Placebo Controlled Trial. PLoS ONE 10(4): e0121651. https://doi.org/10.1371/journal.pone.0121651
Academic Editor: Mark Bellgrove, Monash University, AUSTRALIA
Received: April 29, 2014; Accepted: January 17, 2015; Published: April 6, 2015
Copyright: © 2015 Dovis et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors have no support or funding to report.
Competing interests: P.J.M.P. is member of Stichting Gaming & Training, a nonprofit organization that facilitates the development and implementation of ‘‘Braingame Brian.’’; S.v.d.O. has been a paid consultant for Janssen Pharmaceuticals with regard to ‘‘Healseeker,’’ a serious game for cognitive function training. S.D. and R.W.W. declare no competing interests exist. The statements in the competing interest section do not alter the authors' adherence to PLOS ONE policies on sharing data and materials.
Introduction
Theories of ADHD suggest that deficits in executive functioning are at the core of the ADHD-syndrome, and play a pivotal role in explaining the problems children with ADHD encounter in daily life [1], [2], [3], [4]. Via dorsal frontostriatal brain circuits, executive functions (EF) allow individuals to regulate their behavior, thoughts and emotions, and thereby enable self-control [5]. Evidence indeed suggests that impairments in EF are related to deficits in attention, hyperactivity and impulsivity [6], [7], [8], [9], [10], [11], and with associated problems such as deficient academic functioning [12], [13]. Moreover, research suggests that EF-capacity and its associated levels of brain activity are not static, but may be altered by task-repetition or training [14]. Therefore, in the past few years, EF training interventions aimed at ADHD symptom reduction have received considerable interest.
Nonetheless, these EF interventions have yielded mixed results, especially on ADHD behavior (for an overview see [15], [16], [17], [18], [19], [20]; in addition see [21], [22], [23]). Generally, these interventions focus on training a single domain of cognitive functioning in children with ADHD, such as working memory (WM), inhibition, or cognitive flexibility. However, evidence suggests that most children with ADHD show deficits on multiple EFs [24], and that these EFs are largely related to different brain regions [25], [26], [27]. Therefore, training of multiple EFs might be a potentially more effective strategy to reduce EF related ADHD symptoms.
To date, evidence for multiple EF training interventions is limited. Few studies have investigated the effects of these interventions in children with ADHD [28], [29], [30], [31], [32], [33], and although these studies generally show promising results (e.g., improvement of ADHD behavior as rated by parents and/or a significant other [e.g., the teacher]; an increase of neural activity and gray matter volume in ADHD affected brain areas), none of these studies are placebo-controlled.
Besides EF deficits, children with ADHD have problems with motivation. Motivational models [34], [35], [36], [37], and subsequent research (for an overview see [38], [39]; also see [40], [41], [42], [43]) suggest that children with ADHD are less stimulated by reinforcement (i.e. reward) than typically developing children (probably due to a dopaminergic deficit), and therefore require higher amounts and frequencies of reward in order to perform optimally. This elevated need for reinforcement in children with ADHD may result in motivational problems during EF training: the child has to repeat the same responses over and over again for many trials, making most EF training programs tedious and boring for children with ADHD [44]. Research suggests that motivational problems can decrease the effects of EF training in children with ADHD [45]. However, gamification of an EF training or task (e.g., by using game mechanics and visuals) has been found to optimize both motivation and training-effects in children with ADHD [40], [45], [46]. Gaming increases the release of striatal dopamine [47], [48], promoting long-term potentiation of neural connections within the striatum [49], which is suggested to improve motivation and one’s ability to learn [50] (e.g., during EF training).
In the current double-blind, placebo-controlled study, we investigated the efficacy of a gamified, 5-week, home-based, multiple EF training intervention titled Braingame Brian (BGB; [44]) in children with ADHD (combined-subtype). A previous waitlist-controlled study of BGB [31] showed promising results on reduction of symptoms of ADHD and improvement of EF. BGB targets multiple EFs that are commonly impaired in children with ADHD: visuospatial WM, response inhibition, and cognitive flexibility [51]. To date, most EF-training studies focus on the effects of WM training (e.g., see [15]), whereas very few studies investigate the unique effects (i.e. without WM training) of response inhibition- and/or cognitive flexibility training in children with ADHD. Only Kray et al. [23] investigated effects of a cognitive flexibility training in children with ADHD; they found placebo-controlled effects on untrained EF performance (i.e., interference control), but they did not investigate effects on behavior. Moreover, we are not aware of any studies investigating the unique effects of inhibition training in children with ADHD (for studies of combined WM and inhibition training see [28], [29], [30]). Therefore, participants in the current study were randomized to one of three treatment conditions: (1) a full-active-condition where visuospatial WM, response inhibition and cognitive flexibility were trained, (2) a partially-active-condition where only inhibition and cognitive flexibility were trained and the visuospatial WM training-task was presented in placebo-mode, or (3) to a full placebo-condition. Short-term and long-term (3-months) effects were evaluated across various outcome measures (including performance measures of WM, inhibition, cognitive-flexibility, interference control, and complex reasoning, and rating scales assessing parent- and teacher-rated ADHD behavior, parent-rated EF- and motivational behavior, and parent-, teacher- and child-rated general problem behavior).
We expected that: (1) improvement on outcome measures of WM, inhibition, and cognitive flexibility (i.e., performance measures and EF rating-scales) would be specifically related to the type of treatment received (e.g., greatest improvement on WM if WM was trained), (2) the (far-) transfer of treatment effects to other, untrained, domains of EF (such as interference control or parent-rated planning, organization of materials or self-monitoring) would be limited. We expected that spill-over effects to untrained domains of EF (far transfer) would be limited because different EFs are largely related to different brain regions [25], [26], [27], and because most placebo-controlled EF training studies that investigate children with ADHD do not find such far transfer effects (e.g., see [19]), (3) children in the full-active condition would improve significantly more on ADHD behavior than children in either the partially-active condition or placebo condition, and (4) children in the partially-active condition would improve significantly more on ADHD behavior than children in the placebo condition. Finally, we also investigated other domains of impairment that are associated with ADHD (such as sensitivity to reward and punishment, oppositional defiant behavior, quality of life, and problems in daily situations). However, given the current knowledge-base in the field (e.g., there are no placebo-controlled EF training studies that investigate effects on sensitivity to reward and punishment, quality of life or problems in daily situations, and placebo-controlled studies investigating effects on oppositional defiant behavior show mixed results [21], [23], [84], [85], [86]), we refrained from presenting hypotheses regarding these domains of impairment.
Methods
Trial Design
This was a multicenter (14 sites), double-blind, placebo-controlled, multi-arm parallel-group study conducted in the Netherlands (trial register: http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=2728; registry name: improving executive functioning in children with ADHD: training executive functions within the context of a computer game; registry number: NTR2728). No important changes to methods were made after trial commencement (the trial started April 2011 and ended January 2013). The protocol for this trial and CONSORT checklist are available as S1 Protocol and S1 CONSORT Checklist.
Participants
Study settings.
Children were recruited from 14 outpatient mental-healthcare centers. This study was conducted in the Netherlands, within a predominantly urban type of community.
Eligibility criteria.
Eligible participants were all children aged 8 to 12 years with (a) a prior DSM-IV-TR [52] diagnosis of ADHD combined-type and absence of any autism spectrum disorder according to a child psychologist or psychiatrist, (b) a score within the clinical range (95th to 100th percentile) on the ADHD scales of both the parent and teacher version of the Disruptive Behavior Disorder Rating Scale (DBDRS [53]; Dutch translation: [54]), (c) meeting criteria for ADHD combined-type on the ADHD section of the Diagnostic Interview Schedule for Children, parent version (PDISC-IV [55]). The PDISC-IV is a structured diagnostic interview based on the DSM-IV, with adequate psychometric properties, (d) absence of conduct disorder (CD) based on the CD sections of the PDISC-IV, (e) an IQ score ≥80 established by the short version of the Dutch Wechsler Intelligence Scale for Children (WISC-III; [56]). Two subtests, Vocabulary and Block Design, were administered to estimate Full Scale IQ (FSIQ). This composite score has satisfactory reliability and correlates highly with FSIQ [57], (f) absence of any neurological disorder, sensory (color blindness, vision) or motor impairment as stated by the parents, (g) not taking any medication other than Methylphenidate or Dextroamphetamine. Participants discontinued their Methylphenidate at least 24 hours before each test-session, allowing a complete wash-out [58]. Participants taking Dextroamphetamine discontinued medication 48 hours before each test-session [59], finally, (h) parents had to agree to keep the dose of ADHD medication stable between the intake and the 3-months follow-up session, and had to consent not to initiate or participate in other psychosocial treatments. Group differences in baseline demographics and clinical characteristics are listed in Table 1.
Treatment Conditions
General characteristics of the intervention.
“Braingame Brian” (BGB [44]) is a computerized, home-based EF training, embedded in a game world and is named after its main character “Brian”. Brian is a young inventor who, throughout the game, helps and befriends the game-worlds inhabitants by creating increasingly elaborate inventions (e.g., a delivery-rocket for the grocery-store owner). BGB consists of 25 training sessions. Within each session, the player can create inventions by completing two blocks of three training tasks. Within each block, the first training task is always a WM task (used for drawing a blueprint of the invention), the second and third task, a cognitive flexibility task and an inhibition task, are presented in changing order (and are used for sorting building-materials, and electrically-charging the invention). Each session takes about 35–50 minutes (30 minutes for completing the tasks and an optional amount of time for game-world exploration). An additional standardized external reward system—receiving game-related stickers, reward ribbons and medals for completing sessions (the same for all participants)—is used to even further raise the child’s motivation to do the training (for more details see [44] and S1 Appendix). In the current study BGB was presented in three conditions:
Full-active condition.
In this condition WM, inhibition and cognitive-flexibility were all in training-mode. Training-mode entailed that, after each block of training tasks, the difficulty level of the training task was automatically adjusted to the child’s level of performance. Furthermore, in training-mode (a) the WM task [60] consisted of five training levels: the first level targeted visuospatial short-term memory (STM) only, whereas the other four levels targeted combinations of visuospatial STM, updating and manipulation of information (i.e. these four levels targeted both STM and the central executive). Each level was trained for 5 of the 25 sessions. The difficulty level was increased by increasing the amount of information that had to be remembered, updated and manipulated, (b) the inhibition task [61] was designed to decrease the time needed to inhibit a prepotent response (comparable with the stop signal reaction time measured by the STOP task [62]). On most trials the child had to respond to a go-stimulus by pressing left or right within a specific time-frame (a green colored response window between 550–850 ms; see Fig 1). This created a prepotent response tendency. However, on 25% of the trials, somewhere after the go-stimulus and before the middle of the response window, a stop-signal was presented (a tone and a visual cue) and the child had to inhibit the prepotent response (stop-trials). The difficulty level was increased by shortening the time allowed to inhibit this response, (c) the cognitive-flexibility task [61] was designed to decrease the time a child needs to adapt his/her behavior when task-rules change (i.e. switch cost). Specifically, the child had to sort objects with different shapes and colors (e.g. blue or red colored plungers and wheels) to either the left or the right according to a rule. The rule was either to sort according to shape or to sort according to color. In 25% of the trials the rule switched (switch-trials). The difficulty level was increased by shortening the time allowed to switch between the two rules (for a more detailed description of the three training tasks see [31]).
Partially-active condition.
In this condition the inhibition and cognitive-flexibility tasks were in training-mode, and the WM task was in placebo-mode. Placebo-mode entailed that only the first level of the WM task was presented (for all 25 sessions), and that the difficulty level was not adjusted to the child’s level of performance (no more than two items had to be remembered). The amount of trials in placebo-mode was increased to match the training time in training-mode (10 minutes training per session for each EF domain).
Placebo condition.
In this condition WM, inhibition and cognitive-flexibility were all in placebo-mode. In placebo-mode the inhibition task and the cognitive-flexibility task were presented the same way as in training-mode except that the stop-trials and switch-trials were replaced by go-trials and non-switch trials (i.e., no stop-trials and switch-trials were presented) and the difficulty level was not adjusted.
Process Measures
No important changes to outcome measures were made after trial commencement.
Compliance.
Compliance was defined as completing all of the 25 training sessions within a 5-week period. Using this algorithm, each child was categorized as compliant or noncompliant to treatment.
Blinding.
At post-test, parents were asked to report the condition they thought their child was assigned to (full-active, partially-active, or placebo).
Improvement index during training.
To validate whether the training actually improved task performance on the designated EFs, the improvement on training performance from beginning to end of training was assessed. It was tested whether children improved during training with paired t-tests. For the inhibition training and the cognitive flexibility training the results of day 2 and 3 of training (the Start Index) were compared with the results of their two best training days (the Max Index). The WM training had five levels and each level covered only 5 of the 25 training days. Therefore, to measure improvement on the WM training, within each level, the results of day 2 of training (the Start Index) were compared with the results of the best training day (the Max Index).
Performance Measures
Stop task.
The Stop task was used to measure the time needed to inhibit an ongoing response [62]. Two types of trials were presented: go-trials and stop-trials. During go-trials a go-stimulus (an arrow) that was either pointing right or left was presented. Participants were instructed to press a response button that corresponded to the direction of the stimulus as quickly and as accurately as possible. Stop-trials were identical to the go-trials but in addition a stop-signal was presented (a tone and a visual cue), which indicated that the participant had to withhold his/her ongoing response. The delay between the go- and stop-signal was dynamically varied (in steps of 50ms) so that inhibition was successful in 50% of the stop-trials. At this point, the go-process and stop-process are of equal duration, which makes it possible to estimate the latency of the stop-process: the stop signal reaction time (SSRT [62]). Aside from two practice blocks, four experimental blocks (of 64 trials each) were administered. The SSRT was used as outcome measure of inhibitory processing. Test retest reliability of the SSRT in children with ADHD is. 72 [63].
Stroop.
The Stroop Color and Word Test [64] measures interference control and consists of three pages with words and/or colors. On the first page, word naming is measured by naming the words red, green, yellow, and blue, printed in black ink. On the second page, color naming is measured by naming the colors of small rectangles. The first and second page represent the congruent trials. On the third page, colors are then named when shown as nonmatching color words (incongruent trials). The interference score on the Stroop is the time needed for the third page minus the time needed for the second page, and was used as our outcome measure of interference control. The STROOP has adequate reliability [65].
Corsi Block Tapping Task (CBTT).
The CBTT [66] assesses the capacity of visuospatial STM and WM. The task consists of nine cubes (blocks) that are positioned on a board. In the present study, the same test format (size of board and blocks, distances between blocks) was used as in Kessels, van Zandvoort, Postma, Kappelle, and de Haan [67] (also see [68]), and the same procedure was used as in Geurts, Verté, Oosterlaan, Roeyers, and Sergeant [69]. The experimenter tapped a sequence of blocks that a child then had to reproduce in the same (CBTT-forward) or in reversed order (CBTT-backward). The minimum sequence length was three and the maximum was eight blocks, and each length was presented on three trials. The total amount of sequences that is correctly reproduced is the total score. The total score on the CBTT-forward (max. total score = 18) was used as an outcome measure for visuospatial STM and the total score on the CBTT-backward (max. total score = 18) was used as an outcome measure of visuospatial WM. The CBTT shows good reliability [70].
Digit span.
The scaled score on the Digit-span subtest from the WISC-III testing battery [56] was used as a composite measure of verbal STM and WM. Participants were orally given sequences of numbers and were asked to repeat them, either in the same (i.e. STM) or in reversed order (i.e. WM). Digit span has adequate reliability [56].
Trail Making Test (TMT).
The TMT of the Delis-Kaplan Executive Function System (D-KEFS [71]) measures cognitive flexibility and is a timed task that requires the individual to connect a series of letters and numbers in ascending order while alternating between numbers and letters. The scaled contrast score—the contrast between the scaled non-switch trials (number- and letter sequencing) and the scaled switch trials (number-letter switching)—was used as outcome measure of cognitive flexibility (i.e., switch-cost). Test-retest reliabilities range from. 20 to. 77 [71].
Raven coloured progressive matrices.
Raven’s coloured progressive matrices [72] measures non-verbal reasoning ability. The test consists of 36 items. The total amount of items correct (total score; max. = 36) was used as outcome measure for non-verbal reasoning. Test-retest reliability ranges from. 68 to. 90 [73].
Questionnaires and Rating Scales
DBDRS (parent and teacher versions).
The DBDRS contains four DSM-IV scales; Inattention, Hyperactivity/ Impulsivity, Oppositional Defiant Disorder (ODD), and CD. Parents and teachers rate the child’s behavior on a 4-point Likert-type scale. Adequate psychometric properties have been reported [54]. The scores on the Inattention and Hyperactivity/Impulsivity scales were used as outcome measure of ADHD behavior. The scores on the ODD and CD scales were used as outcome measures of general problem behavior.
Behavior Rating Inventory of Executive Function questionnaire (BRIEF).
[74]. The Dutch version of the BRIEF is used to assess parent-rated EF. The BRIEF consists of 75 questions and includes eight EF sub-domains: Inhibit, Shift, Emotional Control, Initiate, WM, Plan/Organize, Organization of Materials, and Monitor. The test has adequate psychometric properties [75]. T-scores on the EF sub-domains were used as outcome measures.
Sensitivity to Punishment and Sensitivity to Reward Questionnaire for children (SPSRQ-C).
The SPSRQ-C measures parent-rated sensitivity to punishment and reward [76] (Dutch translation: [77]) and contains 33 items, divided in a Punishment Sensitivity scale, and three Reward Sensitivity scales: Reward Responsivity, Impulsivity/Fun-Seeking, and Drive. Each item is scored on a 5-point Likert scale. Adequate psychometric properties are reported [76]. Subscale scores were used as outcome measures.
Pediatric Quality of Life Inventory (PedsQL; parent and child versions).
[78] (Dutch translation: [79]). The PedsQL consists of 23 items, scored on a five-point Likert-scale, and is divided in four subscales: Physical, Emotional, Social, and School Functioning. The Psychosocial Health Summary score (a composite of the Emotional, Social and School Functioning subscales) was used as outcome measure. Adequate psychometric properties are reported [79].
The Home Situations Questionnaire (HSQ).
The HSQ [80] is designed to assess the impact of problem behavior at home and in public situations. Parents report whether each of 16 daily situations (e.g. getting dressed and going to bed) was a problem and rate their severity on a 9-point scale. The mean severity score was used as outcome measure. The HSQ has adequate psychometric properties [81].
Procedure
This study was approved by the faculty’s IRB (the Ethics Review Board of the Faculty of Social and Behavioral Sciences of the university of Amsterdam). After obtaining written informed consent from the parents (on behalf of the participating children), parents and teachers completed the DBDRS. At this first screening the 6-month version of the DBDRS was administered (regarding the child’s behavior over the past 6-months), whereas at the pre-test, post-test and follow-up a two-week version of the DBDRS was administered (regarding the child’s behavior over the past two-weeks). If DBDRS inclusion criteria were met, children and parents were invited to the intake session. During this session questions regarding demographics were asked (see Table 1), and the PDISC-IV, and the short-form of the WISC-III were administered. The Chessboard WM task (for a detailed description see Dovis et al., 2013) was also administered during the intake session. However, this task was part of a different study and its results will therefore be reported elsewhere. If inclusion criteria were met, parent and child were invited to the pre-test session and the startup session, and were independently allocated to one of the three treatment conditions using the process of randomization by minimization [82] on the basis of age, gender, IQ, medication-use (yes/no), and parent- and teacher-rated inattention and hyperactivity/impulsivity symptoms (using the 6-months DBDRS). During the pre-test session the outcome measures were administered, and in the same week the teacher completed the two-week version of the DBDRS. The pre-test occurred approximately 1–2 weeks prior to the startup session (which was the start of the training). During the startup session parent and child were instructed about the training program, the computer, and the external reward system (see S1 Appendix), and a schedule for implementing the intervention and for weekly coaching calls was established. Once a research assistant completed a startup session with a particular family, he/she could not test or have further contact with that family or the teacher (to preserve blinding). During the 5-week, home-based training, a coach (a research assistant blind to the treatment condition) made weekly calls (of about 15 minutes; using a standardized telephone protocol) to the participating families to monitor progress, motivation and compliance, and to solve technical and game-related problems. Parents and children were explicitly instructed not to discuss the content of the training tasks with the coach. If a coach did receive information revealing the treatment condition, he/she was replaced and could no longer have contact with the family or the teacher. 1–2 weeks after the final training session the post-test was scheduled and the teacher completed the DBDRS. 3-months after the final training session the follow-up was scheduled and the teacher completed the DBDRS. At each test-session experimenters were blind to condition.
Statistical Analyses
Sample size was determined by a prospective power analyses for univariate testing (using G*Power) based on the effect sizes of two previous EF-treatment studies [86], [45]. These studies suggested that the treatment effects on our primary outcome measures (i.e., EF measures, ADHD rating-scales) would be medium in size. Groups did not differ with respect to any of the baseline demographics or clinical characteristics (see Table 1). Also, including these baseline demographics and clinical characteristics (i.e., Gender, Age, FSIQ, DBDRS parent and teacher ratings, ODD diagnosis, ADHD medication use, Computergame experience, and Dyslexia) as covariates in the main analyses did not change the pattern of our results. Because repeated-measures were used, covariates were entered after mean centering (see [97]). Multinomial logistic regression was used for assessing the effectiveness of blinding.
An Intent-To-Treat (ITT) approach, using single imputations, was used to compare treatment effects of the three treatment conditions. That is, for each treatment group stochastic regression imputation was used to predict the missing posttest and follow-up values. The missing posttest values were based on the non-missing pretest and posttest scores of each treatment group. The missing follow-up values were based on the non-missing pretest scores, posttest scores, follow-up scores, and pretest-posttest difference scores of each treatment group (although the overall percentage of missing data was low—less than 5% was missing—it must be noted that stochastic regression imputation can increase the probability of making type I errors).
The dependent measures were subjected to four repeated measures MANOVAs (for the performance measures, for ADHD behavior, for EF and motivational behavior, and for general problem behavior; the covariance matrices were assumed to be unstructured), with Treatment condition (full-active, partially-active, placebo) as between-subject factor and Time (pre-test, post-test, follow-up) as within-subject factors. Bonferroni corrections for multiple testing were applied to these MANOVAs: only p-values < .0125 [.05/4] were considered significant. Trends and significant effects were further analyzed with simple contrasts. Bonferroni corrections for multiple testing were applied to these contrasts, in which the amount of dependent variables corrected for was defined per repeated measures MANOVA (7 performance-, 4 ADHD behavior-, 12 EF and motivational behavior-, and 7 general problem behavior variables were each analyzed in 3 pair-wise time comparisons [pre-test vs. post-test, post-test vs. follow-up, and pre-test vs. follow-up], resulting in a required significant level of p = .0024 [.05/21] for the performance measures contrasts, p = .0042 [.05/12] for the ADHD behavior contrasts, p = .0014 [.05/36] for the EF and motivational behavior contrasts, and p = .0024 [.05/21] for the general problem behavior contrasts). For additional within-group analyses paired t-tests were used (Bonferroni corrections were applied). Partial Eta squared effect sizes (ηp2) are reported for all analyses: ηp2 = .01 is regarded a small effect size,. 06 a medium effect size, and. 14 a large effect size [83].
Results
Process Measures
Compliance during training.
Of the 31 participants assigned to the full-active condition, 30 (96.7%) met compliance criteria (25 training days within 5 weeks). All of the 28 participants assigned to the partially-active condition met compliance criteria. Of the 30 participants assigned to the placebo condition, 28 (93.3%) met compliance criteria. Overall, compliance to treatment was high, given that this was a home-based intervention that included a substantial portion of participants with ODD (see Fig 2).
Post-training dropout.
Eight participants (9%) of our total sample (i.e., 3 children in the full-active condition, 2 children in the partially-active condition, and 3 children in the placebo condition) were lost to post-test and follow-up testing (see Fig 2). There were no significant differences on baseline demographics and clinical characteristics (i.e., gender, age, IQ, DBDRS parent and teacher ratings, ODD-diagnosis, medication use, computergame experience, Dyscalculia and Dyslexia) between the children lost to post-test and follow-up testing and the children who participated in these assessments (depending on the level of measurement a MANOVA or Pearson’s chi-squared tests were used). But note that the sample size of the post-training drop out group was small.
Blinding.
There was no significant association between the conditions wherein participants were actually included and the conditions whereof parents afterwards reported (guessed) that their child was assigned to (the multinomial logistic regression model indicated a non-significant model overall, χ2(4) = 1.26, p = .868, -2LL = 18.004). This suggests that, based upon their experience with the actual training condition, parents were not able to guess the condition wherein their child was included. Further, no participant (child, parent, teacher, experimenter, or coach) was unblinded at any point during the conduct of the trial.
Improvement index during training.
It was tested whether children improved during training with paired t-tests (Bonferroni corrections for multiple testing were applied: only p-values < .0024 [.05/21] were considered significant). Within the full-active condition, paired t-tests showed a significant difference (improvement) between the Start Index and Max Index for the Inhibition training, t (30) = -18.66, p < .001, the Cognitive flexibility training, t (30) = -19.14, p < .001, and for all the levels of the WM training (level 1, t (30) = -7.25, p < .001; level 2, t (30) = -7.90, p < .001; level 3, t (30) = -7.19, p < .001, level 4, t (30) = -9.21, p < .001; level 5, t (30) = -7.72, p < .001). Within the partially-active condition (where WM was in placebo-mode), paired t-tests showed significant difference (improvement) between the Start Index and the Max Index for both the Inhibition training, t (27) = -15.86, p < .001, and the Cognitive flexibility training, t (27) = -22.89, p < .001.
Performance Measures
A 3x3 (Treatment condition x Time [pre-test, post-test, follow-up]) repeated measures MANOVA with the main scores of the EF tasks (Stoptask [SSRT], STROOP [interference score], CBTT-fwd [total score], CBTT-bkw [total score], Digit recall [scaled score], TMT [scaled contrast score]) and the Raven (total score) as dependent variables (scores on all seven tasks were analyzed simultaneously), showed a main effect of Time, F (14,334) = 6.74, p < .001, ηp2 = .22, no main effect of Treatment condition, F (14,162) = 1.41, p = .154, ηp2 = .11, and a non-significant trend towards an interaction between Treatment condition and Time, F (28,676) = 1.59, p = .027, ηp2 = .06 (after Bonferroni correction only p-values < .0125 were considered significant). To interpret these effects for each performance based measure, we used simple contrasts:
For each performance based measure, main Time effects and Treatment condition x Time interactions are presented per pair-wise time difference (i.e. pre- vs. post-test, post- vs. follow-up test; pre- vs. follow-up test) in Table 2.
After Bonferroni correction (p < .0024 [.05/21]) results indicate the following: Between the pre- and post-test there was a significant Treatment condition x Time interaction for the CBTT-fwd (p = .002), and a non-significant trend for the Stoptask (p = .037) and the CBTT-bkw (p = .039; see Table 2). Between the pre-test and follow-up there was a non-significant trend towards a Treatment condition x Time interaction for the CBTT-fwd (p = .013) and the STROOP (p = .07; see Table 2). Other pair-wise time differences in Treatment condition x Time interaction effects were non-significant both with- and without Bonferroni correction (investigating Digit recall forward and backward separately [using raw scores] did not change the results). Next, in order to obtain more insight into these two-way interactions, three follow-up repeated measures MANOVAs were performed: one for each combination of treatment conditions (Bonferroni corrections were applied: only p-values < .0167 [.05/3] were considered significant).
Full-active condition versus placebo condition.
A 2x3 (Treatment condition x Time) repeated measures MANOVA with the main scores of the Stoptask, STROOP, CBTT-fwd, and CBTT-bkw as dependent variables, showed a main effect of Time, F (8,232) = 6.22, p < .001, ηp2 = .18, a main effect of Treatment condition, F (4,56) = 5.06, p = .009, ηp2 = .21, and a significant interaction between Treatment condition and Time, F (8,232) = 3.90, p < .001, ηp2 = .12. To further interpret this interaction for each relevant pair-wise time difference and each performance based measure, we used simple contrasts (in the previous contrast analyses no Treatment x Time interactions were found between post-test and follow-up; therefore, only the two-way interactions between pre-test and post-test and between pre-test and follow-up were further explored):
These contrasts are presented in Table 3 and indicate that, compared to pre-test performance, post-test- and/or follow-up performance on the Stoptask, the STROOP and the CBTT forward and backward improved more in the full-active condition than in the placebo condition (p-values ranged from. 002 to. 020; effect sizes ranged from medium to large; see Table 3 and Fig 3A–3D). However, after Bonferroni correction only the Treatment x Time interactions for the CBTT-fwd remained significant (as only p-values < .0063 [.05/8] were considered significant).
Note: SSRT = Stop Signal Reaction Time; CBTT = Corsi Block Tapping Task.
Partially-active condition versus placebo condition
A 2x3 (Treatment condition x Time) repeated measures MANOVA with the main scores of the Stoptask, STROOP, CBTT-fwd, and CBTT-bkw as dependent variables, showed a main effect of Time, F (8,220) = 3.49, p = .001, ηp2 = .11, no main effect of Treatment condition, F (4,53) = 1.44, p = .235, ηp2 = .10, and no significant interaction between Treatment condition and Time, F (8,220) = 1.07, p = .388, ηp2 = .04. However, since we had specific expectations regarding the Treatment condition x Time interactions—we only expected this interaction for the Stoptask and the STROOP, not for the CBTT forward and backward (as WM was not trained in either condition)—simple contrasts were used to further explore the non-significant interaction effect:
These contrasts are presented in Table 3 and indicate that, compared to pre-test performance, post-test performance on the Stoptask improved more in the partially-active condition than in the placebo condition (p = .045; medium effect size; see Table 3 and Fig 3A). However, this difference was no longer significant after Bonferroni correction: as only p-values < .0063 (.05/8) were considered significant.
Full-active condition versus partially-active condition
A 2x3 (Treatment condition x Time) repeated measures MANOVA with the main scores of the Stoptask, STROOP, CBTT-fwd, and CBTT-bkw as dependent variables, showed a main effect of Time, F (8,224) = 9.79, p < .001, ηp2 = .26, no main effect of Treatment condition, F (4,54) = 1.76, p = .151, ηp2 = .12, and a non-significant trend towards an interaction between Treatment condition and Time, F (8,224) = 2.00, p = .048, ηp2 = .07 (after Bonferroni correction only p-values < .0167 [.05/3] were considered significant). To further interpret this interaction, we used simple contrasts:
These contrasts are presented in Table 3 and indicate that, compared to pre-test performance, post-test and/or follow-up performance on the CBTT (forward and backward) improved more in the full-active condition than in the partially-active condition (p-values ranged from. 011 to. 046, effect sizes were medium; see Table 3 and Fig 3C and 3D). However, these differences were no longer significant after Bonferroni correction: as only p-values < .0063 (.05/8) were considered significant.
Within-group analyses.
For each EF task where a Treatment condition x Time interaction was significant with or without Bonferroni correction (Stoptask, STROOP, CBTT-fwd, CBTT-bkw), differences within each treatment group between the pre- and post-test and between the pre-test and follow-up were tested with additional paired t-tests (Bonferroni corrections were applied: only p-values < .0021 [.05/24] were considered significant).
Results are presented in Table 4. After Bonferroni correction we found that: in the full-active condition performance on the Stoptask, the STROOP, the CBTT-fwd and the CBTT-bkw significantly improved between pre- and post-test. Performance on the Stoptask, the STROOP and the CBTT-bkw also significantly improved between pre-test and follow-up (there was a trend for performance on the CBTT-fwd, p = .003). In the partially-active condition performance on the STROOP significantly improved between pre- and post-test (there was a trend for performance on the Stoptask, p = .005), and performance on the Stoptask significantly improved between pre-test and follow-up (there was a trend for performance on the STROOP, p = .016). In the placebo condition none of the differences were significant (although there was a trend for STROOP performance between pre- and post-test, p = .043; see Table 4).
Questionnaires and Rating Scales
ADHD behavior (parent and teacher DBDRS).
A 3x3 (Treatment condition x Time) repeated measures MANOVA with mean scores on the Inattention and Hyperactivity/Impulsivity scales of the parent and the teacher version of the DBDRS as dependent variables, showed a main effect of Time, F (8,340) = 13.32, p < .001, ηp2 = .24, no main effect of Treatment condition, F (8,166) = .33, p = .953, ηp2 = .02, and no significant interaction between Treatment condition and Time, F (16,688) = .77, p = .718, ηp2 = .02. The significant Time effect was further explored using simple contrasts:
For each ADHD scale, main Time effects are presented per pair-wise time difference in Table 2. After Bonferroni correction (p < .0042 [.05/12]) results indicate that: compared to the pre-test, both parents and teachers reported a significant decrease in ADHD symptoms at the post-test and at the follow-up (effect sizes of parent-ratings were large; effect sizes of teacher-ratings ranged from medium to large). However, the non-significant Treatment x Time interaction indicates that this decrease did not differ between the Treatment conditions (in addition see Table 2 & Fig 4).
Parent-rated EF- and motivational behavior (BRIEF and SPSRQ-C).
A 3x3 (Treatment condition x Time) repeated measures MANOVA with mean t-scores on the Inhibit-, Working Memory-, Shift-, Emotional Control-, Initiate-, Plan/Organize-, Organization of Materials-, and Monitor scales of the BRIEF, and the mean scores on the Sensitivity to Punishment-, Impulsivity/Fun Seeking-, Reward Responsiveness-, and Drive scales of the SPSRQ-C as dependent variables, showed a main effect of Time, F (30,318) = 4.91, p < .001, ηp2 = .32, no main effect of Treatment condition, F (30,146) = 1.08, p = .368, ηp2 = .18, and no significant interaction between Treatment condition and Time, F (60,644) = .72, p = .942, ηp2 = .06. The significant Time effect was further explored using simple contrasts:
These contrasts are presented in Table 2. After Bonferroni correction (p < .0014 [.05/36]) results indicate the following: after training, parents reported a significant improvement (with large effect sizes) on almost all scales of the BRIEF (EF behavior; only improvement on the Organization of Materials scale was no longer significant after Bonferroni correction) and on the Impulsivity/Fun Seeking scale of the SPSRQ-C (motivational behavior; medium effect size; improvement on the Punishment Sensitivity scale [p = .022] and the Reward Responsiveness scale [p = .007] was no longer significant after Bonferroni correction). However, the non-significant Treatment x Time interaction indicates that these improvements did not differ between the Treatment conditions (in addition see the Treatment x Time contrasts in Table 2).
General problem behavior (DBDRS, PEDsQL, and HSQ).
A 3x3 (Treatment condition x Time) repeated measures MANOVA with mean scores on the ODD and the CD scales of the parent and the teacher version of the DBDRS, the Psychosocial Health Summary score of the parent and the child version of the PEDsQL, and the mean severity score of the parent-rated HSQ as dependent variables, showed a main effect of Time, F (14,334) = 5.15, p < .001, ηp2 = .18, a non-significant trend towards a main effect of Treatment condition, F (14,162) = 1.83, p = .038, ηp2 = .14 (after Bonferroni correction only p-values < .0125 were considered significant), and no significant interaction between Treatment condition and Time, F (28,676) = 1.10, p = .337, ηp2 = .04. The significant Time effect was further explored using simple contrasts:
These contrasts are presented in Table 2. After Bonferroni correction (p < .0024 [.05/21]) results indicate the following: after training, parents reported a significant improvement on all general problem behavior indices (effect sizes ranged from medium to large), and teachers reported a significant improvement on the ODD scale of the DBDRS (medium effect size). However, the non-significant Treatment x Time interaction indicates that these improvements did not differ between the Treatment conditions (in addition see the Treatment x Time contrasts in Table 2). In contrast to their parents, children reported no significant difference in their Psychosocial Health Summary Score after training.
Treatment Responders
In addition to the overall means, the percentage of children who benefitted from training was calculated for each measure that showed significant (with or without Bonferroni correction) main Time effects and/or Treatment condition x Time interactions on the pairwise comparisons of pre- and post-test scores and/or pre- and follow-up test scores (see Table 2). On each of these measures children were either classified as responders or non-responders by using reliable change indices [98], [99]. Based on classification guidelines by Wise [99], a participant was classified as responder when both the following criteria were met: (1) a reliable change index (RCI) of at least 1.28 (RCI was based on the method of [98]), and (2) an improvement of scores of at least 1 standard deviation [99]. Results for each treatment condition are presented in Table 5. The pattern of these results strongly resembles the pattern of the mean results (see Table 5; in addition see Table A in S2 Appendix).
Discussion
The aim of this study was to determine the short- and long-term effects of a gamified training intervention (BGB) that targets multiple EFs (visuospatial WM, response inhibition and cognitive flexibility) compared to a placebo version of the intervention on various outcome measures in children with ADHD combined-type. In addition, to determine the unique effect of the inhibition and cognitive flexibility training tasks, we compared a full-active condition (where WM, inhibition, and cognitive flexibility were all in training-mode) to a partially-active condition (where only inhibition and cognitive flexibility were in training-mode).
Results indicated that only children in the full-active condition showed improvement on measures of visuospatial STM and WM. Inhibitory performance and interference control only improved in the full-active condition and the partially-active condition. However, no Treatment-condition x Time interactions (with or without Bonferroni corrections) were found for cognitive flexibility, verbal STM and WM, non-verbal complex reasoning, or child-rated psychosocial health, nor for any parent- or teacher-rated ADHD symptoms, EF behaviors, motivational behaviors, or general problem behaviors. Nonetheless, almost all measures showed significant Time-effects, including the teacher-ratings (effect sizes ranged from medium to large).
These findings suggest that improvements on inhibition and visuospatial STM and WM were specifically related to the type of treatment received. However, improvements on untrained EFs and behavior (far transfer effects) were mostly nonspecific (i.e., only interference control improved exclusively in the two conditions where EFs were trained). As such, in this multiple EF training, mainly nonspecific treatment factors—as opposed to the specific effects of training EFs—seem related to the far transfer effects on EF and behavior.
In many ways our findings are similar to those of previous placebo controlled (single) EF training studies in children with ADHD [21], [84], [85], [86], [23] (but note that only one of these studies [21] corrected for multiple testing). Most of these studies find differential treatment effects on outcome measures of trained EFs (although Kray et al. [23], as in the present study, found no significant differences on cognitive flexibility). However, such near transfer effects may not be surprising since many of these outcome measures are very similar to the training tasks themselves and improvement may be the result of a learned strategy instead of improved cognitive capacity [87]. Further, in most placebo controlled studies differential far transfer to untrained EF tasks has been limited, and differential effects on parent- or teacher-rated behavior (e.g., ADHD or EF) are generally not found. Only Klingberg et al. [86] found a differential effect of WM training on parent-rated ADHD. However, the placebo condition used in Klingberg et al. was considerably shorter in time than the training condition. This suggests a difference in parent involvement between the conditions, which may have interacted with the outcome of parent-rated ADHD behavior (e.g., through expectancy effects or inequality of parent-child interactions; see [15]). Another notable feature of the study of Klingberg et al. is that they did not include children with comorbid ODD. However, including ODD diagnosis as a covariate did not change the pattern of our main results. Therefore, the absence of comorbid ODD in the Klingberg et al. study seems an unlikely explanation for their distinctive findings on parent-rated ADHD. This assumption is further substantiated by the findings we presented in Table A (see S2 Appendix): Irrespective of treatment condition, children with comorbid ODD were at least as likely to improve on parent-rated ADHD behavior as children without comorbid ODD.
There are also several important differences between our findings and the findings of previous placebo controlled EF training studies. Although we used more stringent compliance criteria than most previous studies (i.e., completing 100% of the training sessions versus completing 80% of the training sessions), in our study only 3% of the participants failed to meet compliance criteria, whereas in previous studies 15–23% failed to meet compliance criteria. Since most previous studies also used an external reward system, a structured schedule for implementing the intervention, weekly contact with a coach, and performance feedback during training, the most obvious reason for this difference in compliance is the relatively strong gamification of BGB. This hypothesis is consistent with previous findings of increased time-on-training when EF training was gamified [45] (also see [40]), and with the finding that gaming increases the release of striatal dopamine [47], [48], which is associated with increased motivation to continue playing and performing [50].
Moreover, in contrast to the previous placebo-controlled studies, we found a significant improvement on teacher-rated ADHD behavior (effect sizes ranged from medium to large). Although this improvement was unrelated to specific effects of the EF training (as it was also found in the placebo condition), it is still a remarkable finding. Some have argued that EF training studies only find Time effects on parent-ratings but not on teacher-ratings because teachers, in contrast to parents, are only minimally involved in training and thus may be less biased than parents (e.g., by their expectancies of the training outcome) [31]. This suggests that generalization of improvement to teacher-ratings might represent relatively unbiased evidence of treatment induced changes in the child’s behavior. Nonetheless, it is unclear what caused this improvement. It seems unrelated to specific EF training effects, and the only nonspecific treatment factor that clearly distinguishes our study from previous studies appears to be the use of relatively strong gamification (i.e., teachers were not more involved than in previous studies). Is it possible that gamification somehow improved classroom behavior? For example, there is evidence that video game playing can enhance various cognitive skills (e.g., attention; see [88]). However, if playing video games by itself would be sufficient to improve classroom functioning in children with ADHD, it seems illogical that the participants in our study, who play commercial video games for 10 hours per week (see Table 1), did not improve before. Nonetheless, it may be that parents’ positive attitude towards this particular game enhanced its positive effects. For example, sharing the joy of achievement in the game with his/her parents could have enhanced the child’s appraisal of the game’s positive feedback and its effect on his/her self-esteem beyond that of commercial video games (as many parents don’t encourage children to indulge in commercial gaming). Although there is a link between parental praise and children’s self-esteem [89], and self-esteem has been found to mediate the relationship between ADHD and classroom functioning [90], future research should investigate this further. Furthermore, the gamification of BGB may also have impacted classroom functioning by enhancing children’s motivation to comply with treatment. If children were more motivated to comply with treatment than in other EF training studies, which is consistent with the relatively high compliance rate in our study, there may have been less need for parents to discipline their children during training. Evidence suggests that decreased negative parental discipline mediates the effect of ADHD treatment (e.g., medication and behavior therapy) on teacher-rated ADHD behavior [91]. Future EF training studies should use larger samples and appropriate process measures to further investigate these potential mechanisms of mediation.
Although some previous EF training studies in children with ADHD have found differential effects on interference control [23], [30] ([85] and [86] also found differential effects on the STROOP, but they only used the incongruent trials as outcome measure; baseline response times to congruent trials were not controlled for, making it impossible to calculate the interference score), our study is the first to find differential effects on response inhibition. In contrast to the placebo condition, response inhibition was improved in both the full-active condition and the partially-active condition, but no differences were found between these two experimental conditions. This suggests that a combined inhibition and cognitive flexibility training by itself (i.e., without WM) is sufficient to improve response inhibition in children with ADHD. Possibly, previous EF training studies investigating effects on measures of response inhibition in children with ADHD [29], [30], [32] found no improvements because their intervention did not include an inhibition training task (i.e., Hoekzema et al. [32] trained WM, cognitive flexibility, attention, planning and problem solving), or because their inhibition training task was based on a less appropriate response inhibition paradigm; the go/no-go task instead of the stop task [29], [30]. In contrast to the stoptask, the go/no-go task has been criticized as not functionally isolating inhibition (e.g., because of its interaction with selective attention and decision making, and the confounding effects of its prepotent response processes; see [2], [92], [93]). Nonetheless, since we did not investigate effects of the inhibition- and cognitive flexibility training separately, we can only speculate that the improvement on response inhibition was the result of our stop-task-based inhibition training. Additional research is needed to investigate this in more detail.
In contrast to our findings on other near transfer measures, no differential effects of EF training were found on the cognitive flexibility measure (neither with or without Bonferroni correction). This may be the result of the difference between the switch-cost (the index of cognitive flexibility) that was trained, and the switch-cost that was used as outcome measure of cognitive flexibility. Our outcome measure (the scaled contrast score on the TMT) measures global switch-cost (i.e., the difference between a block of switch-trials and blocks of non-switch trials), whereas the cognitive flexibility training focused on training local switch-cost (i.e., the difference between switch-trials and non-switch trials within a block of trials). Although, both types of switch-cost are considered valid measures of cognitive flexibility, evidence suggests that they tap somewhat different cognitive processes and can be differentiated on a neural level [94], [95]. Therefore, it could be argued that our outcome measure of cognitive flexibility was in fact a measure of far transfer. Future studies should investigate this further using more varied measures of cognitive flexibility.
The fact that far transfer was also found in the placebo condition might not (only) be explained by nonspecific treatment effects (e.g., effects of expectancies, self-fulfilling prophecies, attribution, gamification, or improved parent-child interactions), but may be the result of actual cognitive training in the placebo condition. Although the cognitive load in our placebo condition was very low, it could be argued that the requisite of the placebo tasks to focus attention for a substantial amount of time was sufficient to improve cognitive control (e.g., attention) and the behavior of our participants. However, this appears inconsistent with the very limited improvement on EF performance in the placebo condition, and the lack of effects resulting from other activities that require prolonged focused attention (e.g., paying attention in school, playing [educational] video games).
Because no wait-list control condition was utilized, it is not possible to determine to which extent our findings relate to effects of multiple testing, the passage of time, or (nonspecific) treatment factors. However, a previous study investigating BGB [31] found no improvement on parent- and teacher-rated ADHD and EF behavior in a wait-list control group, whilst they did find improvement in the group that was trained. This suggests that the current findings on ADHD and EF behavior are probably not attributable to mere passage of time or multiple testing (for a study of children with autism spectrum disorder see [100]).
In this study different EFs were trained simultaneously within the same training session. However, based on the current state of the literature it is unclear if this is indeed the best strategy for multiple EF training (i.e., there are no studies that directly investigate this). One could assume that training different EFs simultaneously is more effective (especially for transfer to daily life) than training one EF at a time (i.e., training each EF in separate sessions), because functioning in daily life also requires the use of multiple EFs at once. However, our results do not suggest that training three EFs per session (i.e., the full-active condition) has more effect on daily functioning than training two EFs per session (i.e. the partially-active condition). Future studies should further investigate this.
In the current study, far transfer effects were mostly nonspecific. However, we only investigated overall group differences (i.e., disregarding potential subpopulations that show differential responses to treatment), and children were allocated to treatment conditions irrespective of their individual EF deficits. Therefore, before discarding EF training as potential treatment for children with ADHD, future studies should examine moderators (e.g., severity of EF deficits; teacher expectancies) and mediators of treatment success (e.g., improvement on EF performance; parental praise), and should investigate effects of individually tailored EF training (i.e., to make optimal use of the available training-time future studies should match training focus to the specific EF problems of each individual child). Furthermore, to increase chances of finding far transfer that results from EF training specifically, training tasks should be made more ecologically valid (e.g., by using EF training tasks that resemble the complexity of problematic situations in daily-life) and should be intertwined with relevant real-life EF-taxing activities (e.g., completing chores in daily-life could be an additional goal in the EF training; for more suggestions see [96]). Finally, the domains of far transfer that were investigated in this study were limited to direct measures of performance and indirect measures of behavior (e.g., behavior as rated by parents, teachers or children). Future studies should also include direct measures of behavior. For example, a recent placebo-controlled WM training study [84] found no specific treatment effects on parent-rated behavior (teacher-rated behavior was not investigated), but found specific effects on aspects of experimenter-observed off-task behavior during an academic task.
In conclusion, our findings suggest that improvements on inhibition and visuospatial STM and WM were specifically related to the type of treatment received. However, improvements on untrained EFs and behavior were mostly nonspecific. As such, in this multiple EF training (BGB), mainly nonspecific treatment factors—as opposed to the specific effects of training EFs—seem related to the far transfer effects on EF and behavior.
Supporting Information
S1 CONSORT Checklist. CONSORT checklist Dovis et al.
https://doi.org/10.1371/journal.pone.0121651.s001
(DOC)
S1 Appendix. Document containing Appendix 1 (including Fig A.) Appendix 1—Fig A The reward system: an overview of the medals and reward ribbons and the map with all the stickers (photograph courtesy of Arnold Brakenhoff).
https://doi.org/10.1371/journal.pone.0121651.s002
(DOCX)
S2 Appendix. Document containing Appendix 2 (including Table A.).
https://doi.org/10.1371/journal.pone.0121651.s003
(DOCX)
S1 Protocol. Trial protocol 2011-DP-1526 Dovis et al. with attachm.
https://doi.org/10.1371/journal.pone.0121651.s005
(PDF)
Acknowledgments
We are grateful to Jeugdriagg Noord Holland Zuid, GGz Noord Holland Noord (Centrum voor Kinder- en Jeugdpsychiatrie), Regionaal Centrum voor Kinder en Jeugdpsychiatrie Gooi en Vechtstreek (RCKJP), Bosman GGz, Stichting De Praktijk, Stichting Kram, PuntP, Academisch Behandelcentrum UvA Minds, and Kinderpraktijk VIS, to multimedia company ShoSho for the gamification of Brian Game Brian, to Hilde Huizenga and Joost Agelink van Rentergem Zandvliet for their comments and statistical advice, to Marloes van der Arend, Tim van den Broek, Josje de Bont, Annette Brouwer, Tycho Dekkers, Lucie van den Eertwegh, Rebecca Goedee, Roza van der Heide, Lisanne Klink, Astrid Nauta, Inge Meulenberg, Muriël Musa, Pascale Riaskoff, Elise Tilma, Marije Voermans, Ida de Vries, and Pamina Warmbrunn for their help with data collection, and to all participating children and families.
Author Contributions
Conceived and designed the experiments: SD SvdO PJMP RW. Performed the experiments: SD. Analyzed the data: SD SvdO. Contributed reagents/materials/analysis tools: SD SvdO PJMP. Wrote the paper: SD SvdO PJMP RW.
References
- 1.
Barkley RA. Attention-deficit hyperactivity disorder. A handbook for diagnosis and treatment, 3d ed. New York: Guilford Press; 2006.
- 2.
Nigg JT. What causes ADHD? Understanding what goes wrong and why. New York: Guilford Press; 2006.
- 3. Rapport MD, Chung KM, Shore G, Isaacs P. A conceptual model of child psychopathology: implications for understanding attention deficit hyperactivity disorder and treatment efficacy. J Clin Child Psychol. 2001; 30: 48–58. Available: http://www.ncbi.nlm.nih.gov/pubmed/11294077. Accessed 3 April 2013. pmid:11294077
- 4. Sonuga-Barke EJS. The dual pathway model of AD/HD: an elaboration of neuro-developmental characteristics. Neurosci Biobehav Rev. 2003; 27: 593–604. Available: http://dx.doi.org/10.1016/j.neubiorev.2003.08.005. Accessed 12 February 2013. pmid:14624804
- 5. Durston S, van Belle J, de Zeeuw P. Differentiating frontostriatal and fronto-cerebellar circuits in attention-deficit/hyperactivity disorder. Biol Psychiatry. 2011; 69: 1178–1184. Available: http://www.ncbi.nlm.nih.gov/pubmed/20965496. Accessed 14 August 2013. pmid:20965496
- 6. Burgess GC, Depue BE, Ruzic L, Willcutt EG, Du YP, Banich MT. Attentional control activation relates to working memory in attention-deficit/hyperactivity disorder. Biol Psychiatry. 2010; 67: 632–640. Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2953472&tool=pmcentrez&rendertype=abstract. Accessed 2 August 2012. pmid:20060961
- 7. Crosbie J, Arnold P, Paterson A, Swanson J, Dupuis A, Li X, et al. Response inhibition and ADHD traits: correlates and heritability in a community sample. J Abnorm Child Psychol. 2013; 41: 497–507. Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3600128&tool=pmcentrez&rendertype=abstract. Accessed 26 August 2013. pmid:23315233
- 8. Kofler MJ, Rapport MD, Bolden J, Sarver DE, Raiker JS. ADHD and working memory: the impact of central executive deficits and exceeding storage/rehearsal capacity on observed inattentive behavior. J Abnorm Child Psychol. 2010; 38: 149–161. Available: http://www.ncbi.nlm.nih.gov/pubmed/19787447. Accessed 2 August 2012. pmid:19787447
- 9. Raiker JS, Rapport MD, Kofler MJ, Sarver DE. Objectively-Measured Impulsivity and Attention-Deficit/Hyperactivity Disorder (ADHD): Testing Competing Predictions from the Working Memory and Behavioral Inhibition Models of ADHD. J Abnorm Child Psychol. 2012; 40: 699–713. Available: http://www.ncbi.nlm.nih.gov/pubmed/22271141. Accessed 2 August 2012. pmid:22271141
- 10. Rapport MD, Bolden J, Kofler MJ, Sarver DE, Raiker JS, Alderson RM. Hyperactivity in boys with attention-deficit/hyperactivity disorder (ADHD): a ubiquitous core symptom or manifestation of working memory deficits? J Abnorm Child Psychol. 2009; 37: 521–534. Available: http://www.ncbi.nlm.nih.gov/pubmed/19083090. Accessed 18 July 2012. pmid:19083090
- 11. Tillman CM, Eninger L, Forssman L, Bohlin G. The relation between working memory components and ADHD symptoms from a developmental perspective. Dev Neuropsychol. 2011; 36: 181–198. Available: http://www.ncbi.nlm.nih.gov/pubmed/21347920. Accessed 2 August 2012. pmid:21347920
- 12. Biederman J, Monuteaux MC, Doyle AE, Seidman LJ, Wilens TE, Ferrero F, et al. Impact of executive function deficits and attention-deficit/hyperactivity disorder (ADHD) on academic outcomes in children. J Consult Clin Psychol. 2004; 72: 757–766. Available: http://www.ncbi.nlm.nih.gov/pubmed/15482034. Accessed 20 February 2012. pmid:15482034
- 13. Titz C, Karbach J. Working memory and executive functions: effects of training on academic achievement. Psychol Res. 2014; 78: 852–868. Available: http://www.ncbi.nlm.nih.gov/pubmed/24389706. Accessed 28 January 2014. pmid:24389706
- 14. Klingberg T. Training and plasticity of working memory. Trends Cogn Sci. 2010; 14: 317–324. Available: http://www.ncbi.nlm.nih.gov/pubmed/20630350. Accessed 12 July 2012. pmid:20630350
- 15. Chacko A, Feirsen N, Bedard A-C, Marks D, Uderman JZ, Chimiklis A. Cogmed Working Memory Training for Youth with ADHD: A Closer Examination of Efficacy Utilizing Evidence-Based Criteria. J Clin Child Adolesc Psychol. 2013; 42: 769–783. Available: http://dx.doi.org/10.1080/15374416.2013.787622. Accessed 31 July 2013. pmid:23668397
- 16. Evans SW, Owens JS, Bunford N. Evidence-Based Psychosocial Treatments for Children and Adolescents with Attention-Deficit/Hyperactivity Disorder. J Clin Child Adolesc Psychol. 2014; 43: 527–551. Available: http://dx.doi.org/10.1080/15374416.2013.850700. Accessed 18 February 2014. pmid:24245813
- 17. Rapport MD, Orban SA, Kofler MJ, Friedman LM. Do programs designed to train working memory, other executive functions, and attention benefit children with ADHD? A meta-analytic review of cognitive, academic, and behavioral outcomes. Clin Psychol Rev. 2013; 33: 1237–1252. Available: http://www.ncbi.nlm.nih.gov/pubmed/24120258. Accessed 20 October 2013. pmid:24120258
- 18. Rutledge KJ, van den Bos W, McClure SM, Schweitzer JB. Training cognition in ADHD: current findings, borrowed concepts, and future directions. Neurotherapeutics. 2012; 9: 542–558. Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3441933&tool=pmcentrez&rendertype=abstract. Accessed 8 February 2013. pmid:22911054
- 19. Shipstead Z, Redick TS, Engle RW. Is working memory training effective? Psychol Bull. 2012; 138: 628–654. Available: http://www.ncbi.nlm.nih.gov/pubmed/22409508. Accessed 12 July 2012. pmid:22409508
- 20. Toplak ME, Connors L, Shuster J, Knezevic B, Parks S. Review of cognitive, cognitive-behavioral, and neural-based interventions for Attention-Deficit/Hyperactivity Disorder (ADHD). Clin Psychol Rev. 2008; 28: 801–823. Available: http://dx.doi.org/10.1016/j.cpr.2007.10.008. Accessed 27 August 2009. pmid:18061324
- 21. Chacko A, Bedard A-C, Marks DJ, Feirsen N, Uderman JZ, Chimiklis A, et al. A randomized clinical trial of Cogmed Working Memory Training in school-age children with ADHD: a replication in a diverse sample using a control condition. J Child Psychol Psychiatry. 2014; 55: 247–255. Available: http://www.ncbi.nlm.nih.gov/pubmed/24117656. Accessed 29 March 2014. pmid:24117656
- 22. Egeland J, Aarlien AK, Saunes BK. Few effects of far transfer of working memory training in ADHD: a randomized controlled trial. PloS one. 2013; 8: e75660. Available: http://www.plosone.org/article/info:doi/10.1371/journal.pone.0075660#pone-0075660-g001. Accessed 27 November 2013. pmid:24124503
- 23. Kray J, Karbach J, Haenig S, Freitag C. Can task-switching training enhance executive control functioning in children with attention deficit/-hyperactivity disorder? Front Hum Neurosci. 2012; 5: 180. Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3250077&tool=pmcentrez&rendertype=abstract. Accessed 12 July 2012. pmid:22291628
- 24. Fair DA, Bathula D, Nikolas MA, Nigg JT. Distinct neuropsychological subgroups in typically developing youth inform heterogeneity in children with ADHD. Proc Natl Acad Sci U S A. 2012; 109: 6769–6774. Available: http://www.pnas.org/content/109/17/6769.short. Accessed 28 Oktober 2012. pmid:22474392
- 25. McNab F, Leroux G, Strand F, Thorell L, Bergman S, et al. Common and unique components of inhibition and working memory: an fMRI, within-subjects investigation. Neuropsychologia. 2008; 46: 2668–2682. Available: http://dx.doi.org/10.1016/j.neuropsychologia.2008.04.023. Accessed 22 August 2010. pmid:18573510
- 26. Schecklmann M, Ehlis A-C, Plichta MM, Dresler T, Heine M, et al. Working memory and response inhibition as one integral phenotype of adult ADHD? A behavioral and imaging correlational investigation. J Atten Disord. 2013; 17: 470–482. Available: http://jad.sagepub.com/content/early/2012/02/05/1087054711429702.abstract. Accessed 5 August 2013. pmid:22323120
- 27. Smith AB, Taylor E, Brammer M, Toone B, Rubia K. Task-specific hypoactivation in prefrontal and temporoparietal brain regions during motor inhibition and task switching in medication-naive children and adolescents with attention deficit hyperactivity disorder. Am J Psychiatry. 2006; 163: 1044–1051. Available: http://ajp.psychiatryonline.org/article.aspx?articleid=96688&RelatedWidgetArticles=true. Accessed 22 August 2010. pmid:16741205
- 28. Halperin JM, Marks DJ, Bedard A-C, Chacko A, Curchack JT, Yoon CA, et al. Training executive, attention, and motor skills: a proof-of-concept study in preschool children With ADHD. J Atten Disord. 2013; 17: 711–721. Available: http://www.ncbi.nlm.nih.gov/pubmed/22392551. Accessed 6 March 2014. pmid:22392551
- 29. Johnstone SJ, Roodenrys S, Phillips E, Watt AJ, Mantz S. A pilot study of combined working memory and inhibition training for children with AD/HD. Atten Defic Hyperact Disord. 2010; 2: 31–42. Available: http://www.ncbi.nlm.nih.gov/pubmed/21432588. Accessed 21 July 2012. pmid:21432588
- 30. Johnstone SJ, Roodenrys S, Blackman R, Johnston E, Loveday K, Mantz S, et al. Neurocognitive training for children with and without AD/HD. Atten Defic Hyperact Disord. 2012; 4: 11–23. Available: http://www.ncbi.nlm.nih.gov/pubmed/22179720. Accessed 12 August 2013. pmid:22179720
- 31. S Van Der Oord, Ponsioen A, Geurts HM, EL Ten Brink, Prins PJM. A Pilot Study of the Efficacy of a Computerized Executive Functioning Remediation Training With Game Elements for Children With ADHD in an Outpatient Setting: Outcome on Parent- and Teacher-Rated Executive Functioning and ADHD Behavior. J Atten Disord. 2014; 18: 699–712. Available: http://www.ncbi.nlm.nih.gov/pubmed/22879577. Accessed 6 August 2013. pmid:22879577
- 32. Hoekzema E, Carmona S, Tremols V, Gispert JD, Guitart M, Fauquet J, et al. Enhanced neural activity in frontal and cerebellar circuits after cognitive training in children with attention-deficit/hyperactivity disorder. Hum Brain Mapp. 2010; 31: 1942–1950. Available: http://www.ncbi.nlm.nih.gov/pubmed/20336653. Accessed 16 August 2013. pmid:20336653
- 33. Hoekzema E, Carmona S, Ramos-Quiroga JA, Barba E, Bielsa A, Tremols V, et al. Training-induced neuroanatomical plasticity in ADHD: a tensor-based morphometric study. Hum Brain Mapp. 2011; 32: 1741–1749. Available: http://www.ncbi.nlm.nih.gov/pubmed/21365715. Accessed 13 February 2014. pmid:21365715
- 34. Haenlein M, Caul WF. Attention deficit disorder with hyperactivity: a specific hypothesis of reward dysfunction. J Am Acad Child Adolesc Psychiatry. 1987; 26: 356–362. Available: http://www.sciencedirect.com/science/article/pii/S0890856709656914. Accessed 6 March 2014. pmid:3597291
- 35.
Sergeant JA, Oosterlaan J, Van der Meere J. Information processing and energetic factors in attention-deficit/Hyperactivity Disorder. In: Quay HC, Hogan AE, editors. Handbook of disruptive behavior disorders. New York: Kluwer Academic/ Plenum Publishers; 1999. pp. 75–104.
- 36. Sagvolden T, Johansen EB, Aase H, Russell VA. A dynamic developmental theory of attention-deficit/hyperactivity disorder (ADHD) predominantly hyperactive/impulsive and combined subtypes. Behav Brain Sci. 2005; 28: 397–419. Available: http://journals.cambridge.org/abstract_S0140525X05000075. Accessed 13 February 2011. pmid:16209748
- 37. Sonuga-Barke EJS. Editorial: ADHD as a reinforcement disorder—moving from general effects to identifying (six) specific models to test. J Child Psychol Psychiatry. 2011; 52: 917–918. Available: http://www.ncbi.nlm.nih.gov/pubmed/21793824. Accessed 12 January 2012. pmid:21793824
- 38. Luman M, Oosterlaan J, Sergeant JA. The impact of reinforcement contingencies on AD/HD: a review and theoretical appraisal. Clin Psychol Rev. 2005; 25: 183–213. Available: http://www.ncbi.nlm.nih.gov/pubmed/15642646. Accessed 12 October 2011. pmid:15642646
- 39. Luman M, Tripp G, Scheres A. Identifying the neurobiology of altered reinforcement sensitivity in ADHD: a review and research agenda. Neurosci Biobehav Rev. 2010; 34: 744–754. Available: http://dx.doi.org/10.1016/j.neubiorev.2009.11.021. Accessed 15 October 2011. pmid:19944715
- 40. Dovis S, Van der Oord S, Wiers RW, Prins PJM. Can motivation normalize working memory and task persistence in children with attention-deficit/hyperactivity disorder? The effects of money and computer-gaming. J Abnorm Child Psychol. 2012; 40: 669–681. Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3375007&tool=pmcentrez&rendertype=abstract. Accessed 12 July 2012. pmid:22187093
- 41. Dovis S, Van der Oord S, Wiers RW, Prins PJM. What part of working memory is not working in ADHD? Short-term memory, the central executive and effects of reinforcement. J Abnorm Child Psychol. 2013; 41: 901–917. Available: http://www.ncbi.nlm.nih.gov/pubmed/23430155. Accessed 23 February 2014. pmid:23430155
- 42.
Dovis S, Van der Oord S, Wiers RW & Prins PJM. ADHD subtype differences in reinforcement sensitivity and visuospatial working memory. J Clin Child Adolesc Psychol. 2014; In press.
- 43. Strand MT, Hawk LW, Bubnik M, Shiels K, Pelham WE, Waxmonsky JG. Improving working memory in children with attention-deficit/hyperactivity disorder: the separate and combined effects of incentives and stimulant medication. J Abnorm Child Psychol. 2012; 40: 1193–1207. Available: http://www.springerlink.com/content/a886140671369735/. Accessed 19 November 2012. pmid:22477205
- 44. Prins PJM, E Ten Brink, Dovis S, Ponsioen A, Geurts HM, de Vries M, et al. “Braingame Brian”: Toward an Executive Function Training Program with Game Elements for Children with ADHD and Cognitive Control Problems. Games Health J. 2013; 2: 44–49. Available: http://online.liebertpub.com/doi/abs/10.1089/g4h.2013.0004. Accessed 28 February 2013.
- 45. Prins PJM, Dovis S, Ponsioen A, ten Brink E, van der Oord S. Does computerized working memory training with game elements enhance motivation and training efficacy in children with ADHD? Cyberpsychol Behav Soc Netw. 2011; 14: 115–122. Available: http://www.ncbi.nlm.nih.gov/pubmed/20649448. Accessed 13 July 2012. pmid:20649448
- 46. Shaw R, Grayson A, Lewis V. Inhibition, ADHD, and computer games: the inhibitory performance of children with ADHD on computerized tasks and games. J Atten Disord. 2005; 8: 160–168. Available: http://www.ncbi.nlm.nih.gov/pubmed/16110046. pmid:16110046
- 47. Koepp MJ, Gunn RN, Lawrence a D, Cunningham VJ, Dagher a, et al. Evidence for striatal dopamine release during a video game. Nature. 1998; 393: 266–268. Available: http://www.ncbi.nlm.nih.gov/pubmed/9607763. Accessed 22 August 2010. pmid:9607763
- 48. Kühn S, Romanowski a, Schilling C, Lorenz R, Mörsen C, et al. The neural basis of video gaming. Transl Psychiatry. 2011; 1: e53. Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3309473&tool=pmcentrez&rendertype=abstract. Accessed 2 August 2012. pmid:22833208
- 49. Reynolds JN, Hyland BI, Wickens JR. A cellular mechanism of reward-related learning. Nature. 2001; 413: 67–70. Available: http://dx.doi.org/10.1038/35092560. Accessed 10 February 2011. pmid:11544526
- 50.
Gray PO. Psychology. 6th ed. New York: Worth Publishers; 2010.
- 51. Willcutt EG, Nigg JT, Pennington BF, Solanto M V, Rohde LA, Tannock R, et al. Validity of DSM-IV attention deficit/hyperactivity disorder symptom dimensions and subtypes. J Abnorm Psychol. 2012; 121: 991–1010. Available: http://psych.colorado.edu/~willcutt/Willcutt_2012_JAP_ADHD.pdf. Accessed 1 February 2013. pmid:22612200
- 52.
American Psychiatric Association. DSM-IV-TR. Washington, DC: APA; 2000.
- 53. Pelham WE, Gnagy EM, Greenslade KE, Milich R. Teacher rating of DSM-III-R symptoms for disruptive behavior disorders. J Clin Child Psychol. 1992; 8, 259–262.
- 54.
Oosterlaan J, Scheres A, Antrop I, Roeyers H, Sergeant JA. Vragenlijst voor Gedragsproblemen bij Kinderen (VvGK). Lisse, The Netherlands: Swets & Zeitlinger; 2000.
- 55. Shaffer D, Fisher P, Lucas CP, Dulcan MK, Schwab-Stone ME. NIMH diagnostic interview schedule for children version IV (NIMH DISC-IV). J Am Acad Child Adolesc Psy. 2000; 39, 28–38. pmid:10638065
- 56.
Kort W, Compaan EL, Bleichrodt N, Resing WCM, Schittekatte M, Bosmans M, et al. WISC-III NL Handleiding. (Dutch Manual). Amsterdam: NIP; 2002.
- 57.
Sattler JM. Assessment of children: Cognitive applications, 4th ed. San Diego, CA: Author; 2001.
- 58.
Greenhill LL. Childhood ADHD: Pharmacological treatments. In: Nathan P. E., & Gorman J., editors. A guide to treatments that work. New York: Oxford University Press; 1998. pp. 42–64.
- 59. Wong CG, Stevens MC. The effects of stimulant medication on working memory functional connectivity in attention-deficit/hyperactivity disorder. Biol Psychiatry. 2012; 71: 458–466. Available: http://www.sciencedirect.com/science/article/pii/S0006322311011267. Accessed 24 January 2013. pmid:22209640
- 60.
Dovis S, Ponsioen A, Geurts H, Ten Brink E, Van der Oord S, Prins PJM. Executive function training tasks: Working memory. Theoretical background and parameters. Task force ADHD & computer. Amsterdam, the Netherlands: University of Amsterdam; 2008.
- 61.
Dovis S, Geurts H, Ponsioen A, Ten Brink E, Van der Oord S, Prins PJM. Executive function training tasks: Inhibition and cognitive flexibility. Theoretical background and parameters. Task force ADHD & computer. Amsterdam, the Netherlands: University of Amsterdam; 2008.
- 62. Logan GD, Schachar RJ, Tannock R. Impulsivity and Inhibitory Control. Psychol Sci. 1997; 8: 60–64. Available: http://pss.sagepub.com/lookup/doi/10.1111/j.1467-9280.1997.tb00545.x. Accessed 20 February 2014.
- 63. Soreni N, Crosbie J, Ickowicz A, Schachar R. Stop signal and Conners’ continuous performance tasks: test-retest reliability of two inhibition measures in ADHD children. J Atten Disord. 2009; 13: 137–143. Available: http://jad.sagepub.com/content/13/2/137.short. Accessed 20 January 2014. pmid:19429883
- 64.
Hammes JGW. De Stroop Kleur-Woord Test (The Stroop Color-Word Test). Lisse, The Netherlands: Swets and Zeitlinger; 1978.
- 65.
Egberink IJL, Vermeulen CSM, Frima RM. COTAN review 2002, STROOP. 2014. Available: http://www.cotandocumentatie.nl/test_details.php?id=102. Accessed 5 January 2014.
- 66. Corsi PM. Human memory and the medial temporal region of the brain. Dissert Abstr Int. 1972; 34, 819.
- 67. Kessels RPC, van Zandvoort MJ, Postma A, Kappelle LJ, de Haan EH. The Corsi Block-Tapping Task: standardization and normative data. Appl Neuropsychol. 2000; 7: 252–258. Available: http://www.ncbi.nlm.nih.gov/pubmed/18483192. Accessed 29 August 2013. pmid:11296689
- 68. Kessels RPC, van den Berg E, Ruis C, Brands AMA. The backward span of the Corsi Block-Tapping Task and its association with the WAIS-III Digit Span. Assessment. 2008; 15: 426–434. Available: http://www.ncbi.nlm.nih.gov/pubmed/18483192. Accessed 13 August 2013. pmid:18483192
- 69. Geurts HM, Verté S, Oosterlaan J, Roeyers H, Sergeant JA. How specific are executive functioning deficits in attention deficit hyperactivity disorder and autism? J Child Psychol Psychiatry. 2004; 45: 836–854. Available: http://www.ncbi.nlm.nih.gov/pubmed/15056314. pmid:15056314
- 70.
Schellig D. Block-Tapping-Test. Lisse: The Netherlands: Swets and Zeitlinger; 1997.
- 71.
Delis DC, Kaplan E, Kramer JH. Delis-Kaplan Executive Function System: Examiner's Manual, Trial Making Test. San Antonio, TX: Harcourt Assessment; 2007.
- 72.
Raven JC, Court JH, Raven J. Coloured Progressive Matrices. Oxford, UK: Oxford Psychologists Press; 1995.
- 73. Costenbader V, Ngari SM. A Kenya Standardization of the Raven’s Coloured Progressive Matrices. Sch Psychol Int. 2001; 22: 258–268. Available: http://spi.sagepub.com/content/22/3/258.abstract. Accessed 10 February 2014.
- 74.
Gioia GA, Isquith PK, Guy SC, Kenworthy L. Behavior Rating Inventory of Executive Function. Odessa, FL: Psychological Assessment Resources; 2000.
- 75.
Smidts DP, Huizinga M. BRIEF executieve functies gedragsvragenlijst: Handleiding (BRIEF Dutch manual). Amsterdam, The Netherlands: Hogrefe Uitgevers; 2009.
- 76. Colder CR, O’Connor RM. Gray’s Reinforcement Sensitivity Model and Child Psychopathology: Laboratory and Questionnaire Assessment of the BAS and BIS. J Abnorm Child Psychol. 2004; 32: 435–451. Available: http://link.springer.com/10.1023/B:JACP.0000030296.54122.b6. Accessed 6 March 2014. pmid:15305548
- 77. Luman M, van Meel CS, Oosterlaan J, Geurts HM. Reward and punishment sensitivity in children with ADHD: validating the Sensitivity to Punishment and Sensitivity to Reward Questionnaire for children (SPSRQ-C). J Abnorm Child Psychol. 2012; 40: 145–157. Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3268965&tool=pmcentrez&rendertype=abstract. Accessed 26 January 2013. pmid:21789519
- 78. Varni JW, Seid M, Kurtin PS. PedsQL 4.0: Reliability and validity of the Pediatric Quality of Life Inventory version 4.0 Generic Core Scales in healthy and patient populations. Med Care. 2001; 39: 800–812. pmid:11468499
- 79. Bastiaansen D, Koot HM, Bongers IL, Varni JW, Verhulst FC. Measuring quality of life in children referred for psychiatric problems: Psychometric properties of the PedsQL TM 4.0 generic core scales. Qual Life Res. 2004; 13: 489–495. Available: http://link.springer.com/10.1023/B:QURE.0000018483.01526.ab. Accessed 11 February 2014. pmid:15085921
- 80.
Barkley RA, Edelbrock C. Assessing situational variations in children’s behavior problems: The home and school situations Questionnaires. In Prinz R, editor. Advances in behavioral assessment of children and families (vol. 3). Greenwich, CT: JAI press; 1987. pp. 157–176.
- 81. Cunningham CE, Boyle MH. Preschoolers at Risk for Attention-Deficit Hyperactivity Disorder and Oppositional Defiant Disorder: Family, Parenting, and Behavioral Correlates. J Abnorm Child Psychol. 2002; 30: 555–569. Available: http://link.springer.com/article/10.1023/A:1020855429085. Accessed 11 February 2011. pmid:12481971
- 82. Altman DG, Bland JM. Treatment allocation by minimisation. BMJ. 2005; 330: 843. Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=556084&tool=pmcentrez&rendertype=abstract. Accessed 20 February 2014. pmid:15817555
- 83. Kittler JE, Menard W, Phillips KA. Weight concerns in individuals with body dysmorphic disorder. Eat Behav. 2007; 8: 115–120. Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1762093&tool=pmcentrez&rendertype=abstract. Accessed 2 August 2012. pmid:17174859
- 84. Green CT, Long DL, Green D, Iosif A-M, Dixon JF, Miller MR, et al. Will working memory training generalize to improve off-task behavior in children with attention-deficit/hyperactivity disorder? Neurotherapeutics. 2012; 9: 639–648. Available: http://www.springerlink.com/content/n11n34117p1272k7/. Accessed 8 August 2013. pmid:22752960
- 85. Klingberg T, Forssberg H, Westerberg H. Training of working memory in children with ADHD. J Clin Exp Neuropsychol. 2002; 24: 781–791. Available: http://www.ncbi.nlm.nih.gov/pubmed/15689731. pmid:12424652
- 86. Klingberg T, Fernell E, Olesen PJ, Johnson M, Gustafsson P, Dahlström K, et al. Computerized training of working memory in children with ADHD—a randomized, controlled trial. J Am Acad Child Adolesc Psychiatry. 2005; 44: 177–186. Available: http://www.jaacap.com/article/S0890-8567(09)61427-1/abstract. Accessed 17 July 2012. pmid:15689731
- 87. Thompson TW, Waskom ML, Garel KA, Cardenas-Iniguez C, Reynolds GO, Winter R, et al. Failure of working memory training to enhance cognition or intelligence. PloS one. 2013; 8: e63614. Available: http://dx.plos.org/10.1371/journal.pone.0063614. Accessed 22 February 2014. pmid:23717453
- 88. Green CS, Bavelier D. Action video game modifies visual selective attention. Nature. 2003; 423: 534–537. Available: http://www.ncbi.nlm.nih.gov/pubmed/12774121. pmid:12774121
- 89. Felson RB, Zielinski MA. Children’s Self-Esteem and Parental Support. J Marriage Fam. 1989; 51: 727. Available: http://www.jstor.org/stable/352171. Accessed 28 February 2014.
- 90. Shaw-Zirt B, Popali-Lehane L, Chaplin W, Bergman A. Adjustment, social skills, and self-esteem in college students with symptoms of ADHD. J Atten Disord. 2005; 8: 109–120. Available: http://jad.sagepub.com/content/8/3/109.short. Accessed 28 February 2014. pmid:16009659
- 91. Hinshaw SP. Moderators and mediators of treatment outcome for youth with ADHD: understanding for whom and how interventions work. J Ped Psychol. 2007; 32: 664–675. Available: http://jpepsy.oxfordjournals.org/content/32/6/664.full. Accessed 11 March 2014.
- 92. Rubia K, Smith AB, Brammer MJ, Taylor E. Right inferior prefrontal cortex mediates response inhibition while mesial prefrontal cortex is responsible for error detection. NeuroImage. 2003; 20: 351–358. Available: http://www.sciencedirect.com/science/article/pii/S1053811903002751. Accessed 22 January 2014. pmid:14527595
- 93. Schachar RJ, Tannock R, Logan G. Inhibitory control, impulsiveness, and attention deficit hyperactivity disorder. Clin Psychol Rev. 1993; 13: 721–739. Available: http://www.sciencedirect.com/science/article/pii/S0272735805800030. Accessed 6 March 2014.
- 94. Braver TS, Reynolds JR, Donaldson DI. Neural Mechanisms of Transient and Sustained Cognitive Control during Task Switching. Neuron. 2003; 39: 713–726. Available: http://www.sciencedirect.com/science/article/pii/S0896627303004665. Accessed 20 February 2014. pmid:12925284
- 95. White HA, Shah P. Training attention-switching ability in adults with ADHD. J Atten Disord. 2006; 10: 44–53. Available: http://www.ncbi.nlm.nih.gov/pubmed/16840592. Accessed 12 October 2012. pmid:16840592
- 96. Gathercole SE. Commentary: Working memory training and ADHD—where does its potential lie? Reflections on Chacko et al. (2014). J Child Psychol Psychiatry. 2014; 55: 256–257. Available: http://www.ncbi.nlm.nih.gov/pubmed/24438534. Accessed 24 February 2014. pmid:24438534
- 97. Delaney HD, Maxwell SE. On Using Analysis Of Covariance In Repeated Measures Designs. Multivariate Behav Res. 1981; 16: 105–123. Available: http://www.tandfonline.com/doi/abs/10.1207/s15327906mbr1601_6. Accessed 29 July 2014.
- 98. Jacobson NS, Truax P. Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol. 1991; 59: 12–19. Available: http://doi.apa.org/getdoi.cfm?doi=10.1037/0022-006X.59.1.12. Accessed 24 July 2014. pmid:2002127
- 99. Wise EA. Methods for analyzing psychotherapy outcomes: a review of clinical significance, reliable change, and recommendations for future directions. J Pers Assess. 2004; 82: 50–59. Available: http://dx.doi.org/10.1207/s15327752jpa8201_10. Accessed 29 July 2014. pmid:14979834
- 100.
De Vries M, Prins PJM, Schmand BA, Geurts HM. Working memory and cognitive flexibility-training for children with an autism spectrum disorder: a randomized controlled trial. J Child Psychol Psychiatry. 2014; In press. Available: http://www.ncbi.nlm.nih.gov/pubmed/25256627. Accessed 5 December 2014.