Structure and measurement of visuospatial working memory

VSWM is believed to consist of several components, which can be differentiated by distinct experimental tasks. Logie [22] described VSWM as a tripartite model consisting of a passive visual cache for maintaining information, an active inner scribe for enabling rehearsal, and a central executive system, which in turn integrates and controls the other two facets. Each of the three components showed distinctive developmental patterns in Logie and Pearson [23].

Della Sala, Gray, Baddeley, Allamano, and Wilson [24] suggested another two different distinct components of VSWM. Their model distinguishes between visual and spatial components, with the spatial component also potentially consisting of a spatial-sequential subset. The visual component enables maintaining the appearance of objects (shape, colour, etc. as assessed by e.g., the odd-one-out task) and location (e.g., Corsi block-tapping test; [25]). The spatial component enables identifying changes between stimuli (e.g., The Visual Patterns Test; [26]). A similar model differentiating between visual, spatial (simultaneous and sequential) and visuospatial complex span components was suggested by the data in Mammarella and colleagues [27]. Other research suggests somewhat different structures when considering WM more broadly. For example, one study performed a confirmatory factor analysis on 10 visuospatial and 3 verbal WM tasks, with the best model containing four passive components for storage (verbal, visual, simultaneous-spatial, and sequential-spatial) and one active component for processing (central executive) [28]. Active processes are suggested to involve the manipulation of information in addition to information maintenance, while passive processes include only maintenance of information that is not modified after encoding.

Many individual differences tasks which have been employed to measure aspects of VSWM have some notable limitations. In particular, they lack a close relationship to a cognitive model of WM that would bolster the task’s construct validity (e.g., [29]). They may also share the problem of task impurity–measuring several WM components at once. Some research (e.g., [30]) addressed this issue by assessing performance on multiple WM tasks and then adopting a latent variable approach to extract the common variance among the tasks. Yet tasks may still involve other cognitive abilities, such as long-term memory encoding or retrieval [31]. Additionally, while there are some well-established visuospatial tests (e.g., in the Woodcock-Johnson III (WJ III) Tests of Cognitive Abilities); [32], sometimes they might be too expensive to be used by researchers and educators.

Furthermore, many traditional measures of VSWM do not require concurrent processing and storage of visuospatial information, which is the simultaneous engagement of the central executive and visuospatial sketchpad, according to the Baddeley and Hitch WM model. To address this gap in the test literature, Shah and Miyake [33] proposed a new spatial span task using a dual-task paradigm, as typical for experimental WM research. The task requires participants to mentally rotate sequentially presented visual shapes (i.e., letters), while also encoding the shapes’ spatial positions in the correct order. The number of correctly recalled location sequences represents the participant’s VSWM score. The task showed high reliability (Cronbach’s α = .80) and loaded strongly onto a spatial WM factor in their original study [33]. These results were replicated in a later study [34], where the test was named ‘letter rotation task’ and loaded on a complex storage-plus-processing VSWM factor (standardized factor loading = .64), as opposed to a simple-storage visuospatial short-term memory factor or an executive function factor. In sum, the letter rotation task represents an easy-to-understand dual-task paradigm that requires processing and storage of two different attributes (orientation and location) of the same visual shape. It has good psychometric properties, is easy to score, and task difficulty can be easily manipulated via the length of sequences presented.

However, the letter rotation task requires the recognition of letter shapes and hence is not suitable for use with young children, illiterate individuals, or participants with severe reading impairments. Therefore, Alloway and colleagues created a new version of the task (the ‘Mister X’ task) for use with individuals from early childhood (5 years) to adulthood (69 years), as part of The Automated Working Memory Assessment (AWMA; [35, 36]). The Mister X task is considered visually more appealing (to children) due to its use of two illustrated human figures that are presented simultaneously on the screen, each holding a ball in one hand. One Mister X figure is rotated, and participants are required to judge whether both Mister X figures hold the ball in the same hand. In addition, participants must remember the spatial location of the ball of the right figure and recall all ball locations in the correct order, after the end of the sequence of trials. The Mister X task showed good psychometric benchmarks (test-retest reliability, r = .77, [36]) and a single clear loading on a VSWM factor for children as well as adults in an independent validation and application study [37]. The evidence of the task’s reliability and validity for use with adults and children is robust and made the Mister X task a good candidate for further development into a computerised adaptive task with automated item generation.

The present study

The objective of this study is to further adapt the Mister X task to improve the practicality, quality, and efficiency of VSWM measurement. We emphasise that the Mister X task and its predecessors have already been validated and were considered VSWM tasks in the literature. Consequently, our objective is to further validate the task in different settings. For instance, high WM abilities are often associated with obtaining high levels of expertise in given domains, like music [38–40]. Yet, the Mister X task we reproduced was only validated in populations of young children from the general population. Hence, we wanted to extend the domain further to see whether JaJ could also be validated within the context of high performing/specialist populations (Studies 3 and 4).

Here, we document the development and validation of a new adaptive, computerised and open-source WM task–Jack and Jill—based on modern psychometric techniques.

We expanded on the Mister X task in a number of ways. Firstly, in an attempt to engage younger participants, the style of the graphics was chosen to be more cartoon-like. Further, instead of one single character in the task, we created two characters of different genders (i.e., Jack and Jill). This may make the task more widely relatable, especially to younger participants who may find a Mister X character more abstract than the nursery rhyme characters Jack and Jill.

The second change concerns the design and calibration of the test based on item response theory (IRT, see e.g., [41] for an introduction). IRT provides a principled and flexible measurement framework for the Jack and Jill task. The IRT framework allows the computation of estimates of item difficulties and person abilities, as well as of measurement error, on the basis of a single underlying probabilistic model. Furthermore, the use of explanatory item response models [42, 43], which use theoretically motivated item features to predict the difficulty of individual items, can contribute to the construct validity of a test. Moreover, the problem of task impurity, found in previous approaches, can be mitigated by developing an empirically validated cognitive model of task performance via the IRT approach. Despite the theoretical and practical advantages of IRT-based testing, so far only very few established WM tasks make use of an IRT framework. Moreover, the available tests (e.g., [44]) are not available in an open computerised adaptive testing framework at present.

Using IRT as the basis of the Jack and Jill task also enables the automated generation of test items based on their item difficulties; and its implementation using Computerised Adaptive Testing (CAT, [45]). In CAT, the difficulty of items presented is matched dynamically to the current estimate of a participant’s ability. After each trial, the participant’s ability is re-estimated using the response data from the previous trial as well as the prior ability estimate. Because CAT aims to present only items that are as close as possible to the participant’s ability, it is maximally informative for estimating a participant’s ability with increasing precision. Thus, CAT helps to produce an efficient version of the test, able to capture the wide variability of WM found in heterogeneous samples, as well as mitigate fatigue effects. In particular, it offers the possibility to shorten or lengthen a task, guided by the knowledge of how changing the task’s length affects measurement error and reliability.

Our research was taken forward in multiple different settings for a few reasons. Firstly, it allowed us to recruit from a wider pool of participants, yielding a larger overall sample. Secondly, it allowed us to ensure our sample was more heterogenous and to validate our task in different cultural settings, making it more internationally robust. Thirdly, this required translating the test into several languages which are available for use in our release, and hence, makes the test more accessible. Finally, it allowed research teams to address more specific questions in separate work from that presented here.

The following sections report four studies designed to construct, calibrate, and validate a new IRT-based VSWM task that can be used in a computerised adaptive operation mode. Study 1 describes the construction and empirical calibration of the Jack and Jill task, using explanatory item response modelling of data from a large sample of the general adult population in the UK and the US. Study 2 describes the validation of the adaptive version of the Jack and Jill task in a sample of young adults from the UK and Australia. Study 3 reports the validation of the static version of the Jack and Jill task in a sample of adolescent participants from a talent development centre in Russia. Study 4 provides validation against a battery of established spatial tasks, using data from another adolescent sample from the same centre. Table 1 shows the overall design of our study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Design of the study.

https://doi.org/10.1371/journal.pone.0262200.t001

Method

Materials.

The Jack and Jill visuospatial working memory task. The JaJ task presents participants with pictures of a young female (“Jill”) and a young male (“Jack”) on a white background (see Fig 1). The task is divided into trials of different lengths (starting with a sequence of 1 to a maximum of 7 stimuli). Each sequence length is used twice with sequence lengths increasing over time, totalling 14 trials in one testing session. For a detailed task description see Fig 1. Following the scoring procedures of the Letter Rotation task and the Mister X task, participant responses were scored as correct (1) if the whole sequence of a trial was successfully repeated or as incorrect (0) otherwise. ‘Same/different’ judgements of hand positions were not used as the main unit of scoring as this task was used only to additionally load WM processes as in a dual-task paradigm. A participant’s VSWM span is indexed by the length of the sequence of ball positions that a participant is able to recall. The JaJ task can be considered a complex span task because it requires participants to switch between memory encoding (i.e., memorising the ball’s position) and mental rotation with a subsequent decision on visual input, which creates an additional cognitive load.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Schematic of the four consecutive screens of a JaJ trial of length two.

Note: Jill always stays in the same position holding a blue ball in her right (from the participant’s perspective) hand (Panels 1–4), while Jack rotates around his axis on each stimulus presentation and can hold a ball in either his right or left hand (Panels 1–2). Jack’s ball also moves, randomly taking one of the 6 marked possible positions on the screen (orange dots). On each stimulus presentation, participants are required to perform two tasks: a) indicate whether Jack holds the ball in the same hand as Jill; and b) memorise the current ball position. At the end of each trial, participants are asked to recall the sequence of the ball positions by clicking on the marked positions in the correct order (Panels 3–4). Cursor locations represent correct answers.

https://doi.org/10.1371/journal.pone.0262200.g001

A trial is considered an ‘item’ in the language of item response theory. The main parameter hypothesised to affect item difficulty is item length (i.e., the length of the sequence of ball positions).

All items were prepared in advance using independent random sequences of hand and ball positions. Ball position sequences can contain the same position more than once. No items are presented twice within one test session. In the beginning, participants are presented with step-by-step instructions and subsequently two training items of length 1 and 2 with feedback on the correctness of the responses regarding the hand judgements and ball positions. Participants do not receive feedback regarding their performance during the testing phase or after the task.

The computerised version of the JaJ VSWM task was programmed using the psychTestR [51] and Shiny frameworks [52] for the R programming language [53]. It took participants 5–15 minutes to complete.

Results

Whilst we do not use it for formal scoring or modelling, the percentage of correct responses in the secondary “same/different” judgement task was 93.1%, suggesting the task was taken seriously and would have contributed to loading VSWM.

On the main ball position task, averaged across all 14 trials, participants demonstrated an accuracy of 35.5% correct responses (SD = 28.9). Male participants had an average of 33.0% (SD = 28.5) correct responses while females had 39.2% (SD = 29) responses correct. This difference did not reach the common significance level according to a Welch’s t-test (t(195.74) = 1.59, p = .11).

After excluding items of length 1 the relationship between performance accuracy becomes approximately linear as shown in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Regression lines representing the decrease in response accuracy with increasing sequence length.

Note: The black line represents proportion of correct responses by length of sequences to be remembered; Error bars represent 95% CI of the proportion based on the standard error. The blue regression line was fit to sequences from length 1 to 7. The red line was only fit to sequences of length 2 to 7. The red regression line fits the empirical average accuracies connected by the dashed line fairly closely, suggesting an approximate linear trend.

https://doi.org/10.1371/journal.pone.0262200.g002

In order to account for the discontinuity in performance accuracy between items of length 1 and the linear trend for items of length 2 to 7, we created a binary dummy variable that codes items of length 1 as ‘1’ and ‘0’ for all other item lengths.

To formally model participant accuracy, we constructed an explanatory item response model in the form of a binomial mixed-effects logistic regression model [42] using item length and the dummy variable for item length = 1 as fixed effects and participant ID as random intercept effect which traditionally corresponds to the participant ability parameter in IRT models. Participant responses at the item level (correct/incorrect) served as a dependent variable. The function glmer from the R package lme4 [54] was employed for modelling.

According to the model, both the dummy variable and item length contributed significantly to the prediction of the correctness of the responses as shown in Table 2. As expected, longer items decreased the probability of responding correctly and an item of length 1 increased this probability. The model has a predictive accuracy of 86.7% when random effects were included. Without the random effect, prediction accuracy dropped to 71.3%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Logistic regression of item accuracy.

https://doi.org/10.1371/journal.pone.0262200.t002

In sum, the model confirmed our expectations for item length to affect task performance. In addition, the model has good predictive validity and the inclusion of individual differences information (i.e., random participant effect) explains a further considerable amount of variance in the data. Hence, the model was accepted as an explanatory model for the performance of the JaJ task. In a subsequent step, the item difficulty parameters for all item lengths as well as the discrimination parameter were extracted from this model for use with the 1-PL IRT (for technical details of parameter extraction and transformation see [43]).

Simulations using the randomCAT function from the catR package (version 3.16) [55], our derived IRT parameters and true abilities scores in range [-4, 4.15] and test lengths [5. 20] showed that the JaJ task produces scores in the range of [-1.89, 2.86]. The lowest ability value (-1.89) corresponds to the ability of remembering not more than 1 position consistently. This is a meaningful lower bound for a WM test but the assignment of the numerical value -1.89 is rather arbitrary. In the long run, it would be helpful to collect norms from a larger sample drawn from the general population (factoring in age as well), such that values of the IRT scale can be mapped to achievement percentiles in this population sample.

Discussion

Aggregated accuracy scores show that performance on the JaJ task strongly depends on item length with the exception of items of length 1 (see Fig 2). This result may mean that items of length 1 are processed by a different mechanism, possibly visual echoic or short-term memory that does not require the engagement of the central executive and the visuospatial sketchpad, at least not to the same degree as is the case of longer items. In addition, the chance of guessing the correct response for items of length 1 is considerably higher (.167) compared to items with length > 1 (guessing level < .03). This could also have made a (small) contribution to the better performance of length 1 items. In any case, to account for the different behaviour with length 1 items, and a potential change in memory strategy, the explanatory model includes a dummy variable for items of length 1 in addition to the variable that codes item length numerically.

The item difficulty parameters and discrimination parameters extracted from this model form the basis of scoring participant ability in Studies 2 and 3 as well as the computerised adaptive version employed in Study 4. Note that by default participant ability estimates are scaled to have a mean of 0 and a standard deviation of 1 which enables the easy comparison with sample averages from subsequent studies. Item parameters of the 1-PL IRT model are given in Table A1 in the S1 Appendix.

Method

Results

Because the gender of 4 participants was missing (reason unknown), an independent samples t-test using pairwise deletion (resulting in 134 cases with complete test scores) was conducted and did not indicate any gender differences (mean θ_male = 0.68, SD θ_male = 0.95, mean θ_female = 0.61, SD θ_female = 0.98, t = -0.42, p = .68). Further analysis was carried out on the whole sample. Descriptive statistics for the whole sample are presented in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Descriptive statistics for all performance tasks in Study 2.

https://doi.org/10.1371/journal.pone.0262200.t003

Participant IRT ability scores (θ) on the JaJ task ranged [-1.5, 2.36] with a mean of 0.64 (SD = 0.97); the theoretical and empirical range of the sum scores was BDS [0… 12], MUF [0… 14], JaJ [0… 14]. This indicates that the sample of participants tested under laboratory conditions in Study 2 had a higher average but a very similar standard deviation of θ-scores compared to the calibration sample from Study 1, where the mean θ-score was 0 with a standard deviation of 1. Using the explanatory IRT model to convert the θ-scores back to the standard scale of the VSWM tasks indicated that participants in Study 2 had a mean complex memory span of 4.01 (SD = 2.05). However, note that the prediction of complex WM span ability values from IRT θ-scores can produce WM span estimates outside the scale of the task (i.e., < 0 and > 7). Therefore, only 82.7% of participants with a converted WM span estimate between 0 and 7 (after rounding) were included in the calculation of the mean complex memory span. 5% of participants had an estimated score below and 12.2% an estimated score above this range. For comparison, the mean θ-score of 0 in the calibration sample corresponds to a complex span estimate of 2.4.

The average standard error of measurement (SEM) of the θ-scores was .37. The marginal empirical reliability computed from the standard errors of measurement was .83. As expected, all three WM task scores correlated positively and significantly (p-values < .001 after correcting for multiple comparisons using Holm’s procedure [61]). The JaJ had moderate correlations with both the MUF task (r = .44) and the BDS task (r = .37) (see Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Pairwise correlations of all tasks.

https://doi.org/10.1371/journal.pone.0262200.t004

We proceeded to assess the hypothesised measurement model for the VSWM factor with a minimum residual exploratory factor analysis. All factor loadings were > .50 which indicated that the tasks represented the factor well (see Table 5). The VSWM latent variable explained 47% of the variance in the observed VSWM task. JaJ had a uniqueness value (u²) of .66.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Exploratory factor analysis results for the hypothesized VSWM factor.

https://doi.org/10.1371/journal.pone.0262200.t005

Discussion

The data from Study 2 shows that the JaJ task moderately correlates with two other tasks purported to measure aspects of (VS)WM capacity (backwards digit span and a visuospatial memory updating task), suggesting that it is a valid measure of an aspect of WM. The good factor analysis solution suggests that the task set combines well to reflect a latent variable, which was hypothesized to indicate VSWM. Whether or not VSWM is the underlying latent variable, the factor analysis shows there is a latent factor among the three tasks, which is probably more specific than g, since three tasks involve manipulating visually presented information, and is therefore likely to do with VSWM. However, this represents an assumption, rather than a proof.

However, JaJ was correlated more strongly with the visuospatial updating task (measured by MUF; r = .44) than with the memory span task (measured with the BDS; r = .37) in the same sample of participants. This may mean that the JaJ task taps into the visuospatial component of WM more than verbal or numerical components and could serve as a task to assess the capacity of the visuospatial sketchpad construct. Hence, these correlations could also suggest the divergent validity of the newly developed measure within WM modelling. However, the R package cocor v1.1–3 [62] indicated that the correlations were not statistically different from one another, according to Steiger’s [63] procedure (z = 0.94, p = 0.35).

Taken together, the factor analysis suggests the task set indicates shared variance to do with similar VSWM processes (e.g., short-term visual memory/manipulation). Yet, the moderate sizes of correlations and the presence of substantial unexplained variance in the factor solution suggest the three tasks might tap into slightly different constructs within WM more generally (e.g., MUF and BDS measure aspects other general aspects of WM capacity and not specifically VSWM).

The average mean standard error of measurement for the JaJ task was .37, i.e., about 1/3 of a standard deviation of the ability scores, which can be considered acceptably low. The marginal reliability of .83, as computed using the marginal_rxx() function from the R package mirt, was in a good range. This measure of reliability in an IRT-context corresponds to Cronbach’s α from CTT [64, 65]. However, a limitation of our approach is that we do not know the test-retest reliability of our derived score.

Average WM ability estimate, and the corresponding complex span capacity, were considerably higher in Study 2 compared to Study 1. This ability to discriminate between the two samples (Study 1 vs. Study 2.) can be considered as an additional strength of the JaJ task. Several factors may contribute to these differences. First, the testing conditions were different: controlled laboratory setting in Study 2 vs. uncontrolled online conditions in Study 1. Second, different recruitment criteria were applied: predominantly student population with an age cap at 50 in Study 2 (to focus on the demographic our sample of convenience) vs. participants from a market research panel representing the general population without any age cap in Study 1. This result is consistent with previous evidence showing that WM performance differs as a function of college status (e.g., [66]). Besides, because participants in Study 1 completed the study outside of the laboratory, at home, they could have used different devices (e.g., laptops, tablets, or mobile phones), which could also affect accuracy. Finally, a future study should specifically target construct validity with a measure of fluid intelligence and other WM tasks included alongside the JAJ to explore the net of nomothetical relationships of the JAJ more closely. Although, we note that measures of fluid intelligence and WM tend to exhibit reasonably sized correlations.

Method

Results

Descriptive statistics for all measures are presented in Table 6. Skewness and kurtosis of most variables except for JaJ θ varied within an acceptable range (i.e., below the cut-off of 2 as recommended by Field [86]). The relatively high values for kurtosis and skewness of the JaJ sample distribution are expected for this selected sample, which included almost 50% of adolescents, showing high achievement in STEM (Science, Technology, Engineering, and Mathematics). STEM-selected adolescents were previously shown and assumed to demonstrate extremely high VSWM capacity [14, 20, 21].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Descriptive statistics for variables used for analysis in Study 3.

https://doi.org/10.1371/journal.pone.0262200.t006

JaJ θ scores ranged [-1.50, 2.36] with a mean of .94 (SD = .68), corresponding to an average complex WM span of 4.99 (SD = 1.63). This score was computed from 90.1% of the participants with a converted complex span score between 0 and 7 (0.8% scoring < 0 and 8.5% scoring > 7). Thus, the high-achieving adolescents in Study 3 were able to remember one extra position on average, compared with adults in Study 2.

In addition, JaJ θ scores had significant positive associations with all measures of academic achievement and the CBTT as shown in Table 7. In contrast, the correlations between JaJ θ scores and self-reported behavioural difficulties were non-significant at the .05 level.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Pearson’s correlations between behaviour, achievement and VSWM measures.

https://doi.org/10.1371/journal.pone.0262200.t007

The focus of the analysis was on correlations which were shown to stabilise at a sample size of 250 [87]. A post hoc analysis of achieved power showed that with N = 263, a weak-to-moderate effect size of ρ = [0.15–0.53] and the conservative parameters of α = .01 and a two-tailed hypothesis for a bivariate normal correlation model suggested that achieved power was in a range [0.45, 0.99].

An independent samples t-test suggested equal performance of males (mean θ = .96, SD = .74) and females (mean θ = .91, SD = .62; t = .60, p = .54). JaJ θ scores showed a significant positive correlation with the age of participants (r = .19, p < .003). However, the effect of age is confounded by age differences across the three educational tracks. Science (mean age = 15.77, SD = .68) and art (mean age = 15.83, SD = 1.07) students were significantly older (F(2, 259) = 44.43, p < .001) than sport students (mean age = 14.69, SD = .91). A linear model with age and educational track as independent variables and θ as dependent variable showed that the main effect of age and the interaction between age and track were non-significant (p = .30 and p = .56, respectively) while the main effect of track was significant (F(2, 246) = 5.16, p = .006, η_p² = .04). Post-hoc tests showed that participants from the science track had significantly higher JaJ scores (mean θ_Science = 1.23, SD = .44) in comparison to art (mean θ_Art = .84, SD = .70) or sport students (mean θ_Sports = .61, SD = .77), while art students outperformed sport students (see Fig 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Jack and Jill task performance by educational track.

Note: Error bars represent the 95% CI around the mean.

https://doi.org/10.1371/journal.pone.0262200.g003

Measures of reliability and internal consistency from classical test theory as well as IRT indicated that the JaJ task was sufficiently reliable by common benchmarks. The average standard error of measurement for the IRT-based theta scores was sufficiently low (SEM(θ) = .33). Cronbach’s α for the 14-item static version of the test was .78 and the equivalent empirical marginal reliability derived from the underlying IRT model was .79.

Discussion

The aim of Study 3 was to assess the validity and reliability of the JaJ VSWM task in a sample of adolescents selected for high achievement in Science, Arts, or Sports. The test showed sufficient reliability as shown by indicators from classical test theory and IRT. JaJ also showed good concurrent validity as evidenced by the correlation with Corsi Block-tapping test score. The moderate correlation (r = .28) with the CBTT was in a similar range to the correlation (r = .36) between letter rotation scores and CBTT reported by Miyake and collaborators [34]. The moderate correlation between the two tasks may be explained by the fact that JaJ uses a dual-task paradigm, which requires both information processing from the central executive and information storage in the visuospatial sketchpad component, whereas the CBTT only requires encoding and retrieval from visuospatial memory and is not based on a dual-task paradigm.

Furthermore, the data showed a significant and positive association between performance on the JaJ task and age, which disappeared after controlling for the track. The absence of a reliable association between JaJ task performance and age might be due to the narrow age range (14–18) but is consistent with previous literature [23, 66]. In addition, we found correlations of small to moderate size between JaJ scores and measures of academic achievement: Algebra and Russian language. This is consistent with previous research which showed VSWM was related to both maths [14, 88] and reading abilities [14, 18]. JaJ performance explained a small to moderate amount of the variance in the grades data (from 2% in the Russian language to almost 10% in Algebra grades). The stronger associations with maths compared to language grades are in line with previous studies [14, 67]. It is possible that VSWM specifically helps to represent number magnitudes spatially in the form of a mental number line and thus may help to represent maths tasks schematically and solve them efficiently (e.g., [89]). A moderate correlation between JaJ performance and maths achievement is consistent with the correlations of .31 between VSWM and maths reported in the recent meta-analysis by Peng, Namkung, Barnes, and Sun [90]. A correlation between Russian language school grade and JaJ task is lower than the correlation between English achievement and VSWM measures (.29 and .33) reported in Jarvis and Gathercole [91] and a similar correlation between WM and reading abilities of r = .29 reported in the meta-analysis by Peng and collaborators [92]. Overall low correlations between JaJ score and school achievement could be explained in the light of limitations of the achievement measures: the restricted scale [3… 5], restricted variance in this selected population (54% of people with the highest grade on Russian language and 59% of people with the highest grade on Maths) and potential biases and differences in standards, reflected in the teacher ratings [84, 85, 93, 94]).

As in Studies 1 and 2, no significant gender differences were found for JaJ scores in Study 3. This is at odds with some previous studies (e.g., [95]) which report male advantage in VSWM tasks performance. However, numerous other studies similarly report non-significant gender differences (e.g., [66, 96]). Gender differences obtained in some studies may be due to the modality of testing or the parameters measured, reflecting the use of different strategies [97] or differences in other abilities such as spatial [98, 99]. Therefore, the JaJ task can be considered a non-biased test with respect to gender.

The results from Study 3 showed that, on average, STEM-selected students significantly outperformed Arts and Sports students, while Arts students outperformed Sports students. These differences in VSWM may arise from the different expertise profiles of the participants as suggested by previous literature [67, 69, 70]. However, the large variability in the VSWM abilities within the Arts and Sports tracks is worth noting and may be explained by the fact that participants from the Arts track comprised of ballet dancers, musicians, and visual artists, while participants from the Sports track included hockey players, figure skaters, and chess players. Thus, the heterogeneity of expertise within the tracks might have influenced differences in VSWM between tracks. Further research is needed to evaluate the JaJ task performance in specific sub-domains of expertise.

Method

Materials.

Participants filled in the same computerised socio-demographic inventory as in the Study 3.

Computerised adaptive version of the JaJ task. In addition to the programming dependencies listed earlier, the a-JaJ utilised the psychTestRCAT [51] and catR [55, 59,] packages for R. The a-JaJ employs a precalculated item bank with randomly generated items of lengths [1… 7].

The first item administered in each test session is constrained to be a random item of length 2, corresponding to an average ability level of 0.13 on the z-score metric of the IRT model. After responding to an item (i.e., a trial of ball sequences) the participant’s ability is re-estimated using Bayes modal estimation with a Gaussian prior of mean 0 and standard deviation 1 and using the item difficulties from the IRT model described in Study 1 (for additional information see Table A1 in S1 Appendix). The next item is then randomly selected from the subset of items in the item bank that are maximally close in difficulty to the current ability estimate using Urry’s criterion [59]. As in Studies 1–3, the test was terminated after 14 items. Additionally, at the end, the new adaptive test version recomputes the participant ability estimate using weighted maximum-likelihood estimation [60].

Raven’s matrices. Raven’s progressive matrices [49] were used to assess the level of non-verbal intelligence of participants. The test included a set of 30 matrices of increasing difficulty, each of which contains a missing element. Participants choose the missing part out of 6 or 8 possible variants. Elementary sets (A, B) from the original set were dropped from the test. Items 1, 3, 5, 7, 9, 11 from parts C to E and all 12 items from part F were selected to cover the expected high ability range of the participants.

Spatial ability battery. Four tasks from the computerised spatial ability battery King’s challenge [50] (for Russian adaptation see [102, 103]) were used (See Table 8 for the tasks descriptions). A subtest is ended once a participant gives incorrect answers on four consecutive items.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 8. Spatial ability tasks.

https://doi.org/10.1371/journal.pone.0262200.t008

Results

Descriptive statistics for all measures are presented in Table 9. Skewness and kurtosis of most variables except for a-JaJ θ varied with an acceptable range (i.e., below the cut-off of 2 as recommended by Field [86]). Again, the relatively high values for kurtosis and skewness are presumably due to the selected sample used in this study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 9. Descriptive statistics for study variables.

https://doi.org/10.1371/journal.pone.0262200.t009

a-JaJ θ scores varied from -.80 to 2.74 in the current sample, with a mean of 1.16 (SD = .54). This corresponds to an average WM complex span capacity of 5.34 positions (SD = 1.56) computed from the scores of those 77.7% of participants with a score between 0 and 7. Only 2.1% of the participants had an estimated complex span below 0, but 22.2% of participants had estimated span scores > 7. As expected, a-JaJ θ had significant positive associations with all measures of academic achievement (Table 10). It did not show a significant correlation with the age of participants (r = -.01, p = .80) or reveal any significant difference of the JaJ performance (t = .93, p = .33) for females (mean θ = 1.09, SD = .55) and males (mean θ = 1.17, SD = .62). JaJ was quite strongly correlated with the Raven’s matrices task (r = .57) and the non-WM visuospatial measures (r = .46-.54).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 10. Pearson’s correlations between JaJ, general cognitive ability, spatial abilities and achievement measures.

https://doi.org/10.1371/journal.pone.0262200.t010

Fig 4 (Panel A) expands this correlational information for the a-JaJ task at different lengths, (i.e., as if the task had been terminated after 1, 2, 3, …, 14 trials). The figure shows how correlations with the different spatial ability subtests change as the number of trials on the JaJ task increases. For most spatial ability subtests as well as for Raven’s matrices score correlations seem to reach a plateau at around 7 trials, after which the value of the correlation coefficient changes only to a considerably smaller degree.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Indicators of correlational test validity, measurement error and reliability by task length.

A. Correlations with other cognitive performance tasks by length of the a-JaJ test. B. Standard measurement error by test length. C. Marginal reliability of test scores by test length.

https://doi.org/10.1371/journal.pone.0262200.g004

Both indicators of reliability (i.e., the mean standard error of measurement (SEM) of the IRT θ scores and the empirical marginal reliability of the scores) suggest that the a-JaJ task after 14 trials has a good level of reliability. Mean SEM was .27 and hence about a quarter of a standard deviation of the θ scores variability. The corresponding empirical marginal reliability was .80. Fig 4 (Panels B and C) shows how measurement error decreases and reliability of the test increases as the number of trials of the a-JaJ grows. In contrast to the correlations with the spatial ability battery in Fig 4 (Panel A), reliability does not seem to reach a plateau. However, the gain in reliability diminishes gradually as the number of trials increase.

Discussion

The aim of Study 4 was to test the validity and reliability of the adaptive JaJ with a sample of adolescents selected for high achievement. After 14 trials, the a-JaJ had an acceptably low measurement error and showed good reliability. These benchmark indicators were slightly better for the 14-item adaptive version in Study 4 compared to the 14-item static version used in Study 3. The decreased measurement error and slightly larger reliability can be explained by the fact that, for each trial, the adaptive mechanism (described in the Method section) selects the item that maximises the information gathered about an individual participant’s ability. However, the small gain in reliability is only a secondary benefit of the adaptive version. The primary benefit of adaptive IRT-based tasks is the possibility to shorten or lengthen the task guided by the knowledge of how changing the task’s length affects measurement error and reliability. As expected, the task’s reliability increases with the number of trials as depicted in Fig 4 (Panel B and C). The information from these figures provides the basis for principled decisions in practical application scenarios where overall test time is limited and when the trade-off between test length and measurement error needs to be considered. The a-JaJ also demonstrated some concurrent validity through the significant and substantial positive correlations with the spatial ability tasks and the non-verbal intelligence task. The correlations with spatial ability measures varied from .46 to .54, which is similar to the correlations between VSWM measures and spatial visualisation measures (r = .42 –.49) reported in Miyake and collaborators [34] and in Kane and collaborators (r = .39 –.51) [104]. This suggests that the a-JaJ also partially taps into spatial ability (specifically, spatial visualisation) and non-verbal intelligence.

Furthermore, the data of Study 4 replicated the results of Study 3 regarding substantial associations between VSWM performance and academic achievement measures. Lower correlations between a-JaJ scores and achievement measures might be due to narrower variance in variables in comparison to Study 3. This is because the sample in Study 4 consisted of mostly Science students (77.4%) who received high school grades and performed JaJ at a very high level. The data from Study 4 again show no gender and age differences for the task. It is also worth noting that more than 20% of the participants demonstrated estimated span scores > 7, which suggests a possible ceiling effect for this sample. Further research is needed to adapt the task for high-ability individuals.