Study design and setting

We conducted a prospective observational study of patients admitted to two adult tertiary care centre intensive care units (ICUs) in London, Canada. The study was approved by our local research ethics board (WesternREM, study identification number 108156).

Patients

We included adult patients (18 to 80 years of age) who were mechanically ventilated for a minimum of 24 hours. The latter criterion excluded routine post-operative patients and non-critically ill patients requiring a brief period of mechanical ventilation (e.g. for acute intoxication).

Patients were excluded if they had a pre-existing diagnosis of dementia, new or pre-existing diagnosis of neurological disease known to affect cognitive function (e.g. stroke, head trauma, intracranial hemorrhage, traumatic brain injury, or intracranial malignancy), impaired vision or significant upper extremity weakness precluding use of a computer, inability to communicate in English or were unable to or declined to provide informed consent, or active delirium as determined by the Intensive Care Delirium Screening Checklist (ICDSC) [23].

Measurements

Study protocol

One investigator (KH or SM) or a research assistant screened patients for eligibility using medical charts and discussion with bedside nurses. We approached eligible patients and obtained written informed consent prior to commencing the study. Patients were deemed to be adequately alert and capable of providing consent if they were able to participate in the consent discussion.

We recorded demographic and clinical variables including age, sex, admission diagnosis, hospital and ICU admission dates, and Nine Equivalents of Nursing Manpower Use Score (NEMS) [25] on the day of testing from each patient’s chart or electronic medical record.

Patients completed cognitive assessment using a laptop computer with an attached computer mouse in their hospital bed in the ICU or shortly after discharge to the ward. Investigators assisted patients by creating a personalized login and password on the CBS study webpage. A standardized set of written and pictorial instructions and a short instructional video preceded each cognitive test. Patients were provided with as much time as they needed to review the instructions prior to beginning each test. Patients completed each of the tests in sequence until the entire battery of 12 tests was completed or they were unable to continue due to testing related fatigue.

We asked a relative or friend who knew the patient well to complete the IQCODE. Family members responded to the IQCODE questions by telephone if they were not available in person.

Data analysis

Demographic and clinical variables were reported using descriptive statistics, expressed as frequency and percentage for categorical variables and mean and standard deviation (SD) or median and interquartile range (IQR) for continuous variables as applicable.

Patients were defined as having cognitive impairment on a given test if their raw score was ≥ 1.5 SDs below age- and sex-matched controls derived from the CBS normative database [15]. We then compared patients’ cognitive performance with available data from healthy age- and sex-matched control data by converting raw scores into z-scores.

To determine patients’ scores on each of the three cognitive domains (reasoning skills, short-term memory, and verbal processing), the z-score for each individual test was multiplied by a value that reflected the contribution of that test to each cognitive domain (i.e., factor loading) as established by Hampshire and colleagues [15]. Patients’ overall score on each cognitive domain was therefore the sum of the weighted (factor loaded) scores for that domain across all 12 tests. Data imputation was only used to compare domain scores for patients who completed all 12 tests. The scores are designed such that the healthy population mean on each cognitive domain is 0 and the SD is 1.0.

Finally, we defined an abbreviated CBS battery that included six of the 12 CBS tests which most strongly reflect one of the three cognitive domains based on previously published data (Reasoning Skills: Odd One Out and Rotations; Short-term Memory: Paired Associates and Monkey Ladder; and Verbal Processing: Digit Span and Verbal Reasoning) [15]. We replaced the scores for the omitted tests with their expected values given the six observed test scores and the known correlation structure among the tests in the population. The correlation structure between the 12 tests in the CBS battery was derived from a sample of 44,600 [15]. This method has been shown to be most accurate when calculating principle component analysis scores in the presence of missing data [26]. We then calculated a z-score for each patient on each of the three cognitive domains based on the abbreviated CBS and compared this score to the z-scores calculated based on the complete CBS battery.

Discussion

Our results demonstrate that a web-based patient-administered neurocognitive test battery can identify domain-specific cognitive impairment in critical illness survivors. Unlike paper-based cognitive assessment tools, the web-based battery can be self-administered by patients from any location (ward, rehabilitation facility, home) after minimal instructions on logging in and starting the tests, obviating the need for patients to travel to hospital to be tested by specially trained staff. The self-administering nature of web-based battery may enable large-scale studies similar to those completed in other populations [14–16], and provides opportunity for repeated remote monitoring of cognitive recovery in patients who are unable to return to clinic. Given high prevalence of functional disability, post-traumatic stress disorder, and challenging travel logistics, remote web-based testing of cognition can provide a more patient-centred approach to monitoring cognition in ICU survivors.

The majority of previous studies have used traditional, paper-and-pencil testing to assess cognitive function in critical illness survivors [1–9]. Screening tools such as the Mini-Mental State Examination (MMSE) [27], have been designed to screen for overt dementia, but lack comprehensiveness and have poor sensitivity for more subtle, yet consequential cognitive deficits in critical illness survivors [28].

More comprehensive cognitive batteries such as the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) [29] and Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) [30], have well-established psychometric properties (RBANS [31,32]; WAIS-IV [33,34]) have been widely used in various patient populations (RBANS [31,32,35,36]; WAIS-IV [37,38]), but are not optimal for some natural history studies of critical illness survivors due to the need for in-person testing by trained personnel. Our results demonstrate that web-based cognitive testing is both feasible and able to detect subtle changes in cognitive domains similar to traditional paper-and-pencil tests.

There are several advantages to the CBS battery that make it an attractive tool for assessing cognitive dysfunction in ICU survivors. Similar to traditional cognitive assessment tools, the CBS battery used in this study is comprehensive, with 12 individual tests assessing cognitive function across three distinct cognitive domains. It has been used to study cognition in several large patient cohorts [14–16] and has an established large normative database to enable sex- and age-matched comparisons of patients’ performance to healthy controls. Furthermore, the CBS battery can be administered in the functional magnetic resonance imaging scanner, enabling mapping of cognitive results to neuroimaging data [14,39].

Optimization of the CBS battery for ICU needs by addressing feasibility issues identified in this study would enable its use for remote monitoring of cognitive recovery in individual patients, and provide an objective outcome metric for tracking patient’s cognitive health and gauging effectiveness of candidate preventative, therapeutic, and rehabilitative interventions. Given that repeated testing can lead to practice effects, the CBS battery has been optimized for repeated measures studies by incorporating proprietary algorithms that generate a unique set of problems for each assessment, facilitating longitudinal studies with minimal ‘practise effects’. However, whether repeated testing leads to practice effects in ICU survivors requires further dedicated studies. Furthermore, given comparable performance of an abbreviated 6-test version of the CBS battery in our study, its use in future studies would address the issue of test fatigue related to battery length. While we did not administer the abbreviated 6-test battery in this study, we expect that its duration will be approximately half of the 12-test battery (i.e., approximately 20 minutes). This will need to be confirmed in dedicated future studies. In addition, the CBS has recently become available for use on a portable tablet computer, which may further simplify its administration to ICU patients by obviating the need for the hand-computer mouse coordination.

This study has several limitations. Our small sample size allowed us to determine the feasibility of web-based testing using the CBS battery, but prevents us from drawing any definitive conclusions regarding cognitive outcomes in patients. In the future, we will use our feasibility data to optimize CBS battery for ICU use and repeat cognitive assessment in a larger ICU cohort, with repeated measurements in the same patients to assess the ability to track the cognitive recovery process over time.

Web-based cognitive testing in general is not without limitations. The inability to monitor patients during testing to determine their level of engagement, patients’ access to and inexperience with use of a computer, and attrition in follow-up studies are limitations inherent to the use of web-based (remote) cognitive testing that must be addressed prior to large-scale use of such testing platforms in ICU survivors.

In our study, patients’ performance in our study may have been affected by their ability to use a computer mouse to navigate through the tasks. In future studies, we plan to test tablets as an alternative to computer-mouse combination based on patient preference. Finally, in the absence of a “gold standard” test for cognitive impairment in this study, the psychometric characteristics of the CBS could not be assessed, although previous studies in healthy controls [15], and elderly neuropsychiatric patients [40], have confirmed that it is comparable to standard neuropsychological test batteries in terms of its latent structure and relation to age.

We identified that the length of the CBS cognitive battery may be one challenge to its feasibility in larger studies. The proposed abbreviated CBS battery is one potential solution to this challenge. Future studies are needed to assess the feasibility of the abbreviated CBS battery.