Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Pseudo-colouring an ECG enables lay people to detect QT-interval prolongation regardless of heart rate

  • Alaa Alahmadi,

    Roles Conceptualization, Data curation, Formal analysis, Methodology, Visualization, Writing – original draft, Writing – review & editing

    Affiliation Department of Computer Science, The University of Manchester, Manchester, United Kingdom

  • Alan Davies,

    Roles Writing – review & editing

    Affiliation Division of Informatics, Imaging and Data Sciences, School of Health Sciences, The University of Manchester, Manchester, United Kingdom

  • Markel Vigo,

    Roles Supervision, Writing – review & editing

    Affiliation Department of Computer Science, The University of Manchester, Manchester, United Kingdom

  • Caroline Jay

    Roles Conceptualization, Supervision, Writing – review & editing

    caroline.jay@manchester.ac.uk

    Affiliation Department of Computer Science, The University of Manchester, Manchester, United Kingdom

Abstract

Drug-induced long QT syndrome (diLQTS), characterized by a prolongation of the QT-interval on the electrocardiogram (ECG), is a serious adverse drug reaction that can cause the life-threatening arrhythmia Torsade de Points (TdP). Self-monitoring for diLQTS could therefore save lives, but detecting it on the ECG is difficult, particularly at high and low heart rates. In this paper, we evaluate whether using a pseudo-colouring visualisation technique and changing the coordinate system (Cartesian vs. Polar) can support lay people in identifying QT-prolongation at varying heart rates. Four visualisation techniques were evaluated using a counterbalanced repeated measures design including Cartesian no-colouring, Cartesian pseudo-colouring, Polar no-colouring and Polar pseudo-colouring. We used a multi-reader, multi-case (MRMC) receiver operating characteristic (ROC) study design within a psychophysical paradigm, along with eye-tracking technology. Forty-three lay participants read forty ECGs (TdP risk n = 20, no risk n = 20), classifying each QT-interval as normal/abnormal, and rating their confidence on a 6-point scale. The results show that introducing pseudo-colouring to the ECG significantly increased accurate detection of QT-interval prolongation regardless of heart rate, T-wave morphology and coordinate system. Pseudo-colour also helped to reduce reaction times and increased satisfaction when reading the ECGs. Eye movement analysis indicated that pseudo-colour helped to focus visual attention on the areas of the ECG crucial to detecting QT-prolongation. The study indicates that pseudo-colouring enables lay people to visually identify drug-induced QT-prolongation regardless of heart rate, with implications for the more rapid identification and management of diLQTS.

Introduction

Background and significance

An electrocardiogram (ECG or EKG) is a graphical representation of the electrical activity of the heart, widely used as a clinical tool for monitoring heart function and detecting cardiac pathologies [1]. An important measurement on the ECG is the QT-interval, which represents the duration of the ventricular depolarization and repolarization cycle. It is measured from the beginning of the QRS complex (illustrating ventricular depolarization) to the end of the T-wave (showing subsequent repolarization) [1, 2], as shown in Fig 1.

thumbnail
Fig 1. An illustration of electrocardiogram (ECG or EKG) waveforms and the QT-interval.

Different ‘waves’ (peaks and troughs) are labelled with letters and represent different stages of a heartbeat. The QT-interval is measured from the beginning of the Q-wave to the end of the T-wave, identified here using the tangent method drawn at the maximum down-slope of the T-wave.

https://doi.org/10.1371/journal.pone.0237854.g001

The QT-interval is of considerable clinical importance, primarily because its prolongation can increase the risk of a life-threatening arrhythmia known as Torsades de Pointes (TdP), a form of polymorphic ventricular tachycardia that is the leading cause of sudden cardiac death in young, otherwise healthy people [36]. Prolongation of the QT-interval indicates a cardiac disorder known as ‘long QT syndrome’(LQTS), which is caused by the malfunction of cardiac ion channels impairing ventricular repolarization; it manifests as a longer QT-interval than normal on the ECG [7, 8]. The TdP arrhythmia is often precipitated by triggers such as physical activity (notably swimming) [9], and stress-related emotion [10]. Athletes with LQTS are thus particularly at risk of TdP episodes [9, 11, 12]. A major difficulty with identifying LQTS is that it is often asymptomatic; sudden cardiac death can be the first clinical manifestation, and therefore it may go undiagnosed, or underdiagnosed, without ECG assessment [1315].

LQTS can be congenital—a result of genetic mutations in cardiac channelopathies, as seen in Romano–Ward syndrome [16]—or acquired, resulting from the clinical administration of certain drugs [3, 17]. There are many commonly prescribed medications that can prolong the QT-interval, including antihistamines, antibiotics, antidepressants and antiarrhythmic drugs [3, 17]. Drug interactions that block the human ether-a-go-go-related gene (hERG) potassium channel are the most common cause of drug-induced long QT syndrome (diLQTS) [18, 19]. These drugs are associated with large and increasing numbers of sudden deaths, and preventive strategies could therefore save many lives [3, 20, 21].

Frequent ECG monitoring is strongly recommended for people at risk of acquiring diLQTS, including patients on a known QT-prolonging drug and patients participating in a clinical trial testing a new drug [22, 23]. A baseline ECG, before taking a QT-prolonging drug, and follow-up ECGs to monitor the QT-interval are advisable [24, 25]. Recent innovations in home healthcare technologies have made it possible to record high-quality, clinically reliable ECG data outside of the clinical environment [2629], but interpreting such data correctly in a timely manner still remains a major challenge [30, 31]. One approach involves monitoring patients’ ECGs remotely, where the ECG report is referred to a clinician who specialises in ECG interpretation [2629]. However, a single dose of a QT-prolonging drug could dramatically prolong the QT-interval within 24 hours for some patients, with the risk of TdP increasing with continued use [32]. At present, remote expert monitoring is too infrequent and costly to be an effective way of managing this issue for everyone at risk of acquiring diLQTS.

While there are computerised methods for interpreting the ECG, the accuracy of these methods remains limited [30, 31]. Challenges of automating ECG interpretation include the correct recognition of the ECG waveforms, in particular the amplitude and duration characteristics (which differ substantially across individuals), and the precise determination of the onset and offset of the different waves and complexes (P-wave, QRS complex, T-wave) [30].

Automated QT measurement algorithms have proved unsatisfactory for detecting LQTS in particular [3339]. Garg and Lehmann [33] found that even a widely used computerized ECG analysis system was not able to detect QT-interval prolongation in 52.5% of patients affected. Research has also shown that drug-induced QT-prolongation can be underestimated and under-reported by computerised methods in patients on Methadone, a drug that is infamous for prolonging the QT-interval and increasing the risk of TdP [38]. A major challenge for automated QT algorithms is identifying the precise end of the T-wave (the terminal point), especially when the T-wave’s morphology is abnormal [2, 40, 41]. This is particularly problematic, as QT-prolonging drugs also often affect the morphology of the T-wave; for example, patients on pure hERG-blocking drugs can develop flat, asymmetric, and notched T waves, whilst patients on multi-channel blocking drugs (e.g. hERG-blocking with lesser calcium and late sodium block) can develop more distorted, bizarre, T-waves [42]. In fact, specific T-wave patterns can aid detection of LQTS [43], and large T-U waves are known to precede TdP [44]. These indicators are lost by automated QT algorithms, however, as they work by abstracting the ECG data into a number (the calculated length of the QT-interval). As such, the human visual validation of QT-prolongation on the ECG remains mandatory in clinical practice, and provides the richest information for recognizing LQTS [30, 3339].

Giving laypeople the means to detect when the QT-interval is prolonged without having to rely on an external interpreter could lead to a step-change in the detection and management of diLQTS, as well as helping to reduce the development of life-threatening complications in situations that place individuals at risk of QT-interval prolongation. ECG interpretation is complex, however, and detecting QT-prolongation on the ECG is particularly difficult, even for clinicians [45, 46]. From a perceptual-cognitive perspective, this may be related to the fact that people are poor at perceiving quantity represented along a horizontal scale [4749]. Research has also shown that changes in the T-wave morphology and/or artifacts in the ECG signal can cause misinterpretation of the QT-interval length [40, 41, 50, 51]. The effect of heart rate on the QT-interval is another challenge, as it is the proportionate rather than absolute length that is important, and it is common to misinterpret the QT-interval at heart rates that differ from the ‘standard’ 60 bpm [52].

Visualisation techniques have the potential to highlight abnormal changes within the ECG, by supporting intuitive visual perception of the issues. Pre-attentive processing theory, which outlines a set of visual properties known to be detected rapidly and accurately by the human eye, is considered to be especially powerful for designing effective visualisations [5355]. Colour is one of the most effective pre-attentive attributes that is noticed without conscious effort [56, 57]. A useful technique is pseudo-colouring, which encodes continuously varying values using a sequence of colours [55]. It has been widely used to support diagnosis from medical images, including breast disease [58], and for highlighting details in organs and bones structures that would otherwise be difficult to perceive [59, 61]. It is also used extensively in geographic and time series visualisations, where applications include encoding elevation in the data or showing changes over time [55, 60]. To our knowledge, no prior work has used pseudo-colouring to support ECG interpretation.

In this paper, we investigate whether a pseudo-colouring technique can reliably show prolongation of the QT-interval on an ECG, such that it be identified by a lay person. In an earlier feasibility study, we found that superimposing pseudo-colouring on the ECG using a spectrum-approximation colour sequence significantly improved people’s ability to detect increases in the QT-interval at a low normal heart rate, when compared with a reference ECG stimulus showing a normal QT baseline [51]. This initial investigation had several limitations that affected the generalisability of the results. Firstly, the ECGs all had a single, regular heart rate of 60 bpm, and it is unknown how the technique would perform at higher or lower heart rates. The ECGs belonged to the same patient, who was taking the QT-prolonging drug (‘Dofetilide’, an antiarrhythmic drug with a pure hERG blocker) and subsequently experienced a gradual increase in the QT-interval from the normal QT baseline. It is well-known that the ECG differs from one individual to another, and that different types of drug can affect the ECG in different ways. The study examined only sensitivity (identifying true positives); for self-monitoring, it is important also to examine specificity (identifying true negatives). Finally, and most importantly, the study design allowed people to compare the ECG with a normal baseline. Such a baseline would be hard to produce in self-monitoring situations, not least because it would have to be adjusted according to heart rate. Therefore, this study tests the technique by presenting the ECGs one by one, asking participants to judge whether QT-prolongation has occurred when viewing just a single image, rather than comparing it with a baseline.

Objective

The objective of the study was to evaluate the effectiveness of the pseudo-colouring technique in displaying QT-prolongation risk threshold on the ECG, when the data is displayed on two coordinate systems: Cartesian and Polar. Accuracy was quantified in terms of both sensitivity and specificity. The technique was tested with QT-intervals across several heart rates for multiple patients on different QT-prolonging drugs. No comparison baseline ECG was provided.

Materials and methods

In this section, we provide a detailed description of how the ECGs images were produced and displayed, and the experiment designed to evaluate the effectiveness of the technique. The ECGs shown in the figures are reduced in size for inclusion in the paper. All of the full size ECG images used in the experiment, along with the R scripts used to create them, can be found in [62].

ECG data acquisition

The ECG datasets were acquired from a clinical trial study that was conducted to assess the effects of different types of known QT-prolonging drugs, including a pure hERG potassium channel blocker and multichannel blockers, on the ECG of healthy subjects [63]. The open ECG datasets are available online from the PhysioNet database [64].

Visualisation design

The visualisation was created in two layers. The first layer plotted the ECG using the standard method, which is a two dimensional line graph displaying the amplitude, or voltage, of the electrical signal along the Y-axis and the time in milliseconds (ms) along the X-axis. We used the ECG data from lead II, as the QT-interval is conventionally measured in this lead [2, 17]. The pseudo-colouring was then applied in a second layer, following a series of steps: identifying individual heartbeats via R-peak detection; applying pseudo-colouring to each heartbeat; adjusting the colouring according according to heart rate; displaying the data on either a Cartesian (current standard) or Polar coordinate system.

Identifying individual heartbeats.

The R-wave peaks (points with greatest amplitude) were detected in the raw ECG datasets using an automated numerical math function that finds the greatest peaks (maxima) in a time series [65]. A solid vertical line was superimposed on the image at this point perpendicular to the X-axis to show the location of the R-waves on the ECG, and delineate individual heartbeats. Detecting the R wave in the vast majority of ECGs is straightforward, as it consistently has the greatest amplitude. This is in contrast to the other waveforms which vary considerably across ECGs. As high quality ECG signals were used in the study, this method proved accurate and efficient. Where signal data is lower quality, it may be necessary to use alternative detection methods that can reliably distinguish R-peaks from high amplitude noises (see e.g. [6669]).

Applying the pseudo-colouring.

The traditional way to measure the QT-interval clinically is to count the small squares (each representing 40ms) on the standard ECG background grid, from the beginning of the Q-wave to the end of the T-wave. For the purposes of applying the pseudo-colouring, we added sequential time in ms as a third dimension to each heartbeat, from the R-peak minus 20ms (the estimated start of the Q-wave), to the clinically significant point of maximum risk for TdP—the method for estimating this is provided in the next section. We then applied pseudo-colouring to the area under the curve of the ECG signal by mapping every 40ms of this sequential time to a colour code or hue, with the intensity of the hue changing every millisecond. We used a spectrum-approximation sequence as it is effective for supporting people in reading continuous values [70]; it ranges from cool colours (purple to blue to green), which were used to indicate normal QT-interval levels, to warm colours (yellow to orange to red), used to show prolonged QT-interval levels.

Adjusting pseudo-colouring according to heart rate using the QT nomogram.

To apply the pseudo-colouring accurately for different heart rates, it was necessary to identify the value of QT-prolongation that determines being ‘at risk’ for TdP. In the previous study [51], where all ECGs had a 60 bpm heart rate, we used the half R-R interval rule to identify at risk QT-prolongation. This states that the QT-interval is prolonged if it is equal to or longer than the half R-R interval (midpoint between two consecutive R-peaks) [52, 71]. This rule is accurate with a heart rate of 60 bpm, but lacks sensitivity in detecting at risk QT-prolongation with both lower and higher heart rates [52, 72]. In clinical practice it is common to apply a QT correction formula (QTc), and then use a ‘cut off’ value to identify at risk QT-prolongation. Examples of such QTc formulae include Bazett’s, Fredericia’s, Framingham’s and Hodge’s [52]. However, recent research has shown that these correction formulae are inaccurate in identifying patients at risk of drug-induced TdP, particularly for fast and slow heart rates [52, 73].

An alternative approach, known as the ‘QT nomogram’, is a risk assessment method designed specifically for identifying patients at risk of drug-induced TdP according to heart rate [74, 75]. It was developed after screening 129 ECG cases that reported drug-induced TdP and comparing these with control cases (i.e. when no TdP was reported). The actual QT-interval value (not the QTc) and the heart rate for each case were plotted as coordinates on a graph; this produced a line showing the upper bound of the QT-interval value at risk for TdP as a function of heart rate. If the QT/HR value falls on or above this risk line, the patient is at risk of TdP; below the line the patient is not considered at risk of TdP [74]. The nomogram plot can be seen in Fig 2. Further evaluation of the QT nomogram showed that it had higher sensitivity and specificity than widely accepted QTc formulas [76] and the half R-R interval rule [72]. We therefore used it as the foundation for modifying the pseudo-colour as a function of heart rate.

thumbnail
Fig 2. The QT-nomogram for identifying QT-prolongation at risk of TdP over heart rate.

If the QT/HR value falls on or above the risk line, the patient is at risk of TdP; below the line the patient is not considered at risk of TdP [74].

https://doi.org/10.1371/journal.pone.0237854.g002

For each ECG, the heart rate (HR) was calculated and then the corresponding QT value for the ‘at risk’ threshold was acquired from the QT nomogram and plotted on the ECG as a dashed vertical line. To apply the pseudo-colouring, we used nine indices, where each was mapped to a colour code and represented by a small square on the standard ECG background grid. A time value on the nomogram threshold line was mapped to dark orange, and values 40ms and 80ms above the nomogram line were mapped to red and dark red respectively, showing a higher risk for TdP. Time values below the nomogram line were mapped to progressively cooler colours, such that a time value below the line by five small squares (200ms), was mapped to blue. Fig 3 illustrates how the pseudo-colouring technique was applied according to the standard ECG background grid. The mappings between colours and time values according to the nomogram is shown in Table 1. Note that this approach does not provide a numerical value for the QT-interval, but rather colours the area under the curve of the signal over a specific time interval (40ms × 9 indices = 360ms) according to the nomogram. The T-wave area lies within this time interval: if the T-wave is in a cool colour location, then the QT-interval is in the normal range; if it is in a warm colour location, the interval is prolonged.

thumbnail
Fig 3. Mapping the pseudo-colouring to the ECG, according to the QT-nomogram.

A small square on the standard ECG background grid is equal to 40ms.

https://doi.org/10.1371/journal.pone.0237854.g003

thumbnail
Table 1. The nine indices on the pseudo-colouring scale with their corresponding time values and colour codes.

https://doi.org/10.1371/journal.pone.0237854.t001

Coordinate system.

Our previous work indicates that people may be able to detect QT-prolongation more accurately on a Polar coordinate system [51]. To see whether this held when ECGs with varying heart rates are presented in isolation, we compared two coordinate systems (Cartesian and Polar) with and without pseudo-colouring. We used the R programming language [77] using RStudio software version 1.1.447 to create the visualisations. Fig 4 shows examples of ECGs with pseudo-colouring that have different heart rates, but similar QT-interval risk levels for TdP, on Cartesian coordinates. Fig 5 shows the same ECGs on Polar coordinates.

thumbnail
Fig 4. Pseudo-coloured ECGs that have different heart rates, but similar QT-interval risk levels, on Cartesian coordinates.

(A) QT-intervals are below the nomogram line by 80ms for both ECGs (no risk of TdP). The top stimulus has a low heart rate (HR = 57, QT = 401) and the bottom has a high heart rate (HR = 95, QT = 339). (B) QT-intervals are on the nomogram line in both ECGs (risk of TdP). The top stimulus has a low heart rate (HR = 46, QT = 487) and the bottom has a high heart rate (HR = 94, QT = 410).

https://doi.org/10.1371/journal.pone.0237854.g004

thumbnail
Fig 5. Pseudo-coloured ECGs that have different heart rates, but similar QT-interval risk levels, on Polar coordinates.

(A) QT-intervals are below the nomogram line by 80ms for both ECGs (no risk of TdP). The left stimulus has a low heart rate (HR = 57, QT = 401) and the right has a high heart rate (HR = 95, QT = 339). (B) QT-intervals are on the nomogram line in both ECGs (risk of TdP). The left stimulus has a low heart rate (HR = 46, QT = 487) and the right has a high heart rate (HR = 94, QT = 410).

https://doi.org/10.1371/journal.pone.0237854.g005

Experimental design

We used a multi-reader, multi-case (MRMC) receiver operating characteristic (ROC) study design within a psychophysical paradigm. The MRMC ROC design is an evaluation method commonly used to assess diagnostic performance in medical imaging studies, where multiple human observers (readers) interpret multiple patient images (cases) [78]. The area under the ROC curve, a plot of sensitivity versus 1-specificity, is used to measure diagnostic accuracy. The MRMC design helps to increase the generalisability of study results and enhance statistical power, particularly when evaluating different computer-assisted detection (CAD) systems [78]. Psychophysical experiments use detection and discrimination tasks to investigate the relationship between the intensity of a physical stimulus and human perception and sensation, by systematically varying the properties of the stimulus along one or more physical dimensions [79]. Here the paradigm was used to systematically evaluate the impact of the visualisation technique on people’s sensitivity to increases in the QT-interval. We also used eye-tracking, a non-invasive sensor technology that measures human eye movements, to help us interpret the visual behaviour underpinning the results on a post hoc basis [80].

The study used a counterbalanced repeated measures design with two independent variables, each with two levels:

  1. Colour-coding (no colouring and pseudo-colouring).
  2. Coordinate system (Cartesian and Polar).

The within-subjects factorial design yielded a total of 4 (2×2) experimental conditions for each participant: Cartesian no-colouring; Cartesian pseudo-colouring; Polar no-colouring; Polar pseudo-colouring. We counterbalanced the order of visualisation presentation using a balanced Latin square to minimize practice effects. The dependent variables were accuracy (broken down into sensitivity and specificity), reaction time in ms, satisfaction with the visualisation on a five point Likert-type scale from low (1) to high (5), mean fixation duration, total fixation duration and spatial fixation distribution.

Stimulus design

The heart rates of the ECGs acquired from the clinical trial ranged from 40 to 96 bpm, and the QT-interval values ranged from 300 to 579ms. As such, bradycardia (HR <60 bpm) was included in the range of heart rates, but not tachycardia (HR >100 bpm).

We followed psychophysical experimental design principles to systematically select the study’s stimuli. We considered QT interval value, relative to the nomogram line, to be the physical dimension along which the stimulus was varied, in units of 40ms. Six levels of the QT-interval were identified: three levels below the nomogram line (no/low risk to borderline), one level on the nomogram line (moderate risk), and two levels above the nomogram line (high risk). The six QT levels were represented by indices 4 to 9 on the pseudo-colouring scale respectively (see Table 1). Fig 6 shows examples of ECGs with the first QT-level (normal), and the sixth QT-level (severely prolonged), while Figs 4 and 5 show examples of ECGs with the second QT-level (normal) and the fourth QT-level (prolonged). Table 2 shows the corresponding estimated QT-interval prolongation for the six levels.

thumbnail
Fig 6. Pseudo-coloured ECGs that have similar heart rates, but different QT-interval risk levels, on Cartesian coordinates.

(A) Normal QT-interval (HR = 55, QT = 361). (B) Severely prolonged QT-interval (HR = 52, QT = 579).

https://doi.org/10.1371/journal.pone.0237854.g006

thumbnail
Table 2. The six QT-levels with their corresponding QT-value relative to the nomogram and an estimated range.

https://doi.org/10.1371/journal.pone.0237854.t002

ECG case selection

Forty ECG cases were selected from multiple patients (n = 17) to match the six QT-levels. Twenty ECGs were below the nomogram line (i.e. no risk of TdP) and 20 ECGs were on or above the nomogram line (i.e. at risk of TdP). Six ECG cases were on a placebo, and had values corresponding to the first QT-level, across heart rates ranging from 48 to 90 bpm. The other 34 ECG cases (18 on Dofetilide, a known QT-prolonging drug with a pure hERG potassium channel blocker, and 16 on Quinidine, a multichannel blocker with a strong hERG block, and lesser calcium and late sodium blocks), represented the second to sixth QT-levels across heart rates from 42 to 96 bpm. Table 3 shows the metadata of each ECG case in terms of QT value, heart rate, drug type, patient ID, the corresponding QT-level relative to the nomogram line and whether the case was at risk for TdP or not based on the nomogram plot.

thumbnail
Table 3. Characteristics of the 40 selected ECG cases used in the study.

https://doi.org/10.1371/journal.pone.0237854.t003

Participants

A total of forty three participants, with no experience in ECG interpretation, were recruited from a university campus (30 students and 13 staff). Eligibility for the study was determined by asking participants to rate their knowledge of ECG interpretation, and only people who identified as having no knowledge at all were included in the study. There were 18 male and 25 female participants between 20 and 56 years old (Mean = 31, SD = 8). Table 4 shows the participants’ demographic data. Participants’ sight was normal or corrected-to-normal and they reported no motor or neurological disorders. The research was approved by the University Research Ethics Committee). All participants provided written informed consent.

Apparatus

Stimuli were displayed on a 23.8 inch (diagonal) wide-screen Spectrum eye-tracking monitor, with a resolution of 1920 × 1080 pixels. Eye gaze was recorded using the Tobii Pro Spectrum eye-tracker and Tobii Pro Lab 1.95 software with a sampling rate of 600Hz. Key press events were recorded during the experiment to measure reaction times. Each Cartesian coordinate ECG stimulus measured 32.31cm × 6.14cm, and each Polar coordinate stimulus was 15.61cm × 12.93cm.

Task and procedure

Participants completed a 15 minute training session prior to starting the experiment, where they were introduced to the ECG trace, and taught how to identify the QT-interval using the tangent method to identify the end of the T-wave (see Fig 1 and [81]). Participants were instructed to use the tangent method visually on the screen (i.e. without any additional equipment) to roughly identify the end of the T-wave; the training session did not involve any medical terms or high-level training techniques typically associated with clinical ECG interpretation and focused solely on identification of the QT interval. Participants were then introduced to the four visualisation techniques, and shown how to use the pseudo-colouring scale and the risk threshold (represented by a dashed line) to assess the QT-interval. Participants were told that a QT-interval is considered prolonged if the T-wave ends on or exceeds the risk threshold dashed line, and when the pseudo-colouring within the QT-interval, and particularly in the Tpeak–Tend interval, contains warm colours (yellow/orange/red), with a greater visibility of warm colours indicating a higher level of QT-prolongation. Finally, participants completed an assessment task, in which they were asked to highlight the start and end points of the QT-intervals on two different ECGs and assess whether it is normal or abnormal using the visualisation techniques, to ensure that they understood how to perform the task. Participants were tested individually. They completed four tests, one for each visualisation technique, in which they read all ECG cases for one test before moving to the next. Participants read the same ECG cases using each visualisation technique. ECGs were presented at random within a test. Each test began with a five-point calibration of the eye-tracking apparatus. ECGs were presented one at a time, and participants completed two tasks: the first task used a psychophysical one alternative forced-choice paradigm (also known as a yes-no detection task), in which participants indicated verbally, as quickly as possible, whether they perceived the QT-interval as (‘normal’ or ‘abnormal’), whilst pressing the space-bar on the keyboard to collect reaction time; the second task was to rate their confidence in their response using a 6-point scale of ‘very likely normal’ (1), ‘probably normal’ (2), ‘possibly normal’ (3), ‘possibly abnormal’ (4), ‘probably abnormal’ (5), and ‘very likely abnormal’ (6). There was no time limit imposed, but participants were encouraged to do the tasks as quickly as possible. The binary responses and the confidence scores were recorded on a spreadsheet during the experiment by the researcher. At the end of the experiment, participants were asked to provide their satisfaction with the visualisations by rating how much each helped them to assess the QT-interval using a five point Likert-type scale ranging from ‘not very much’ (1) to ‘a lot’ (5).

Statistical analysis methods

Response accuracy was measured by calculating the average area under the receiver operating characteristic (ROC) curves (AUC) with 95% confidence intervals (CIs). QT-levels 1-3 inclusive were classified as negative (normal), and levels 4-6 as positive (prolonged). We used the Dorfman-Berbaum-Metz multi-reader multi-case (DBM MRMC) software from the University of Iowa [82], to calculate and compare the AUCs of the four visualisation techniques, based on the methods of Dorfman, Berbaum and Metz [83] and Obuchowski and Rockette [84] and later unified and improved by Hillis and colleagues [78, 85, 86]. The DBM MRMC method uses jackknifing and analysis of variance (ANOVA) methods and we considered both readers and cases as random variables; this allows the results to be generalised to the population of readers and cases. The ROC curve was fitted using the trapezoidal area and Wilcoxon method. The statistical power of ROC MRMC was greater than 90%, and the sample size estimation was performed using [87].

All statistical tests were performed at a 5% significance level (α = 0.05); two-sided 95% confidence intervals (CIs) were used to quantify uncertainty. Within-participant comparisons of sensitivity and specificity were performed using McNemar’s chi-squared test. We measured perceptual sensitivity to QT-levels using the psychometric function and just noticeable difference (JND) threshold, along with the signal detection analysis method R packages [88, 89]. Differences in reaction time and satisfaction scores were determined via a Friedman test and post hoc pairwise comparisons performed with a Wilcoxon signed-rank test utilising a Bonferroni correction (α = 0.008). Visual behaviour was analysed using eye tracking metrics including mean fixation duration, total fixation duration and fixation distribution using the nearest neighbour index (NNI) method, as implemented by Davies et al. [90].

Results

All anonymised data and related metadata underpinning the findings reported in this article can be found in [62].

Accuracy

Area under the ROC curve.

Pseudo-colouring significantly increased the average area under the ROC curve (AUC) for both coordinate systems (Tables 5 and 6). When Cartesian coordinates were used, the average AUC increased from 0.895 (standard error (SE) = 0.024) to 0.935 (SE = 0.020), while the increase was from 0.878 (SE = 0.029) to 0.934 (SE = 0.023) for Polar coordinates (Table 5). The average increase in AUC as a result of pseudo-colouring was therefore 0.04 for Cartesian and 0.056 for Polar coordinates, a statistically significant increase in both cases (p = 0.014, p < 0.001 for Cartesian and Polar coordinates respectively, see Table 6). There was no significant difference in average AUC as a function of coordinate system when pseudo-colouring was used (p = 0.978, see Table 6). Although the average AUC was higher for Cartesian coordinates than for Polar coordinates when no-colouring was used, the difference was not statistically significant (p = 0.273, see Table 6).

thumbnail
Table 5. The average area under the ROC curve across all participants and for each visualisation technique.

https://doi.org/10.1371/journal.pone.0237854.t005

thumbnail
Table 6. Pairwise comparisons of the area under the ROC curve for the four visualisation techniques.

https://doi.org/10.1371/journal.pone.0237854.t006

Specificity.

Specificity was calculated as the proportion of true negative (i.e. ‘normal’) responses for QT-levels 1 to 3. The mean specificity across all participants was high and similar for both coordinate systems regardless of pseudo-colouring was used (Table 7). Although pseudo-colouring increased the mean specificity for Polar coordinates from 0.88 (SE = 0.002) to 0.90 (SE = 0.002), the increase was not statistically significant (p = 0.460). The highest mean specificity was 0.94 (SE = 0.002) for Cartesian coordinates when pseudo-colouring was not used, which was significantly higher than when pseudo-colouring was used with this coordinate system (p < 0.001). There was no difference in the mean specificity between the two coordinate systems when pseudo-colouring was used (p = 1.000).

thumbnail
Table 7. Comparisons of mean specificity between the visualisation techniques.

https://doi.org/10.1371/journal.pone.0237854.t007

Sensitivity.

Sensitivity was measured using the ROC method as the proportion of true positive (i.e. ‘abnormal’) responses for QT-levels 4 to 6. The mean sensitivity increased with pseudo-colouring from 0.63 (SE = 0.004) to 0.83 (SE = 0.003) with Cartesian coordinates, and from 0.72 (SE = 0.004) to 0.82 (SE = 0.003) with Polar coordinates (Table 8). The increase in sensitivity of 0.2 (SE = 0.004) with Cartesian and 0.1 (SE = 0.003) with Polar coordinates was statistically significant (p < 0.001). Sensitivity was higher when Cartesian coordinates were used than when Polar coordinates were used by 0.01, a statistically significant difference (p = 0.001). When pseudo-colouring was not used, the mean sensitivity was 0.09 (SE = 0.003) higher with Polar coordinates than with Cartesian coordinates, a difference that was statistically significant (p < 0.001).

thumbnail
Table 8. Comparisons of mean sensitivity between the visualisation techniques.

https://doi.org/10.1371/journal.pone.0237854.t008

We also measured perceptual sensitivity across QT-levels using the following psychophysical detection methods:

(1) Psychometric function and just noticeable difference (JND). We used a psychometric function, an inferential model used in psychophysical detection and discrimination tasks, to model the relationship between QT-level increases and participants’ sensitivity across the four visualisation techniques. The psychometric function was plotted as the proportion of responses that indicated a QT-prolongation as a function of the QT-level. The results show that pseudo-colour significantly improved the perception of QT-interval increases regardless of heart rate with both coordinate systems (see Fig 7A). We estimated the just noticeable difference (JND) threshold, which is defined in psychophysics as the smallest change required in a dimension of a stimulus in order for a difference to be perceived [79]. In this study, we defined it as the minimum increase in the QT-interval with respect to the nomogram line required for it to be detectable. We estimated the 75% JND threshold as the value of the QT-interval increase from the nomogram line at which the proportion of responses indicating QT-prolongation is equal to 0.75. The JND thresholds, determined by fitting the psychometric function using a logistic function with maximum likelihood estimation (MLE) (Fig 7B), were 51.2, 6.9, 28.8 and 7.3 milliseconds for Cartesian no-colouring, Cartesian pseudo-colouring, Polar no-colouring and Polar pseudo-colouring respectively. Pseudo-colour thus reduces the JND in both coordinate systems, showing that 75% of people are able to perceive a QT-interval as prolonged when the QT-value above the nomogram line increases by approximately 6.9 and 7.3ms with Cartesian and Polar coordinates respectively. When pseudo-colouring was not used, the JND was 51.2 and 28.8ms for Cartesian and Polar coordinates respectively, showing that sensitivity was higher for Polar than Cartesian coordinates.

thumbnail
Fig 7. Psychophysical detection measures of sensitivity.

(A) The fitted psychometric function plot shows the proportion of responses indicating QT-prolongation as a function of the QT-interval difference from the nomogram line, which corresponds to the six QT-levels. The QT value of the nomogram line (i.e. the fourth QT-level) is equal to 0 on the X-axis. Cartesian no-colouring is represented by a red line, Cartesian pseudo-colouring by a green line, Polar no-colouring by a turquoise line and Polar pseudo-colouring by a purple line. (B) The just noticeable difference (JND) thresholds plot. The error bars represent bootstrap confidence intervals.

https://doi.org/10.1371/journal.pone.0237854.g007

(2) Signal detection analysis. The psychometric function and JND threshold quantify sensitivity in terms of a binary response (here, ‘normal’ or ‘abnormal’). In psychophysics, signal detection theory recognises that there is uncertainty in the task, such that the decision may be affected by noise either externally, for example where a stimulus changes between presentations, or internally, as people may be biased towards saying ‘normal’ or ‘abnormal’ more frequently [79]. We attempted to minimise the effect of decision bias by increasing sample size, counterbalancing experimental conditions, and including confidence scores, but some uncertainty is likely to remain. Signal detection theory measures the sensitivity, while taking response bias into account, using a sensitivity index d′ [79]. This was calculated for each participant and then averaged across all participants. Table 9 shows the mean sensitivity index (d′) with the standard deviation and standard error for each visualisation technique. The results show that pseudo-colouring increased the mean sensitivity with both coordinate systems independent of decision bias.

thumbnail
Table 9. The mean sensitivity index for each visualisation technique.

https://doi.org/10.1371/journal.pone.0237854.t009

Reaction time

We measured reaction time as the period from the appearance of the stimulus on the screen to the key press event when people made their binary decision. The mean reaction times across all stimuli were 7.9 and 8.6 seconds for Cartesian coordinates with and without pseudo-colouring respectively, and 8.1 and 8.9 seconds for Polar coordinates with and without pseudo-colouring respectively. A Friedman test was conducted for each QT-level stimulus and overall for each condition. People were significantly faster when pseudo-colouring was used for each QT-level and over all ECG stimuli cumulatively (p ≤ 0.001)(Table 10, Fig 8). A post hoc pairwise comparison using a Wilcoxon signed-rank test with Bonferroni correction (α = 0.008) showed that although participants responded faster when pseudo-colouring was used with Cartesian coordinates than with Polar coordinates, the difference was not significant (p = 0.808).

thumbnail
Fig 8. Mean reaction time in seconds for each QT-level and all stimuli combined.

Error bars represent 95% confidence intervals.

https://doi.org/10.1371/journal.pone.0237854.g008

thumbnail
Table 10. Results of the Friedman test comparing reaction times across the four visualisation techniques.

https://doi.org/10.1371/journal.pone.0237854.t010

Satisfaction

A Friedman test showed there was a statistically significant difference in participants’ satisfaction as a function of visualisation technique (χ2(3) = 50.954, p < 0.001). A post hoc pairwise comparison using a Wilcoxon signed-rank test with Bonferroni correction (α = 0.008) showed that people preferred pseudo-colouring to no-colouring (p < 0.008); and Cartesian coordinates to Polar coordinates whether pseudo-colouring was used (Z = −3.340c, p = 0.001), or not (Z = −3.029c, p = 0.002). Although people were more sensitive to QT-prolongation when the ECG was presented on Polar coordinates with pseudo-colouring, people preferred Cartesian coordinates without pseudo-colouring to Polar coordinates with pseudo-colouring (Z = −2.142b, p = 0.032).

Eye movement analysis

An area of interest (AOI) was created on the whole ECG for each stimulus. As such, 160 AOIs were created (40 ECGs × 4 visualisation techniques).

Mean fixation duration.

The mean fixation duration metric, which is an indicator of cognitive load [91, 92], was calculated across all participants for each ECG stimulus and then averaged over each QT-level. Mean fixation duration increased as the QT-level increased when pseudo-colouring was used (Fig 9A). When considered alongside the psychometric function (Fig 7A), this indicates that pseudo-colour helped people to focus visual attention. When pseudo-colouring was not used, mean fixation duration continued to increase at higher QT-levels for Cartesian coordinates, but this effect was not seen with Polar coordinates, indicating a difference in how people interpreted the ECG as a function of coordinate system.

thumbnail
Fig 9. Eye movement analysis results.

(A) Mean fixation duration in milliseconds on the ECG stimuli averaged for each QT-level. (B) Total fixation duration in milliseconds on the ECG stimuli averaged for each QT-level.

https://doi.org/10.1371/journal.pone.0237854.g009

Total fixation duration.

The total fixation duration (also known as the dwell time) quantifies the amount of time spent fixating on the stimulus [91]. It was calculated across all participants for each stimulus and then averaged for each QT-level. The results show that total fixation duration was lower when pseudo-colouring was used for both coordinate systems (Fig 9B). People spent less cumulative time fixating on the ECG stimuli with pseudo-colouring, but individual fixations were more focused (Fig 9A and 9B). When pseudo-colouring was not used, there were more, shorter fixations, indicating a less-focused interpretation strategy.

Fixation distribution.

Figs 10 and 11 show heatmaps of fixations across an ECG stimulus with and without pseudo-colouring, for Cartesian and Polar coordinate systems respectively. We hypothesised that pseudo-colour helped to draw people’s attention to the coloured T-wave area, and that fixations would therefore be more clustered when pseudo-colour was used. We used the Nearest Neighbour Index (NNI) to determine if the fixations were randomly spaced or clustered. The NNI provides a ratio of the distribution pattern of points in space from 0 (clustered) to 1 (dispersed) [90]. We calculated the average Nearest Neighbour Index (NNI) for each ECG stimulus across all participants and for all visualisation techniques. Fig 12 shows a histogram of the distribution of the Nearest Neighbour Index (NNI), along with a box-plot for Cartesian and Polar coordinates, with and without pseudo-colouring. When Cartesian coordinates were used (Fig 12A), pseudo-colouring resulted in clustering in more stimuli than no-colouring. When Polar coordinates were used (Fig 12B), however, pseudo-colouring did not appear to affect the fixation distribution.

thumbnail
Fig 10. A heatmap of fixation count shows the cumulative number of fixations across all participants on Cartesian coordinates.

With pseudo-colouring, people made fewer, more clustered fixations on the coloured T-wave area compared with no-colouring on the same ECG.

https://doi.org/10.1371/journal.pone.0237854.g010

thumbnail
Fig 11. A heatmap of fixation duration shows the cumulative mean fixation duration across all participants on Polar coordinates.

With pseudo-colouring, fixations were longer on the coloured T-wave area and the vertical dashed line that represents the QT risk threshold, while with no-colouring fixations were longer on the vertical dashed line only.

https://doi.org/10.1371/journal.pone.0237854.g011

thumbnail
Fig 12. Nearest Neighbour Index (NNI) of fixations on ECGs with and without pseudo-colouring.

(A) Cartesian coordinates. (B) Polar coordinates.

https://doi.org/10.1371/journal.pone.0237854.g012

Discussion

This study demonstrates that applying pseudo-colouring to an ECG relative to the QT nomogram significantly increases lay people’s accuracy in visually assessing QT-intervals regardless of heart rate (Tables 5 and 6). In particular, pseudo-colouring increased sensitivity to prolongation (true positive cases); whilst people were able to identify normal QT-intervals without colour, adding pseudo-colouring resulted in more accurate identification of at risk QT-prolongation cases, shown by both the ROC curve analysis and the psychophysical detection measures of sensitivity (Tables 8 and 9; Fig 7). The psychometric function shows that pseudo-colour helped increase sensitivity to QT-interval increases even before a patient is at risk of TdP (Fig 7A), specifically in the borderline range (i.e. the third QT-level). This level could be critical from a clinical perspective as just a 40ms increase from this point could put someone at risk of TdP. Clinical research has shown that even small (∼10ms) QT-interval increases from the baseline should be regarded as a significant side effect of a QT-prolonging drug [9395]. From the perspective of our study design, this meant that pseudo-colouring with Cartesian coordinates reduced specificity (true negative cases) at this level, as we considered it to be within the normal range. From a practical perspective, however, considering this level to be abnormal may serve a useful purpose.

Pseudo-colour significantly reduced reaction times and helped to focus visual attention on the areas of the ECG crucial for detecting QT-prolongation. Eye-tracking data showed that pseudo-colour consistently increased mean fixation duration as the QT-level increased (Fig 9A); when the ECG was presented on Cartesian coordinates, fixations were more clustered when pseudo-colouring was used (Fig 12A).

Obtaining a precise measurement of the QT-interval is known to be challenging. An important QT-interval increase may be just a few milliseconds, less than one small square on the ECG. This would be particularly difficult to detect at high heart rates, as the QT-interval length shrinks with the R-R interval length [40, 41]. Here, we demonstrate that pseudo-colour can support detection of QT prolongation regardless of heart rate without needing to measure the QT-interval (Figs 4 and 5). Fig 11 shows a heat map of absolute fixation duration across all participants on an ECG with a QT-interval 40ms above the nomogram line. When pseudo-colouring was not used, people’s fixations appear to be more focused on measuring the gap between the end of the T-wave and the QT risk threshold dashed line. When pseudo-colouring is present on the same ECG, fixations appear more focused on the T-wave. Focusing attention in this area is particularly useful, as long QT syndrome (LQTS) is associated with prolonging ventricular repolarization [7, 8], which is represented by the T-wave on the ECG [1, 2]. Research examining the effects of QT-prolonging drugs is investigating precisely how the T-wave responds to them to provide further insights of drug ion channel interactions and TdP risk [42]. Pseudo-colour might ultimately help clinicians to determine which part of the T-wave—the initial half of the T-wave (J-T-peak) or the second half (T-peak T-end)—underpins QT-prolongation.

Limitations and future work

Whilst this work indicates the potential for pseudo-colouring to assist in the self-monitoring of QT-interval, the study took place in a controlled setting with a limited number of stimuli, and the transferability of the technique to practice remains an open question. Cartesian and Polar coordinates support the same level of accuracy, but people expressed a preference for pseudo-colour displayed on Cartesian coordinates. It may be, however, that the ECG trace would be better presented with Polar coordinates on smaller screens like smart watches, and future work should thus examine the effects of screen size and lighting setting on accuracy.

In this study, we only investigated the assessment of QT-interval, and it is not clear whether these visualisation techniques would generalise to interpretation of other ECG abnormalities, such as changes in ST-segment elevation. Research has shown that a large number of clinicians lack the skills to interpret QT-prolongation accurately [45, 46]; pseudo-colouring could also be used to help clinicians, especially within emergency departments, to visually assess and monitor patients’ QT-intervals before or during the provision of a QT-prolonging medication. Future work should evaluate the visualisation techniques in clinical trials with more diverse clinical populations.

Conclusion

Applying pseudo-colouring to ECGs according to the QT-nomogram supports lay people in detecting QT-prolongation visually, regardless of heart rate. The results indicate that self-monitoring ECGs for drug-induced LQTS is feasible, with the potential to prevent the development of life threatening complications.

Acknowledgments

We would like to thank the Digital Experimental Cancer Medicine Team (digitalECMT), Manchester, UK, for their feedback and support.

References

  1. 1. Schamroth L. An introduction to electrocardiography. Wiley-Blackwell; 1971.
  2. 2. Goldenberg IL, Moss AJ, Zareba W. QT interval: how to measure it and what is “normal”. Journal of cardiovascular electrophysiology. 2006 Mar;17(3):333–6.
  3. 3. Yap YG, Camm AJ. Drug induced QT prolongation and torsades de pointes. Heart. 2003 Nov 1;89(11):1363–72. pmid:14594906
  4. 4. Anderson ME, Al-Khatib SM, Roden DM, Califf RM, Institute DC. Cardiac repolarization: current knowledge, critical gaps, and new approaches to drug development and patient management. American heart journal. 2002 Nov 1;144(5):769–81. pmid:12422144
  5. 5. Moss AJ, Schwartz PJ, Crampton RS, Tzivoni D, Locati EH, MacCluer J, et al. The long QT syndrome. Prospective longitudinal study of 328 families. Circulation. 1991 Sep;84(3):1136–44. pmid:1884444
  6. 6. Goldenberg I, Moss AJ. Long QT syndrome. Journal of the American College of Cardiology. 2008 Jun 17;51(24):2291–300. pmid:18549912
  7. 7. Moss AJ, Kass RS. Long QT syndrome: from channels to cardiac arrhythmias. The Journal of clinical investigation. 2005 Aug 1;115(8):2018–24. pmid:16075042
  8. 8. DICM JD, Sharma P, Rathi S. Long QT syndrome. Continuing Education in Anaesthesia, Critical Care Pain. 2008;8(2).
  9. 9. Behere SP, Shubkin CD, Weindling SN. Recent advances in the understanding and management of long QT syndrome. Current opinion in pediatrics. 2014 Dec 1;26(6):727–33. pmid:25313972
  10. 10. Vukmirović M, Tomašević-Vukmirović I, Angelkov L, Vukmirović F. Emotional stress as a cause of syncope and torsade de pointes in patients with long QT syndrome. Vojnosanitetski pregled. 2015;72(2):192–5. pmid:25831915
  11. 11. Johnson JN, Ackerman MJ. Return to play? Athletes with congenital long QT syndrome. Br J Sports Med. 2013 Jan 1;47(1):28–33. pmid:23193325
  12. 12. Gomez AT, Prutkin JM, Rao AL. Evaluation and Management of Athletes With Long QT Syndrome: An Evolved Paradigm. Sports health. 2016 Nov;8(6):527–35.
  13. 13. Khan IA. Clinical and therapeutic aspects of congenital and acquired long QT syndrome. The American journal of medicine. 2002 Jan 1;112(1):58–66. pmid:11812408
  14. 14. Shah M, Carter C. Long QT syndrome: A therapeutic challenge. Annals of pediatric cardiology. 2008 Jan;1(1):18. pmid:20300233
  15. 15. Roden DM, Viswanathan PC. Genetics of acquired long QT syndrome. The Journal of clinical investigation. 2005 Aug 1;115(8):2025–32. pmid:16075043
  16. 16. Priori SG, Schwartz PJ, Napolitano C, Bianchi L, Dennis A, Fusco MD, et al. A recessive variant of the Romano-Ward long-QT syndrome?. Circulation. 1998 Jun 23;97(24):2420–5. pmid:9641694
  17. 17. Camm AJ, Malik M, Yap YG. Acquired long QT syndrome. John Wiley Sons; 2008 Apr 15.
  18. 18. Thomas D, Karle CA, Kiehn J. The cardiac hERG/IKr potassium channel as pharmacological target: structure, function, regulation, and clinical applications. Current pharmaceutical design. 2006 Jun 1;12(18):2271–83. pmid:16787254
  19. 19. Recanatini M, Poluzzi E, Masetti M, Cavalli A, De Ponti F. QT prolongation through hERG K+ channel blockade: current knowledge and strategies for the early prediction during drug development. Medicinal research reviews. 2005 Mar;25(2):133–66.
  20. 20. Wysowski DK. Surveillance of prescription drug-related mortality using death certificate data. Drug Safety. 2007 Jun 1;30(6):533–40. pmid:17536879
  21. 21. Brown AM. Drugs, hERG and sudden death. Cell Calcium. 2004 Jun 1;35(6):543–7. pmid:15110144
  22. 22. Shah RR. The significance of QT interval in drug development. British journal of clinical pharmacology. 2002 Aug;54(2):188–202. pmid:12207642
  23. 23. Kozik TM, Wung SF. Acquired long QT syndrome: frequency, onset, and risk factors in intensive care patients. Critical care nurse. 2012 Oct 1;32(5):32–41. pmid:23027789
  24. 24. Nachimuthu S, Assar MD, Schussler JM. Drug-induced QT interval prolongation: mechanisms and clinical management. Therapeutic advances in drug safety. 2012 Oct;3(5):241–53. pmid:25083239
  25. 25. Drew BJ, Ackerman MJ, Funk M, Gibler WB, Kligfield P, Menon V, et al American Heart Association Acute Cardiac Care Committee of the Council on Clinical Cardiology, American College of Cardiology Foundation. Prevention of torsade de pointes in hospital settings: a scientific statement from the American Heart Association and the American College of Cardiology Foundation endorsed by the American Association of Critical-Care Nurses and the International Society for Computerized Electrocardiology. Journal of the American College of Cardiology. 2010 Mar 2;55(9):934–47.
  26. 26. Solosko T, Gehman S, Herleikson E, Lyster T, Fong S, Hansen K, et al inventors. Continuous outpatient ECG monitoring system. United States patent US 9,615,793. 2017 Apr 11.
  27. 27. Steinberg C, Philippon F, Sanchez M, Fortier-Poisson P, O’Hara G, Molin F, et al. A Novel Wearable Device for Continuous Ambulatory ECG Recording: Proof of Concept and Assessment of Signal Quality. Biosensors. 2019 Mar;9(1):17.
  28. 28. Wu T, Redouté JM, Yuce M. A Wearable, Low-Power, Real-Time ECG Monitor for Smart T-shirt and IoT Healthcare Applications. InAdvances in Body Area Networks I 2019 (pp. 165–173). Springer, Cham.
  29. 29. Randazzo V, Ferretti J, Pasero E. ECG WATCH: a real time wireless wearable ECG. In2019 IEEE International Symposium on Medical Measurements and Applications (MeMeA) 2019 Jun 26 (pp. 1-6). IEEE.
  30. 30. Schläpfer J, Wellens HJ. Computer-interpreted electrocardiograms: benefits and limitations. Journal of the American College of Cardiology. 2017 Aug 21;70(9):1183–92. pmid:28838369
  31. 31. Hongo RH, Goldschlager N. Status of computerized electrocardiography. Cardiology clinics. 2006 Aug 1;24(3):491–504. pmid:16939838
  32. 32. Li K, Vo K, Lee BK, Addo N, Coralic Z. Effect of a single dose of iv ondansetron on QTc interval in emergency department patients. The Bulletin of the American Society of Hospital Pharmacists. 2018 Mar 1;75(5):276–82.
  33. 33. Garg A, Lehmann MH. Prolonged QT interval diagnosis suppression by a widely used computerized ECG analysis system. Circulation: Arrhythmia and Electrophysiology. 2013 Feb;6(1):76–83.
  34. 34. Miller MD, Co-burn JP, Ackerman MJ. Diagnostic accuracy of screening electrocardiograms in long QT syndrome I. Pediatrics. 2001 Jul 1;108(1):8–12. pmid:11433047
  35. 35. Rautaharju PM, Surawicz B, Gettes LS. AHA/ACCF/HRS recommendations for the standardization and interpretation of the electrocardiogram: part IV: the ST segment, T and U waves, and the QT interval a scientific statement from the American Heart Association Electrocardiography and Arrhythmias Committee, Council on Clinical Cardiology; the American College of Cardiology Foundation; and the Heart Rhythm Society Endorsed by the International Society for Computerized Electrocardiology. Journal of the American College of Cardiology. 2009 Mar 17;53(11):982–91.
  36. 36. Estes NM III. Computerized interpretation of ECGs: supplement not a substitute. Am Heart Assoc. 2013.
  37. 37. Tyl B, Azzam S, Blanco N, Wheeler W. Improvement and limitation of the reliability of automated QT measurement by recent algorithms. Journal of electrocardiology. 2011 May 1;44(3):320–5. pmid:21163494
  38. 38. Talebi S, Alaleh A, Sam Z, Sandeep S, Visco F, Totouom-Tangho H, et al. Underestimated and unreported prolonged QTc by automated ECG analysis in patients on methadone: can we rely on computer reading?. Acta cardiologica. 2015 Apr 1;70(2):211–6. pmid:26148382
  39. 39. Kligfield P, Badilini F, Denjoy I, Babaeizadeh S, Clark E, De Bie J, et al. Comparison of automated interval measurements by widely used algorithms in digital electrocardiographs. American heart journal. 2018 Jun 1;200:1–0. pmid:29898835
  40. 40. Postema PG, Wilde AAM. The measurement of the QT interval. Current cardiology reviews. 2014 Aug 1;10(3):287–94.
  41. 41. Morganroth J. Focus on issues in measuring and interpreting changes in the QTc interval duration. European Heart Journal Supplements. 2001 Sep 1;3(suppl_K):K105–11.
  42. 42. Vicente J, Johannesen L, Mason JW, Crumb WJ, Pueyo E, Stockbridge N, et al. Comprehensive T wave morphology assessment in a randomized clinical study of dofetilide, quinidine, ranolazine, and verapamil. Journal of the American Heart Association. 2015 Apr 13;4(4):e001615. pmid:25870186
  43. 43. Chorin E, Havakuk O, Adler A, Steinvil A, Rozovski U, van der Werf C, et al. Diagnostic value of T-wave morphology changes during “QT stretching” in patients with long QT syndrome. Heart Rhythm. 2015 Nov 1;12(11):2263–71.
  44. 44. Kirchhof P, Franz MR, Bardai A, Wilde AM. Giant T–U waves precede torsades de pointes in long QT syndrome: a systematic electrocardiographic analysis in patients with acquired and congenital QT prolongation. Journal of the American College of Cardiology. 2009 Jul 7;54(2):143–9.
  45. 45. Viskin S, Rosovski U, Sands AJ, Chen E, Kistler PM, Kalman JM, et al. Inaccurate electrocardiographic interpretation of long QT: the majority of physicians cannot recognize a long QT when they see one. Heart Rhythm. 2005 Jun 1;2(6):569–74. pmid:15922261
  46. 46. Solomons L, Treloar A, Noronha R. Competence of psychiatric clinicians in interpreting electrocardiograms and QT intervals: can they do this? Does it matter?. Psychiatric Bulletin. 2008 Aug;32(8):291–4.
  47. 47. Papadopoulos K, Koustriava E. Adult brains don’t fully overcome biases that lead to incorrect performance during cognitive development: an fMRI study in young adults completing a Piaget‐like task. Developmental Science. 2009 Mar;12(2):326–38
  48. 48. Leroux G, Spiess J, Zago L, Rossi S, Lubin A, Turbelin MR, et al. Piaget’s water-level task: The impact of vision on performance. Research in developmental disabilities. 2011 Nov 1;32(6):2889–93.
  49. 49. Wittig MA, Allen MJ. Measurement of adult performance on Piaget’s water horizontality task. Intelligence. 1984 Oct 1;8(4):305–13.
  50. 50. Alahmadi A, Davies A, Vigo M, Jay C. Can laypeople identify a drug-induced QT interval prolongation? A psychophysical and eye-tracking experiment examining the ability of nonexperts to interpret an ECG. Journal of the American Medical Informatics Association. 2019 Mar 8;26(5):404–11. pmid:30848818
  51. 51. Alahmadi A, Davies A, Royle J, Vigo M, Jay C. Evaluating the Impact of Pseudo-Colour and Coordinate System on the Detection of Medication-induced ECG Changes. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems 2019 Apr 18 (p. 123). ACM.
  52. 52. Luo S, Michler K, Johnston P, Macfarlane PW. A comparison of commonly used QT correction formulae: the effect of heart rate on the QTc of normal ECGs. Journal of electrocardiology. 2004 Oct 1;37:81–90. pmid:15534815
  53. 53. Solomons L, Treloar A, Noronha R. Competence of psychiatric clinicians in interpreting electrocardiograms and QT intervals: can they do this? Does it matter?. Psychiatric Bulletin. 2008 Aug;32(8):291–4.
  54. 54. Ware C. High-speed visual estimation using preattentive processing. ACM Transactions on Computer-Human Interaction (TOCHI). 1996 Jun 1;3(2):107–35.
  55. 55. Healey CG, Booth KS, Enns JT. Information visualization: perception for design. Elsevier; 2012 May 21.
  56. 56. Nothdurft HC. The role of features in preattentive vision: Comparison of orientation, motion and colour cues. Vision research. 1993 Sep 1;33(14):1937–58.
  57. 57. Gutwin C, Cockburn A, Coveney A. Peripheral popout: The influence of visual angle and stimulus intensity on popout effects. InProceedings of the 2017 CHI Conference on Human Factors in Computing Systems 2017 May 2 (pp. 208-219). ACM.
  58. 58. Zahedi Z, Sadri S, Soltani M, Tehrani MK. Breast diseases detection and pseudo-colouring presentation for gray infrared breast images. In2011 Asia Communications and Photonics Conference and Exhibition (ACP) 2011 Nov 13 (pp. 1-8). IEEE.
  59. 59. Quweider MK. Adaptive Pseudocolouring of Medical Images Using Dynamic Optimal Partitioning and Space-Filling Curves. In2009 2nd International Conference on Biomedical Engineering and Informatics 2009 Oct 17 (pp. 1-6). IEEE.
  60. 60. Saito T, Miyamura HN, Yamamoto M, Saito H, Hoshiya Y, Kaseda T. Two-tone pseudo colouring: Compact visualization for one-dimensional data. InIEEE Symposium on Information Visualization, 2005. INFOVIS 2005. 2005 Oct 23 (pp. 173-180). IEEE.
  61. 61. Semary NA. A Proposed HSV-BASED PSEUDO-colourING SCHEME FOR ENHANCING MEDICAL IMAGES. Computer Science Information Technology.:81.
  62. 62. Alahmadi A, Vigo M, Jay C. Visualising ECGs with Pseudo-colouring to detect LQTS regardless of heart rate. Version 1.0.0. [Computer Software]. https://doi.org/10.5281/zenodo.3940377
  63. 63. Johannesen L, Vicente J, Mason JW, Sanabria C, Waite‐Labott K, Hong M, et al. Differentiating drug‐induced multichannel block on the electrocardiogram: randomized study of dofetilide, quinidine, ranolazine, and verapamil. Clinical Pharmacology Therapeutics. 2014 Nov 1;96(5):549–58.
  64. 64. Goldberger AL, Amaral LA, Glass L, Hausdorff JM, Ivanov PC, Mark RG, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation. 2000 Jun 13;101(23):e215–20. pmid:10851218
  65. 65. Hans W. Borchers. pracma: Practical Numerical Math Functions. R package version 2.1.8. 2018. https://CRAN.R-project.org/package=pracma
  66. 66. Fujii T, Nakano M, Yamashita K, Konishi T, Izumi S, Kawaguchi H, et al. Noise-tolerant instantaneous heart rate and R-peak detection using short-term autocorrelation for wearable healthcare systems. In2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) 2013 Jul 3 (pp. 7330-7333). IEEE.
  67. 67. Portet F, Hernández AI, Carrault G. Evaluation of real-time QRS detection algorithms in variable contexts. Medical and Biological Engineering and Computing. 2005 Jun 1;43(3):379–85. pmid:16035227
  68. 68. Lee WK, Lee HJ, Lee JS, Yoon HN, Sim SY, Lim YG, et al. Validation of algorithm with noise tolerance methods to detect R-wave. InThe 15th International Conference on Biomedical Engineering 2014 (pp. 539-542). Springer, Cham.
  69. 69. Sathyapriya L, Murali L, Manigandan T. Analysis and detection R-peak detection using Modified Pan-Tompkins algorithm. In2014 IEEE International Conference on Advanced Communications, Control and Computing Technologies 2014 May 8 (pp. 483-487). IEEE.
  70. 70. Ware C. colour sequences for univariate maps: Theory, experiments and principles. IEEE Computer Graphics and Applications. 1988 Sep;8(5):41–9.
  71. 71. Dubin D. Rapid interpretation of EKG’s: an interactive course. Cover Publishing Company; 2000.
  72. 72. Berling I, Isbister GK. The half RR rule: a poor rule of thumb and not a risk assessment tool for QT interval prolongation. Academic Emergency Medicine. 2015 Oct;22(10):1139–44. pmid:26375169
  73. 73. Desai M, Li L, Desta Z, Malik M, Flockhart D. Variability of heart rate correction methods for the QT interval. British journal of clinical pharmacology. 2003 Jun;55(6):511–7. pmid:12814443
  74. 74. Chan A, Isbister GK, Kirkpatrick CM, Dufful SB. Drug-induced QT prolongation and torsades de pointes: evaluation of a QT nomogram. QJM: An International Journal of Medicine. 2007 Oct 1;100(10):609–15.
  75. 75. Fossa AA, Wisialowski T, Magnano A, Wolfgang E, Winslow R, Gorczyca W, et al. Dynamic beat-to-beat modeling of the QT-RR interval relationship: analysis of QT prolongation during alterations of autonomic state versus human ether a-go-go-related gene inhibition. Journal of Pharmacology and Experimental Therapeutics. 2005 Jan 1;312(1):1–1. pmid:15306635
  76. 76. Waring WS, Graham A, Gray J, Wilson AD, Howell C, Bateman DN. Evaluation of a QT nomogram for risk assessment after antidepressant overdose. British journal of clinical pharmacology. 2010 Dec;70(6):881–5. pmid:21175443
  77. 77. Ihaka R, Gentleman R. R: a language for data analysis and graphics. Journal of computational and graphical statistics. 1996 Sep 1;5(3):299–314.
  78. 78. Hillis SL, Berbaum KS, Metz CE. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Academic radiology. 2008 May 1;15(5):647–61. pmid:18423323
  79. 79. Prins N. Psychophysics: a practical introduction. Academic Press; 2016 Jan 4.
  80. 80. Duchowski AT. Eye tracking methodology. Theory and practice. 2007;328(614):2–3.
  81. 81. Postema PG, De Jong JS, Van der Bilt IA, Wilde AA. Accurate electrocardiographic assessment of the QT interval: teach the tangent. Heart Rhythm. 2008 Jul 1;5(7):1015–8. pmid:18598957
  82. 82. Kevin M. Schartz, Stephen L. Hillis and Lorenzo L. Pesce, and Kevin S. Berbaum. OR-DBM MRMC 2.51, Medical image perception laboratory department of radiology, The University of Iowa. https://perception.lab.uiowa.edu/OR-DBM-MRMC-program-manual.
  83. 83. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Investigative radiology. 1992 Sep 1;27(9):723–31.
  84. 84. Obuchowski NA Jr, Rockette HE Jr. Hypothesis testing of diagnostic accuracy for multiple readers and multiple tests an anova approach with dependent observations. Communications in Statistics-simulation and Computation. 1995 Jan 1;24(2):285–308.
  85. 85. Hillis SL, Obuchowski NA, Schartz KM, Berbaum KS. A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data. Statistics in medicine. 2005 May 30;24(10):1579–607.
  86. 86. Hillis SL. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Statistics in medicine. 2007 Feb 10;26(3):596–619. pmid:16538699
  87. 87. Hillis SL, Schartz KM. Multireader sample size program for diagnostic studies: demonstration and methodology Journal of Medical Imaging. 2018 Nov;5(4):045503. Retrieved from http://perception.radiology.uiowa.edu. pmid:30525062
  88. 88. Linares D, López i Moliner J. quickpsy: An R package to fit psychometric functions for multiple groups. The R Journal, 2016, vol. 8, num. 1, p. 122–131. 2016. https://journal.r-project.org/archive/2016-1/linares-na.pdf.
  89. 89. Makowski D. The psycho Package: an Efficient and Publishing-Oriented Workflow for Psychological Science. J. Open Source Software. 2018;3(22):470.
  90. 90. Davies A, Vigo M, Harper S, Jay C. The visualisation of eye-tracking scanpaths: what can they tell us about how clinicians view electrocardiograms?. In2016 IEEE Second Workshop on Eye Tracking and Visualization (ETVIS) 2016 Oct 23 (pp. 79-83). IEEE.
  91. 91. Holmqvist K, Nyström M, Andersson R, Dewhurst R, Jarodzka H, Van de Weijer J. Eye tracking: A comprehensive guide to methods and measures. OUP Oxford; 2011 Sep 22.
  92. 92. Debue N, Van De Leemput C. What does germane load mean? An empirical contribution to the cognitive load theory. Frontiers in psychology. 2014 Oct 1;5:1099. pmid:25324806
  93. 93. Rock EP, Finkle J, Fingert HJ, Booth BP, Garnett CE, Grant S, et al. Assessing proarrhythmic potential of drugs when optimal studies are infeasible. American heart journal. 2009 May 1;157(5):827–36. pmid:19376308
  94. 94. Brell JM. Prolonged QTc interval in cancer therapeutic drug development: defining arrhythmic risk in malignancy. Progress in cardiovascular diseases. 2010 Sep 1;53(2):164–72. pmid:20728704
  95. 95. Food and Drug Administration, HHS. International conference on harmonisation; guidance on E14 Clinical Evaluation of QT/QTc interval prolongation and proarrhythmic potential for non-antiarrhythmic drugs; availability. notice. Federal register. 2005 Oct 20;70(202):61134.