Correction
28 Oct 2015: Kohyama T, Moriyama K, Kanai R, Kotani M, Uzawa K, et al. (2015) Correction: Accuracy of Pulse Oximeters in Detecting Hypoxemia in Patients with Chronic Thromboembolic Pulmonary Hypertension. PLOS ONE 10(10): e0141944. https://doi.org/10.1371/journal.pone.0141944 View correction
Figures
Abstract
Purpose
Pulse oximetry is routinely used to continuously and non-invasively monitor arterial oxygen saturation (SaO2). When oxygen saturation by pulse oximeter (SpO2) overestimates SaO2, hypoxemia may be overlooked. We compared the SpO2 - SaO2 differences among three pulse oximeters in patients with chronic thromboembolic pulmonary hypertension (CTEPH) who spent their daily lives in a poor oxygen state.
Material and Method
This prospective observational study recruited 32 patients with CTEPH undergoing elective cardiac catheterization. As we collected arterial blood samples in the catheter laboratory, SpO2 values were simultaneously recorded. Three pulse oximeters were used on each patient, and SpO2 values were compared with oximetry readings using a blood gas analyzer. To determine the optimal SpO2 value by which to detect hypoxemia (SaO2≦90%), we generated receiver operating characteristic (ROC) curves for each pulse oximeter.
Result
The root mean square of each pulse oximeter was 1.79 (OLV-3100), 1.64 (N-BS), and 2.50 (Masimo Radical). The mean bias (SpO2 - SaO2) for the 90%–95% saturation range was significantly higher for Masimo Radical (0.19 +/- 1.78% [OLV-3100], 0.18 +/- 1.63% [N-BS], and 1.61 +/- 1.91% [Masimo Radical]; p<0.0001). The optimal SpO2 value to detect hypoxemia (SaO2≦90%) was 89% for OLV-3100, 90% for N-BS, and 92% for Masimo Radical.
Citation: Kohyama T, Moriyama K, Kanai R, Kotani M, Uzawa K, Satoh T, et al. (2015) Accuracy of Pulse Oximeters in Detecting Hypoxemia in Patients with Chronic Thromboembolic Pulmonary Hypertension. PLoS ONE 10(5): e0126979. https://doi.org/10.1371/journal.pone.0126979
Academic Editor: Masataka Kuwana, Nippon Medical School Graduate School of Medicine, JAPAN
Received: January 12, 2015; Accepted: April 9, 2015; Published: May 15, 2015
Copyright: © 2015 Kohyama et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Pulse oximeters measure oxyhemoglobin saturation (SpO2). SpO2 is routinely used worldwide not only in the intensive care unit (ICU) [1] and operating room, but also in outpatients [2] to detect patients at risk for hypoxemia. Because a reduction of FIO2 benefits patients with respiratory failure, it is recommended to maintain a SpO2 in mechanically-ventilated patients at approximately 90% [3]. The mechanical ventilation protocol summary by the ARDS network states that the oxygenation goal of patients with ARDS is 55–80 mmHg of PaO2 or 88–95% of SpO2.
Because such recent oxygen therapies substitute the SpO2 for SaO2, the accuracy of pulse oximeters around 90% is crucial to avoid hypoxemia, but some studies suggest that SpO2 overestimates SaO2, especially in patients with critically illnesses. [4] [5]. In a retrospective study including patients with septic shock, Wilson et al. [6] reported that the mean bias (SpO2-SaO2) was positive and 2.75 +/- 3.1%. Wilson et al. [6] also showed that among patients with 90%-93% SpO2 value, 50% of patients were with hypoxemia (SaO2< = 90%). Jubran et al. [7] retrospectively evaluated patients in the ICU and found that the cut-off value of SpO2 to detect hypoxemia (SaO2< = 90%) should be 94%. These results alert the possibility that SpO2 overestimates SaO2 in the ICU, and a cut-off value of SpO2< = 90% may leave patients at risk for hypoxemia.
Because each pulse oximeter follows different algorithms, it is necessary to define optimal SpO2 values to avoid hypoxemia by gathering SaO2 and SpO2 data prospectively from patients in the ICU. Of note, patients in the ICU have different backgrounds and frequently have hemodynamic instability and hypoxemia. Studies have shown that these factors influence the SpO2 values [6] [8]. Therefore, patients with poor oxygenation and without hemodynamic instability, hypercapnia and acidosis may be ideal candidates for defining optimal SpO2 values to avoid hypoxemia.
In the current study we hypothesized that the accuracy of pulse oximeters can be evaluated in hypoxic patients without hemodynamic instability. For this purpose we recruited patients with chronic thromboembolic pulmonary hypertension (CTEPH) because these patients were considered to live with poor oxygenation. When patients with CTEPH underwent elective cardiac catheterization, we attached 3 different pulse oximeters on their fingers. We measured SpO2 using three pulse oximeters and SaO2 simultaneously, and calculated biases (SpO2-SaO2 differences).
Materials & Methods
Study Design and Data Collection
This prospective observational study was conducted at Kyorin University Hospital in Tokyo, Japan. This study protocol was approved by our Institutional Review Board on Human Research (number H25-028). Written informed consent of this study was obtained from all patients. The study period was between September 2013 and February 2014. Thirty-two patients with CTEPH who underwent elective cardiac catheterization were recruited in this study.
When right heart catheterization and percutaneous transluminal pulmonary angioplasty [9] are performed in the catheterization laboratory, we routinely monitor SpO2 and SaO2 in all patients. For the current study, to compare the accuracy of different pulse oximeters, we used three pulse oximeters from three different companies, as follows: OLV-3100 (Nihonkohden, Nishiochiai, Tokyo, Japan); N-BS (Nellcor Puritan Bennett, Pleasonton, CA, USA); and Masimo Radical (Masimo, Irvine, CA, USA). The finger probes used were TL-273T3 (Nihonkohden), D-25 (Nellcor Puritan Bennett), and LNOP Neo-L (Masimo Radical). Each patient was randomly mounted with a total of 3 finger probes from each pulse oximeter on the 2nd, 3rd, and 4th fingers.
The arterial catheter was placed into the radial artery ipsilateral to the pulse oximeter probe. Arterial blood samples were collected anaerobically through the arterial line when patient’s condition was stable and SpO2 values were stable for > 30 seconds. Each SpO2 value was recorded simultaneously as we collected blood samples. The functional SaO2 (HbO2 / [hemoglobin+HbO2]) was determined using a blood gas analyzer (ABL 825; Radiometer, Copenhagen, Denmark). Fractional SaO2 was calculated from the functional SaO2 and the measured levels of carboxyhemoglobin and methemoglobin. Quality control standards were run each day. We tried to obtain 3 arterial blood samples from each patient at different time points.
In addition to patients with CTEPH, we enrolled 5 healthy volunteers to collect control data. Under room air condition, 5 healthy volunteers were mounted with 3 finger probes from 3 pulse oximeters, and arterial blood samples were collected anaerobically from their radial artery.
Statistical Analysis
Assuming an SD of SpO2 to be 2 to 3%, we estimated that a sample of 30 patients would need to be enrolled in order for the study to have 90% power, at a two-tailed significance level of 0.05, to detect a mean between-group difference of 1 SD. Data are expressed as the mean +/- standard deviation (SD) and root-mean-square (RMS), as the international standard of pulse oximeter equipment (ISO 9919) states the accuracy of the pulse oximeter equipment in terms of the RMS difference [10]. Data were also analyzed in 3 subgroups of SaO2 (85%<SaO2≦90%, 90%<SpO2≦95%, and 95%<SaO2≦100%). Statistical analysis was performed by one-way analysis of variance followed by Tukey’s test. To assess agreement between SpO2 and SaO2, a Bland-Altman plot was created, and the mean difference as bias, SD as precision, and the 2SD of difference as 95% limits of agreement were calculated. To assess the accuracy of each pulse oximeter to detect hypoxemia, the sensitivity, specificity, and positive and negative predictive values (PPV and NPV) were calculated. Receiver operating characteristic (ROC) curves were constructed. The closest value to the best specificity and sensitivity point on the ROC curve was identified, and the optimal SpO2 value for each pulse oximeter was determined. All statistical analyses were performed with EZR (Saitama Medical Center, Jichi Medical University, Omiya, Japan) which is a graphical user interface for R (The R Foundation for Statistical Computing, version 3.1.1) [11]. The levels of significance were set at a p = 0.05.
Results
Patient characteristics
We enrolled 32 patients in this study. Table 1 shows the demographics and results of blood gas sampling obtained immediately after arterial catheter placement. Although we tried to obtain three SpO2 data by three pulse oximeters at the same time when we collected blood samples, we failed to collect some SpO2 values. Therefore, we obtained 92 arterial blood samples, 88 SpO2 data points by OLV-3100 (Nihonkohden), 80 data points by N-BS (Nellcor), and 75 data points by Masimo Radical (Masimo). At the beginning of right heart catheterization, the mean values with SD were 93.1% +/-3.5% for SaO2. Fig 1 represents the SaO2 data obtained immediately after arterial catheter placement. Of 32 patients with CTEPH, 21 (65.6%) had an initial SaO2 < = 95%.
Before right heart catheterization, each patient had an arterial catheter placed into their radial artery ipsilateral to the pulse oximeter probes. Arterial blood was sampled, and the SaO2 was measured by blood gas analyzer (ABL 825). Closed squares indicate initial SaO2 values of 32 patients enrolled in this study.
Calculated biases (SpO2—SaO2)
We collected control data from 5 healthy volunteers. Table 2 summarizes data from healthy controls mounted with 3 finger probes from 3 pulse oximeters under room air condition. In 5 patients tested, all the 3 pulse oximeters qualified RMS≦4 (the ISO 9919).
Table 3 shows the calculated mean biases (SpO2—SaO2) measured by 3 pulse oximeters for the 85%–100% saturation range. All three pulse oximeters showed positive biases, suggesting that SpO2 overestimated SaO2. The mean bias by Masimo Radical was significantly higher than the mean bias by Nihonkohden OLV-3100 and Nellcor N-BS (P<0.0001). Although all 3 pulse oximeters qualified for the ISO 9919 (RMS≦4); the RMS by Masimo Radical had the highest value. Fig 2 is the Bland Altman plot indicating a bias and limits of agreement.
For each data point, the mean value ([SpO2 +SaO2]/ 2) is presented on the x-axis, and the difference value (SpO2—SaO2) on the y-axis. Black lines represent the 95% confidence interval for SpO2 [bias ± 2 standard deviation (SD)]. The mean difference value (SpO2—SaO2) represents the bias, and SD represents the precision. Oximeters are: A. Nihonkohden OLV-3100, B. Nellcor N-BS, and C. Masimo Radical. Bias was 0.19% +/- 1.79% (mean +/- SD) by Nihonkohden oximeter, 0.18% +/- 1.64% by Nellcor oximeter, and 1.61% +/- 1.93% by Masimo oximeter.
Some patients had oxygen administered to avoid hypoxemia during the procedure. Actually, 35/92 blood samples were collected from patients with oxygen inhalation. Table 4 shows data from patients under room air condition.
Subgroup analysis
We questioned whether or not the biases of pulse oximeters are influenced by the SaO2 range. Fig 3 shows the calculated biases (SpO2—SaO2) in the 5% range of SaO2. Because the number of samples was small (n = 17), SD values tended to be higher in the 85%< SaO2≦90% range. The mean bias by the Masimo Radical for the 90%< SpO2≦95% saturation range was significantly higher compared with the mean bias by the Nihonkohden OLV-3100 and Nellcor N-BS (0.19 +/- 1.99 for Nihonkohden OLV-3100, 0.34 +/- 1.52 for Nellcor N-BS, and 2.26 +/- 1.53 for Masimo Radical; p<0.0001 vs. Nihonkohden 3100 and Nellcor N-BS). There were no significant differences in the other 2 ranges (85%< SpO2≦90% and 95%< SpO2≦100% saturation range).
Bias is calculated as SpO2 (oximeter-measured value of oxyhemoglobin saturation) minus SaO2. SpO2 measured by Nihonkohden 3100 are indicated with closed circles, SpO2 by Nellcor N-BS with gray circles, and SpO2 by Masimo radical with open circles. Data are presented as the mean ± SD and analyzed by one-way analysis of variance followed by Tukey’s test: * p<0.0001 vs. Nihonkohden 3100 and Nellcor N-BS.
Cut-off value of SpO2
We hypothesized that the cut-off value of each pulse oximeter to detect hypoxemia (SaO2< = 90% or PaO2< = 60 mmHg) was different. We calculated the sensitivity, specificity, PPV, and NPV of each pulse oximeter to detect hypoxemia (Table 5 and Table 6). As shown in Table 5, the specificity ([patients with SaO2>90 and SpO2>90] / [patients with SaO2>90]) by Masimo Radical was 45.5% and the NPV ([patients with SaO2>90 and SpO2>90] / [patients with SpO2>90]) by Nihonkohden 3100 was 65.0%.
Next, we calculated the optimal SpO2 value to detect hypoxemia by constructing ROC curves. The optimal SpO2 value was 89% for Nihonkohden 3100, 90% for Nellcor N-BS, and 92% for Masimo Radical. Using the optimal SpO2 value of each pulse oximeter, the sensitivity, specificity, PPV, and NPV were re-calculated. The specificity by Masimo Radical increased from 45.5% to 81.8%, although the NPV decreased from 100% to 69.2%. The NPV by Nihonkohden 3100 increased from 65.0% to 86.7%, and the sensitivity also increased from 90.3% to 97.2%.
Discussion
Key findings
In this study we evaluated the accuracy of three pulse oximeters to detect hypoxemia in patients with CTEPH who spend their daily life in a poor oxygen state. Among 32 patients enrolled in this study, 21 (65.6%) had a SaO2 < = 95%. We found that all three pulse oximeters had positive biases, suggesting that all SpO2 values overestimated SaO2 values. Among 3 pulse oximeters, significant differences were detected in calculated biases (SpO2—SaO2), especially in the 90%< SaO2≦95% range. We also found that the optimal cut-off values to detect hypoxemia were slightly different among the 3 pulse oximeters (89% for Nihonkohden 3100, 90% for Nellcor N-BS, and 92% for Masimo Radical). Our results suggest that when we substitute SpO2 for SaO2, the optimal SpO2 should be determined for each pulse oximeter to avoid hypoxemia.
Relationship to previous studies
Several data suggest that SpO2 overestimates SaO2 in patients with critically illnesses. In an observational prospective study involving patients admitted to the ICU, Van de Louw et al. [1] reported that the accuracy of SpO2 appeared to be influenced by the type of oximeter, the presence of hypoxemia, and the requirement for vasoactive drugs. Van de Louw et al. [1] suggested that a SpO2 > 94% appears necessary to ensure a SaO2 of 90%. In a retrospective study, Wilson et al. [6] showed that pulse oximetry overestimated the SaO2 by a mean of 2.75% in emergency department patients with severe sepsis and septic shock. Wilson et al. [6] found that the overestimation was exacerbated by the presence of hypoxemia. Of ICU patients, Ghayumi et al. [12] showed that a SpO2 cut-off value ≦94% could predict hypoxemia (PaO2<60 mmHg) with a sensitivity of 100% and a specificity of 95% in liver transplant candidates. The Ghayumi et al. [12] study included patients with a mean SaO2 value of 95.19%, in contrast to the 93.1% mean SaO2 value in our study, with more stable conditions of patients. Our results are consistent with previous studies that showed SpO2 overestimates SaO2. (1,6)
Significance and implications
In this study we put more emphasis on the ability of pulse oximeters to detect hypoxemia rather than RMS differences (SpO2-SaO2) that is an international standard parameter of the accuracy of SpO2. We found that Masimo Radical had 100% of sensitivity and NPV with lowest specificity (Table 5). These statistical parameters suggest that Masimo Radical is most reliable when patients have a SaO2< = 90%. In the clinical setting, especially in ICU, the purpose of monitoring SpO2 is to avoid hypoxemia, keeping the SaO2 > 90%. After applying a calculated optimal SpO2 (92% for the Masimo Radical), the specificity increased from 45.5% to 81.8%, while the NPV decreased from 100% to 69.2%. These changes in statistical parameters are more favorable in keeping the SaO2 > 90%, and beneficial in the clinical setting. These results imply that when we define a statistical “optimal value” for medical monitors, we must consider the clinical purpose of each monitor.
Strengths and Limitations
There were several limitations in this study. First, because this study was performed in Japan and only included patients with CTEPH, all of the patients were Asian (Japanese) and 75% of the patients were female. Because Feiner et al. [13] reported that skin color and gender are predictive of errors in SpO2 estimates at low SaO2 levels (< 80%), our results may not be applicable to patients of general. In addition, some data were gathered during the catheter procedure. These procedures might have affected hemodynamics and resulted in increased biases. Although several factors might have influenced the accuracy of SpO2, we speculate that the unified backgrounds and stable conditions of the patients trump these drawbacks; however, further studies are needed including patients with different races.
Second, we put too much emphasis on detecting SaO2 values < 90%. SpO2 is simply a non-invasive monitor substituting SaO2. In the clinical situation, especially in the general ward, clinicians have a tendency to expect a safer SpO2 range, not approximately 90%, but > 95%. With that point of view, there were no significant differences among the 3 pulse oximeters in the 95<SaO2≦100% range. Thus, keeping patients in the 95<SaO2≦100% range, all the pulse oximeters were reliable and there was no need for detecting the optimal SpO2. However, recent evidence suggests that conservative oxygen (targeting a SpO2 between 90% and 92%) therapy may be beneficial to critically ill patients [14]. Indeed, the accuracy of a SpO2 of approximately 90% may be important for future oxygen therapy. We suggest that further studies are warranted to evaluate “optimal” oxygen therapy on the assumption that SpO2 overestimates SaO2, and optimal SpO2 to detect hypoxemia differs among pulse oximeters.
Third, because this study was limited to patients with CTEPH, these results should be limited to normo/or hypocapnic patients, and may not be applicable to patients with hypercapnia, acidosis and hemodynamic instability. As oxygen dissociation curve clearly illustrates, the SaO2 sure changes according to PaCO2 level and pH. It is also well-known that hemodynamic instability affects the accuracy of pulse oximeters [6]. Further studies are recommended including patients with COPD, who spend their daily life with hypercapnia, and patients with septic shock.
Conclusions
In conclusion, we found that SpO2 measured by 3 pulse oximeters overestimated the SaO2, and the optimal cut-off value to detect hypoxia was slightly different among 3 pulse oximeters (89% for Nihonkohden 3100, 90% for Nellcor N-BS, and 92% for Masimo Radical). We suggest that when SpO2 is substituted for SaO2, optimal SpO2 should be determined for each pulse oximeter to avoid hypoxemia.
Author Contributions
Conceived and designed the experiments: KM TY. Performed the experiments: TK RK MK KU TS. Analyzed the data: KM. Contributed reagents/materials/analysis tools: KM. Wrote the paper: TK KM TS TY.
References
- 1. Van de Louw A, Cracco C, Cerf C, Harf A, Duvaldestin P, Lemaire F, et al. (2001) Accuracy of pulse oximetry in the intensive care unit. Intensive Care Med 27:1606–1613 pmid:11685301
- 2. Majumdar SR, Eurich DT, Gamble JM, Senthilselvan A, Marrie TJ (2011) Oxygen saturations less than 92% are associated with major adverse events in outpatients with pneumonia: a population-based cohort study. Clin Infect Dis 52:325–31 pmid:21217179
- 3. Vadász I, Sznajder Jl (2010) Update in acute lung injury and critical care. Am J Respir Crit Care Med 183:1147–52 pmid:21531954
- 4. Perkins GD, McAuley DF, Giles S, Routledge H, Gao F (2003) Do changes in pulse oxymeter oxygen saturation predict equivalent changes in arterial oxygen saturation? Crit Care 7:R67–R71 pmid:12930558
- 5. Seguin P. Le Rouzo A, Tanguy M, Guillou YM, Feillu A, Malledant Y (2000) Evidence for the need of bedside accuracy of pulse oximetry in an intensive care unit. Crit Care Med 28:703–706 pmid:10752818
- 6. Wilson BJ1, Cowan HJ, Lord JA, Zuege DJ, Zygun DA (2010) The accuracy of pulse oximetry in emergency department patients with severe sepsis and septic shock: a retrospective cohort study. BMC Emerg Med 10:9 pmid:20444248
- 7. Jurbran A, Tobin MJ (1990) Reliability of pulse oximetry in titrating supplemental oxygen therapy in ventilator-dependent patients. Chest 97:1420–1425 pmid:2347228
- 8. Secker C1, Spiers P (1997) Accuracy of pulse oximetry in patients with low systemic vascular resistance. Anaesthesia 52:127–30. pmid:9059094
- 9. Kataoka M, Inami T, Hayashida K, Shimura N, Ishiguro H, Abe T, et al. (2012) Percutaneous transluminal pulmonary angioplasty for the treatment of chronic thromboembolic pulmonary hypertension. Circ Cardiovasc Interv. 5:756–62. pmid:23132237
- 10. Ikeda K, MacLeod DB, Grocott HP, Moretti EW, Ames W, Vacchiano C (2014) The accuracy of a near-infrared spectroscopy cerebral oximetry device and its potential value for estimating jugular venous oxygen saturation. Anesth Analg 119:1381–92 pmid:25313967
- 11. Kanda Y (2012) Investigation of the freely available easy-to-use software 'EZR' for medical statistics. Bone Marrow Transplant 48:452–458 pmid:23208313
- 12. Ghayumi SM, Khalafi-Nezhad A, Jowkar Z (2014) Pulse oximeter oxygen saturation in prediction of arterial oxygen saturation in liver transplant candidates. Hepat Mon 14:e15449 pmid:24748894
- 13. Feiner JR1, Severinghaus JW, Bickler PE (2007) Dark skin decreases the accuracy of pulse oximeters at low oxygen saturation: the effects of oximeter probe type and gender. Anesth Analg 105:S18–23 pmid:18048893
- 14. Suzuki S, Eastwood GM, Glassford NJ, Peck L, Young H, Garcia-Alvarez M, et al. Crit Care Med (2014) Conservative oxygen therapy in mechanically ventilated patients: a pilot before-and-after trial. 42:1414–22 pmid:24561566