Figures
Abstract
Background
In March 2004, the OraQuick® rapid HIV antibody test became the first rapid HIV test approved by the US Food and Drug Administration for use on oral fluid specimens. Test results are available in 20 minutes, and the oral fluid test is non-invasive. From August 2004–June 2005, we investigated a sudden increase in false-positive results occurring in a performance study of OraQuick® oral-fluid rapid HIV tests in Minnesota.
Methodology/Principal Findings
In a field investigation, we reviewed performance study data on oral-fluid and whole-blood OraQuick® rapid HIV test device lots and expiration dates and assessed test performance and interpretation with oral-fluid and whole-blood specimens by operators who reported false-positive results. We used multivariate logistic regression to evaluate client demographic and risk characteristics associated with false-positive results. Next, we conducted an incidence study of false-positive OraQuick rapid HIV tests in nine US cities and tested both oral-fluid and finger-stick whole-blood specimens from clients; reactive tests were confirmed with Western blot. Sixteen (4.1%) false-positive oral-fluid results occurred in the performance study from April 15, 2004 through August 31, 2004 with unexpired devices from six test lots among 388 HIV-uninfected clients (specificity, 95.9%; 95% CI: 93.4–97.6). Three test operators who had reported false-positive results performed and interpreted the test according to package-insert instructions. In multivariate analysis, only older age was significantly associated with false-positive results (adjusted odds ratio = 4.5, 95% CI: 1.2–25.7). In the incidence study, all valid oral-fluid and whole-blood results from 2,268 clients were concordant and no false-positive results occurred (100% specificity).
Conclusions/Significance
The field investigation did not identify a cause for the increase in false-positive oral-fluid results, and the incidence study detected no false-positive results. The findings suggest this was an isolated cluster; the test's overall performance was as specified by the manufacturer.
Citation: Jafa K, Patel P, MacKellar DA, Sullivan PS, Delaney KP, Sides TL, et al. (2007) Investigation of False Positive Results with an Oral Fluid Rapid HIV-1/2 Antibody Test. PLoS ONE 2(1): e185. https://doi.org/10.1371/journal.pone.0000185
Academic Editor: Douglas Nixon, University of California, San Francisco, United States of America
Received: October 16, 2006; Accepted: January 8, 2007; Published: January 31, 2007
This is an open-access article distributed under the terms of the Creative Commons Public Domain declaration which stipulates that, once placed in the public domain, this work may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose.
Funding: Funding for the investigation described in the manuscript was provided by the Centers for Disease Control and Prevention.
Competing interests: The authors have declared that no competing interests exist.
1
In this paper we refer to both OraQuick and OraQuick Advance as OraQuick.
Introduction
In March 2004, the OraQuick® rapid HIV-1 antibody test (Orasure Technologies, Bethlehem, Pennsylvania, USA) became the first rapid Human Immunodeficiency Virus (HIV) test approved by the US Food and Drug Administration (FDA) for use on oral fluid specimens. In June 2004, the FDA approved the test for HIV-2 antibody detection in oral fluid and a name change to OraQuick® Advance Rapid HIV-1/2 Antibody Test [1]. OraQuick1 is a qualitative, visually read lateral-flow immunoassay intended for point-of-care use and is waived for use on oral fluid, finger-stick, and venipuncture whole blood specimens under CLIA (the Clinical Laboratory Improvement Amendments of 1988) [2]. The test is performed on an oral fluid specimen, collected by swabbing the flat pad of the test device once around the outer surface of the upper and lower gums, or on 5 µL of whole blood [1]. A reactive test develops reddish-purple lines in both the test and control zones of the device, while a non-reactive test develops a reddish-purple line in the control zone of the device and no line in the test zone [1]. A test is considered invalid if the control or test line develops outside either the control or test zone, if no control line develops, or if a red background in the result window makes it difficult to read the result after 20 minutes [1]. Results are read in 20–40 minutes [1]. Data submitted by the manufacturer as part of an FDA review for approval indicated a specificity of 99.8% (95% confidence interval [CI]: 99.6–99.9) on oral fluid and 100% (95% CI: 99.7–100) on whole blood [1].
As part of its Advancing HIV Prevention initiative, the Centers for Disease Control and Prevention (CDC) supports rapid HIV testing methods to increase HIV testing and awareness of HIV status [3]. In July 2002, the University of Minnesota's Department of Family Practice initiated a CDC-sponsored study to evaluate the performance of OraQuick to test persons with unknown HIV status in outreach settings [4]. In July 2004, the Minnesota Department of Health requested CDC to investigate a sudden increase in false-positive results with the oral-fluid rapid HIV tests. Test operators described the false-positives as having qualitatively different test lines, but test-line intensity and color were not recorded [4]. In this paper we report on: (1) a field investigation to evaluate false-positive device characteristics, to assess test operator practices, and to retrospectively evaluate performance study client characteristics associated with false-positive test results; and (2) an incidence study of false-positive OraQuick test results to prospectively evaluate client characteristics associated with false-positive test results and device characteristics including test-line intensity and color.
Methods
Field Investigation
Definitions.
A false-positive rapid HIV test result was defined as a reactive oral-fluid OraQuick test result followed by a negative FDA-approved serum or oral-fluid Western blot.
Device investigation.
We reviewed performance study data for two time periods: July 1, 2002–April 14, 2004 and April 15, 2004–August 31, 2004. The latter corresponded to the time period during which 16 false-positive oral-fluid results were reported. To assess device characteristics associated with false-positive tests, we reviewed test storage conditions at the performance study site and examined expiration dates and manufacturer lot numbers of the tests yielding false-positive results. We requested the manufacturer to report any manufacturing discrepancies for implicated lots.
Operator investigation.
We interviewed performance study operators, all of whom had performed and interpreted tests yielding false-positive results, about any unusual device characteristics for those tests. To assess operator practices while performing and interpreting the test we observed operators who reported ≥1 false-positive result as they prepared OraQuick devices, performed one oral-fluid and one finger-stick whole-blood test each on volunteer specimens, inserted devices into the developer solution, timed the test, and interpreted results.
Epidemiologic investigation.
We constructed an epidemic curve of false-positive results that occurred during the performance study. To evaluate demographic and risk characteristics potentially associated with false-positive results, we examined client data collected at the time of testing during two time periods: July 1, 2002–April 14, 2004 and April 15, 2004–August 31, 2004. In univariate analyses, we used exact Mantel-Haenszel chi-square tests and exact 95% confidence limits for odds ratios. We included variables associated with false-positive results at P≤0.25 in univariate analysis in a multivariate logistic regression model. We performed all analyses with SAS (version 9.1, SAS Institute, Cary, North Carolina, USA).
Incidence Study
Rationale.
From February 1, 2005 through May 31, 2005, we conducted a multi-site study of the incidence of false positive results from oral-fluid OraQuick tests because (1) the false-positive results occurred at the end of the Minnesota performance study and it was unknown whether similar incidence would continue; (2) the initial field investigation was retrospective and information on host factors (e.g., pre-existing medical conditions and interfering substances) that might be associated with false-positive results could not be elicited; and, (3) the performance study did not record test line intensity and color, thereby limiting our ability to accurately describe false-positive results read within the 20–40 minute reading window and to determine the degree of under-ascertainment of faintly reactive devices among operators at other sites who had not reported any false positives. If a sufficient number of false-positive results occurred during the incidence study, we planned a case-control study to assess the associations of host factors with false-positive results.
Definitions.
For the purpose of the incidence study, we defined a negative HIV test result as one in which the client's whole-blood rapid HIV test result was non-reactive. We defined a positive HIV test result as one with a reactive whole-blood rapid test followed by an FDA-approved positive serum Western blot.
Sample size.
We based the sample size calculation for the incidence study on the FDA requirement that, in clinical trials of rapid HIV tests, the lower bound of the 95% confidence interval (CI) for specificity must be at least 98% [5]. With a false-positive incidence of 1.5%, a sample size of 1800 would yield a 95% CI for specificity of 97.9–99.0 [6]. If we observed a false-positive incidence ≥1.5%, the test would not meet FDA specificity requirements and there would be sufficient false-positives for us to proceed with the case-control study.
Study settings and methods.
Three state health departments conducted the incidence study from February 1, 2005 through May 31, 2005 at HIV testing sites already using rapid tests in nine cities: Minneapolis, Minnesota; Madison and Milwaukee, Wisconsin; and Atlantic City, East Orange, Hackensack, Newark, New Brunswick and Trenton, New Jersey. Minnesota was invited to participate because the sudden increase in false-positives was reported in Minneapolis; other states volunteered to participate. Clients seeking HIV testing were eligible for enrollment into the incidence study if they were ≥18 years old and had no history of antiretroviral therapy, HIV vaccination, or a previous HIV diagnosis.
All incidence study participants consented to and received both whole-blood and oral-fluid OraQuick tests. Participants with concordant non-reactive rapid test results received a negative result, while those with concordant reactive rapid test results received a preliminary positive result. Participants with discordant rapid test results were informed of both results, offered confirmatory testing, and invited to participate in the case-control study. To assess rapid test specificity, any reactive rapid test result was followed by a ‘gold standard’ Genetic Systems HIV-1 Western blot (Bio-Rad Laboratories, Inc., Hercules, California, USA) on serum specimens. Since sensitivity was not under investigation in the incidence study, no additional testing was conducted on concordant non-reactive whole-blood and oral-fluid OraQuick tests (such additional testing was conducted in the performance study and is reported elsewhere [4]). All study operators had received previous training on administration and interpretation of OraQuick. To ensure identification of all false-positive results, we re-trained operators to interpret and record the intensity and color of all test lines observed.
Data collection and analysis.
We used QDS™ Questionnaire Development System Version 2.3 (Nova Research Company, Bethesda, Maryland, USA) to record rapid-test results, color and intensity of test and control lines, invalid results, manufacturer lot number, and rapid-test operator initials and read times. Sites also reported Western blot results and routinely collected demographic and risk characteristics of testing clients enrolled as study participants. We calculated specificity with a standard formula and performed all analyses with SAS version 9.1.
Human subjects.
As a public-health response to an increase in false-positive test results from an FDA-approved device, CDC determined the study did not require Institutional Review Board (IRB) review under human-subjects-protection guidelines. The Minnesota and Wisconsin IRBs of record concurred. To comply with human-subjects guidelines in New Jersey, the protocol was approved by the New Jersey Department of Health & Senior Services IRB of record.
Results
Field Investigation
Device investigation.
From July 1, 2002 through April 14, 2004 seven (0.3%) false-positive oral-fluid OraQuick results occurred with test devices from four different lots among 2017 HIV-uninfected clients (Fig. 1). From April 15, 2004 through August 31, 2004, 16 (4.1%) false-positive oral-fluid OraQuick results occurred with test devices from six different lots among 388 HIV-uninfected clients (specificity, 95.9%; 95% CI: 93.4–97.6) (Fig. 1). All false-positive tests were stored and used within the manufacturer's temperature specifications, and used before their expiration date. The manufacturer reported that all implicated lots were manufactured and shipped according to standard procedures and that all device components were within manufacturer specifications.
Epidemic curve of false-positive OraQuick rapid HIV antibody test results, oral fluid, by Month and Lot, University of Minnesota study, July 2002 through August 2004.
Operator investigation.
Three of the four operators who had observed the 16 false-positive test results were interviewed; one had left the study and could not be contacted. All were involved in the performance study from start to finish and reported that most test lines they observed in devices with false-positive oral-fluid results were faint; some test lines were described as gray. When test lines were faint operators, who worked in teams conducting outreach testing, reported that they occasionally sought the opinion of another operator on whether the test was reactive or non-reactive. Operators did not report any unusual environmental conditions during the period when the increased number of false-positive tests was reported. We observed that all operators used the rapid test on volunteer oral-fluid and whole-blood specimens in accordance with the package insert and interpreted results correctly.
Epidemiologic investigation.
No client demographic or risk factors were associated with false positive oral-fluid test results during July 1, 2002–April 14, 2004. Further host-factor analyses were restricted to 382 (94%) clients tested in the performance study during April 15, 2004–August 31, 2004 for whom complete demographic and risk data were available. In univariate analyses age, non-Black race, injection drug use, and male-to-male sex were associated with false-positive results at a threshold of P≤0.25 (Table 1). In the multivariate logistic regression model, only older age was significantly associated (P = 0.02) with false-positive oral-fluid results (Table 1).
Incidence Study
Participant characteristics.
Of 3067 eligible clients at all study sites who received a whole-blood rapid test, 2283 (74%) consented to an oral-fluid rapid test. Of 2283 clients who provided consent, 15 (0.6%) were excluded from analyses because either their test results were read before 20 minutes (n = 9) or their records had missing data (n = 6). Demographic and risk characteristics of the remaining 2268 clients are summarized in Table 2.
Test results.
Of the 2268 clients tested, results of one (0.04%) test using whole blood and six (0.3%) using oral fluid were invalid. This difference in the number of invalid results by specimen type was not statistically significant (Fisher's exact test, P = 0.12). Nine (0.4%) clients had reactive tests on both oral fluid and whole blood, and 2253 (99.3%) had non-reactive tests on both specimens (Table 3). Valid oral-fluid and whole-blood test results were 100% concordant, and all reactive rapid tests were confirmed as positive by serum Western blot (100% specificity).
Device characteristics.
No participant had an invalid test on both whole-blood and oral-fluid specimens. Of the six invalid oral-fluid results, one had a partial reddish-purple test line, three had a red background that did not clear at 20 minutes, and two developed no control lines. All six clients with invalid oral-fluid results had non-reactive whole-blood results and non-reactive repeat oral-fluid results. Two oral fluid tests classified as non-reactive were described by the same reader as having a faint gray test line (Table 3); both clients had non-reactive finger-stick results. The test devices were used on two different days, were read at 20 and 21 minutes respectively, and were from two different lots. The first client whose oral-fluid test device had a gray line was a 62-year old White male who reported multiple sex partners and tested anonymously. The second client was a 47-year old Asian male who reported sex with a known HIV-infected male partner six weeks prior to the current HIV test; he had a negative HIV test result three months prior to the current test. No further HIV test results are available for either client as they did not return to the study site for subsequent testing.
Discussion
As part of a two year rapid HIV test performance study, a cluster of 16 false-positive oral fluid OraQuick tests was reported from April 15, 2004 through August 31, 2004 during which the specificity of the test fell below the FDA required minimum of 98% [4], [5]. However, in the first 18 months of the performance study, and in three other contemporaneous studies, oral-fluid OraQuick tests demonstrated a specificity ≥99.6% [4]. The overall specificity of the oral-fluid test for the entire duration of the performance study was 99.0% (95% CI: 98.6–99.4) and there were 23 oral-fluid false positives, 1 oral-fluid false negative, and 2 whole blood false positives [4]. Our field investigation did not suggest a specific cause for the cluster, and our subsequent incidence study detected no false-positive tests. The lack of any false-positive tests despite testing 2268 clients at nine sites during the 4-month incidence study suggests that the observed cluster was isolated and that, even as false-positive results occur at expected rates (we expected two to nine false-positives in the incidence study), future occurrences of false-positive tests may not be evenly distributed over time. While false-positive oral fluid results reported in the performance study occurred in outreach sites, such results have also been reported in a static clinical site (Shelley Facente, personal communication) [7].
The retrospective nature of the field investigation imposed certain limitations. First, we did not observe test operators performing and interpreting the tests at the time they identified the false-positive results. Although we did not note these practices when the operators performed the tests under our observation, over-collection of oral-fluid specimens (swabbing the outer gums more than once), over-interpretation of the test line, or other practices such as specimen contamination and erroneous recording of results might have contributed to false-positive results. The manufacturer intends to systematically evaluate the influence of oral fluid over-collection on test specificity (Stephen R. Lee, OraSure Technologies, personal communication). Second, we were unable to accurately capture the appearance of the false-positive devices at the time they were identified. We used optical densitometry and detected a peak in the test zone of false-positive devices when compared to a non-reactive device but these readings were obtained days to weeks after the test was run and the devices may have changed appearance in the interim. We were therefore unable to make valid comparisons between interpretations of false-positive devices among operators who had identified them versus operators who had not. Third, we were unable to obtain data retrospectively from persons with false-positive results about medical conditions (e.g., Epstein-Barr virus infection, hepatitis A and B infection, rheumatoid factor, or multiparity) which might be associated with false-positive OraQuick Advance results [1].
Older age was significantly associated with false-positive results in the performance study, but age may be a marker for unmeasured conditions that increase with age and that are associated with false-positive results (e.g., prevalence of markers of hepatitis B infection among injection drug users and men who have sex with men). Since no oral-fluid false positives were detected in the incidence study, we were unable to compare age differences between false positives in the performance and incidence study populations. Despite the lack of statistical significance, false-positive results in the performance study occurred more often in specific groups such as injection drug users and men who have sex with men. These findings suggest one or more unmeasured host and site-specific factors may have interacted to produce the cluster. For example, higher rates of viral hepatitis are known to occur in these groups. Both the incidence study and the performance study had similar representation from these groups.
As with any diagnostic test, comprehensive instructions for performing and interpreting the OraQuick test are essential. The OraQuick Advance package insert does not specify how to interpret partial test lines or visible test lines that do not have a discernable reddish-purple color [1]. In the incidence study, two test lines described by the same operator as very faint or gray were interpreted as non-reactive, and one incomplete reddish-purple test line was considered invalid. All three of these test lines occurred among clients with a negative finger-stick rapid test result; since OraQuick is more sensitive with finger-stick whole blood than with oral fluid it is unlikely these were true positives [1], [4]. In the original Minnesota performance study and perhaps elsewhere [7], these types of test results may have been reported as reactive. Without guidance from the manufacturer other clusters of false-positive results might occur, in part because of differential interpretation. The manufacturer should revise the package insert to specify how to interpret partial lines and visible lines in the test zone that do not appear reddish-purple.
With proper training to ensure correct use and interpretation, the OraQuick Advance oral-fluid rapid HIV test can be used successfully as a point-of-care test, although the test has a lower specificity with oral fluid than with whole blood [1], [4], [7]. Regardless of the specimen type, clients should be counseled that a reactive rapid HIV test result is preliminary and must be confirmed [8], [9].
Acknowledgments
We gratefully acknowledge the following institutions and affiliated staff that contributed to this investigation: Atlantic City Health Department; Bergen County Department of Health; City of Milwaukee Health Department; East Orange Health Department; Henry J. Austin Health Center, Trenton; Madison Department of Public Health; Minnesota Department of Health; New Jersey Department of Health and Senior Services; New York State Department of Health; North Jersey Community Research Initiative; Red Door Clinic, Hennepin County Human Services and Public Health Department; Robert Wood Johnson Medical School, University of Medicine and Dentistry of New Jersey; STD Control Program, Los Angeles County Department of Public Health; and the Wisconsin Division of Public Health.
The OraQuick Study Group consists of the authors; Stephen J. Swanson and Jeffrey B.Wiener, Centers for Disease Control and Prevention; Amy Doczy, City of Milwaukee Health Department; Deborah Persell, Hennepin County Human Services and Public Health Department; Mary Jo Hussey, Madison Department of Public Health; Peter Carr, Minnesota Department of Health; and Kathleen Krchnavek and James Vergeront, Wisconsin Division of Public Health.
Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the views of the Centers for Disease Control and Prevention or the US Department of Health and Human Services.
Author Contributions
Conceived and designed the experiments: KJ BB PP DM PS KD TS SP EC EM PK SS DP PC KK JV. Performed the experiments: KJ BB PP DM PS KD TS AN EC EM AD DP MH. Analyzed the data: KJ PP DM PS KD JW. Contributed reagents/materials/analysis tools: PK. Wrote the paper: KJ BB PP DM PS KD TS AN SP EC EM PK SS JW AD DP MH PC KK JV. Other: Lead investigator and coordinated all study sites: KJ.
References
- 1.
OraSure Technologies Inc. (2004) OraQuick Advance rapid HIV-1/2 antibody test [package insert]. Bethlehem, Pennsylvania: Orasure Technologies. Available: http://www.orasure.com/uploaded/398.pdf. Accessed 2007 January 11.
- 2.
Centers for Disease Control and Prevention (2006) CLIA certificate of waiver fact sheet. Available: http://www.cdc.gov/hiv/rapid_testing/materials/roltCLIA.htm. Accessed 2007 January 11.
- 3.
Centers for Disease Control and Prevention (2003) Advancing HIV Prevention: new strategies for a changing epidemic—United States, 2003. MMWR Morb Mortal Wkly Rep. 52. (15): pp. 329–332. Available: http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5215a1.htm. Accessed 2007 January 11.
- 4.
Delaney KP, Uniyal A, Kerndt PR, Gardener A, Keenan PA, et al. (2006) Performance of an oral fluid rapid HIV 1/2 test: experience from four CDC studies. AIDS. 20. : 1655–1660. Available: http://www.cdc.gov/hiv/topics/testing/rapid/pdf/Delaney_OraQuick_AIDS06.pdf. Accessed 2007 January 11.
- 5.
US Department of Health and Human Services, Food and Drug Administration, Center for Drug Evaluation and Research (2000) Development of rapid HIV tests. Presented at: Blood Products Advisory Committee Sixty-Sixth Meeting; Volume I; Silver Spring, Maryland. Available: http://www.fda.gov/ohrms/dockets/ac/00/transcripts/3620t1.pdf. Accessed 2007 January 11.
- 6. Clopper C, Pearson E (1934) The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26([4]): 404–413.
- 7.
Wesolowski LG, MacKellar DA, Facente SN, Dowling T, Ethridge SF, et al. (2006) Post marketing surveillance of OraQuick® whole blood and oral fluid rapid HIV testing. AIDS. 20. : 1661–1666. Available: http://www.cdc.gov/hiv/topics/testing/rapid/pdf/Wesolowski_PMS2_AIDS06.pdf. Accessed 2007 January 11.
- 8.
Centers for Disease Control and Prevention (2006) HIV counseling with rapid tests. Available: http://www.cdc.gov/hiv/pubs/rt-counseling.htm. Accessed 2007 January 11.
- 9.
Centers for Disease Control and Prevention (2004) Notice to readers: Protocols for confirmation of reactive rapid HIV tests. MMWR Morb Mortal Wkly Rep. 53. : 221–2. Available: http://www.cdc.gov/mmwr/preview/mmwrhtml/mm5310a7.htm. Accessed 2007 January 11.