http://orcid.org/0000-0003-4814-7776
Figures
Abstract
Objective
Treatment decisions in inflammatory bowel diseases are increasingly based on longitudinal tracking of calprotectin results. Many hospital laboratories measure calprotectin levels in sent-in stool samples with an enzyme-linked immunosorbent assay (ELISA). Several manufacturers introduced a lateral flow–based test with software application that turns a smartphone camera into a reader for quantitative measurements. We compared three home tests (IBDoc, QuantonCal and CalproSmart) and companion ELISA tests (fCAL, IDK-Calprotectin and Calprotectin-ALP) to see if measurement pairs agreed sufficiently.
Design
A method comparison study was conducted with stool samples from patients with active or quiescent inflammatory bowel disease. Medical students without any specific laboratory training carried out the home tests with two iOS (iPhone 6 and 7) and two Android devices (Samsung Galaxy S6 and Motorola Moto G5 Plus). Two experienced laboratory technicians measured the calprotectin concentration with the ELISA method. Primary outcome was test agreement (defined as percentage of paired measurements within predefined limits of difference). Secondary outcome included reading error rate (RER) per smartphone type.
Results
We performed 1440 smartphone readings and 120 ELISA tests. In the low calprotectin range (≤500 μg/g) IBDoc, QuantOnCal and CalproSmart showed 87%, 82% and 76% agreement with their companion ELISAs. In the high range (>500 μg/g) the agreement was 37%, 19% and 37%, respectively. CalproSmart and QuantOnCal had significantly higher RERs than IBDoc (respectively 5.8% and 4.8%, versus 1.9%). Forty-three percent of reading errors was on the Motorola device, in particular with the QuantOnCal application.
Conclusions
All three calprotectin home tests and companion ELISAs agreed sufficiently when concentrations are ≤500 μg/g. To minimize wrongful interpretation of calprotectin changes over time it is essential to always use the home test and companion ELISA of one and the same manufacturer. Manufacturers should explicitly evaluate and report the suitability of commonly used smartphones for quantitative calprotectin readings.
Citation: Haisma S-M, Galaurchi A, Almahwzi S, Adekanmi Balogun JA, Muller Kobold AC, van Rheenen PF (2019) Head-to-head comparison of three stool calprotectin tests for home use. PLoS ONE 14(4): e0214751. https://doi.org/10.1371/journal.pone.0214751
Editor: Pal Bela Szecsi, Holbæk Hospital, DENMARK
Received: November 27, 2018; Accepted: March 19, 2019; Published: April 18, 2019
Copyright: © 2019 Haisma et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are available in the Dataverse NL repository at https://hdl.handle.net/10411/CVKZ2F and The Groningen Data Catalogue (catalogus.helpdesk@umcg.nl) at https://groningendatacatalogue.com/menu/groningendatacatalogue/dataexplorer/details/umcg_collections/aaaac2knlrv2t6qwh3rgblaaae.
Funding: This work was funded by BÜHLMANN Laboratories AG. Immundiagnostik AG donated reagents and extraction devices for unrestricted use. Neither company had a role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: We have read the journal’s policy and the authors of this manuscript have the following competing interests: This work was supported by BÜHLMANN Laboratories AG, manufacturer of the IBDoc home test and the fCAL ELISA assay used in this study. Immundiagnostik AG, manufacturer of the QuantOn Cal home test and IDK-Calprotectin ELISA assay, kindly donated reagents and extraction devices for unrestricted use. Neither company had a role in the design of this study, nor in the execution, analyses, interpretation of the data or decision to submit results. This does not alter the authors’ adherence to PLOS ONE policies on sharing data and materials.
Introduction
Treatment decisions in patients with inflammatory bowel disease (IBD) are increasingly based on longitudinal tracking of stool calprotectin test results. There are indications that calprotectin-guided escalation of therapy results in better short-term clinical and endoscopic outcomes than symptom-guided decisions alone.[1] Repeated calprotectin concentrations below 250 μg/g are considered to be a useful treatment target and are synonymous with a low risk for disease flare.[2]
Many hospital laboratories measure the calprotectin concentration in sent-in stool samples with an enzyme-linked immunosorbent assay (ELISA). It was generally believed that the protein was stable in stool samples for up to seven days, [3] but recent experiments have shown that calprotectin values gradually decline to approximately 65% of baseline levels after 6 days storage at room temperature.[4, 5] To prevent underestimation of disease activity stool samples collected at home should be refrigerated until arrival at the hospital laboratory, but this complicates transport. A more practical solution would be to measure the calprotectin concentration immediately after stool sample collection at home.
A new measurement method has recently been introduced commercially. The method itself and convenience of use are similar to those of a pregnancy test. By taking a photo of the test strip with a smartphone, the patient can perform quantitative measurements at home. We compared three of these home tests (IBDoc, Quanton Cal and CalproSmart) with the ELISA method of the same manufacturer (fCAL, IDK-Calprotectin and Calprotectin-ALP) to see which of the pairs has the best agreement. In addition, we evaluated the reading error rate (RER) per smartphone type.
Methods
Study design and pre-analytical sample handling
This method comparison study was performed at the University Medical Center Groningen (UMCG, the Netherlands). Between March and May 2018 IBD patients treated at the Department of Paediatric Gastroenterology volunteered in donating stool samples for this study. They defecated onto a stool collection sheet held above the toilet water at home, and transferred a sample of stool into a classical screw top container with a spatula. They described the stool consistency by completing the Bristol Stool Form Scale (BSFS), [6] and send both the screw top container and BSFS in a resealable biomaterial envelope to the Department of Laboratory Medicine. For the purpose of this study the patients themselves were not involved in the measurement of stool calprotectin. Upon arrival at the laboratory, the stool sample was homogenized and then aliquoted for analysis with two separate methods. Both aliquots were stored at -20°C until analysis.
Measurement of stool calprotectin with home test
Three third year medical students without any specific laboratory training carried out the home tests of three manufacturers (Table 1). Each manufacturer-specific test cassette was read with four different smartphones which were purchased especially for this project:: two iOS (iPhone 6 and 7) and 2 Android devices (Samsung Galaxy S6 and Motorola Moto G5 Plus) (Fig 1). The sequence of use of smartphone type and home test was randomized to prevent bias. An hour before the testing session the students defrosted one of two aliquots. They all dipped the assay-specific sampling pin in the sample and transferred the fecal material into the assay-specific tube with extraction buffer. After observing the advised extract processing time, they then applied a drop of extraction fluid in the measurement window of the test cassette as per manufacturer’s instructions. After the appropriate incubation time the cassette was read with the smartphone by placing the camera above the cassette. The image was automatically analyzed by the smartphone application and the quantitative calprotectin result was directly shown on the smartphone screen.
We used 40 stool samples that covered the whole range of potential calprotectin values and performed 1440 smartphone readings and 120 ELISA measurements.
Measurement of stool calprotectin with ELISA
Experienced laboratory technicians, who were blinded for the smartphone readings, thawed the second aliquot and carried out batch ELISA testing with three assays that corresponded with the home tests (see Table 1). The samples were manually weighted and the measurements were done on a Dynex DS2 automated ELISA system (Alpha Labs, Easleigh, UK).
Outcome measures
The agreement between the home test and the companion ELISA measurement was considered as the primary outcome measurement, and was analyzed with a Bland-Altman plot. As described previously [7], we reasoned that disagreement in the lower range of the tests (i.e., below 500 μg/g) could lead more easily to misclassification of disease activity than disagreement in the higher range (>500 μg/g). We therefore used predefined acceptable limits of difference, which were arbitrary set at ±100 ug/g for the lower range, and ±200 μg/g for the higher range. Secondly, concordance of home test and companion ELISA test were determined in each of 3 calprotectin ranges commonly used in our clinical practice (i.e., <250 μg/g for target range, 250–500 μg/g for undecisive range, and >500 μg/g for active disease).[7] Other outcome measures included the reading error rate (RER) per smartphone type, with reading error defined as an image of the test cassette not leading to quantitative result, and evaluation of the usability of the home test and the smartphone application on the first, seventh and last day of the home test experiments. For that purpose, the students completed the system usability scale (SUS).[8] The SUS is a simple, ten item scale giving a global view of subjective assessments of usability and learnability of a system. Scores have a range of 0 to 100, and should be considered as grade scales rather than percentages.
Sample size calculation
The primary outcome was taken as the overall percentage of agreement between the calprotectin home test and the corresponding ELISA test per manufacturer. We wished to detect a difference in agreement of at least 15% by a two-sided test. With the level of significance set at 5% and a study power of 80%, we aimed to include at least 110 paired measurements per manufacturer.
Statistical analysis
In Bland-Altman analysis, a scatter plot is constructed in which the difference between the paired measurements is plotted on the vertical axis and the average of the measures of two methods on the horizontal axis. The mean difference in values obtained with the two methods is called the bias and is represented by a central horizontal line on the plot. The standard deviation (SD) of differences between paired measurements is used to construct horizontal lines above and below the line to represent 95% limits of agreement (LOA). The plot enables the reader to visually assess the bias, data scatter and the relationship between magnitude of difference and size of measurement. ELISA measurements out of the measurable range of the home test were rounded to the upper limit of the home test range. In addition, we provide a distribution histogram of the differences between the paired measurements in the low range calprotectin concentrations. Concordance of home test and companion ELISA readings in each of the three calprotectin ranges are presented in a scatterplot. RERs per home test and smartphone type are reported as percentages with confidence intervals. Data were recorded electronically by using SPSS version 23.0 for Apple Mac (IBM Corporation, Armonk, NY). Graphs were constructed with GraphPad Prism version 7 for MacBook (GraphPad Software, San Diego, California USA). P-values <0.05 were considered statistically significant.
Ethical consideration
The Medical Ethics Review Committee of the University Medical Center Groningen waived consent requirement for this specific study, as no human subjects as meant in the Dutch Medical Research Involving Human Subjects Act (WMO) were involved. Nonetheless legal guardians from all participants, as well as the children aged 12 and above, gave informed consent to use their voluntary stool samples for research. This study was conducted in compliance with the Clinical Trial Agreement, the study protocol, designated Standard Operating Procedures and the international standard for studies for In Vitro Diagnostic Medical Devices. (ISO 22870: 2016 Point-of-care testing (POCT)–Requirements for quality and competence). All authors had access to the study data and reviewed and approved the final manuscript.
Results
Between March and May 2018, we selected 40 stool samples, of which 23 (58%) were in the low calprotectin range (<500 μg/g) and 17 (42%) in the high range (≥500 μg/g). Per homogenized stool sample 36 smartphone readings and 3 ELISA measurements were done.
Bland-Altman analysis
In Table 2 we show the agreement between home test and corresponding ELISA per manufacturer using the Bland-Altman plot analysis. IBDoc underestimated the fCAL ELISA results over the whole range of potential calprotectin values with a mean bias of 105 μg/g (agreement interval 942 μg/g). QuantOn Cal overestimated IDK-Calprotectin ELISA results with a mean bias of 137 μg/g (agreement interval 1574 μg/g). CalproSmart overestimated Calprotectin-ALP ELISA results with a mean bias of 141 μg/g (agreement interval 914 μg/g). Fig 2 shows that in all comparisons the paired measurements were more clustered and closer to the zero-difference line in the low calprotectin range and more scattered in the high range.
The grey zone corresponds with the predefined acceptable limits of difference, which were arbitrary set at ±100 μg/g for the lower range and at ±200 μg/g for the higher range.[7] The dotted line is the bias (mean of the differences), the dashed lines are the upper and lower 95% limits of agreement (LOA).
Low calprotectin range (≤ 500 μg/g).
In the low calprotectin range the percentage of paired measurements within acceptable limits of difference for IBDoc, QuantOn Cal and CalproSmart were 87% (263/304), 82% (285/349) and 76% (235/310), respectively. Fig 3 shows that the difference scores for IBDoc, QuantOn Cal and CalproSmart were distributed in an approximately normal pattern around the bias of -19 μg/g (agreement interval 560 μg/g), -4 μg/g (agreement interval 455 μg/g) and 72 μg/g (agreement interval 458 μg/g), respectively. The agreement interval of all three comparisons exceeded the acceptable limits of difference in the low range (i.e. 200 μg/g).
High calprotectin range (>500 μg/g).
In the high calprotectin range the percentage of paired measurements within acceptable limits of difference for IBDoc, QuantOn Cal and CalproSmart were 37% (61/167), 19% (20/108) and 37% (52/142), respectively. The difference scores were not normally distributed around the bias (data not shown). The bias for IBDoc, QuantOn Cal and CalproSmart was -262 μg/g (agreement interval 1160 μg/g), 591 μg/g (agreement interval 2392 μg/g) and 293 μg/g (agreement interval 1310 μg/g), respectively. The agreement interval of all three comparisons exceeded the acceptable limits of difference in the high range (i.e. 400 μg/g).
Concordance
Fig 4 shows the concordance between home tests and corresponding ELISA tests in each of three ranges (i.e., <250, 250–500 and >500 μg/g) that we use in our clinical practice. Of the IBDoc-fCAL pairs 358 of 471 (82%) were concordant. Discordance between the IBDoc-fCAL pairs leading to serious misclassification of disease activity (i.e., calprotectin >500 μg/g with one method and <250 μg/g with the other) were observed in 24 of 471 stool samples (5%). QuantOn Cal and IDK-Calprotectin had 361 of 457 (79%) concordant test pairs, and 35 of 457 (8%) discordant test pairs leading to serious misclassification of disease activity. CalproSmart and Calprotectin-ALP had 330 of 452 (73%) concordant test pairs, and 11 of 452 (2%) seriously discordant test pairs.
Reading error rate
The CalproSmart and QuantOn Cal smartphone applications had significantly more reading errors than the IBDoc application, with rates of respectively 5.8% and 4.8% versus 1.9% (P = 0.002 and P = 0.012). Forty-three percent of the total amount of reading errors was with the Motorola device, in particular in combination with the QuantOn Cal application (Fig 5). Common reason for reading errors was an out of focus image.
System usability scale
Fig 6 shows that mean SUS scores per home test were lowest on the first day of testing. On the last day of testing IBDoc was awarded the highest grade (B) of all home tests, mainly because the smartphone application was error-friendly and therefore less cumbersome to use.
SUS scores above 90 should be converted to “A”; 80–90 to “B”; 70–80 to “C”; 60–70 to “D”; and below 60 to “F”.
Discussion
This is the first head-to-head method comparison study investigating three calprotectin home tests and their companion ELISA tests. We have shown that the majority of measurements performed with a lateral flow immunoassay and smartphone reader agreed sufficiently with the ELISA-based quantification of the same manufacturer, provided that calprotectin levels are below 500 μg/g. In the high calprotectin range a substantial proportion of measurement pairs exceeded the predefined limits of difference (±200 μg/g). QuantOn Cal performed significantly poorer in the high range than IBDoc and CalproSmart. IBDoc had significantly fewer reading errors than the other two applications. The Motorola device turned out to be less suitable for calprotectin readings than the other smartphone types. A complete summary of the test performance per manufacturer is shown in S1 Table.
The possibility to measure calprotectin with a smartphone at home is a novelty in IBD-care. We identified three recently published studies on the IBDoc home test, [7, 9, 10] and one on the CalproSmart home test.[11] All studies showed acceptable agreement between home test and companion ELISA in the low range and lack of agreement in the high range. These studies differ from ours in several important ways. First, in three of the aforementioned studies, home tests were performed before shipping the unrefrigerated stool samples to the laboratory.[7, 10, 11] The possible decline in calprotectin during transport may have affected the agreement with the ELISA results, while in our experiment the transport of stool samples was taken out of the equation. Second, none of the studies performed a subgroup analysis per smartphone type, while we compared several iOS and Android devices. By doing so, we were able to show that not all smartphone cameras were suitable for calprotectin readings. Third, in this study we report an a priori decision of acceptable limits of difference for adequate interpretation of the Bland-Altman plot, a feature that was absent in three of the aforementioned studies.[9–11] Precision is of utmost importance in the low ranges of calprotectin values, where small deviations can lead to misclassification of disease activity and wrong treatment decisions. In our practice, however, imprecision in the high range is less of an issue, since we consider each calprotectin result >500 μg/g to reflect active disease, irrespective of the absolute concentration. A twofold increase in calprotectin, e.g. from 1000 to 2000 μg/g, does not necessarily mean that the inflamed surface area of the gastrointestinal tract doubled, neither that the intensity of the inflammation increased. Any shift of calprotectin values out of the target range, and into the action range (≥ 500 μg/g) is a trigger for us to change the treatment plan. In order to appreciate the true value of the home tests in the high range, we also evaluated the concordance with companion ELISAs. We observed that 119 of 125 IBDoc readings ≥ 500 μg/g were concordant with fCAL results (95%), compared to 35% and 49% for the QuantOnCal—IDK-Calprotectin and CalproSmart—Calprotectin-ALP pairs.
There are some limitations in our study that need to be addressed. First, the home tests were not performed by real IBD patients. Instead, we had three 3rd year medical students (AG, SA, JAAB) without any specific laboratory training to fill the lateral flow cassettes and execute the smartphone readings. They only had access to the instructions for use and video tutorials on the internet. We acknowledge that their level of education is higher than the “average IBD patient”. Second, although we aimed to collect stool samples that covered the whole range of potential consistencies, we missed the liquid samples. For that reason we were unable to evaluate the effect of stool consistency on the precision of the lateral flow-based method.
The evidence base for calprotectin-guided treatment escalation and de-escalation is accumulating.[1, 2, 12] Simultaneously, the number of telemonitoring initiatives for IBD care is rising.[13–16] Telemonitoring with calprotectin home testing can make the service to IBD patients more efficient, as the technology allows to select and target patients that benefit from a face-to-face encounter with their IBD-team at short notice.
We evaluated by how much the home tests differed from the trusted ELISA method, and found that in the lower ranges the difference was small enough not to cause problems in interpretation. We conclude that the flow-based home-test and the companion ELISA method can be used interchangeably. To minimize wrongful interpretation of calprotectin changes over time it is essential to always use the home test and companion ELISA of one and the same manufacturer. This advice was already in force for the use of ELISA tests.[17–20] Ultimately, patients and health care providers would both benefit from better co-calibration of stool calprotectin assays.
Supporting information
S1 Table. Summary of performance of calprotectin home tests.
https://doi.org/10.1371/journal.pone.0214751.s001
(DOCX)
Acknowledgments
We thank Lucie Wagenmakers and Koos van de Belt (laboratory technicians, University Medical Center Groningen) for performing the ELISA measurements.
References
- 1. Colombel JF, Panaccione R, Bossuyt P, Lukas M, Baert F, Vañásek T, et al. Effect of tight control management on Crohn’s disease (CALM): a multicentre, randomised, controlled phase 3 trial. Lancet 2018;390:2779–2789. pmid:29096949
- 2. Heida A, Park KT, van Rheenen PF. Clinical Utility of Fecal Calprotectin Monitoring in Asymptomatic Patients with Inflammatory Bowel Disease: A Systematic Review and Practical Guide. Inflamm Bowel Dis 2017;23:894–902. pmid:28511198
- 3. Røseth AG, Fagerhol MK, Aadland E, Schjønsby H. Assessment of the neutrophil dominating protein calprotectin in feces. A methodologic study. Scand J Gastroenterol 1992;27:793–8. pmid:1411288
- 4. Padoan A, D’Incà R, Scapellato ML, De Bastiani R, Caccaro R, Mescoli C, et al. Improving IBD diagnosis and monitoring by understanding preanalytical, analytical and biological fecal calprotectin variability. Clin Chem Lab Med 2018;56:1926–1935. pmid:29729144
- 5. Haisma SM, van Rheenen PF, Wagenmakers L, Muller Kobold AC. Calprotectin Instability May Lead to Treatment Errors in Patients with IBD. Arch Dis Child 2019. [Epub ahead of print].
- 6. O’Donnell LJD, Virjee J, Heaton KW. Detection of pseudodiarrhoea by simple clinical assessment of intestinal transit rate. BMJ 1990;300: 439–40. pmid:2107897
- 7. Heida A, Knol M, Muller Kobold AC, Bootsman J, Dijkstra G, van Rheenen PF. Agreement Between Home-Based Measurement of Stool Calprotectin and ELISA Results for Monitoring Inflammatory Bowel Disease Activity. Clinical Gastroenterology and Hepatology 2017;15:1742–1749. pmid:28606846
- 8. Brooke J. SUS: a retrospective. Journal of Usability Studies 2013;8:29–40.
- 9. Piekkala M, Alfthan H, Merras-Salmio L, Puustinen Wikström A, Heiskanen K, Jaakkola T, et al. Fecal Calprotectin Test Performed at Home: A Prospective Study of Pediatric Patients With Inflammatory Bowel Disease. JPGN 2018;66: 926–931. pmid:29240011
- 10. Bello C, Røseth A, Guardiolac J, Reenaers C, Ruiz-Cerulla A, Van Kemseke C, et al. Usability of a home-based test for the measurement of fecal calprotectin in asymptomatic IBD patients. Digestive and Liver Disease 2017;49:991–996. pmid:28587751
- 11. Vinding K, Elsberg H, Thorkilgaard T, Belard E, Pedersen N, Elkjaer M, et al. Fecal Calprotectin Measured By Patients at Home Using Smartphones—A New Clinical Tool in Monitoring Patients with Inflammatory Bowel Disease. Inflamm Bowel Dis 2016;22:336–344. pmid:26535869
- 12. Doherty G, Katsanos KH, Burisch J, Allez M, Papamichael K, Stallmach A, et al. European Crohn’s and Colitis Organisation Topical Review on Treatment Withdrawal ['Exit Strategies'] in Inflammatory Bowel Disease. J Crohns Colitis 2018;12:17–31. pmid:28981623
- 13. Aguas Peris M, Del Hoyo J, Bebia P, Faubel R, Barrios A, Bastida G, et al. Telemedicine in inflammatory bowel disease: opportunities and approaches. Inflamm Bowel Dis 2015;21:392–9. pmid:25437818
- 14. De Jong MJ, van der Meuleren-de Jong AE, Romberg-Campus MJ, Becx MC, Maljaars JP, Cilissen M, et al. Telemedicine for management of inflammatory bowel disease (myIBDcoach): a pragmatic, multicentre, randomised controlled trial. Lancet 2017;390:959–968. pmid:28716313
- 15. Heida A, Dijkstra A, Muller Kobold AC, Rossen JW, Kindermann A, Kokke F, et al. Efficacy of Home Telemonitoring versus Conventional Follow-up: A Randomized Controlled Trial among Teenagers with Inflammatory Bowel Disease. J Crohns Colitis 2018;12:432–441. pmid:29228230
- 16. Patil SA, Cross RK. Current Landscape of Telemedicine Practice in Inflammatory Bowel Disease. Inflamm Bowel Dis 2018; Epub ahead of print.
- 17. Labaere D, Smismans A, van Olmen A, Christiaens P, D’Haens G, Moons V, et al. Comparison of six different calprotectin assays for the assessment of inflammatory bowel disease. United European Gastroenterology Journal 2014;2:30–37. pmid:24918006
- 18. Whitehead SJ, French J, Brookes MJ, Ford C, Gama R. Between-assay variability of faecal calprotectin enzyme-linked immunosorbent assay kits. Ann Clin Biochem 2013;50: 53–61. pmid:23129721
- 19. Kittanakom S, Shajib S, Garvie K, Turner J, Brooks D, Odeh S, et al. Comparison of Fecal Calprotectin Methods for Predicting Relapse of Pediatric Inflammatory Bowel Disease. Canadian Journal of Gastroenterology and Hepatology 2017:1450970. pmid:28491862
- 20. Amcoff K, Stridsberg M, Lampinen M, Magnuson A, Carlson M, Halfvarson J. Clinical implications of assay specific differences in f-calprotectin when monitoring inflammatory bowel disease activity over time. Scand J Gastroenterol 2017;52:344–350. pmid:27881032