Confidence intervals in the classical analysis - using a reference test.

It is still common in medical literature, the classical statistical approach which admits the microscopy as a gold standard. This approach has been criticized also in a malaria context [14], [19]. An important related issue is also the confidence intervals that accompany the point estimate for the sensitivity or the specificity (or other proportions). This problem is not much addressed in medical literature but is still present even when a true gold standard is considered. Usually a 95% confidence interval (95% CI) is obtained by the Wald method that has been strongly criticized due to the poor coverage probability, even for large sample sizes, and the possibility of lower and upper limits outside [26]–[28]. To avoid the latter drawback, we recommend the version of the Wald CI, and other methods, given by Pires and Amado [26]. However, as poor coverage probability remains, other alternative methods for constructing confidence intervals should be used. There are a lot of alternative methods that are re-emerging, for example, the Clopper-Pearson (or exact binomial), Wilson (or score), Agresti-Coull and Jeffreys methods that provide more reliable coverage probabilities than the Wald method. In the context of diagnostic tests, Wilson method was recommended by [29]. In risk situation, when a coverage probability must be guaranteed, a conservative method (e.g. Clopper-Pearson) may present advantages. Nevertheless, these and other recommended methods may also present coverage problems near the boundaries (0 or 1) [26]–[28]. Some of these are available from R Packages or Epitools [30] (caution should be taken regarding the problem of limits outside ). The key to avoid troubles is to use several recommended methods to understand if they provide consistent information. The mathematical expressions of the methods used in this work can be found in Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Lower – – and upper – – bounds of a confidence level for a two-sided confidence interval – – for a proportion (, where is the number of successes) using different methods.

https://doi.org/10.1371/journal.pone.0040633.t002

Latent class models.

Ignoring the limited precision of a reference test can incur serious bias in the performance of other medical diagnostic tests and also in the prevalence estimation. Frequentist and Bayesian latent class models are important mathematical frameworks to study the prevalence and the performance of diagnostic tests in the absence of a gold standard test. In a Bayesian analysis, data are combined with the prior information that expresses expert opinions and other sources of knowledge. The elicitation of an informative prior is a hard and subjective process that needs a careful dialogue between statisticians and experts. Despite the existence of a broad and diverse literature in elicitation of prior distributions, it is mainly oriented to statisticians and not to experts in other fields. However, a vast literature has emphasized the importance of prior information. Speybroeck et al. [14] and the references therein point out the merits of the Bayesian paradigm in the estimation of the parameters associated with three diagnostic tests and the prevalence of malaria infection.

In a frequentist perspective, the parameters of latent class models can be obtained by the well-known Expectation Maximization (EM) algorithm. In a Bayesian approach, the parameters are usually estimated by Markov Chain Monte Carlo (MCMC) methods, via Gibbs sampling. The simplest model is the Two Latent Class Model (2 LCM). In this model, the true disease/infection status of an individual is considered a latent variable, , with two mutually exclusive categories (1. diseased/infected and 0. non-diseased/non-infected). The manifest binary variables, , that express the diagnostic tests results, only give an indication on disease/infection status. The 2 LCM assumes that, given the true state of the disease or infection, the results of the diagnostic tests are independent. This assumption is known as Hypothesis of Conditional Independence (HCI) and it will be discussed in the next subsection. Frequentist and Bayesian latent class models, and their extensions to more complex settings, require a careful analysis of several points to ensure reliable results.

Hypothesis of conditional independence.

According to the parsimony principle, mathematical models with the smallest number of parameters are preferred to the more complex ones. However, to investigate if the simplest and most parsimonious 2 LCM describes the data adequately, we need to check if the HCI is or not violated. The HCI in some medical problems may not be a realistic assumption, for example, when the two tests are based on a similar biological phenomenon (e.g. [20], [31]). The diagnostic of local dependence has been discussed by several authors [31]–[35] and different methods have been proposed. Among others, Hagenaars [32] suggests the analysis of the standardized residuals for each pair of manifest variables. Garrett and Zeger [34] developed a graphical method, the log odds ratio check (LORC) plot, to compare the log odds ratio for the observed and predicted two-way cross classification tables for each pair of manifest variables. Qu et al. [35] also propose a graphical method, the correlation residual plot, which is obtained by plotting residuals of pairwise correlation coefficients, defined as the difference between the observed and expected correlations. Sepúlveda et al. [31] propose the use of Biplot representations based on generalized linear models to identify conditional dependence between pairs of manifest variables within each latent class. In this field, Subtil el al. [36] simulated data incorporating local dependence between pairs of manifest variables and applied different local dependency diagnostic methods and found some problems in the detection of the violation of the principle of conditional independence. In case of failure of HCI, there are alternative approaches to 2 LCM. Alternative models that accommodate conditional dependencies have been proposed in the last decade. Albert and Dodd [37] present an overview of some modeling approaches to incorporate conditional dependence between tests. Qu et al. [35] and Hadgu and Qu [38] developed a general latent class model with random effects to incorporate possible conditional dependencies among diagnostic tests. Additionally, Dendukuri and co-authors presented models in a Bayesian perspective [39], [40]. The accessibility of MCMC methods provide solutions to complex models in evaluation of diagnostic tests [41]. On the other hand, the knowledge transfer from other areas may also contribute to this medical field. In particular, sociology and psychology offer solid methodological developments in the latent class models that may be useful in the context of multiple diagnostic tests, as pointed out by Formann [42].

Non-identifiability and label-switching problem.

The non-identifiability of latent class models is a sensitive issue that requires careful attention. If models are not identified, there will not be a unique computational solution. Jones et al. [43] and the references therein give an overview on identifiability of models for multiple binary diagnostic tests in the absence of a gold standard. Apart from checking trivial conditions, such as that the number of parameters has to be smaller than the number of different patterns, in general, for complex models it is not possible to say a priori whether a model is or not identifiable [42].

An advantage of the Bayesian approach is the incorporation of the prior information to avoid the non-identifiability. When the model is identifiable, non-informative prior distributions can be used for all parameters [40]. When the model is not identifiable, it may still be possible to obtain a solution, adding constraints on the parameters or/and by using informative prior distributions for some parameters (e.g. [44] and [40] and the references therein). In practice, a special attention should be given to the non-identifiability under symmetric priors that leads to the label switching in the MCMC output [45] produced in the parameter estimation process. Label switching occurs when latent classes change meaning over the estimation chain in the context of MCMC. Other types of estimation (e.g., maximum likelihood estimation) can exhibit this problem [46]. Machado et al. [47] show the graphical behavior of the traceplots and posterior densities for the latent class probabilities with a label-switching problem. Stephens [48] points out that the common strategy of removing label switching by imposing artificial identifiable constraints on the model parameters does not always provide a satisfactory solution. In fact, there is an active scientific debate in many fields and other solutions have been proposed in the literature (e.g. [45], [48]–[51]). On the other hand, for the situations in which two diagnostic tests are applied in two populations (the Hui-Walter paradigm), Gustafson [52], [53] demystifies the conventional view of identifiability – “identifiability good, non-identifiability bad”–, presenting realistic scenarios where a moderate amount of prior information leads to reasonable inferences from a non-identified model, and scenarios where large sample sizes may be required to obtain reasonable inferences from an identified model.

Sampling strategies or stratified analysis.

The product-multinomial model appears naturally when we collect independent samples on a number of subpopulations corresponding to the traditional stratified sampling [54]. Gustafson [52] explores one way to develop an identifiable model through pre- or post-stratification of the sample/population according to some categorical variable. Dohoo [55] and Gardner et al. [56] argue that it is acceptable to artificially construct populations with a practical meaning. The post-stratification is a way to overcome some situations, where it is inconvenient or impossible to stratify a population into strata before sampling because the value of the variable of interest is only observed after the individual is sampled. In our application, described before, the variable fever could be an example of this type.

In medical problems, the relevance of distinguishing between subsets is very important to understand if the performance of a diagnostic test varies across smaller groups. As an example, the World Malaria Report 2010 [5] emphasizes that “the clinical sensitivity of an RDT to detect malaria is highly dependent on the local conditions, including parasite density in the target population, and so will vary between populations with differing levels of transmission”. In order to estimate the prevalence and the performance measures of several diagnostic tests in the absence of a gold standard, in two or more distinct populations, BLCM are widely used by the veterinary community [57], [58], where subpopulations (e.g. herds) appear naturally or are created (e.g. [59]). On the other hand, Martinez et al. [16] present a Bayesian approach to estimate the disease prevalence, and the accuracy of three screening tests in the presence of two covariates (age, pregnancy) in the absence of a gold standard for cervical cancer. A logit link function was used to relate the covariates linearly to the screening performance measures to provide a meaningful and well-known measure of association - odds ratio. Posterior odds ratios as association measures between pregnancy and age and the performance measures of the three tests and prevalence are presented. This approach is of great importance in the discovery of potential effects of covariates in the sensitivity, specificity and prevalence. If these effects are already known, it seems to be appropriate to choose a stratified analysis, providing the performance measures of each test in each stratum. In a first view the study design - a random survey - seems to suggest a Latent Class Model with covariates [60], however, as described before, the technicians that applied the RDT knew the age groups and the fever status of the individuals. Thus, the stratified analysis is the chosen approach to the malaria dataset.

M1.

The typical model (with constraints) admits a different prevalence for each subpopulation and the sensitivities and specificities of each test are the same across subpopulations (i.e., for , , , , , , and ),

M2.

The general model (without constraints) that assumes possible differences across subpopulations in terms of prevalence, sensitivities and specificities of each test,

M3.

This model with constraints admits a different prevalence across subpopulations, the specificity of microscopy is equal across subpopulations and also the specificity of PCR (i.e., for , and ). All the remaining parameters vary across subpopulations,

M4.

The general model - M2 - adding: ,

M5.

The same constraints of the M3 adding: .

In a first step, we explored our malaria dataset with non-informative prior distributions for all parameters related with test characteristics ( and and ), using Beta(1,1) distributions, equivalent to Uniform distributions over the interval [0,1]. For the prevalence in the four subpopulations ( and ), the Uniform distribution was considered – U(0,0.5). For the five selected models (M1, M2, M3, M4, and M5), no convergence problems were found and some of the measures that we have been discussing are presented in Table 3.

The assumption of constant test accuracy across subpopulations with different malaria infection prevalence was evaluated though model M1 and it seems to reveal a poorer fit. M2 admits differences across subpopulations in terms of prevalence, sensitivities and specificities of each test and compared with the model M1 seems to fit better. M4 adds only the possibility of febrile and afebrile under-five children and febrile with at least five years old having similar prevalence, but the test’ characteristics varying across subpopulations. This model presents a DIC similar to M2. M3 and M5 present yet better DICs. However, M3 presents a DIC not substantially different from M4. Between M3 and M5, the difference in DICs is also less than 5. Following the recommendations of the BUGS Project [68], we present the estimated parameters according to the three models to investigate possible discrepancies in estimates given by these models (see Table 4).

The posterior inferences, which combine prior information (or lack of it) with data information via Bayes’ theorem, are summarized in Table 4, presenting the posterior means and 95% credibility interval. Additional to the original parameters of the models, the positive predictive values () and negative predictive values () were also indirectly estimated using their relationship with the prevalence, sensitivities and specificities (see expressions, for example, in [29]). M3 and M5 produce similar results. M4 presents some discrepancies at least in some predictive values. According to the parsimony principle, M5 is the simplest model and all criteria of selection and goodness-of-fit are satisfactory, consequently, it is elected as the final model to fit the malaria dataset. Further analysis will be needed to see how the inferences change with different types of informative priors.

RDT.

Table 5 shows a range of values for sensitivities and specificities of the RDT test (ICT Diagnostics, Cape Town, South Africa), according to local area and age groups or fever status. Bendezu et al. [74] describe that the same RDT used in different places showed different results (probably related to different conditions like temperature, humidity, characteristics of the malaria parasites, etc.). The study design, the sample size and statistical analysis of each study also contribute to different findings across different studies.

Microscopy.

As microscopy is usually the reference test, very few papers present its sensitivity and specificity. Speybroeck et al. [14], using a Bayesian approach, found the following posterior means and 95% credibility intervals for sensitivity by survey: 53.0% ([42.0–70.0] in Vietnam, 90.0% [72.0–100.0] in Peru Iquitos, 89.0% [71.0–100.0] in Peru Jaen, Cambodia - Survey 1, and Cambodia - Survey 2. In terms of specificities the lower bounds of credibility intervals were higher than 94%. These authors give details about their prior distributions (Uniform) based on expert opinion. Through the classical analysis, using the PCR as a reference test, Batwala et al. [75] explored the performance of microscopy as a function of laboratory experience – health centre (HC) microscopy and expert microscopy. The point estimates and the 95% CI are reported in both cases. In patients with years, the specificity of HC microscopy was 95.7% [90.8–98.4] and the expert microscopy was 98.6% [94.9–99.8]. In children under-five, the specificities of HC microscopy and the expert microscopy were 89.0% [79.5–95.1] and 94.5% [86.6–98.5], respectively. The overall sensitivity of HC microscopy was 47.2% (36.5–58.1) and the sensitivity of expert microscopy was 46.1% [35.4–57.0].

PCR.

The Bayesian analysis performed by Speybroeck et al. [14], in the absence of a reference test, highlighted that PCR is more sensitive than microscopy and the estimates for sensitivity vary from 95.0% [89.0–100.0] in Vietnam to 98.0% [95.0–100.0] in Peru Iquitos and Peru Jaen. In terms of specificity the results are the following: Vietnam –97.0% [95.0–100.0], Peru Iquitos –99.0% [98.0–100.0] and Peru Jean –100.0% (99.0–100.0). Coleman et al. [76] studied the performance of PCR at different parasite densities relative to expert laboratory microscopy, for active surveillance of Plasmodium falciparum and Plasmodium vivax, and reported that PCR was sensitive 95.7% [84.3–99.3] and specific 98.1% [97.8–98.4] for malaria at parasite densities 500/l. However, the sensitivity of PCR dropped off markedly for parasite densities 500/l. The specificity was constantly high, with a minimum lower bound of the CI equal to 97.4%.

Based on expert opinions on malaria diagnosis in STP and published works focusing on similar diagnostic tests, we consider Beta distributions to represent a pessimistic or skeptical, a optimistic and our prior beliefs distribution. The theoretical parameters and quantiles of these distributions are presented in Table 6. Our prior distributions for each subpopulation express: (i) In general, the RDT test (ICT Diagnostics) presents a similar behavior in each subpopulation; (ii) PCR is usually more sensitive than microscopy; (iii) Microscopy is usually more specific than PCR; (iv) Sensitivity of microscopy is slightly better in the febrile individuals. Except for the specificity of microscopy, we consider the same prior distribution for each parameters across the four subpopulations (see Table 6), even though M5 only admits that the specificity of microscopy and the specificity of PCR are equal across subpopulations.

In Table 7, we present again the the posterior means and HPD intervals for each parameters through model M5 with a skeptical, a optimistic and our prior beliefs distributions. We check the convergence of all parameters, and not just those of interest, before proceeding to make any inference, using the trace plots and Gelman-Rubin and Raftery-Lewis convergence diagnostics measures (the last one is presented in Table 7). DIC, AIC, BIC and Bayesian p-values are also indicated in Table 7. These measures favors our prior beliefs distribution, but the more optimistic prior is yet admissible. The Bayesian p-value (0.007) associated to model M5 with a skeptical prior distribution reveals a prior-data conflict. Compared with the results obtained using M5 with non-informative priors, it can be seen (last columns in Table 4) that our prior information contributes to an increase of the sensitivities of microscopy and PCR, in afebrile children under-five. In the febrile children under-five, the sensitivities of RDT and PCR are also improved. The rest of the parameters are quite similar. In afebrile children under-five, the sensitivity of microscopy (even under an optimistic prior) is very low. This finding is not unexpected since other previous studies have reported low values, when this test is not considered as a gold standard, pointing out that asymptomatic cases often have undetectable malaria parasites by microscopy [14], [75].

Our study is associated to a small region composed of two islands, where an intensive malaria control programme aimed at pre-elimination of malaria was developed with success, where prevalences were highly reduced in general and many positive cases had no malaria clinical signs associated. Therefore, the data and results may not be comparable to other regions elsewhere, with higher prevelances obtained by microscopy or RDT. As mentioned before different results may reflect different factors and the word “comparison” may be too strong. Nevertheless, Chinkhumba et al. [77] state that malaria RDTs must have both sensitivity and specificity above 95% in field setting. These authors report that the sensitivity of the RDTs evaluated in their study are similar to the results of other published studies. However, they found a low specificity in febrile patients above 5 years of age. Kyabayinze et al. [78] alert to the low specificity of the ICT rapid test especially in children below 5 years of age. In our study, in terms of the RDT test, for the afebrile children under-five, the specificity estimated by posterior mean is 93.8% and in the remaining subpopulations is above 97.5%. In terms of sensitivity, for febrile children under-five, we find: 94.1% [87.5–99.4]. In all subpopulations, the positive predictive values of RDT are lower than other tests. The PCR yields reliable results in four subpopulations.