2.1 Biological context

We consider the general case of a bacterial pathogen inoculated into an animal host where it can potentially reach n anatomical compartments—which can be distinct organs, tissues, lumens, or predefined sections thereof. All our examples are motivated by IT experiments in which a set of identical animals receive the same initial inoculum dose in one compartment (e.g., mouth, nose, blood, peritoneum, etc) at time t = 0. The inoculum is composed of an even mix of T tagged strains. At given times τ₁, τ₂ etc, a subset of A₁, A₂, etc animals are chosen at random and euthanised. The abundance of each tagged strain in each of the n anatomical compartments of interest is measured. Thus, at a given time τ_i, the data consist of a matrix D_i with n rows by A_iT columns, filled with observed bacterial numbers. From this matrix, we can calculate the observed moments, namely the mean and variance of strain abundance within each compartment, and the covariance between each pair of compartments.

Depending on the experimental procedures, these observations are usually subject to some degree of uncertainty, due to observational error. In general we assume that this error is random with a mean of zero, so that there is no systematic bias; this should be assessed by the researchers who conducted the experiments. As a result, we assume that the observed means are unbiased, but the observed (co)variances may be incorrect. In Section 2.7, we describe how known sources of error can be accounted for as part of the data processing procedure. In addition, there usually is some uncertainty about the actual inoculum dose received by each animal. Variations in the abundance of each strain should be assessed experimentally by testing several inoculum doses: this provides estimates for the initial mean and variance of the number of bacteria with each tag that are present in the target compartment at t = 0.

We emphasise a few key assumptions and caveats of the present study, which we review in further detail in the Discussion. First, the variable of interest from a modelling perspective is the abundance of a single tagged strain, rather than the total bacterial load per animal (as the latter can be obtained by adding up individual strains). Indeed, our model framework assumes that, over the time period considered and for a given set of initial conditions, the rates of bacterial division, death and migration per capita are independent of the total bacterial load. Second, we assume that all the bacterial cells are governed by identical rates of division, death and migration. While this excludes the case of so–called persister cells (i.e., a subset of bacteria with a much lower division rate than the rest) or similar discrete partition, it is worth noting that the model we describe below does generate stochastic variations in the generation time of bacteria, consistent with empirical distributions of bacterial replication in vivo [25].

2.2 Stochastic model framework

As a function of time t since inoculation, the vector of positive integer state variables N(t) = {N₁(t), …, N_n(t)} represents the simultaneous abundance of bacteria in compartments 1 to n. In the context of IT studies, this represents a single tagged strain. Three types of stochastic events drive the bacterial dynamics: division (which adds one bacterium to a given compartment), death (which removes one bacterium from a given compartment) and migration (which moves one bacterium from one compartment to another). Assuming linear transition rates, we have a total of n division rates r_iN_i and death rates k_iN_i within organ i, and n(n − 1) migration rates m_i,jN_i from compartment i to j. Note that specific models may assume that some of the parameters are equal to zero, for example if there is no physical connection between given pairs of compartments.

In particular, we consider two geometries, illustrated in Fig 1, corresponding to two typical anatomical topologies of relevance to bacterial infection: a radial network with a central compartment (e.g., bloodstream supplying every organ), and a linear network (e.g., digestive track).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Diagram illustrating the two types of network structure we consider.

https://doi.org/10.1371/journal.pcbi.1005841.g001

2.3 Computation of the first- and second-order moments

The method we propose for parameter inference relies on the first two moments of the stochastic system. That is, we use only the expected number of bacteria within each compartment, the variance of the number of bacteria within each compartment, and the pair-wise covariances. A simple approach to generating these moments for a particular stochastic system in terms of the model parameters, is given by [26]. Letting λ = {r₁N₁, …, r_nN_n, k₁N₁, …, k_nN_n, m_1,2N₁, …} be the vector of transition rates, we can write the h^th non-central moment of the state of the i^th compartment as: (1) where ϕ_j is a function describing the change of the state for the jth transition. In Supplementary Materials (S1.8), we show that this leads to a closed, linear system of differential equations for the first moments. Letting be the vector of first moments as a function of time, we can express these differential equations in matrix form as: (2) which leads to the solution, (3) where M₁(0) are the initial conditions of the system, A is a time-independent matrix containing the model parameters, and exp is the matrix exponential function.

Next, let be the vector of second moments. By applying Duhamel’s formula to the differential equations obtained from Eq (1), we obtain the following expression for the second-order moments: (4) where B and C are time-independent matrices containing model parameters, and M₁(0) and M₂(0) are vectors containing the initial moments. Using the numerical method for matrix exponential in [27], we can evaluate the first- and second-order moments at any time point. Remarkably, no moment-closure approach is required as the expressions for the second-order moments are independent of higher-order moments. See Supplementary Materials S1.8 for a full derivation.

2.4 Parameter inference by divergence minimisation

Given a dataset consisting of one or more matrices D of bacterial counts (as per section 2.1), and a stochastic model (as per section 2.2), we now describe methods to estimate the parameter values of the model that minimise the divergence between the predicted and observed distributions of bacterial abundance, using only the lower moments of those distributions. Specifically, we evaluate the means, variances and covariances of the N_i variables at a given time t. From the corresponding matrix D, we calculate the vector of observed means μ^(D) and the matrix of observed variance-covariance V^(D); and from the model’s solution given a set of parameters θ, we compute the vector of predicted means μ^(θ) and the matrix of predicted variances-covariances V^(θ) which can be derived from M₂(t). We compared four common divergence measures: a Chi-Squared metric, and normal approximations to the Mahalanobis distance, the Hellinger distance, and the Kullback-Leibler divergence.

Note that none of the measures below is designed to deal with the particular situation when any of the organs is reported void of bacteria (i.e., N_i = 0) in all replicates at a given time point, i.e. if all the observed moments related to that organ are equal to zero. In some experimental systems, this may occur as an artefact of the observation method, e.g., when only a small sample is measured: in this case, it is possible to “correct” the data for sampling biases (see Section 2.7). Otherwise, a simple solution would be to remove the moments relative to that organ from the inference procedure. In some cases, it may make sense to completely remove the empty organ from the model if no meaningful inference can be expected from its inclusion, as illustrated in Section 3.5.

The Chi-Squared metric adds up the squared pairwise-differences between each predicted moment and its corresponding observed moment, each term being scaled by the magnitude of the observed moment. As a result, all moments are effectively treated equally. The expression for this divergence is: (5)

The other three divergence expressions we tested make use of the fact that our chosen statistics μ and V are moments of distributions. Since we do not establish complete characterisations of the distributions, we decided to borrow the expressions of well-known divergences measures for multivariate normal distributions, as these only require the knowledge of their first and second-order moments. In other words, we compute the divergence between two multivariate normal distributions with respective moments (μ^(D), V^(D)) and (μ^(θ), V^(θ)).

The Mahalanobis distance is measured between each point in the observed data, and the distribution described by the predicted moments. That is, by estimating the parameters we are trying to find the distribution that these data are most likely to have been sampled from. Specifically, each observation X_j is given by column j of the data matrix D. The divergence is obtained by summing the distances to the predicted distribution from every observation: (6)

The squared Hellinger distance is given by: (7) where μ = μ^(θ) − μ^(D) and .

The Kullback-Leibler divergence from the predicted distribution to the observed one is given by: (8) where tr(⋅) is the trace operator. The derivations of the Hellinger and Kullback-Leibler divergences are given in Supplementary Materials S1.2.

Even though these expressions may not provide correct estimates of the actual “Mahalanobis distance”, “Hellinger distance” and “Kullback-Leibler divergence” between the data and the predicted distributions (as these are not normally distributed), they still provide adequate divergence measures: they all return positive values which are only equal to zero (hence are minimised) when μ^(θ) = μ^(D) and V^(θ) = V^(D), which occurs when the data are drawn from the predicted distribution. Furthermore, we only require that the calculated divergence here is ordered in the same way as the true divergence between these distributions, so that the minimum occurs at the same parameter values. We use these properties for the purpose of parameter inference, given a dataset and a model. The set of parameter values θ that minimises the divergence measure is termed the “minimum divergence estimate” (MDE).

All code was written in R [28]. We used the UObyQA optimisation routine in the powell package [29].

2.5 Bootstrap variance estimate

In order to quantify the uncertainty in our parameter estimates conditional on the dynamic model considered, we utilise the parametric bootstrap method (e.g., [30]), which can be exploited simply here due to the computational efficiency of the inference approach. Having obtained an MDE, θ* for a given data matrix D and a given model, we simulate B data sets from the model at these parameters (i.e., x^b ∼ f(x ∣ θ*), b = 1, …, B). For each simulation, we estimate the corresponding parameters using the MDE technique (θ^b, b = 1, …, B). These B estimated parameters are used to estimate the variance-covariance matrix of the parameter estimates. Subsequently, one can use this matrix to estimate confidence intervals in a number of ways [31]. For simplicity, the analysis in Section 3.5 uses the ellipse package in R [32] to obtain approximate 95% confidence intervals of the model parameters (that is, assuming the bootstrapped parameter estimates follow a multivariate normal distribution).

2.6 Model goodness-of-fit

In order to assess the model goodness-of-fit, we once again utilise the parametric bootstrap. Concurrent to calculating the uncertainty estimates using the MDE method, the divergences corresponding to each simulated data set at their estimated parameter values are recorded. These bootstrapped divergences can thus be used to represent the null distribution of divergences for the model at the estimated parameter values—giving a representation of the divergences we should expect from the model at these values. The divergence estimated for the observed data is then compared to the null distribution to obtain a p-value for the hypothesis that the data could have been generated by the model.

2.7 Data analysis with observational noise

A common source of error when fitting a model to experimental data is the observation process. In the type of microbiology experiments considered here, where the data represent bacterial loads in infected animals, there are at least two steps that affect the accuracy of the measurements: sampling (when only part of the bacterial loads are recovered) and and quantitation (the process by which the number of bacterial cells in the samples is measured). For example, in a recent IT experiment [11], sampling error was modelled as a binomial process (as known fractions of each homogenised organ were plated) and quantitation error was modelled using a log-normal distribution which was estimated empirically using an independent control experiment (in which known numbers of bacterial colonies were processed by qPCR in the same way as bacterial samples extracted from animals in the main experiment). Although both error distributions were centred on the true bacterial numbers (i.e. the mean numbers of bacteria were not biased), the variance and covariance in the reported data would not have been accurate estimates of the variance and covariance within the animals: hence our MDE could be biased if we did not account for observational errors.

In our present reanalysis of those data, we integrate both error terms before performing parameter inference, using the following procedure. Our goal is to propose a simple heuristic which could be applied to any experimental dataset with known (or assumed) observational error distributions. First, we generate a large number of stochastic simulations of the model under a biologically reasonable range of parameter values (i.e. using uniform prior distributions across sensible ranges) to generate “perfect” observations (1. in Fig 2). We then calculate the corresponding “perfect moments” from every simulated dataset (1M. in Fig 2), which we cannot observe from our experimental procedure. Next, we apply the observation noise to the simulated data (2. in Fig 2), as is the case in the experimental procedure, and calculate the corresponding “observed moments” (2M. in Fig 2). We then use linear regression models to establish a relationship between each of the perfect and observed moments (with transformations where appropriate), with weights given by the simulations proximity to the actual observed moments from the experiment. This calibrated regression model is then applied to the moments of the experimental data of interest, in order to estimate the moments of the true, unobserved bacterial loads in the animals. We eventually compute MDE using these corrected moments.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Schematic demonstrating the experimental procedure to obtain the observed moments.

Ideally, we want to observe the “true” moments, however this is not possible due to the observation process. In order to understand the effect of the observation noise on the moments, we simulate the system, add noise consistent with the experimental system, and evaluate the relationship between the “true” and “observed” moments of these simulations. We can thus infer the “true” moments of the system, given our “observed” moments.

https://doi.org/10.1371/journal.pcbi.1005841.g002

We note that for some models of the observation process, it may be possible to establish analytic relationships between the “perfect” and “observed” moments using conditional expectation and variance theory (e.g., [33]). The choice of suitable correction methods for a given system will depend on both the model complexity and the level of empirical quantification of observation noise.

3.1 Moment computation

As a proof-of-concept, we compare our proposed computation method of the means, variances and covariances of bacterial loads, with the values derived from large numbers of Gillespie simulations, across a range of model structures and parameter values. Our results illustrate the computational effort required to obtain the same level of accuracy with each method. For each model and each parameter set, we simulated 100 experiments, each consisting of 10, 50, 100, 250, 500, or 1000 replicates (representing the product of the number of animals A by the number of tagged strains T as per section 2.1) at a given time, without observational error. The initial bacterial loads at t = 0 in each replicate experiment were drawn at random from a Poisson distribution, to mimic typical variability in inoculum doses in experiments [11]. We considered three model structures, illustrated in Fig 3: (a) basic migration-birth-death model, (b) four-compartment radial network, and (c) four-compartment linear network. For each virtual experiment, we calculated the first- and second-order moments of the bacterial loads in two ways at a given time point (t = 6 for the basic migration-birth-death model, and t = 4 for the four-compartment radial and linear network).

We randomly drew the stated number of initial conditions (ranging from 10 to 1000), and evaluated the moments at the future time point in the following two ways. First, we used Gillespie simulations to progress each initial condition forward to the stated observation time, and then calculated the moments from the collection of simulations. Second, we took the stated number of initial conditions to estimate the moments at time zero, and used these to evaluate the moments at the observation time using Eqs (3) and (4); we refer to this as the direct moment-calculation method.

Results are included in the Supplementary Materials. Figure S1 in S1 Supplementary Materials shows that there is still a greater amount of variation in the moments from 1000 Gillespie simulations, compared to the direct approach. Both methods produce more accurate estimates of the moments as experiment size increases, as a direct result of more accurate estimates of the initial distribution of bacteria. Furthermore, the computation time of the Gillespie approach increases exponentially with the number of simulations, while the direct method is consistently more efficient, independent of the size of the experiment (Figure S2 in S1 Supplementary Materials). Similar patterns are observed for the four-compartment linear (Figure S3 in S1 Supplementary Materials) and radial (Figure S4 in S1 Supplementary Materials) network models.

3.2 Divergence measures: Simulation study

Next, we compare the accuracy of MDE among the four candidate divergence measures, using data simulated from the same three model structures as in the previous section and across a range of parameter values. For each model structure and each parameter combination, we ran 100 series of 100 Gillespie simulations from t = 0 to each of 8 observation times, representing 800 experiments (each of size 100) with “perfect” observations of the system (i.e., without experimental noise). This process was repeated for a set of different parameter values for each model to represent different scenarios—three representative values for each parameter in the basic model (3³ = 27 scenarios), and five randomly generated sets of parameters for the four compartment linear and radial networks with parameters. The initial conditions for each simulation, in every scenario, were randomly generated from a Poisson distribution with mean parameter 200. The optimisation routine was initiated at randomly generated conditions each time, with each parameter values drawn independently from a Uniform(0,1) distribution.

From the results of each simulated experiment, we computed the MDE using each of the Chi-Squared, Mahalanobis, Hellinger, and Kullback-Leibler divergences. The performance of each divergence was measured by the mean absolute relative error (MARE). That is, under scenario s, with p target parameters θ_s = (θ_1s, θ_2s, …, θ_ps), and estimated parameters for the j^th simulation, the MARE is given by:

Fig 4 displays the average error across each scenario for the four divergence measures, at a range of observation times, for each of the three models under consideration. We can see that both the Hellinger and Kullback-Leibler divergence measures perform considerably better than the Chi-Squared and Mahalanobis divergences for the Basic, Linear and Radial networks. The similar performance of the Hellinger and Kullback-Leibler divergences is not unexpected, due to their relationship. We observe a marginal advantage in favour of the Kullback-Leibler divergence for the basic and radial network, and thus we use the Kullback-Leibler divergence in analysing experimental data in Section 3.5. The results aggregated by scenario for each model are shown in Supplementary Materials S1.4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The average mean absolute relative error (MARE) across a number of observation times, for the two-compartment network (top panel), the linear four-compartment network (middle panel) and the radial four-compartment network (bottom panel).

Each point is the average for that measure, at that observation time, with error bars representing the standard error.

https://doi.org/10.1371/journal.pcbi.1005841.g004

Fig 5 further illustrates the distribution of parameter estimates for a few selected scenarios (i.e. combinations of parameter values) and time points. For each time point, the variation was generated by the stochastic birth-death-migration process, demonstrating the level of uncertainty in parameter estimates due to the biological process itself. In this case, each simulated experiment was made up of 100 simulations; fewer replicates would increase the parameter uncertainty. At each time point, the estimates of the migration rate m₁₂ are much less variable than those of the killing rate k₂ and replication rate r₂, reflecting properties of the particular model structure rather than the inference method itself. Indeed, in the absence of division and death in compartment 1, the number of bacteria observed in compartment 1 contains complete information on the migration rate m₁₂. Meanwhile, the net growth rate in compartment 2 is given by r₂ − k₂, producing a strong positive correlation between the estimates of these two parameters (Fig 6).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Box plots of MDE corresponding to simulated data for two different choices of parameter values (Scenario 4 and 12), at a number of different observation times (1, 4 and 8) of the simple, migration-birth-death model.

The box plots represent 100 parameter estimates corresponding to the 100 simulations, and the red crosses denote the true parameter value.

https://doi.org/10.1371/journal.pcbi.1005841.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Bivariate plot of MDE corresponding to simulated data for two different choices of parameter values (Scenario 4 and 12), at a number of different observation times (1, 4 and 8) of the simple, migration birth-death model.

The points represent 100 parameter estimates of r₂ and k₂ corresponding to the 100 simulations, and the red crosses denote the true parameter value.

https://doi.org/10.1371/journal.pcbi.1005841.g006

We also remark that the earlier observation time typically contains more information about the migration rate, whereas the later observation time better captures the replication and killing rates. This is a result of the particular dynamic we are considering, whereby bacteria begin in the first compartment, and steadily move out to the other compartment(s). For a relatively large migration rate, the first compartment is typically close to depleted at later observation times, and thus we obtain a poorer estimate (e.g., Scenario 12, Fig 5). For a smaller migration rate, there will be an optimal observation time that has allowed sufficient numbers of bacteria to have moved out of the first compartment, improving the parameter estimate (e.g., Scenario 4, Fig 5). Results of all scenarios are shown in Supplementary Materials S1.7. Figs 7 and 8 show similar results to Fig 5, for the linear and radial examples, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Box plots of MDE corresponding to simulated data for a random set of parameter values (Scenario 4), at a number of different observation times (1, 4 and 8) of the linear model.

The box plots represent 100 parameter estimates corresponding to the 100 simulations, and the red crosses denote the true parameter value.

https://doi.org/10.1371/journal.pcbi.1005841.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Box plots of MDE corresponding to simulated data for a random set of parameter values (Scenario 4), at a number of different observation times (1, 4 and 8) of the radial model.

The box plots represent 100 parameter estimates corresponding to the 100 simulations, and the red crosses denote the true parameter value.

https://doi.org/10.1371/journal.pcbi.1005841.g008

3.3 Migration in two directions

Many biological systems allow for migration in both directions between any two compartments in the system (e.g., later observations of the Salmonella system in [11] have shown that bacteria migrate back to the blood from the liver and spleen). Here, we consider a three compartment model where there is migration in both directions between two of the compartments, and we are trying to estimate the parameters, m₁₂, m₁₃, m₂₁, k₂, k₃, r₂, r₃. Note that this example can be represented as either a radial or linear network. Similar to the migration birth-death model in previous examples, we consider all combinations of two representative parameter values (0.25, 0.50), and observing the process at times t = 1, 2, …, 8. Initial conditions are drawn from a Poisson distribution with mean parameter 200, and each experiment consists of 100 replicates (i.e., A × T). For each scenario (i.e., parameter combination and observation time), 100 experiments were simulated. As we established above that the Kullback-Leibler divergence was best, we only consider parameter estimation with respect to this divergence measure. The performance of the parameter inference is evaluated using the MARE, as before. Fig 9 shows the MARE across each observation time for the Kullback-Leibler divergence, and Fig 10 demonstrates the parameter estimates for three randomly selected scenarios. The MARE here is consistent with the Kullback-Leibler divergence estimates in the previous example (Fig 4), noting the different number of parameters being estimated. The accuracy of inference on the two reciprocal migration rates (m₁₂ and m₂₁) depends on the choice of observation time, as illustrated on Fig 10 and S8: in particular, at t = 1, very few bacteria have yet migrated both ways (since they all started in compartment 1 at t = 0), and the estimation of m₂₁ is very sensitive to the model’s stochasticity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. The average mean absolute relative error (MARE) across a number of observation times, for the three-compartment network with migration in both directions between two compartments.

Each point is the average for that measure, at that observation time, with error bars representing the standard error.

https://doi.org/10.1371/journal.pcbi.1005841.g009

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Box plots of MDE corresponding to simulated data for a random set of parameter values, at a number of different observation times of the three compartment model with migration in both directions between two compartments.

The box plots represent 100 parameter estimates corresponding to the 100 simulations, and the red crosses denote the true parameter value.

https://doi.org/10.1371/journal.pcbi.1005841.g010

3.4 Comparison with maximum likelihood estimation

In order to assess the overall quality of the inference achieved from this minimum divergence estimation technique, we compare the parameter inferences to those obtained via a Maximum Likelihood approach. The full details of the maximum likelihood estimation can be found in [11]. Briefly, a numerical procedure is used to solve the master equation for a given set of parameter values up to the required observation time. The probability of each observed value is then used to evaluate the likelihood of the observed data. The huge size of the state space in models with two or more compartments poses computational challenges, creating a critical trade-off between computational speed and accuracy. This was addressed by implementing a Runge-Kutta integration algorithm capable of solving large systems (over a million) of simultaneous differential equations. The same optimisation routine as above (UObyQA optimisation routine in the powell package [29]) was then applied to search across the parameter space for the maximum likelihood estimates (MLE). In this section, we compare the speed and accuracy of the two estimation methods (MDE v. MLE) when applied to simulated data from the simple migration-birth-death model (Fig 3a), before turning to experimental data in the next section.

We generated multiple series of simulations corresponding to different numbers of experimental replicates (A × T = 24, 40, 80) observation times (t = 0.5, 2, 4), with various combinations of three numerical values (0.25, 0.50, 0.75), for parameters m₁₂, k₂, r₂. For each scenario (i.e., parameter values, experiment size, observation time), 30 series of simulations were produced, and initial conditions were randomly sampled from a Poisson distribution with mean parameter 50. We computed MDE and MLE separately from each simulated dataset, hence producing 30 estimates for each combination of parameters (see Supplementary Materials for complete set of results). We did not consider more complex models, due to the computational burden of the likelihood-based method.

Fig 11 shows the difference in computation time between the moments-based method, and the likelihood-based method, across the different observation times. In particular, solving the moment equations by matrix exponentiation results in computation times independent of the duration of the simulated experiments. In contrast, the likelihood-based method requires solving a very large system of differential equation forward through time, and so the later the observation time, the longer it takes to compute the parameter estimate. Although this could be avoided in the case of the single-compartment migration-birth-death model by deriving the likelihood at the observation time through inversion of the probability generating functions [12], such semi-analytical solutions are not available for more complex models. As a result, likelihood-based methods become impractical for data collected several days after inoculation [11].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. Comparison of computation times to obtain MDE and MLE from simulated data.

Each box plot comprises of 2430 runs of each estimation procedure: 30 simulated datasets (of size either 24, 40 or 80) were generated from each of 27 combinations of parameter values (see main text for complete list of combinations). The size of the experiment does not impact the computation time for either method.

https://doi.org/10.1371/journal.pcbi.1005841.g011

Fig 12 shows that, across the sets of simulated data, MDE achieves similar or greater accuracy than MLE. Unexpectedly, while the mean absolute relative error (MARE) of MDE decreases as the size of the dataset (number of simulations) increases, the improvement is much less pronounced with MLE. As is visible in Fig 12, there are large subsets of simulations which lead to much better accuracy by MDE than MLE, especially as the number of simulations and their duration increase. Importantly, both methods used the same optimisation routine, either to minimise the divergence or to maximise the likelihood. We should reiterate the algorithm used to compute the likelihood was designed to trade off accuracy for speed: while it should be possible to reduce the MARE discrepancy between the two methods, this could only be achieved by increasing the running time of MLE, which is already far greater than that of the MDE. It is also worth keeping in mind that, no matter how accurate each estimation method is, our aim is to apply them to experimental datasets that typically comprise of under 100 replicate observations per time point [7, 11, 12]. As illustrated above in Fig 5, multiple sets of 100 simulations vary sufficiently from each other to generate wide variance in the parameter estimates. In other words, the system’s natural stochasticity hampers the benefits of seeking greater accuracy.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 12. Pairwise comparison of the mean absolute relative error (MARE) achieved by MLE (x axis) or MDE (y axis).

Each dot shows the MARE values for one simulated dataset, grouped by parameter values (colour scale), observation time and sample size (panels). Note that the axes are square-root transformed to improve clarity.

https://doi.org/10.1371/journal.pcbi.1005841.g012

3.5 Analysis of experimental data

Parameter estimation with observation noise.

We applied the MDE method to estimate the parameters corresponding to these systems, and compared the results to the estimates obtained via the MLE approach, presented in [11]. Since data were recorded from independent samples at two time points (0.5 and 6h post inoculation), parameter estimation was performed by minimising the sum of the divergences between observed and predicted moments at both time points (see Supplementary Materials section S1.6 for a comparison of alternative methods). Having obtained the MDE parameter estimates, we used the bootstrap approach to calculate uncertainty intervals on the estimates, and assess the model goodness-of-fit.

As described in [11], the recorded data were subject to observation noise. In particular, only a fraction of some organs were sampled (e.g., as we are not able to fully recover the total amount of blood from an individual mouse), and there was noise introduced via the qPCR. The observed moments were adjusted using the simulation-based, pre-processing procedure described in Section 2.7: we assumed that the form of the sampling was binomial, and the qPCR noise has been previously modelled using a log-Normal distribution [11]. For example, for the naive group, we sampled 10,000 parameters (c_L, c_S, k_L, k_S, r_L, r_S) each independently from U[0, 3], and initial inoculum i₀ ∼ Poisson(30) (the mean WITS population calculated from plating experiments [11]). “Perfect moments” were calculated from simulations of the same size as the experimental data. The corresponding observed moments were then calculated by applying the observation noise model—binomial sampling to represent the fractional sampling of organs within each mouse, and log-Normal noise to represent the qPCR noise—to each simulation, and calculating the moments. The variances were log-transformed for the purpose of the regression models to predict the corresponding perfect moments from those observed from the experiment.

Fig 13 provides a comparison of the MLE and MDE estimation procedures for the naive and LV experimental groups. The box plots represent the bootstrapped parameter estimates, with the MDE parameter estimate (blue dots), and MLE parameter estimates (red crosses). Fig 14 shows the estimated replication and killing rates in both the liver and spleen, for both groups. The cross and dashed ellipse (red) represent the MLE, and the dot and solid ellipse (blue), represent the MDE (where the ellipse is calculated from 1000 bootstrap samples, represented as grey dots). As explained in the previous section, the strong positive correlation between the replication and killing rates within each organ, clearly visible on Fig 14, is characteristic of this system.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 13.

Left: Diagram of each model illustrating the relevant compartments, initial conditions, and rates of interest. Right: MDE parameter estimate (blue dot) with box plots of the bootstrapped parameter estimates, and MLE parameter estimate (red cross) with 95% confidence interval (red bars).

https://doi.org/10.1371/journal.pcbi.1005841.g013

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 14. Bivariate distributions of replication and killing rates in liver and spleen, for both naive and live-vaccinated groups.

Blue circles are the MDE values, with the blue (solid) ellipses representing the 95% confidence ellipses calculated using the 1000 bootstrap samples (grey points). The red crosses are the MLE values, with red (dashed) ellipse calculated using the hessian evaluated at the MLE.

https://doi.org/10.1371/journal.pcbi.1005841.g014

Model goodness-of-fit.

In calculating the MDE parameter estimates for the bootstrap samples, we evaluated divergence measures for datasets simulated from the model. Thus, the collection of these divergences provide a suitable representation of the null distribution of divergences under this model. We compared the divergence for our observed data set to this null distribution in order to assess the model goodness-of-fit. Fig 15 demonstrates the goodness-of-fit measure for the models fit to the two experimental groups.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 15. Figures demonstrating the goodness-of-fit of the two models to the respective data sets.

The histogram bars are the bootstrapped estimate of the null distribution of divergences under the model at the estimated parameter values for the respective model. The (blue) vertical dashed-line is the divergence corresponding to the observed data set.

https://doi.org/10.1371/journal.pcbi.1005841.g015

The approximate p-values associated with the goodness-of-fit tests for the naive and LV experimental groups are 0.002 and 0.882, respectively. This suggests that the model provides a suitable fit to the vaccinated experimental group; however, there is a non-negligible discrepancy between the model output for the naive group fit, and the observed data. This suggests that some of the assumptions in the original dynamic model may be erroneous; however, revisiting them goes beyond the scope of the present study.

As a further demonstration of the model goodness-of-fit, we compare the model output at each of the estimated parameter sets, to the observed data. Fig 16 shows these simulations from each of the MDE and MLE approaches, at each of the 0.5h and 6h observation times, for both the naive and vaccinated experimental groups. Observational noise consistent with that in the experimental data was added to the simulated data (i.e., binomial sampling for observed counts, and log-normal noise consistent with qPCR). The plots indicate that the main source of discrepancy between data and model in the naive-mice group is the distribution of bacterial loads in the blood after 6 hours.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 16. Observed experimental data for both naive and vaccinated groups, at both observation times, compared to simulations from the model at parameters estimated by both the MLE and MDE methods, in each of the blood (B), liver (L) and spleen (S).

Numbers of bacteria are the sums of eight identical WITS.

https://doi.org/10.1371/journal.pcbi.1005841.g016