Simulated isothermal titration calorimetry data

To assess the accuracy of BCIs in a context where all sources of error are known, we simulated ITC data in which the integrated heat is given by, (2) where is the model integrated heat at injection i for an ITC without instrument noise, given by Eq S3 in S1 Appendix. It is a function of several parameters,

• ΔG: the free energy of binding,
• ΔH: the enthalpy of binding,
• ΔH₀: the enthalpy of dilution and stirring per injection, and
• [L]_s and [R]₀: the concentrations of titrant in the syringe and of the titrand in the cell, respectively.

In the simulated curves, ΔG = −10 kcal/mol, ΔH = −5 kcal/mol, and ΔH₀ = 0.5 μcal. The concentrations were drawn from lognormal distributions with the stated values of 0.1 mM for [R]₀ and 1 mM for [L]_s and with uncertainty 10%. ϵ_i represents error in the measurement of q_i and was modeled as normal variable with zero mean and standard deviation of 1 μcal. The number of injections was 24 (i ∈ [1, 24]). A total of 50 simulated heat curves were generated.

Titration of Mg(II) into EDTA

In order to assess the effectiveness of the Bayesian approach in describing the true uncertainty in the experimental measurements, we studied a simple complexation reaction—the 1:1 binding of Mg(II) to the chelator EDTA—for which multiple experimental replicates can be easily collected. The entire ITC experiment was repeated from scratch—with all solutions prepared completely independently so that any concentration errors would be fully independent—a total of 14 times. This is critical, as simply repeating the experimental measurement with the same stock solutions would not capture the true experimental variability. For each trial, the titrant (MgCl₂), titrand (EDTA), and buffer (50 mM Tris-HCl pH 8.0) were weighed and dissolved to prepare solutions at the two planned concentrations for the titrant MgCl₂ and the titrand EDTA. In the first five trials, we prepared the titrant and titrand concentrations as 1.0 mM and 0.1 mM, respectively. In the other nine trials, the titrant and titrand concentrations were prepared as 0.5 mM and 0.05 mM, respectively.

Magnesium chloride hexahydrate [MgCl₂⋅(H₂O)₆] was purchased from Fisher Scientific (Catalog No. BP214-500, Lot No. 006533) and anhydrous ethylenediaminetetraacetic acid (EDTA) was purchased from Sigma-Aldrich (Catalog No. E6758-500G, Batch No. 034K0034). Tris base was purchased from Fisher Scientific (Catalog No. BP154-1, Lot No. 082483). Buffer was prepared by weighing Tris base, adding MilliQ water, and adjusting the final pH to 8.0 by dropwise titration with HCl or NaOH. Solutions were prepared by weighing powder and adding the appropriate amount of buffer, neglecting the volume occupied by powder, to make a concentrated solution (15 mM for MgCl₂ and 1.0 mM for EDTA). To maximize the number of significant figures, at least 0.1 g of MgCl₂ and 0.01 g of EDTA were weighted out. The solutions were then further diluted with buffer to prepare the titrant and titrand. For example, to prepare a 0.1 mM solution of EDTA, a pipetman was used to measure 9 parts buffer to 1 part of 1.0 mM EDTA.

ITC measurements were performed on a MicroCal VP-ITC calorimeter. The experiments consisted of a total of 24 injections, with the first injection programmed to deliver 2 μL of titrant (MgCl₂) into the sample cell, and the remaining 23 injections programmed to deliver 12 μL. Data was collected for 60 s prior to the first injection and 300 s for each injection. The injection rate for all injections was 0.5 μL/s. All experiments were conducted at 298.1 K, and the reference power was fixed at 5 μcal/s.

The baseline was corrected and injection heats integrated using NITPIC [33].

Titration of phosphonamidate-type inhibitors into thermolysin

To demonstrate our approach on protein:ligand systems, we also analyzed titrations of phosphonamidate-type inhibitors into thermolysin initially described in Krimmer et. al. [34]. For each individual measurement, lyophilized thermolysin powder was freshly weighed (1.5–2 mg) and dissolved in an appropriate volume of buffer to achieve a concentration of 30 μM. The concentration was confirmed by ultraviolet absorption at 280 nm. Prior to measurement, the thermolysin soluton was centrifuged for 8 min at 8150 g. In contrast, one solution was prepared for all measurements with each ligand by dissolving the pure powder (0.3–0.4 mg) in buffer without the addition of DMSO. A MX5 microbalance from Mettler Toledo (Switzerland) with a readability of 1 μg and a repeatability of 0.8 μg was used for the sample weighting. Measurements were repeated in this fashion at least nine times. Results reported in [34] were based on three repetitions with a fresh batch of thermolysin and after optimizing ITC parameters. In contrast, our present analysis was based on all available data for each system except for a small subset with a large baseline shift in the middle of an injection.

Lyophilized thermolysin (EC number 3.4.24.2) from Bacillus thermoproteolyticus was purchased from Calbiochem (EMD Biosciences). The inhibitors (Fig 1) P-((((benzyloxy)carbonyl)amino)methyl)-N-((S)-4-methyl-1-oxo-1-(propylamino)pentan-2-yl)phosphonamidicacid (ligand 1), P-((((benzyloxy)carbonyl)amino)methyl)-N-((S)-1-(isobutylamino)-4-methyl-1-oxopentan-2-yl)phosphonamidicacid (ligand 2), and P-((((benzyloxy)carbonyl)amino)methyl)-N-((S)-4-methyl-1-(((S)-2-methylbutyl)amino)-1-oxopentan-2-yl)phosphonamidicacid (ligand 3), were generously provided by Nader Nasief and David G. Hangauer (University of Buffalo, Buffalo, New York, USA), who synthesized and purified them as previously described [35]. In the paper [35], they claimed that the compounds were at least 95% pure. (Crystal structures of ligands 1 (PDB ID 4MXJ), 2 (PDB ID 4MTW), and 3 (PDB ID 4MZN) in complex with thermolysin have been previously reported [34]). All measurements were performed with a buffer composed of 20 mM HEPES (pH 7.5), 200 mM NaSCN, and 2 mM CaCl₂ ⋅ 6H₂O. HEPES was purchased from Carl Roth (Catalog No. 9105.3, Batch no. 192184596), NaSCN was purchased from Fluka Analytical (Catalog No. 71938-1KG, Lot no. BCBC9384V), and CaCl₂ ⋅ 6H₂O was purchased from Carl Roth (Catalog No. T886.2, Lot no. 433205269). Prior to measurement, the buffer was filtered through a 0.22 μm filter and degassed under reduced pressure.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Chemical structures of thermolysin ligands used in this study.

https://doi.org/10.1371/journal.pone.0203224.g001

ITC measurements with thermolysin were performed on an MicroCal ITC₂₀₀ calorimeter from GE Healthcare (Piscataway, New Jersey). After an initial delay 170 or 180 sec, the initial injection (0.3 to 0.5 μL) was followed by 19 to 26 main injections (1.2 to 1.5 μL). The duration of the injection (in sec) was twice the value of the volume (in μL). All measurements were performed at a temperature of the measurement cell of 298.15 K, a stirring speed of 1000 rpm, titrand (thermolysin) concentration of 30 μM, and a titrant (ligands 1-3) concentration of 400 μM. For details on each protocol, see S1 Table.

As with the Mg(II):EDTA data, the baseline was corrected and injection heats integrated using NITPIC [33]. Representative differential power and integrated heat curves for Mg(II):EDTA binding and thermolysin:ligand binding are shown in S1 Fig. Some data were excluded due to large baseline shifts that could not be readily corrected.

CBS:CAII dataset from the ABRF-MIRG’02 study

Finally, we considered a protein:ligand ITC dataset from a previously published study which demonstrated large interlaboratory variation far in excess of reported error estimates [23]. As data were unavailable from the authors, injection heat data were digitized from Fig 4 in [23], the ABRF-MIRG’02 paper, which includes 14 ITC datasets measured fully independently on identical source material (aliquots of CAII and dry powder stocks of CBS) by independent laboratories. Dataset 2 was generated by an instrument called the CSC 4200 ITC (see Table 2 in [23]) for which we could not find the user’s manual to obtain information such as the cell volume. Therefore, we excluded this dataset. We also excluded dataset 4 because we were not able to reliably digitize the large number of injections. For other datasets, the experimental design parameters were taken from Table 2 in [23], while the reported thermodynamic parameters and standard errors were taken from Table 3 in [23]. In the ABRF-MIRG’02 study [23], most experiments obtained standard errors were using a nonlinear least squares fit. The exceptions were datasets 10 and 14, in which the standard deviation was obtained by repeating the same experiment 3 and 5 times, respectively. In these datasets, it was not clearly specified whether the entire experiment or just the titration was repeated in each replicate.

Digitization leads to some error. As a quantitative estimate of this error, consider the experiment from Group 1 in Fig 4 in [23], the ABRF-MIRG’02 paper. When zoomed in, the center of the axes and the markers can be identified to within a single pixel. The markers are 8 pixels high, and the plot is 182 pixels high. This translates to a maximum error of 1 pixel (marker center) on an axis conversion of 11 kcal/mol / 182 ± 2 pixels 0.060 ± 0.001 kcal/mol. Hence, the error (0.06 kcal/mol/injection) is rather small, even if done by eye.

Frequentist confidence intervals

Origin software was used to perform nonlinear least squares fit of the heat data to obtain the binding constant K_a, enthalpy ΔH, and the stoichiometry number n, and their corresponding standard errors. Each parameter was assumed to be normally distributed and the standard error was used as a standard deviation. The lower and upper bounds of the α% confidence interval were the 1 − α/2 and 1 + α/2 percentile, respectively, of the normal distribution with a mean as the point estimate and standard deviation as the reported uncertainty.

Sampling from the Bayesian posterior

Our Bayesian model is constructed to infer the unknown true parameters, (3) In addition to the parameters described above in “Simulated ITC data”, this model includes σ, the standard error of heat measurement per injection. σ is a nuisance parameter in that it is not the objective of a measurement but is necessary to calculate the likelihood. The assumption of a constant σ is most reasonable if all injections include the same number of power measurements.

Priors.

The prior p(θ) was a product of priors for each parameter, p(θ) = ∏_j p(θ_j). Uniform priors were chosen for ΔG, ΔH, and ΔH₀: (6) (7) (8) where q_min = min{q₁, q₂,…, q_N}, q_max = max{q₁, q₂,…, q_N} and Δq = q_max − q_min, usually reported in units of cal.

We used three different sets of priors for the true concentrations of titrant in the syringe, [L]_s, and receptor in the cell, [R]₀ (Table 1): General, Flat [R]₀, and Comparison. All of the concentration models make use of the fact that concentrations must be positive. In the General model, both concentrations are assigned lognormal priors with the mean and standard deviation given by their stated experimental values and corresponding experimental uncertainties due to preparation steps, (9) The lognormal prior prevents negative values and the lognormal distribution is the maximum entropy distribution when the mean and variance of the logarithm of the parameter is specified. In the absence of specific quantification of the titrant concentration uncertainty, we assumed a value of δ[X]₀ equal to 10% of the provided [X]₀. This specified uncertainty is in line with quantification of typical laboratory titrant concentration errors observed by Myszka et. al. [23]. In cases where the practitioner uses an orthogonal method to quantify titrant concentration or carefully tracks the uncertainty during preparation steps, as described in Boyce et. al. [25], this more precise concentration uncertainty could be used instead. Alternatively, δ[X]₀ could be treated as a free nuisance parameter. Although the parameter may not be precisely determined from a single ITC experiment, it could potentially be elucidated by sampling from a Bayesian posterior based on multiple measurements with the same titrand solutions, analogous to a global fitting procedure possible with nonlinear least squares [36].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of concentration priors used in this manuscript.

https://doi.org/10.1371/journal.pone.0203224.t001

In the Flat [R]₀ model, a uniform prior was used for [R]₀ such that, (10) This model is useful in cases where the receptor concentration is not clearly known, such as when the sample is impure or partially degraded. Due to potential degradation of protein used in some ITC measurements, we used this model in our analysis of data for thermolysin. Finally, in the Comparison model, we used a uniform prior for [R]₀ and a sharply peaked prior for [L]_s such that .

The Comparison model mimics the treatment of concentrations in standard nonlinear least squares fitting. In the standard procedure, [L]_s is assumed to be precisely the stated value while [R]₀ can take any positive value that minimizes the total residual sum of squares. There is no penalty for changing [R]₀ from its stated value. This is consistent with flat prior for [R]₀ a sharp prior for [L]_s. On the other hand, the General model allows for but penalizes deviations from the stated values. In the absence of further information, we believe that the General model is the most justified of the three models because concentrations are likely to be close to their stated values. Our main reason for performing calculations with the Comparison model was to isolate the effects of concentration models from other aspects of the Bayesian analysis.

Finally, since even its order of magnitude may be unknown, an uninformative Jeffreys prior [37] was assigned to the noise parameter σ, (11) where σ₀ ≡ 1 cal is a reference quantity that simply renders the ratio σ/σ₀ dimensionless. This model assumes that the injection heat measurement uncertainty σ is constant for all injections. This may be a good approximation when the same number of power measurements are integrated for each injection (i.e., when injections are of identical duration), but when experiments contain injections of different durations, the noise variance σ² should be proportional to the number of power measurements summed to give the injection heat (with all other things being held constant). More complex noise variance models (such as those considered in [38]) could also be considered. The noise model could also be improved using calibration experiments based on the same protocol (such as blank titrations), or even other data collected on the instrument for other systems; in these cases, likelihoods from independent experiments are simply multiplied.

While we used uninformative priors (except for our concentrations) in this study, alternative priors for other parameters can be used. If some knowledge of thermodynamic parameters or concentrations is available from another type of experiment, e.g., spectrophotometric measurements, then these can be incorporated into their respective priors. In such cases, the prior could be normally-distributed with the sample mean and standard deviation as parameters. Another way to parameterize the prior for concentrations is by careful propagation of error during the sample preparation process (from estimates of known pipetting error magnitudes, known analytical balance accuracies, and reported compound purities). Alternatively, the posterior from a previous (e.g., pilot) ITC experiment can be used as the prior to integrate the information from a second ITC experiment with different experimental parameters.

The Kullback-Leibler divergence quantifies differences between thermodynamic parameter distributions obtained from Bayesian and nonlinear least-squares approaches

To compare posterior marginal distributions, we computed the Kullback-Leibler divergence (KL-divergence), between the posterior marginal densities in the two most important thermodynamic quantities of interest, (ΔG, ΔH), (12) and are the posterior marginal densities specified by two different experiments with associated datasets and . This metric, commonly used as a measure of deviation between two probability densities, can be interpreted as the amount of information lost when is used to approximate . The marginal posterior density for each experiment was estimated by using a Gaussian kernel density estimate (KDE) based on MCMC samples for ΔG and ΔH (ignoring other parameters). We used the KernelDensity package implemented in scikit-learn [43] to estimate the density p(ΔG, ΔH). The bandwidth for Gaussian kernel was set to 0.03 kcal/mol. Although the Kullback-Leibler divergence can be analytically computed for Gaussian densities, we also used the same KDE method to estimate probability densities p(ΔG, ΔH) for nonlinear regression. Samples for ΔG and ΔH were drawn from Gaussian distributions with the mean and standard deviation based on nonlinear regression point estimates and errors, respectively.

Markov chain Monte Carlo sampling leads to precise estimates of Bayesian credible intervals

Our MCMC sampling protocol appears to yield precise estimates of 95% BCIs (Fig 2 and S2–S5 Figs). In all of the selected systems, the estimated 95% BCIs do not substantially change after considering about 2000 samples. The standard deviation of estimated upper and lower bounds over the five independent simulations in each system was less than 5% of the length of the average interval. Therefore, we are confident that the number of MCMC samples and mixing of the MCMC chain is sufficient to yield consistent estimates of the BCIs and other statistics of interest.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Convergence of 95% Bayesian credible intervals (BCIs) with MCMC sampling.

5000 MCMC samples were generated from the Bayesian posterior (General model) for several variables based on one ITC experiment measuring Mg(II):EDTA binding. For five independent repetitions of the MC simulations, the black lines are running estimates, as the number of samples is increased, of the upper and lower limits of 95% BCIs. The red line and error bars are the average and standard deviation across the five independent simulations. Similar plots for ligands 1-3 binding to thermolysin and CBS:CAII are available as S2–S5 Figs.

https://doi.org/10.1371/journal.pone.0203224.g002

Bayesian analysis yields unimodal distributions of linearly correlated parameters

Bayesian analysis permits multimodal posteriors and nonlinear parameter correlations to be investigated. Qualitative trends in the posterior density may be visualized by generating histograms of MCMC samples drawn from the posterior. For our systems, representative 1D marginal distributions of key parameters (Fig 3) are unimodal. Although some skew is evident in ΔH, the Gaussian distribution could be considered a reasonable approximation for most of these parameters. Our observation is consistent with previous analyses of nonlinear regression which showed that a Gaussian assumption is appropriate when the magnitude of statistical error is less than 10% of a parameter [28, 29].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Representative 1D marginal distributions of thermodynamic parameters from Bayesian ITC analysis.

1D marginal probability densities for thermodynamic parameters of interest were estimated based on 5000 MCMC samples generated from the Bayesian posterior (General model) for one ITC experiment measuring Mg(II):EDTA binding. Horizontal bars show 95% Bayesian credible intervals. The triangle in density plot of [R]_o indicates the stated value.

https://doi.org/10.1371/journal.pone.0203224.g003

Representative 2D marginal distributions (Fig 4) show that some pairs of parameters are nearly independent and others are highly correlated, with varying degrees of correlation in between. Of particular interest is the fact that while the free energy ΔG and enthalpy ΔH are mostly uncorrelated (top left of Fig 4), there is high correlation between the enthalpic (ΔH) and entropic (TΔS) contributions to binding (top right of Fig 4) and between ΔH and the receptor concentration [R]₀ (bottom right of Fig 4). These correlations are not considered in the standard nonlinear regression analysis.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Representative 2D marginal distributions of pairs of thermodynamic properties from Bayesian ITC analysis.

2D joint marginal probability densities were estimated based on 5000 MCMC samples generated from the Bayesian posterior (General model) for one ITC experiment measuring Mg(II):EDTA binding. TΔS was derived from the sampled parameters ΔG and ΔH to aid our discussion of enthalpy-entropy compensation.

https://doi.org/10.1371/journal.pone.0203224.g004

Given that the correlations appear to be linear, they can be succinctly summarized via the correlation coefficient. The estimated correlation matrix shown in Table 2 indicates that the titrant [L]_s and titrand concentrations [R]₀ are highly correlated with each other and with the enthalpy ΔH but only weakly with ΔG. This result is consistent with Tellinghuisen [44], who evaluated the sensitivity of the binding constant and enthalpy to changes in concentration.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Correlation matrix estimated from the Bayesian posterior (General model) for an Mg(II):EDTA binding dataset.

Numbers in parentheses denote the uncertainty in the last digit.

https://doi.org/10.1371/journal.pone.0203224.t002

Estimates of concentrations and ΔH are correlated because the effect of changing one of the parameters can be largely counteracted by changing another. When samples from a Bayesian posterior for MG(II):EDTA binding were used to parameterize a simple linear model for [L]_s and ΔH as a function of [R]₀, different parameter values led to essentially the same integrated heat curve (Fig 5). An important implication of this enthalpy-concentration compensation is that given a measured integrated heat curve, the precise values of the three parameters are underdetermined; by itself, ITC cannot simultaneously determine the titrant or titrand concentration and the enthalpy of binding.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Integrated heat for different values of titrand concentration [R]₀ for Mg(II):EDTA binding.

Corresponding values of [L]_s and ΔH were based on a simple linear regression of [L]_s and of ΔH versus [R]₀. The other parameters (ΔG, ΔH₀) took the last value from the MCMC time series (General model). The legend shows the titrand concentration [R]₀, in mM, corresponding to each line.

https://doi.org/10.1371/journal.pone.0203224.g005

Due to the observed correlations, apparent enthalpy-entropy compensation [45] is a possible consequence of simple concentration error. If multiple measurements of the same receptor-ligand pair were performed with different disturbances to the receptor concentration (e.g. small dilutions), ΔH would be perturbed more significantly than ΔG, leading to an apparent trend between ΔG and TΔS.

Median enthalpy estimates are sensitive to the titrand concentration model

Even though nonlinear least squares fitting and Bayesian analysis are based on the same binding model, other variations in the analysis procedure may lead to different estimates of ΔG and ΔH. We compare different analysis methods by considering how the median (which is less sensitive to outliers than the mean) of each quantity within a dataset. For all datasets, the median estimate of ΔG is largely consistent across the different analysis methods. In contrast, with the thermolysin datasets, ΔH estimates are consistent between all models except for the General model, which differ by as much as 1.5 kcal/mol (Table 3).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Median estimates of ΔH (kcal/mol).

For nonlinear least squares, the value is the median of the different point estimates across different measurements. For Bayesian analysis, it is the median of the median sample from each Bayesian posterior. The numbers in parentheses are standard deviations estimated by bootstrapping: resampling the datasets (for nonlinear least squares) or the MCMC samples (for Bayesian analysis) with replacement using 1000 replicates.

https://doi.org/10.1371/journal.pone.0203224.t003

The consistency between all models except for the General model indicates that the major reason for discrepancy is the prior on the receptor concentration. In all but the General model, the titrand concentration freely changes (subject to the constraint [R]₀ > 0) from stated concentration without penalty. In the General model, the prior penalizes deviations from the stated value of [R]₀. Estimates of the concentration affect ΔH but not ΔG because concentrations are highly correlated with ΔH but not with ΔG.

It is also evident that the titrand concentration is the determining factor for the shift in median ΔH because the titrant concentration is lognormal in all the Bayesian priors. Modifying the standard deviation in the lognormal distribution affects credible intervals but does not change the median. By elimination, the factor that leads to the shift in the median is usage of a lognormal instead of uniform prior for the titrand concentration.

One result that is insensitive to the choice of concentration model is the one-to-one stoichiometry of these binding processes. Even when the prior on [R]₀ is completely flat, concentrations sampled from the posterior are fairly close to the stated concentration. Sampled [R]₀ that are a factor of two (or more) greater than the stated concentrations could indicate multiple ligands binding to each receptor molecule. Since sampled concentrations are similar to stated concentrations, the selected one-to-one binding model is suitable for these systems.

Bayesian confidence intervals are more consistent with each other than those from the standard analysis

In addition to the median enthalpy, the width and and consistency of intervals is also dependent on the concentration model (See Table 4, Fig 6, and S8–S19 Figs). For ΔH and [R]₀ in particular, NlRCIs and BCIs based on the Comparison model are narrower and correspondingly less consistent with one another than BCIs based on other concentration models. BCIs based on the General model are substantially broader and those based on the flat [R]₀ model are broader still. However, all the BCIs and NlRCIs for ΔG are of comparable magnitude.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Figure numbers for confidence interval plots in this manuscript.

https://doi.org/10.1371/journal.pone.0203224.t004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Uncertainty estimates from Bayesian and nonlinear least squares analyses of Mg(II):EDTA ITC replicates.

95% credible intervals estimated from the Bayesian posterior based on the General model (left) and confidence intervals from nonlinear least squares (right) for parameters specifying magnesium binding to EDTA. The vertical green lines are the median of the median MCMC samples. There are two median estimates for R because the experiments were done at two different concentrations. Red bars denote the standard deviations of the lower and upper bounds, estimated by bootstrapping, and are a total of two standard deviations wide.

https://doi.org/10.1371/journal.pone.0203224.g006

In contrast with the dependence of the shift in the median enthalpy on the titrand concentration model, the change in interval size is primarily driven by the titrant concentration model. The Comparison and Flat [R]₀ model have the same uniform prior for the titrand concentration. However, the size of the ΔH and [R]₀ intervals for the Flat [R]₀ model is much larger because the standard deviation in the lognormal model for [L]_s is larger. (Data are in figures noted in Table 4).

On a note related to the width and consistency between confidence intervals, nearly every pair of 95% BCIs for ΔG and ΔH from the General and Flat [R]₀ model have at least some overlap with one another. (The 95% BCIs for [R]₀ do not overlap when the stated concentrations differ, as in the Mg(II):EDTA and CBS:CAII datasets.) As with other statistics, BCIs based on the Comparison model are very similar to NlRCIs (S7, S10, S13, S16 and S18 Figs).

One complication with assessing confidence interval estimates is that we do not know the “true” value. Because we do not know the “true” value, we used the median value from repeated experiments as an approximation. The mean value is also a suitable choice, but the median is less sensitive to outliers.

Most of the 95% BCIs for ΔG, ΔH, and [R]₀ from the General and Flat [R]₀ models contain the median. One exception is for the CBS:CAII dataset, in which BCIs for ΔG capture the median less consistently. In contrast, while most 95% NlRCIs for ΔG contain the median (except in the CBS:CAII dataset), the 95% NlRCIs for ΔH and [R]₀ generally do not. BCIs from the Comparison model behave similarly to NlRCIs (S5, S8, S11, S14 and S16 Figs). The size of these intervals appear to be significantly underestimated in all of our systems.

A better way to visualize the performance of confidence intervals is to compare the fraction of intervals that contain the true value with the stated confidence level. If stated levels are accurate, they should reflect the probability that the interval contains the true value. In this type of plot, therefore, data points should lie along the diagonal, the solid line of Figs 7 and 8, and S19–S22 Figs Points below the diagonal indicate that stated confidence intervals are too small. Conversely, points above the diagonal indicate that they are too large.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Uncertainty validation for Bayesian analysis of simulated data.

The predicted versus observed rate (%) in which BCIs contain the true value (red circles), the mean (blue leftward triangles) or the median (green rightward triangles) for binding parameters are shown. Error bars are standard deviations based on bootstrapping.

https://doi.org/10.1371/journal.pone.0203224.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Uncertainty validation for Bayesian and nonlinear least squares analyses of Mg(II):EDTA data.

For the Mg2:EDTA binding experiments, the predicted versus observed rate (%) in which intervals contain the median value for binding parameters is shown. Intervals were BCIs based on the General model (blue leftward triangles), Comparison model (green rightward triangles), or nonlinear least squares confidence intervals (red circles). Error bars are standard deviations based on bootstrapping.

https://doi.org/10.1371/journal.pone.0203224.g008

Simulated data are useful for performing this type of assessment in a context where true parameter values and all sources of error are known. We generated 50 simulated heat curves (S6 Fig) and estimated BCIs for each. As an assessment of the BCIs, we then plotted the fraction of BCIs that contain the true value against their stated levels. For the simulated data, the observed fraction of BCIs containing the true value (red dots) is very close to the stated levels (Fig 7), indicating that the Bayesian approach produces accurate confidence intervals.

The simulated data are also valuable for assessing our choice of medians as approximations to true parameter values. For the simulated data, the mean and median give almost the same estimate for the true value, but the median is slightly closer (Table 5). The blue and green dots in Fig 7 correspond to the observed fraction of BCIs containing the mean and the median, respectively. Except for the fraction containing the mean ΔG at low confidence (blue dots on left panel of Fig 7), they are very close to the observed fraction of BCIs containing the true value. Because the performance of the median and mean is similar and the median is more robust with respect to outliers, the median is a reasonable approximation to the true value.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Comparison of Bayesian estimates of mean and median with the true values for simulated data.

The numbers in parentheses denote uncertainty in the last digit, which are standard deviations estimated by bootstrapping by resampling the MCMC samples and the datasets with replacement 1000 times.

https://doi.org/10.1371/journal.pone.0203224.t005

Based on uncertainty validation, BCIs based on the General model perform nearly ideally for Mg(II):EDTA and less reliably for the other experimental datasets. In the cases of Mg(II):EDTA and ligand 2:thermolysin binding, the observed fraction of BCIs (General model) for ΔG and ΔH that contain the median is very close to the ideal line (Fig 8 and S20 Fig). For the other datasets, BCIs based on the General model are less consistent with observed rates. In the cases of ligand 1:thermolysin and ligand 3:thermolysin, the median-containing frequency of ΔG BCIs is also very close the ideal line whereas that ΔH BCIs deviates from ideality, especially for larger confidence intervals (S19 and S21 Figs). In the CBS:CAII dataset, however, BCIs for ΔH are more consistent with observed rates than for ΔG.

Intervals from other models had variable performance. NlRCIs of ΔG have similar performance to BCIs but the observed rate at which NlRCIs for ΔH contain the median is significantly less than ideal. This deviation from ideality is consistent with the poorly overlapping 95% confidence intervals for ΔH. BCIs from the Comparison model behave similarly to NlRCIs. In contrast with intervals from other models, BCIs based on the flat [R]₀ model generally overestimate the width of intervals for the thermolysin model. The overestimation of intervals suggests that the uniform prior employed in this analysis is too uninformative.

Overall, our Bayesian method (with the General model) led to reasonable BCIs for multiple measurements performed by a single individual within a single laboratory. The performance of BCIs in accounting for laboratory-to-laboratory variability in the CBS:CAII datasets digitized from the ABRF-MIRG’02 paper [23] was weaker. In this dataset, there must be one or more significant sources of error that the present approach fails to account for.

The strong correlation between concentrations and ΔH explains the dramatic improvement of the credible intervals of ΔH (e.g. Fig 8) when the uncertainty in [L]_s is included in the Bayesian analysis. In the same vein, the weak correlation between concentrations and ΔG explains why NlRCIs for ΔG are reasonable (Fig 8) even if the titrant concentration was treated as exactly known in the fit. Trends in the accuracy of confidence intervals are consistent with previous analyses based on error propagation [24, 25, 44, 45], which showed that titrant concentration errors propagate to small relative errors in ΔG but large relative errors in ΔH. If the error in titrant concentration is correctly propagated, it may be possible to make NlRCIs more accurate [25], but testing this is beyond the scope of the present work. In subsequent analysis, we will only consider the General model.

Binding parameter distributions are more consistent with Bayesian analysis than nonlinear regression

In most datasets, the estimated Kullback-Leibler divergence between pairs of Bayesian posteriors is smaller than those estimated for nonlinear regression (Fig 9 and S23–S26 Figs). For the thermolysin datasets where the flat [R]₀ model was tested, the Kullback-Leibler divergence for the flat [R]₀ model was even smaller than for the general model. Therefore, marginals of the Bayesian posteriors are more consistent with one another than the Gaussian distributions from nonlinear regression. This finding agrees with above analyses that the Bayesian posterior captures the variance among experiments better than nonlinear least squares. The one exception is with the CBS:CAII dataset, in which the Kullback-Leibler divergence matrix based on the Bayesian method is comparable to the one from nonlinear regression (S26 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. The natural logarithm of Kullback-Leibler divergence between Bayesian approach (General model) and nonlinear least-squares.

The natural logarithm of the KL-divergence between posterior marginal distributions (top) and between Gaussian distributions of nonlinear least squares errors (bottom) is shown. Each column and row corresponds to one of the 14 datasets of Mg(II):EDTA binding. The diagonal elements should be ln0 = −∞ but were set to 1 for visualization.

https://doi.org/10.1371/journal.pone.0203224.g009