Surveillance data

Recording every case of influenza is not practicable; infections are often asymptomatic [38] or symptomatic but not clinically attended [39], laboratory testing may not be performed for clinically attended cases or give false negative results, and reporting of lab-confirmed cases is not mandatory in most instances. Forecast comparisons are instead based on syndromic clinical surveillance data from the U.S. Outpatient Influenza-like Illness Surveillance Network (ILINet) [7, 40], a group of health care providers that voluntarily report statistics regarding ILI, where ILI is defined as a 100°F (37.8°C) fever with a cough and/or sore throat without a known cause other than influenza. CDC aggregates these reports and estimates the weekly percentage of patients seen that have ILI, %ILI, across all health care providers using a measure called weighted %ILI (wILI).

• Geographical resolution: CDC reports wILI for each of the 10 U.S. Department of Health & Human Services (HHS) regions, as well as for the nation as a whole; the wILI for each of these locations is a weighted average of the ILINet %ILI for state-level units based on population.
• Temporal resolution: wILI is available on a weekly basis; weeks begin on Sunday, end on Saturday, and are numbered according to the epidemiological week (epi week) convention in the United States.
• Timeliness: Initial wILI estimates for a given week are typically released on Friday of the following week; additional reports and revisions from participating health care providers are incorporated in updates throughout the season and after it ends.
• Specificity: Influenza is just one of many potential causes of ILI. Laboratory testing data [7] suggest that influenza is responsible for a significant portion of ILI cases during the flu season, especially for weeks when wILI is high, but only for a very small fraction of cases in the typical flu off-season. Much of the variance and “peakiness” in wILI can be associated with influenza epidemics, but wILI trajectories do not taper off to near-zero values as one might expect in a direct measurement of influenza prevalence.
• Influence of non-ILI cases: Since wILI depends on records of both ILI cases and total cases, patterns in non-ILI cases can impact wILI trajectories. We discuss one such pattern in the Holiday effects section.

CDC hosts the latest ILINet report and other types of surveillance data through Fluview Interactive, a collection of web modules [41]; we provide current and historical ILINet reports and some other data sources through our delphi-epidata API [42] and epivis visualizer [43].

Forecasting targets

The FluSight project focuses on in-season distributional forecasts and point predictions of key targets of interest to public health officials:

• Short-term wILI: the four wILI values following the last available observation (incorporating all data revisions through some week well after the season’s end)
• Season onset: the first week in the first run of at least three consecutive weeks with wILI values above a location- and season- specific baseline wILI level set by CDC [7], or “none” if no such runs exist; describes whether and when an influenza epidemic started in a given season
• Season peak percentage: the maximum of all wILI values for a given season
• Season peak week: the week or weeks in which wILI takes on its maximum value, or “none” if there was no onset in the 2015/2016 comparison

When making distributional forecasts, wILI values are discretized into CDC-specified bins and a probability assigned to each bin, forming a histogram over possible observations. The width of the bins was set at 0.5 %wILI for the 2015/2016 comparison and 0.1 %wILI for the 2016/2017 comparison; we use a width of 0.5 %wILI for analysis of the 2015/2016 comparison prospective forecasts, and a width of 0.1 %wILI for retrospective evaluation. CDC typically presents wILI values rounded to a resolution of 0.1 %wILI; some targets and evaluations are based on these rounded values.

Evaluation metrics

We focus on three metrics for evaluating performance of a forecast for a given target:

• Unibin log score: , where is the probability assigned to i, the bin containing the observed value. We use this score for ensemble weight selection and most internal evaluation as it has ties to maximum likelihood estimation, and is “proper score” [44]. A score for a (reported) distributional prediction is called “proper” if its expected value according to any (internal) distributional prediction is maximized when reporting , i.e., forecasters can maximize their expected scores by reporting their true beliefs. We refer to the “unibin log score” simply as the “log score” except for when comparing it with the multibin log score, which is defined next. The exponentiated mean log score is the (geometric) average probability assigned to events that were actually observed. The exponentiated difference in the mean log scores of method A and method B is an estimate of the (geometric) expected winnings of unit-sized bets of the form “this bin will hold the true value” when bets are placed optimally according to the forecasts of A, and (relative) prices are set optimally according to the forecasts of B.
• Multibin log score: , where the i’s considered are typically bins within 0.5 %wILI of observed values for a wILI target, or within 1 week for a timing target. The multibin log score was designed by FluSight hosts in consultation with participants, and the judgment “near observed value” was selected as a level of error that would not significantly impact policymakers’ decisions. The exponentiated mean multibin log score is the (geometric) average amount of mass a forecaster placed within this margin for error of observed target values.
• Absolute error: , where is the point prediction and y is an observed value. (In the case of onset, we consider point predictions for the value of onset conditioned on the fact that an onset actually occurs. We do not consider absolute error for onset in instances where no onset occurred. Some methods considered would sometimes fail to produce such conditional onset point predictions when they were confident that there was no onset, but these methods are not included in any of the figures containing absolute errors.)

The FluSight 2015/2016 forecast comparison evaluations were based solely on the multibin log score [45].

Terminology and notation

The “flu season” is typically defined as epi week 40 of one year through 20 of the next; we also include data from the rest of the year as part of the season for the purpose of fitting models. In all mathematical notation, we will number the first week of the season as 1 rather than using the corresponding epi week. Let

• denote the t-th CDC report of the current season, containing wILI values for weeks 1 through t, inclusive, which is normally published on Friday of week t + 1;
• T be the number of weeks in the current season (either 52 or 53); we omit all details regarding differing season lengths, presenting forecasting methods and labeling epi week plot axes as if all seasons were of length T;
• Y_1..T be the ground truth wILI for the current season: the wILI values used for forecast evaluation, specifically the epi week 28 report for the FluSight comparison, or later revisions as they are available for cross-validation analysis;
• be the ground truth wILI for past season s; and
• Z^t be a vector containing the forecasting targets of interest at the t-th wILI report of the current season: Y_t+1..t+4 and the seasonal onset, peak week, and peak percentage; for the FluSight comparison, forecasts for these targets were typically due on Monday of week t + 2, and allowed to use ILINet and any other data released before the deadline.

Our goal is to forecast Z^t given and previous reports. This can be broken down into multiple steps, such as:

1. “Backcast” updates to the data through time t, producing a distribution over Y_1..t based on the value of and previous reports.
2. Connect the backcast for Y_1..t with corresponding forecasts for Y_t+1..T, yielding a distribution for the entire trajectory Y_1..t.
3. Calculate the distribution for Z^t corresponding to this distribution over Y_1..t.

We first introduce the delta density method, which forecasts Y_t+1..T given Y_1..t (step 2). We then discuss a separate procedure for combining multiple forecasts into an adaptively weighted ensemble, forecasting Z^t given either Y_1..t or (steps 2–3 or 1–3). We also outline a method for estimating the distribution of Y_1..t given (step 1), and analyze its performance when used in conjunction with the delta density method.

Delta density method

Consider the task of estimating the density function using an instance-based approach. Kernel density estimation and kernel regression use smoothing kernels to produce flexible estimates of the density of a random variable (e.g., ) and the conditional expectation of one random variable given the value of another (e.g., ), respectively; we can combine these two methods to obtain estimates of the conditional density of one random variable given another. One possible approach would be to use the straightforward estimate where {1..S} is the set of fully observed historical training seasons, and I^1..t and O^t+1..T are smoothing kernels describing similarity between “input” trajectories and between “output” trajectories, respectively. However, while basic kernel smoothing methods can excel in low-dimensional settings, their performance scales very poorly with growing dimensionality. During most of the season, neither Y_1..t nor Y_t+1..T is low-dimensional, and the current season’s observations are extremely unlikely to closely match any past or . This, in turn, can lead to kernel density estimates for Y_t+1..T based almost entirely on the single season s with the closest when conditioning on Y_1..t, and excessively narrow density estimates for Y_t+1..T even without conditioning on Y_1..t. So, instead of applying kernel density estimation directly, we first break the task down into a sequence of low-dimensional sub-tasks. We avoid the high-dimensional output problem by chaining together estimates of for each u from t + 1 to T, where ΔY_u = Y_u − Y_u−1; estimating these single-dimensional densities requires relatively little data. However, this reformulation exacerbates the high-dimensional input problem since we are conditioning on Y_1..u−1, which can be considerably longer than Y_1..t. We address the high-dimensional input problem by approximating with , where R^u is some low-dimensional vector of features derived from Y_1..u−1. Smoothing kernel methods are used to approximate the conditional density functions using data from past seasons.

We use two sets of choices for the approximate conditional density function and summary features to form two versions of the method.

• Markovian delta density: approximates the conditional density of ΔY_u given Y_1..u−1 with its conditional density given just the previous (real or simulated) observation, Y_u: where I^u and O^u are Gaussian smoothing kernels. The first equality corresponds to the chain rule of probability on the actual (not estimated) densities; the second incorporates the Markov assumption (i.e., selects R^u = [Y_u−1]); and the third gives our choice of estimators for the conditional densities for each u. The bandwidth of each I^u and O^u is chosen separately using bandwidth selection procedures for regular kernel density estimation of Y_u−1 and ΔY_u, respectively. (Specifically, we use the bw.SJ function from the R [46] built-in stats package, with bw.nrd0 as a fallback in the case of errors. These functions do not accept weights for the inputs; it may be possible to improve forecast performance by incorporating these weights or by using other approaches to select the bandwidths.) Note that density estimates for ΔY_u are based on data from past seasons on week u only, allowing the method to incorporate seasonality and holiday effects (for holidays that consistently occur at the same time of year).
  Forecasts are based on Monte Carlo simulations of Y_t+1..T ∣ Y_1..t. The following procedure draws a single possibility for Y_t+1..T:
  
  1. Draw from its conditional density estimate given Y_1..t.
  2. Draw from its conditional density estimate given Y_1..t and .
  3. Continue in the same manner for all remaining weeks.
  This process is illustrated in Fig 1. Repeating this procedure many times yields a sample from the model for Y_t+1..T ∣ Y_1..t; stopping at 2000 draws seems sufficient for use in our ensemble forecasts, while at least 7000 are needed to smooth out noise when displaying distributional target forecasts for the delta density method in isolation. Any negative simulated wILI values in these trajectories are clipped off and replaced with zeroes.
• Extended delta density: approximates the conditional density of ΔY_u given Y_{1..u − 1} with its conditional density given four features:
  • the previous wILI value, Y_u−1;
  • the sum of the previous k^u wILI values, roughly corresponding to the sum of wILI values for the current season;
  • an exponentially weighted sum of the previous k^u wILI; values, where the weight assigned to time v is 0.5^t′−v; and
  • the previous change in wILI value, ΔY_u−1.
  
  The approximate conditional density assigns each of these features a weight (0.5, 0.25, 0.25, and 0.5, respectively) in order to reduce overfitting and emphasize some relative to the others, and incorporates data from other weeks close to u (specifically, withinl^u weeks; the choice of l^u is discussed in a later section) with a truncated Laplacian kernel. We selected these weights and other settings, such as kernel bandwidth selection rules, somewhat arbitrarily based on intuition and experimentation on out-of-sample data; a cross-validation subroutine could be used to make the selection as well, but would multiply the amount of computation required. In case the resulting product of Gaussian and Laplacian kernels is too narrow, we mix its results with a wide boxcar kernel which evenly weights all data from time u − l^u to u + l^u:
  Using data from v ≠ u incorporates additional reasonable outcomes for ΔY_u by incorporating past wILI patterns with different timing, but risks including some very unreasonable possibilities produced by repeatedly drawing from the same v rather than following seasonal trends with increasing v’s. For example, when a portion of a past season that is more similar to itself with a slight time shift than to any other past season, it may be selected for multiple consecutive u’s and produce an unreasonable trajectory. This could potentially occur when drawing data from the relatively flat regions of wILI trajectories of many seasons, or when incorporating observations around an unusually early, late, high, or low peak. To prevent this possibility, we combine the natural estimate for Y_u arising from the density estimate for ΔY_u with a random draw from the unconditional density estimate for Y_u (using a Gaussian kernel and only data from week u):

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. The delta density method conditions on real and simulated observations up to week u − 1 when building a probability distribution over the observation at week u.

This figure demonstrates the process for drawing a single trajectory from the Markovian delta density estimate, ignoring the data revision process. The latest ILINet report, , which incorporates observations through week 48, is shown in black. Kernel smoothing estimates for future values at times u from t + 1 to T are shown in blue, as are simulated observations drawn from these estimates. Past seasons’ trajectories are shown in red, with alpha values proportional to the weight they are assigned by the kernel I^u.

https://doi.org/10.1371/journal.pcbi.1006134.g001

Residual density method

This same approach can be applied to estimate the distribution of residuals of a wILI point predictor. Suppose that we have observed

• , the partial wILI trajectory up to time t₁, and
• , point estimates of the trajectory up to some later time t₂;

our goal is to estimate the conditional distribution of

• , the unknown residuals,

given and , using data from past seasons. This can be achieved by chaining together draws from conditional density estimates of (Y − X)_u ∣ R^u for u from t₁ + 1 to t₂, where R^u is a function of Y_1..u−1 and . The delta density method can be seen as a special case where t₁ = t; t₂ = T; X_1..t = Y_1..t, past values of Y which are treated as known and are simply duplicated in the simulated trajectories; and X_t+1..T = Y_t..T−1, values of Y which begin as unknown but are filled in as needed by previous simulation steps, giving (Y − X)_t+1..T = ΔY_t+1..T. We use the residual density method to backcast Y_1..t from and as the basis for another forecaster in the ensemble.

Fig 2 shows sample forecasts over wILI trajectories generated by each of these approaches and compares them to some alternatives described in S1 Appendix.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Delta and residual density methods generate wider distributions over trajectories than methods that treat entire seasons as units.

These plots show sample forecasts of wILI trajectories generated from models that treat seasons as units (BR, Empirical Bayes) and from models incorporating delta and residual density methods. Yellow, the latest wILI report available for these forecasts; magenta, the ground truth wILI available at the beginning of the following season; black, a sample of 100 trajectories drawn from each model; cyan, the closest trajectory to the ground truth wILI from each sample of 100.

https://doi.org/10.1371/journal.pcbi.1006134.g002

Combining multiple methods: Stacking approach to model averaging

Forecasting systems that select effective combinations of predictions from multiple models can improve on the performance of the individual components, as demonstrated by their successful application in many domains. For each probability distribution and point prediction in a forecast, we treat the choice of an effective combination as a statistical estimation problem, and base each decision on the models’ behavior in leave-one-out cross-validation forecasts. Additional cross-validation analysis indicates that this approach achieves performance comparable to or better than the best individual component.

Considering details of the data generation process

Two important features of ILINet data to consider in models and forecast evaluation are 1. timeliness and accuracy of initial wILI values for each week and subsequent updates to these values, and 2. changes in behavior on and around major holidays. We examine these details of the data generation process, describe how they are addressed in the delta density model, and demonstrate the importance of considering the update procedure when performing retrospective evaluation and prospective forecasting.

Holiday effects.

On average, wILI tends to be higher on holidays and during the winter holiday season than would be expected based on interpolating observations from nearby weeks [21, 50], as seen in Fig 3. Sharp rises and drops in wILI are common from early or mid-December to early January (roughly coinciding with a four week period beginning with epi week 50), with either the season’s peak or a lower, secondary peak commonly occurring on epi week 52. This pattern appears to arise from two factors:

• spikes downward in the number of non-ILI visits during the holiday season (corresponding to increases in wILI), perhaps caused by patients choosing not to visit the doctor for less serious issues on holidays, and
• decreases in the average number of ILI visits at the end of the holidays, perhaps due to decreased transmission of ILI during holidays, which make the preceding increases in wILI appear even sharper.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. On average, wILI is higher on holidays than expected based on neighboring weeks.

Weekly trends in wILI values, as expressed by the contribution of a each week to a sum of wILI values from seasons 2003/2004 to 2015/2016, excluding 2008/2009 and 2009/2010 (which include portions of the 2009 influenza pandemic), show spikes and bumps upward on and around major holidays. (U.S. federal holidays are indicated with event lines.) The number of non-ILI visits to ILINet health care providers spikes downwards on holidays (disproportionately with any drops in the number of ILI visits), contributing to higher wILI. The number of ILI visits generally declines in the second half of the winter holiday season, causing winter holiday peaks to appear even higher relative to nearby weeks. In addition to holiday effects, we see that average ILINet participation jumps upward on epi week 40, and gradually tapers off later in the season and in the off-season.

https://doi.org/10.1371/journal.pcbi.1006134.g003

Similarly, there are spikes or minor blips downward in the average number of non-ILI visits (which can result in small increases in wILI) associated with Thanksgiving Day; Labor Day; Independence Day; Memorial Day; Birthday of Martin Luther King, Jr.; Washington’s Birthday; Columbus Day; and perhaps other holidays. The spike upward in wILI at Thanksgiving can push wILI unexpectedly over the onset threshold, and holiday effects may help explain the surprising frequency at which peaks occur on epi week 7 but not neighboring weeks. Additional age-specific patterns may be obscured by this analysis of aggregate ILI and non-ILI visit counts.

2015/2016 FluSight comparison

During the 2015/2016 FluSight comparison, we submitted weekly, prospective forecasts from three forecasting systems:

• Delphi-Stat: an adaptively weighted ensemble of instance-based statistical forecasting methods, and the topic of this paper;
• Delphi-Archefilter: forms an empirical (rather than mechanistic) process model describing wILI trajectories, and incorporates both wILI and multiple forms of digital surveillance data using statistical filtering techniques [50]; and
• Delphi-Epicast: wisdom-of-crowds approach based on combining predictions submitted by several human participants [72].

Our past and ongoing forecasts, as well as Python [73] and R [46] code for components of the systems used to generate them, are publicly available online [74–76]. Changes made to Delphi-Stat throughout the 2015/2016 season are described in S5 Appendix. These three forecasting systems were ranked as the top three in the 2015/2016 comparison in terms of overall multibin score, with Delphi-Stat at the top. Fig 4 shows the performance of the three Delphi forecasting systems, broken down by evaluation metric and forecasting target. S1 and S2 Figs. show the multibin scores broken down by location and by forecasting week. Delphi-Stat had consistently strong aggregate multibin scores across different targets, locations, and forecasting weeks, and the best overall multibin log score of all FluSight 2015/2016 submissions. Delphi-Stat’s unibin log score evaluations relative to the other two Delphi systems seem similar to or better than the corresponding multibin log score evaluations, as Delphi-Stat has the best unibin score of the three for each target rather than just overall; this observation seems natural since Delphi-Stat was developed to optimize unibin log score, and may suggest that optimizing for multibin log score rather than unibin log score when selecting ensemble weights or as a post-processing step could produce multibin log score improvements. However, the system’s point predictions, while optimized for the mean absolute error metric, were less accurate than (but still competitive with) the other two when averaged across all predictions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The three Delphi systems had similar overall scores; Delphi-Stat gave the best distributional forecasts overall, while Delphi-Epicast gave the best point predictions overall.

These bar plots contain evaluations for the 2015/2016 season, averaged across 11 locations and 29 forecast weeks, for each target and evaluation metrics. Shorter bars indicate better performance. Each entry for a specific target is an average of 319 evaluations, giving a total of 2233 evaluations overall for each system. This figure’s data is shown in tabular form in S2 Appendix.

https://doi.org/10.1371/journal.pcbi.1006134.g004

Cross-validation analysis

Prospective forecast evaluation ensures that performance estimates are truly out-of-sample, not inflated by design decisions or model fits that are influenced by the evaluation data; however, such evaluation data is not readily generated, as it is expensive in terms of physical time: new wILI observations arrive once per week, and performance can vary significantly from season to season and from week to week. The evaluations from the 2015/2016 comparison may be noisy due to these season-to-season fluctuations. To address this issue, we use pseudo-out-of-sample retrospective analysis to provide more stable estimates of performance. Specifically, we use leave-one-season-out cross-validation: for each evaluation season s, we form and evaluate retrospective forecasts for s at every evaluation week using all training seasons except for s as inputs to the forecasting methods as if they were past seasons. (We exclude seasons prior to 2010/2011 from the evaluation set because records of HHS region ILINet data revisions are only available beginning in late 2009. We exclude seasons prior to 2003/2004 from the training set because year-round ILINet observations, which are required by some of the ensemble components, started in 2003. The 2009/2010 season—containing the peak of the 2009 pandemic according to our adjusted definition of “season”—is also removed from the training set. Finally, we do not include the season currently underway (S + 1) in evaluation or training as it has not been completely observed.) Using cross-validation prevents most direct model fitting to evaluation data, and basing design decisions on motivations other than the effects on cross-validation evaluation helps limit fitting through iterative design.

Fig 5 shows the distribution of log scores for several forecasting methods, described earlier in the text and in S1 Appendix, and the three ensemble approaches specified earlier in the text. Except for the uniform distribution and ensembles, all forecasting methods miss some possibilities completely, reporting unreasonable probabilities less than exp(−10) ≈ 0.0000454 for events that actually occurred. In these situations, the log score has been increased to the cap of −10 (as CDC does for multibin log scores). Delta and residual density forecasting methods (Delta density, Markovian; Delta density, extended; and BR, residual density) are less likely to commit these errors than other non-ensemble, non-uniform approaches, and have higher average log scores. Ensemble approaches combine forecasts of multiple components, missing fewer possibilities, and ensuring that a reasonable log score is obtained by incorporating the uniform distribution as a component. For the full Delphi-Stat ensemble, the main advantage of the ensemble over its best component appears to be successfully filling in possibilities missed by the best component with other models to avoid -10 and other low log scores appears, while for ensembles of subsets of the forecasting methods, there are other benefits; S3 Appendix shows the impact of these missed possibilities and the log score cap.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Delta and residual density methods cover more observed events and attain higher average log scores than alternatives operating on seasons as a unit; ensemble approaches can eliminate missed possibilities while retaining high confidence when justified.

This figure contains histograms of cross-validation log scores for a variety of forecasting methods, averaged across seasons 2010/2011 to 2015/2016, all locations, forecast weeks 40 to 20, and all forecasting targets. A solid black vertical line indicates the mean of the scores in each histogram, which we use as the primary figure of merit when comparing forecasting methods; a rough error bar for each of these mean scores is shown as a colored horizontal bar in the last panel, and as a black horizontal line at the bottom of the corresponding histogram if the error bar is wider than the thickness of the black vertical line.

https://doi.org/10.1371/journal.pcbi.1006134.g005

Fig 5 also includes estimates of the mean log score for each method and rough error bars for these estimates. We expect there to be strong statistical dependence across evaluations for the same season and location, and weaker dependencies between different seasons and locations; thus, the most common approaches to calculating standard errors, confidence intervals, and hypothesis test results will be inappropriate. Properly accounting for such dependencies and calibrating intervals and tests is an important but difficult task and is left for future investigation. We use “rough standard error bars” on estimates of mean evaluations: first, the relevant data (e.g., all cross-validation evaluations for a particular method and evaluation metric) is summarized into one value for each season-location pair by taking the mean of all evaluations for that season-location pair; we then calculate the mean and standard error of the mean of these season-location values using standard calculations as if these values were independent. Under some additional assumptions which posit the existence of a single underlying true mean log score for each method, these individual error bars—or rough error bars for the mean difference in log scores between pairs of ethods—suggest that the observed data is unlikely to have been recorded if the true mean log score of the extended delta density method were greater than that of the adaptively weighted ensemble, or if the true mean log score of the “Empirical Bayes A” method were greater than the extended delta density method. The mean and rough standard error estimates in Fig 5 also appear in tabular form in S4 Appendix.

Methods that model wILI trajectories and “pin” past wILI to its observed values have a large number of log scores near 0 because they are often able to confidently “forecast” many onsets and peaks that have already occurred; ensemble methods also have a large number of log scores near 0. Note that these scores are closer to 0 for ensembles that optimize weighting of different methods than for the ensemble with uniform weights. For this particular set of forecasting methods, targets, and evaluation seasons:

• the uniformly weighted ensemble has lower average log score than the best individual component (extended delta density),
• using the stacking approach to assign weights to ensemble components improves ensemble performance significantly and gives higher average log score than the best individual component,
• the adaptive weighting scheme does not provide a major benefit over a fixed-weight scheme using a single set of weights for each evaluation metric.

When given subsets of these forecasting methods as input, with regard to average performance:

• the uniformly weighted ensemble often outperforms the best individual, but is sometimes slightly (≈ 0.1 log score) worse;
• the stacking approach improves upon the performance of the uniformly weighted ensemble; and
• the adaptive weighting scheme’s performance is equal to or better than that of the fixed-weight scheme, sometimes improving on the log score by ≈ 0.1. The adaptive weighting scheme’s relative performance appears to improve with more input seasons, fewer ensemble components, and increased variety in underlying methodologies and component performance. These trends suggest that using wider RelevanceWeight kernels, regularizing the component weights, or considering additional data from 2003/2004 to 2009/2010, for which ground truth wILI but not weekly ILINet reports are available, may improve the performance of the adaptive weighting scheme. In addition to these avenues for possible improvement in ensemble weights for the components presented in Fig 5, the adaptive weighting scheme provides a natural way of incorporating forecasting methods that generate predictions for only a subset of all targets, forecast weeks, or forecast types (distributional forecast or point prediction). For example, in the 2015/2016 season, we incorporated a generalized additive model that provided point predictions (and later, distributional forecasts) for peak week and peak height given at least three weeks of observations from the current season.

Fig 6 shows a subset of the cross-validation data used to form the ensemble and evaluate the effectiveness of the ensemble method, for two sets of components: one using all the components of Delphi-Stat, and the other incorporating three of the lower-performance components and a uniform distribution for distributional forecasts. The Delphi-Stat ensemble near-uniformly dominates the best component, extended delta density, in terms of log score, and has comparable mean absolute error overall. The ensemble approach produces greater gains for the smaller subset of methods, surpassing not only its best components, but all forecasting methods in the wider Delphi-Stat ensemble except for the delta density approaches.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. The ensemble method matches or beats the best component overall, consistently improves log score across all times, and, for some sets of components, can provide significant improvements in both log score and mean absolute error.

These plots display cross-validation performance for two ensembles and some components broken down by evaluation metric, target type, and forecast week; each point is an average of cross-validation evaluations for all 11 locations, seasons 2010/2011 to 2015/2016, and all targets of the given target type; data from the appropriate ILINet reports is used as input for the left-out seasons, while finalized wILI is used for the training seasons. Top half: log score evaluations (higher is better); bottom half: mean absolute error, normalized by the standard deviation of each target (lower is better). Left side: full Delphi-Stat ensemble, which includes additional methods not listed in thelegend; right side: ensemble of the three methods listed in the legend, plus a uniform distribution component for distributional forecasts. Many components of the full ensemble are not displayed. The “Targets, uniform” method is excluded from any mean absolute error plots as it was not incorporated into the point prediction ensembles.

https://doi.org/10.1371/journal.pcbi.1006134.g006

Fig 7 shows cross-validation performance estimates for the extended delta density method based on three evaluation schemes:

• Ground truth, no nowcast: the ground truth wILI for the left-out season up to the forecast week is provided as input, resulting in an optimistic performance estimate;
• Real-time data, no nowcast: the appropriate wILI report is used for data from the left-out season, but no adjustment is made for possible updates; this performance estimate is valid, but we can improve upon the underlying method;
• Backcast, no nowcast: the appropriate wILI report is used for data from the left-out season, but we use a residual density method to “backcast” updates to this report; this performance estimate is valid, and the backcasting procedure significantly improves the log score;
• Backcast, Gaussian nowcast: same as “Backcast, no nowcast” but with another week of simulated data added to the forecast, based on a Gaussian-distributed nowcast; and
• Backcast, Student t nowcast: same as “Backcast, Gaussian nowcast” but using a Student t-distributed nowcast in place of the Gaussian nowcast.
• Backcast, ensemble nowcast: same as the previous two but using the ensemble nowcast (which combines “no nowcast” with “Student t nowcast”).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Using finalized data for evaluation leads to optimistic estimates of performance, particularly for seasonal targets, “backcasting” improves predictions for seasonal targets, and nowcasting can improve predictions for short-term targets.

Mean log score of the extended delta density method, averaged across seasons 2010/2011 to 2015/2016, all locations, all targets, and forecast weeks 40 to 20, both broken down by target and averaged across all targets (“Overall”). Rough standard error bars for the mean score for each target (or overall) appear on the right, in addition to the error bars at each epi week.

https://doi.org/10.1371/journal.pcbi.1006134.g007

For every combination of target and forecast week, using ground truth as input rather than the appropriate version of these wILI observations produces either comparable or inflated performance estimates.

Using the “backcasting” method to model the difference between the ground truth and the available report helps close the gap between the update-ignorant method. The magnitude of the performance differences depends on the target and forecast week. Differences in mean scores for the short-term targets are small and may be reasonably explained by random chance alone; the largest potential difference appears to be an improvement in the “1 wk ahead” target by using backcasting. More significant differences appear in each of the seasonal targets following typical times for the corresponding onset or peak events; most of the improvement can be attributed to preventing the method from assigning inappropriately high probabilities (often 1) to events that look like they must or almost certainly will occur based on available wILI observations for past weeks, but which are ultimately not observed due to revisions of these observations. The magnitude of the mean log score improvement depends in part on the resolution of the log score bins; for example, wider bins for “Season peak percentage” may reduce the improvement in mean log score (but would also shrink the scale of all mean log scores). Similarly, the differences in scores may be reduced but not eliminated by use of multibin scores for evaluation or ensembles incorporating uniform components for forecasting.

Using the heavy-tailed Student t nowcasts or nowcast ensemble appears to improve on short-term forecasts without damaging performance on seasonal targets. The performance of the nowcast ensemble is further explored in S5, S6, S7, S8, S9, S10, S11 and S12 Figs. The Gaussian nowcast has a similar effect as the other nowcasters except on the “1 wk ahead” target that it directly predicts: its distribution is too thin-tailed, resulting in lower mean log scores than using the forecaster by itself on this target.

Conclusion

The delta density forecasting method, stacking-based adaptively weighted ensemble, distributional “backcasts” of wILI updates, and nowcasts from ILI-Nearby provide significant improvements upon other individual forecasting approaches that we considered. Promising avenues for further improvements include refining the methodology to rely less on arbitrary and heuristic feature, kernel, bandwidth, and parameter selections, regularization of ensemble weights, incorporating conditional density estimators from statistical literature, and using additional data sources and finer-resolution data models.