Data and population

This retrospective, county-level observational study leveraged data from a variety of sources. We first obtained daily changes in population mobility from Google Community Mobility Reports for February 15, 2020 through April 21, 2020. These reports contain aggregated, anonymized data from Android cellphone users who have turned on location tracking. Daily mobility statistics are presented for all U.S. counties, unless the data do not meet Google’s quality and privacy thresholds [5]. We did not exclude any additional counties. Our analytic sample included 128,856 county-day observations, accounting for 1,184 counties and 125.3 million individuals.

We then identified dates of implementation for state SIP orders from the COVID-19 US State Policy (CUSP) database at Boston University [2]. Changes in state policies are identified and recorded from state government websites, news reports, and other publicly-available sources.

Lastly, we obtained a variety of county-level characteristics including daily data on COVID-19 deaths and cases from the Johns Hopkins Coronavirus Resource Center [6], results of the 2016 presidential election from the MIT Election Lab [7], and demographics from the Health Resources and Services Administration’s Area Health Resource Files [8].

Study variables

They key outcome in our analysis was changes in population-level mobility. Our data include the number of visits and time spent at various locations such as grocery stores or workplaces. Mobility is reported as the percent change relative to a median baseline value for the corresponding day of the week during Jan 3–Feb 6, 2020 [5]. We then computed an index measure by taking the mean of percent changes for all non-residential categories which included: retail and recreation, groceries and pharmacies, parks, transit stations, and workplaces.

Our key predictor of interest was the number of days, positive or negative, relative to SIP implementation. While SIP orders sometimes took effect at midnight (13), their effects on mobility may be delayed. In addition, when assessing other policies, there may be expectation of enactment. SIP orders, however, appear to have been announced with little to no delay between announcement and enactment [9]. Thus the announcement of these orders were associated with spikes in mobility as residents rushed to stock up on essential goods [10]. To account for this, and to capture actual response to the SIP order (rather than noise) we allowed for a one-day washout period. We thus dropped observations on the day of implementation.

We included additional variables in our analysis which may plausibly moderate the association between SIP orders and population mobility. First, the stage of the pandemic affects residents’ exposure to COVID-19 and may change the effectiveness of policy efforts. Indeed, the WHO recommends different actions depending on pandemic phase [11]. We operationalized the stage of pandemic as measured by early vs. late enactment of a SIP (pre or post April 1^st) and by the state-level cumulative number of COVID-19 deaths and cases on each SIP’s effective date.

Second, population-level factors such as income have been found to be associated with a higher likelihood and ability to engage in social-distancing measures [12, 13]. In addition, structural racism [14] and disproportionate representation among “essential workers” [15, 16], may predispose Black individuals or individuals of Hispanic descent to additional risk. We extracted the following county-level measures from the AHRF: population estimate (F11984-18); per-capita income (F09781-17); poverty rate (F13321-17); Black population (F13910-17); total area in square miles (F13874-10). We created a population-density measure by dividing population by total area in square miles, and a share of Black residents by dividing Black population by total population [8]

Third, recent research demonstrates that political beliefs affect individual-level responsiveness and perception of the pandemic and the likelihood of adoption of state policies [17–19]. We used county-level vote share for Donald J. Trump in the 2016 presidential election as a proxy measure for political partisanship. We believe this is an appropriate proxy of partisan political affiliation, as it captures not only political alignment but also potential agreement with Donald J. Trump’s negative expressed views on lockdowns and SIP orders (for example, the President allegedly considered “reopening” as early as March) [20].

And lastly, most states saw dramatic declines in mobility before any SIP orders were enacted. It is possible that this so-called “private response” to the pandemic is related to the eventual enactment of SIP orders and/or their effectiveness. To operationalize this, we measured average state-level mobility change from February 15 to each state’s date of SIP enactment.

All continuous variables were standardized and effect sizes reflect a one standard deviation change in the independent variable.

Statistical analysis

We estimated both overall and state-specific local linear regression discontinuity (RD) models following the methods outlined by Calonico, Cattaneo, & Farrell (2020) [21]. We used the number of days relative to SIP orders as our running variable; e.g. a value of -1 indicates the day immediately preceding SIP implementation, and a value of 1 indicates the first day post-implementation. All estimates used robust bias-corrected standard errors clustered by county, with a triangle kernel and data-driven optimal bandwidth selection which is considered a best practice to reduce analyst subjectivity [21, 22]. This approach mimics randomization and allows us to observe the effects of SIP orders on mobility, since we would not expect counties to change day-to-day in other ways which would impact mobility. In sensitivity analyses, we also weighted our models by county population.

To investigate whether SIP order effects varied with state characteristics, we aggregated our county-level covariates to the state level by summing across counties within the state (cumulative deaths and cases) or taking either population-weighted averages (all other variables). To investigate whether our state-level RD estimates are statistically different from one another, we conduct a series of z-tests for each state-pair in our sample [23]. We then estimated a series of bivariable linear regressions with several state characteristics as our independent variables and our RD estimates for the effects of state SIP orders as the dependent variable, with robust standard errors.

First, to assess whether the population’s prior exposure to the epidemic moderated SIP response, we compared state-level effect estimates by whether the order was early vs. late and by log-deaths and log-cases on each SIP’s effective date. Second, to investigate whether SIP effect size was related to population risk factors for COVID transmission and adverse outcomes, we assessed whether SIP effect size varied with demographic and socioeconomic variables. Third, we assessed whether political alignment moderated the effect of SIP orders. Fourth, we assessed whether the pre-SIP decline in mobility was associated with the estimated effect of the SIP order.

Lastly, we also assessed whether SIP impact differed by whether the state had implemented a range of other measures at the time of the SIP order. SIP orders were implemented alongside complementary policies in some states, thus we investigated the possibility that other policies beyond SIP orders (enacted at the same time) may have had effects on mobility. To do so, we ran bivariable state-level models regressing the estimates RD coefficients on indicators for whether other policies (school closures, daycare closures, nursing home restrictions, business closures, restaurant closures, and gym closures) were enacted in the state. These indicators were obtained from the COVID-19 US State Policy (CUSP) database

Lastly, to understand how different categories of mobility (e.g., workplace, transit use etc.) responded to SIP orders, we re-ran our primary state-level RD specification with each mobility category as the outcome. We investigated variation in these coefficients across states, and investigated their association with the epidemiological and socioeconomic factors discussed earlier.

Robustness checks

We conducted several robustness checks in our analysis. First, enactment of SIP orders was likely endogenous with respect to deaths and/or cumulative disease burden. Unlike standard linear or logistic regression approaches, effect estimates from regression discontinuity designs are not biased by covariates unless there are discontinuities in average covariate values at the threshold [22]. We investigate this by running our primary national RD specification with a 7-day lagged measure of deaths per million population as the outcome.

Second, while data-driven bandwidths tend to be superior to alternative bandwidth selection procedures [24], it is possible for variation in state-level results to arise due to variation in bandwidths. To account for this, we re-ran our RD model for each state with a fixed bandwidth taken from our primary model. We then examined whether the estimated coefficients are correlated with the estimated from our primary state-level RD specifications.

Third, to compare our estimates with existing work, we also implemented an event-study specification as an alternative to our primary RD model, following the approach in Allcott et al. which includes date and county fixed effects [25].

We considered a P-value of less than .05 as significant for RD models, and 0.10 as significant for the bivariate linear regressions due to small sample size (N = 39) [26]. Analyses were performed using Stata MP version 13.1 and the rdrobust package.

Ethics statement

Our study involved secondary analysis of non-identifiable, aggregate data and therefore is not human-subjects research. No IRB approval was sought.

Robustness checks

Below, we present the results of our robustness checks as discussed in the statistical analysis section.

Disease burden

To test whether disease burden is a confounder in our RD models, we examined whether it has a discontinuity with the running variable. Our results (0.48, 95% CI: -0.04 to 0.99) indicate that while it is plausible that SIP enactment is endogenous to disease burden, it is nonetheless not a confounder for our analysis.

Variable bandwidth

While estimates from state-level models with a fixed bandwidth differed from estimates that allowed the bandwidth to vary by state, the estimated coefficients were strongly correlated (r = 0.77) (S2 Fig).

Event study design

Replicating prior work that uses an event study design [25] we find that the estimate for one day after SIP implementation (-7.9, 95% CI: -8.5 to -7.5) is similar to our primary RD result (S3 Fig). We note that whereas the event study compares means on either side of the threshold, the RD approach leverages data points away from the threshold in order to obtain more precise estimates of state-specific effects.

Limitations

There are limitations to our study. First, we focused on the short-run, immediate impact of SIPs on mobility. While SIPs may have had longer-run impacts too, we could not separate these effects from other secular trends in our RD study design. Second, it is unclear whether the Android users who contribute data to Google Community Mobility Reports are representative of individuals living in that county. Unfortunately, this information is unavailable. Indeed, there are observed demographic differences between Android and iPhone users across age groups and gender [29]. To the extent they are different, our results may not generalize to other individuals. Third, other policies (such as transportation or business closures) were implemented either in tandem with or prior to SIP order enactment. These policies likely had their own effects, and may have affected compliance with SIP orders, which may explain some of the variation between states. Fourth, while some other work has relied on county-level SIP orders as well, our analysis has not accounted for these changes. Any pre-existing county-level SIP orders may attenuate our effect estimates. Thus, our results may be a conservative estimate. Lastly, small sample sizes in our bivariable regressions may limit our ability to detect associations with state-level effect estimates. Despite this limitation, we are able to identify associations between our effect estimates and certain important measures such as political affiliation and some demographics. Moreover, the heterogeneity that we find is consistent with prior evidence on heterogeneity in mobility and responsiveness to SIP orders.”