Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

The effects of scientific messages and narratives about vaccination

  • Ozan Kuru ,

    Roles Conceptualization, Formal analysis, Writing – original draft, Writing – review & editing

    okuru@nus.edu.sg

    Affiliation Department of Communications and New Media, National University of Singapore, Singapore, Singapore

  • Dominik Stecula,

    Roles Conceptualization, Data curation, Formal analysis, Writing – original draft, Writing – review & editing

    Affiliation Department of Political Science, Colorado State University, Fort Collins, Colorado, United States of America

  • Hang Lu,

    Roles Conceptualization, Data curation, Writing – review & editing

    Affiliation Department of Communication and Media, University of Michigan, Ann Arbor, Michigan, United States of America

  • Yotam Ophir,

    Roles Conceptualization, Formal analysis, Writing – review & editing

    Affiliation Department of Communication, University at Buffalo, State University of New York, Buffalo, New York, United States of America

  • Man-pui Sally Chan,

    Roles Conceptualization, Formal analysis, Writing – review & editing

    Affiliation Department of Psychology, University of Illinois, Champaign, Illinois, United States of America

  • Ken Winneg,

    Roles Conceptualization, Data curation, Methodology, Writing – review & editing

    Affiliation Annenberg Public Policy Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States of America

  • Kathleen Hall Jamieson,

    Roles Conceptualization, Formal analysis, Funding acquisition, Resources, Supervision, Writing – original draft, Writing – review & editing

    Affiliation Annenberg Public Policy Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States of America

  • Dolores Albarracín

    Roles Conceptualization, Funding acquisition, Supervision, Writing – original draft, Writing – review & editing

    Affiliations Department of Psychology, University of Illinois, Champaign, Illinois, United States of America, Annenberg Public Policy Center, University of Pennsylvania, Philadelphia, Pennsylvania, United States of America

Abstract

A fundamental challenge complicates news decisions about covering vaccine side effects: although serious vaccine side effects are rare, less severe ones do occur occasionally. The study was designed to test whether a side effect message could induce vaccine hesitancy and whether that could be countered by pro-vaccine messages about vaccine safety. A large (N = 2,345), nationally representative experiment was conducted by randomly exposing participants to one of six videos about the measles, mumps, and rubella (MMR) vaccine edited from news programs produced during the 2019 measles outbreak in the United States. The design was a 2x3 factorial crossing the presence or absence of a hesitancy-inducing narrative message with a pro-vaccine science-supporting message (i.e., no message, science-supporting expert message, or pro-vaccine narrative message), leading to a total of six conditions. A general linear model was used to assess the effects of these videos on respondents’ (1) vaccine risk perceptions, (2) policy views on vaccination, (3) willingness to encourage others to vaccinate their children, and (4) intention to send a pro-vaccine letter to their state representative. Findings indicated that the science-supporting expert message about vaccine safety led to higher pro-vaccine evaluations relative to other conditions [e.g., b = -0.17, p < .001, a reduction in vaccine risk perceptions of 0.17 as compared to the control]. There was also suggestive evidence that the hesitancy-inducing narrative may limit the effectiveness of a science-supporting expert message, although this finding was not consistent across different outcomes. When shown alone the hesitancy-inducing narrative did not shift views and intentions, but more research is needed to ascertain whether exposure to such messages can undercut the pro-vaccine influence of science-supporting (expert) ones. All in all, however, it is clear that science-supporting messages are effective and therefore worthwhile in combating vaccine misinformation.

The effects of scientific messages and narratives about vaccination

In 2018–19, the U.S. experienced the largest measles outbreak in over a quarter-century, a disease for which, like COVID-19, an effective and safe vaccine exists [1]. The increasing number of measles cases, largely driven by inadequate vaccination rates in several mostly-religious communities [2], prompted news attention attempting to explain the re-emergence of a disease declared eliminated in the U.S. nearly two decades ago [3, 4]. As cases multiplied, the volume of news coverage, as well as Google searches about measles and the measles, mumps, and rubella (MMR) vaccine, spiked (Supporting Information A in S1 File). Responding to these circumstances, the news media produced stories about the outbreaks, the safety and efficacy of the MMR vaccine, and the reluctance of some in affected communities to vaccinate their children.

Although the media have the potential to educate the public about vaccines, some news practices and values may produce unintended negative consequences. Among these is the journalistic practice that reduces abstract topics, such as vaccination refusal or hesitancy, to personal stories. News accounts about a parent who fears or reports a child’s experience with a side effect cannot be simply debunked as inaccurate or treated as misinformation in and of itself [5]. The U.S. Centers for Disease Control reports that the possible risks of the MMR vaccine include (a) soreness, redness, or rash, (b) fever or swelling of the glands in the cheeks or neck, or (c) in more serious cases, seizures, pneumonia, swelling of brain, and unusual bleeding [6]. Unfortunately, because of the availability heuristic [7], news stories about redness or a rash may produce vaccine hesitancy [8] because the audience overgeneralizes the vivid and sometimes dramatic narrative [9] or misjudges the severity of the vaccine side effects.

In this paper, we treated parental reports of potentially real side effects as Hesitancy-Inducing Narratives because, even when accurate, their portrayal in media can lead to overgeneralization and fuel vaccine hesitancy by leading the public to draw inaccurate inferences about the prevalence and severity of side effects. In short, individual cases of vaccine side effects, even if true, may elicit false inferences, and the media’s reliance on dramatic and vivid cases may lead to overestimation of risks that are relatively rare. More importantly, individual cases are not the results of randomized control trials (RCTs) conducted during vaccine development and approval processes, and hence do not constitute scientifically valid information about the risk of a vaccine. Hence, they should not undercut the generalized results of RCTs.

This dual aspect of vaccine side effects—their rare existence and people’s tendency to overgeneralize from individual stories—places their portrayal in the media in what we consider a gray zone between accurate and misleading information. We thus posit that media coverage of such stories without proper contextualization can be misleading [10] and has the potential to influence public opinion. Such stories have even influenced decisions by national health organizations as was the case in June 2013 when the “Japanese Ministry of Health, Labour, and Welfare suspended proactive recommendations for the HPV vaccine after unconfirmed reports of adverse events following vaccination appeared in the media” [11].

Influence of vaccine hesitancy-inducing narratives

Even though prior experimental work has shown that visiting websites that are misleadingly critical of hypothetical vaccines can increase vaccine risk perceptions [12], no prior work has established whether media coverage of stories about side effects also induces (or reinforces) vaccine hesitancy. However, the existing literature offers important hints. Specifically, a number of narrative biases explain how individual accounts of vaccine side effects could be more influential than statistical information about their prevalence [1315]. For example, narrative information about side effects leads to lower intentions to vaccinate than does statistical information about them [13].

Past research on narrative bias has shown that curbing the influence of narratives is often difficult and finding ways of reducing this bias is important [5]. Previous work on individual stories of vaccine side effects has focused on the effects of the type of information (statistical or narrative) about side effects on vaccine beliefs and intentions but has not established how best to counteract the negative influences of messages about them. Thus, our aim was to examine whether stories of vaccine side effects may be counteracted by other news intended to convey the safety and benefits of vaccination. Studies of the positive [1618] and counter-productive [1921] effects of attempts to reduce misinformation have generally concluded that correcting misinformation is challenging [13, 14]. Moreover, by eliciting more thinking about the earlier misinformation, corrections can make it more familiar and accessible [22]. In that situation, corrections can amplify the initial effect of the misinformation, causing message recipients to be more misinformed after than before the correction. In other cases, simply providing correct information fails to counter the emotional effects of misinformed narratives [5]. Indeed, a meta-analysis found that the misinformation effects are, on average, much larger than the debunking ones [22]. Taken together, studies on misinformation have revealed that corrections do not necessarily remove misperceptions and misinformation and that their effectiveness depends on individual differences in motivation and educational level [19, 20, 23, 24].

More specifically, hesitancy-inducing narratives may be counteracted with two types of correction strategies: (a) statistical information that is usually delivered by experts or (b) narratives about people who are pro-vaccine and share their experiences. Expert-delivered statistical information is typically effective at changing beliefs, attitudes, and behaviors [17, 18, 25] but can sometimes backfire [26]. Correctives that use emotional personal narratives are often successful as well [5, 2729], in part because misinformation persistence is often linked to affect [30], especially when embedded in narratives [5]. Yet, how these interventions operate in individuals’ processing of vaccine side effect stories in the media remains to be established.

The context in which the public encounters messages that induce hesitancy could create additional challenges. One of these contextual matters is whether an audience receives pro-vaccine messages only or sees them juxtaposed with hesitancy-inducing narratives. Prior work on the process of correction has compared the standalone misinformation with the misinformation followed by correct information using either a between- or within-subjects design [22]. Examining such combined messages is important because an audience that receives a hesitancy-inducing narrative followed by a claim that the vaccine is safe might process the second claim differently than would those exposed only to the narrative or to the correct information [31, 32].

Similarly, information presented as categorical (e.g., “the vaccine is safe”) can be refuted by a single contrary example (of a real side effect). In such a circumstance, general and statistically supported claims about the safety of vaccines delivered by an expert such as Dr. Fauci may lose their effectiveness. Finally, examining combined messages is important because they better reflect news ecosystems in which people are exposed to conflicting messages due to what is known as media “false-balance” [31]. It is also important to examine epistemologically different forms of scientific information (e.g., aggregate conclusions drawn from randomized placebo-controlled trials on vaccine safety) vs. individualized narratives about side effects.

Current study

This study examined the individual and combined effects of exposure to a news account of a parent’s story of a child who experienced a negative vaccine reaction and news accounts about the value and safety of the MMR vaccine during the 2018–2019 U.S. measles outbreak. We relied on a between-subjects factorial experimental design. Our factorial approach manipulated 1) absence or presence of the hesitancy-inducing narrative message (a message describing vaccine side effects) and 2) presence of an expert pro-vaccine science message containing statistical information, a narrative pro-vaccine message from the point of view of parents, or no pro-vaccine message. This 2x3 factorial design allowed us to compare a total of six conditions. It also allowed us to test both the effects of hesitancy-inducing narratives and those of correction strategies on vaccine views and intentions. Analyses thus included main effects and the interaction between the two factors.

Although we utilized a factorial approach in the analysis, we formulated specific expectations and research question as well, which were preregistered. First, we hypothesized that exposure to the hesitancy-inducing narrative compared to the control would yield (a) higher perceptions of vaccine risk, (b) less support for pro-vaccine policy positions, (c) weaker intentions to encourage parents to vaccinate their children, and (d) lower probability of agreeing to send a pro-vaccine letter to a state representative (H1). Second, exposure to either the science-supporting (statistical) message or the science-consistent narrative would yield lower perceptions of vaccine risk, more support for pro-vaccine policies, stronger intentions to encourage parents to vaccinate, and higher probability of agreeing to send a pro-vaccine letter to a state representative than would the control message and the hesitancy-inducing narrative (H2). Third, both combined conditions would produce lower perceptions of risk, higher levels of pro-vaccine policy support, stronger intentions to encourage parents to vaccinate their children, and higher probability of agreeing to send a letter to a state representative than would the hesitancy-inducing narrative alone (H3). Finally, we compared the influence of pro-vaccine messages (both the science-supporting message and the science-consistent narrative) with and without the initial presentation of the hesitancy-inducing narrative (RQ1). This comparison allowed us to test for the influence of combined messages that include both hesitancy-inducing and pro-vaccine elements.

Our study also makes a novel contribution by focusing on the influence of coverage of vaccine side effect stories in the context of ecologically valid, real news videos. By contrast, most of the prior work in this domain focused on fictitious diseases and hypothetical vaccines and did not characterize public reaction to actual information about the MMR vaccine. In addition, we investigate effects on a variety of attitudinal and intentional outcome measures.

Methods

Data

We collected data through the AmeriSpeak panel of NORC at the University of Chicago. AmeriSpeak is a probability-based nationally representative sample of American adults (see Supporting Information B in S1 File). The NORC recruitment panel was a two-stage stratified sampling frame that covers 97% of U.S. households. To prevent sampling bias (e.g., some parents might be more interested in the topic of vaccine), the invitation did not indicate that the study would focus on vaccination. They were invited to “a survey about health and wellness.” More details about the NORC sampling procedures are provided in Supporting Information B in S1 File.

Two different nationally representative samples were collected concurrently between February 28 and March 18, 2019. We combined one from an ongoing longitudinal panel on vaccination (N = 1,682), and a supplemental, cross-sectional sample (N = 864) for a total of 2,546 random assignments (see the exclusion procedures section below for the effective sample size). This sampling procedure helped us achieve the appropriate sample size determined by the power analysis (see Supporting Information B in S1 File). The demographic composition of the samples as well as comparisons to the latest U.S. Census estimates also appear in Supporting Information B in S1 File.

The first sample was part of a larger panel study of infectious diseases and vaccination attitudes focusing on the flu, measles-mumps-rubella, and human papillomavirus. The experimental module was within the fourth wave of the panel and focused on the measles to make it possible to study the outbreak occurring in the country at the time of data collection. The number of participants was 1,803. Most responded to the survey over the internet (N = 1,682, 93.3%), although a smaller group did so over the phone (N = 121, 6.7%). Among the Internet respondents, 766 took the survey on a desktop computer (42.5%), 807 (44.8%) on a smartphone, and 109 on a tablet (6%). The retention from Wave 1 (September 2018) to Wave 4 (March 2019) was 60% (exactly).

The second sample was collected by the same survey company and is made up of 1,006 “fresh” respondents who were unrelated to the panel sample (to check for the panel effects of the larger study). Most respondents in this new sample also took the survey over the internet (N = 864, 85.9%); a smaller proportion completed it over the phone (N = 142, 14.1%). Because our experiment contained audio-visual stimuli, those respondents who took the survey on the phone (listening and answering questions vocally) were not part of the experiment. Phone respondents had a slight overrepresentation of older respondents. Among Internet respondents, 324 respondents took the survey over a desktop computer (32.2%), 496 (49.3%) on a smartphone, and 44 on a tablet (4.4%).

Procedure

After the informed consent, respondents answered a series of pre-treatment questions. In the manipulation section, the news clips were designed to ensure exposure; respondents were not able to fast-forward but were able to re-watch the video if they chose to do so. Also, participants could not move to the next page of the survey without watching the video at least once. After the exposure, we asked another set of quality control questions. We asked whether respondents were able to hear the video. Since we conducted the study shortly after the news clips aired, we also asked them if they had seen the video before (partially or fully). Exclusion criteria and results based on these quality checks are explained below. The outcome measures, manipulation checks, and debriefing of respondents about the experiment followed.

Manipulations

Participants viewed one of the six videos. The conditions were created to test the potential hesitancy-inducing influence of the side-effects story in relation to a control group, as well as potential countering forces of the statistical or narrative science-supporting messages either presented alone or presented in combination with the hesitancy-inducing narrative message. Real, contemporary news videos were obtained from the websites of NBC, ABC, CBS, and PBS and then professionally edited for length. Videos were stripped of channel logos to minimize source effects. Full details on each video, as well as a link to each news clip, are provided in Supporting Information B in S1 File. A brief description of video contents is provided below.

The first video condition (C1), hesitancy-inducing narrative (Hes), showed a mother expressing regret about vaccinating her first child against MMR and indicating that she would not vaccinate her second child because her older boy suffered pictured side-effects. The narrator quotes the mother saying that the child suffered severe side effects. In response to the reporter’s question about whether she believes that the CDC. is lying when it says vaccination is safe, the mother asserts that scientists do not know or adequately understand all of the potential hazards of vaccinations. We named this condition hesitancy-inducing narrative because it is a personal story without statistics. The second condition (C2), science-supporting (statistical) message (SSM), showed an expert, Dr. Anthony S. Fauci, Director of the National Institute of Allergy and Infectious Diseases, providing scientifically based statistical information about the contagiousness of measles and the safety and effectiveness of the vaccine. Because the video includes an expert and statistical information, we named this condition science-supporting (expert) message. In the third condition (C3), science-consistent narrative (SCN), participants were presented with the stories of several families with children who either would be endangered by exposure to measles (e.g., who cannot get the vaccine because of other health conditions or age limits) or who had caught measles and experienced complications. These parents blamed their situation on those who lowered herd immunity because they refused vaccination. We named this condition the science-consistent narrative because its personal stories deliver pro-vaccine information without using pro-vaccine statistics. In the fourth condition (C4), hesitancy-inducing narrative + science-supporting statistical message (Hes+SSM), participants watched the hesitancy-inducing narrative followed by the science-supporting message. The two components of this combined video were slightly shortened to make the length comparable with the first three conditions. The two parts are presented uninterrupted in a single continuous video. In the fifth condition (C5), hesitancy-inducing narrative + science-consistent narrative (Hes+SCN), participants watched the hesitancy-inducing narrative followed by the science-consistent narrative. The two components of this combined video were slightly shortened as well and the two parts are presented uninterrupted in a single continuous video. Finally, participants in the control condition (C6) watched a video of comparable length about aspirin. This video was edited to be equivalent in length and format and addressed a health-related topic not related directly to vaccines. These conditions were categorized based on the 2x3 factorial design as mentioned before.

Exclusion procedures

Not all participants from the sampling procedure were included in the experiment. Since the study was conducted online using a self-administered survey experiment with video messages, we administered the manipulations carefully to ensure exposure. One hundred twenty-five respondents, homogenously distributed across conditions, were excluded for one of three reasons: not being able to hear the audio check, reporting having been exposed to the video in full outside of the study, or not being able to play the experiment video: (1) 85 reported they could not hear, and others entered the wrong number during an audio check at the beginning of the study. The 85 participants who could not hear were asked if they had another audio-working device. Of those 85, the 64 who said they did not were dropped from the experiment; others joined with a new device. (2) After viewing the video, 32 respondents stated that they had seen it fully somewhere and were thus dropped from analysis because prior exposure would confound our experimental manipulation. However, 80 respondents who said they remembered seeing part of the video were retained. (3) 22 participants indicated they could not view the entire video and were dropped from analysis due to inadequate exposure to the experimental manipulation. (4) 7 participants who indicated they could not hear the experiment video were dropped as well due to inadequate exposure to the experimental manipulation. Additionally, those who left the audio check question blank (N = 56) at the very beginning of the survey were excluded.

Due to a survey progression (display logic) error, some participants were able to answer questions without having been shown the experimental module; this reduction in sample size was also random across conditions. Hence, the overall experiment sample size was 2,345. A small number of participants with missing data in dependent and independent variables were also dropped. Final Ns are reported along with the results in Table 1.

thumbnail
Table 1. General linear model, fixed effects ANCOVA and binary logit models predicting four outcome variables with experimental conditions and covariates.

https://doi.org/10.1371/journal.pone.0248328.t001

Manipulation checks

Manipulation checks asked respondents to identify whether any of the eight listed content items appeared in the video. Four of the items (individually or in different combinations) were included and the remaining four were distractors that did not appear in any of the conditions. This arrangement allowed us to assess how well the manipulations worked across conditions. The results showed that our manipulations worked. (For details, see Supporting Information B in S1 File).

Ethics statement

The study received approval from the Institutional Review Board (IRB) of the University of Pennsylvania. The study team declares full compliance with the ethical guidelines. Participants were appropriately asked for informed consent about the study at the beginning and they were debriefed at the end. The study participants were told that they could contact the research team with questions or if they wanted to see the original full versions of the news clips. Note that the videos they were shown were edited versions of news segments that had been nationally aired a few weeks before the study. Our study prioritized using actual news videos and not manipulating their content such that no information was changed or added during an epidemic at the time of data collection. The only stylistic change added a print overlay for the statistical information offered by Dr. Fauci. Hence, we did not expect or encounter adverse conditions or ethical problems, and there was no deception.

Measures

Outcome measures included evaluations of the MMR vaccine risk perceptions, pro-vaccine policy views, encouraging others to vaccinate their children, and agreeing to send a pro-vaccine letter to state legislators. Full question and response option wordings for all questions are provided in Supporting Information B in S1 File.

MMR vaccine risk perceptions index (outcome measure).

We measured MMR vaccine evaluations with four items that tapped various but closely related beliefs and evaluations about the vaccine, which were averaged into an index (e.g., “Just your best guess, how risky, if at all, do you think the measles, mumps, and rubella (MMR) vaccine is?” with response options, “Not risky at all,” “Not too risky,” “Somewhat risky,” “Very risky,” Supporting Information B in S1 File). The items had high reliability (α = .81) and an exploratory factor analysis suggested a single factor solution was reasonable (Maximum-Likelihood estimation, variance explained = .54). Higher scores in the index represent more negative evaluations of the MMR vaccine. Given the subjective nature of risk perceptions and variety of ways to measure the construct [33], tapping these closely related vaccine evaluations with multiple items provides broader applicability and ecological validity for our conclusions about vaccine risk perceptions and prevents bias from a single item. We also provide the main results for each item separately in Supporting Information C in S1 File.

Pro-vaccine policy index (outcome measure).

Pro-vaccine policy views were measured with six items about ongoing policy discussions around various types of exemption to vaccination [34]. To capture the different dimensions of the policy debates surrounding vaccination, such as forms of exemption (e.g., religious or medical) and mandatory school vaccinations, we asked these multiple questions, such as “It should be mandatory for parents to vaccinate their children against preventable diseases such as measles, mumps, and rubella,” with response options “Strongly support,” “Somewhat support,” “Neither support nor oppose,” “Somewhat oppose,” “Strongly oppose,” Supporting Information B in S1 File). The items had high reliability (α = .85) and an exploratory factor analysis supported a single factor solution (Maximum-Likelihood estimation, variance explained = .52). The measure, which was the average of all six items, provides a battery of overall vaccine-related policy positions. We also provide the main results for each item separately in Supporting Information C in S1 File.

Intention to encourage other parents to vaccinate their children (outcome measure).

A single item measured respondents’ willingness to encourage other parents to vaccinate their children against measles, mumps, and rubella: “How likely are you to encourage parents to give the measles, mumps, and rubella (MMR) vaccine to children?” with response options “Not likely at all,” “Not too likely,” “Somewhat likely,” “Very likely” (Supporting Information B in S1 File). Given that this was a specific behavioral intention regarding the MMR vaccine, a single item was deemed sufficient.

Agreeing to send a pro-vaccine letter to state representative (outcome measure).

We also used the single item shown for this specific behavioral intention of agreeing to send a pro-vaccine letter. The short letter statement can be seen in Supporting Information B in S1 File. We imputed the respondents’ state of residence into this statement where it reads “current state of residence” to increase engagement, relevance, and realism.

Covariates.

Our theoretical covariates included (a) respondents’ vaccine misinformation (prior vaccine misinformation could influence reaction to hesitancy-inducing narrative), measured with four items (e.g., “It is better to develop natural immunity by getting the disease than by receiving the vaccine”), and averaged into an index (α = .80); (b) trust in medical authorities (trust could influence reaction to science-supporting or science-consistent messages), measured with two items (“U.S. Centers for Disease Control and Prevention (CDC)” and “Your primary care doctor or primary medical provider”) and averaged into an index (α = .80). We also measured (a) conservative-liberal ideology, which we included as a covariate for policy-related outcomes, with a single item; (b) exposure to news about measles (as an indication of issue salience) measured with a single item; and (c) being a parent of someone under 30. We decided to identify parents with children under 30 years of age because the MMR vaccine is mostly a childhood vaccine with available booster shots for later ages. This non-optimal age cut-off point for parental status was chosen because the study was designed as part of a larger time-sharing study of various other vaccines. Methodological covariates included gender of the respondents (because random assignment did not produce equivalent conditions, female coded = 1, male coded = 0) and sample type (methodological check given we ran the tests on the aggregated sample, see Data section for further details, main sample coded = 0, supplemental sample coded = 1). Supporting Information B in S1 File contains more details on question wording.

Analytical strategy

Although we preregistered pairwise contrasts across our six conditions, in response to the feedback during revisions, we grouped the six conditions into two factors for analytical efficiency. The first factor, which we named hesitancy-inducing message factor, had two levels, which are hesitancy-inducing narrative and control. The second factor, which we named science message factor, had three levels: control, science-supporting message, and science-consistent narrative. Because of the large N, we used conservative Bonferroni adjustments for 15 possible pairwise comparisons.

To assess experimental effects, we utilized both general linear model (GLM) regression and analysis of covariance (ANCOVA) to predict the MMR vaccine risk perceptions and pro-vaccine policy views, because these outcome measures were scales (continuous variables composed of multiple items). We utilized logit models for encouraging others to vaccinate their children and agreeing to send a pro-vaccine letter to state representative because these measures were single-item ordinal/categorical measures. The models included the two factors and introduced the two main effects along with the combined effect (interaction) of the two factors, and other covariates. The results show mean differences for MMR risk perceptions (ranging from 1 to 4.25) and pro-vaccine policy views (ranging from 1 to 5) and shift in the predicted probabilities (0 to 1, see the two coding strategies below) for encouraging others and sending a pro-vaccine letter (0 or 1). Balanced (equally weighted) estimated marginal means are calculated as homogeneity of variances assumption was met. Cell-weighted ANCOVA models produced the same results. We included the same set of covariates for all models (see section above) with the exception of ideology, which was included in only two policy-related models (pro-vaccine policy support and sending a pro-vaccine letter to state representative), because it is a policy-relevant variable. Various statistical assumptions were checked, and are reported in Supporting Information B in S1 File. The assumption of normality was violated, although this is less problematic for comparisons involving random assignments (experiment), especially when the sample size is large and cell sizes are similar.

Recoding of encouraging others to vaccinate their children.

Given the violations of some logit regression assumptions and the strong negative skew in intentions to encourage others to vaccinate their children (Supporting Information A in S1 File for further details), we recoded this outcome measure into a binary option consisting of lower likelihood (= 2 or 1, coded 0) and higher likelihood (= 3 or 4, coded 1) and conducted a binary logistic regression. Given that about half of respondents selected the highest likelihood (originally coded = 4), we also recoded it into a second/alternative binary option consisting of lower likelihood (= 3 or lower, coded 0) and highest likelihood (= 4, coded 1). We present these results for both coding strategies.

Finally, preregistration details that also include itemized deviations from the preregistration are in Supporting Information D in S1 File, additional analyses that relied on non-factorial analytical approach (preregistered) are in Supporting Information E in S1 File (of which results did not differ substantially), and finally, additional robustness checks are in Supporting Information F in S1 File.

Results

Table 1 contains coefficients for the main and interaction effects as well as coefficients for the covariates. The table is organized with vertical panels for each of our dependent measures. Both GLM regression and ANCOVA results are provided for the continuous outcome variables (for comparison purposes with the logit models of the categorical outcomes). For logit models, odd ratios are provided too. Fig 1 shows the estimated marginal means for the continuous dependent variables and the predicted probabilities for the categorical dependent variables. Bonferroni-corrected post-hoc contrasts were performed among the means and probabilities in Fig 1 and different letters indicate means or probabilities that are significantly different from each other. Full statistical details about the contrasts appear in Supporting Information B in S1 File.

thumbnail
Fig 1. Hes–Hesitancy-inducing narrative, SSM–Science-Supporting Message, SCN–Science-Consistent Message.

Shared letters denote that no significant differences exist among the conditions (which was based on Bonferroni correction for a total of 15 pairwise comparisons among the six conditions). Non-overlapping CIs do not imply significant difference due to the Bonferroni correction.

https://doi.org/10.1371/journal.pone.0248328.g001

We first examined the effects of our experimental messages on MMR vaccine risk perceptions, policy views, intentions to encourage others to vaccinate their children against measles (for both coding strategies), and intentions to send a pro-vaccine letter to a state representative. As shown in the second row of Table 1, exposure to the hesitancy-inducing narrative had no significant effect on our outcomes, except for a small negative effect on vaccine policy views, F(1,2280) = 3.96, p = .046. This generally null effect can also be seen in the five plots in Fig 1, which show no differences between the hesitancy-inducing narrative condition and the control condition after Bonferroni corrections were performed.

In contrast to the null main effects of the hesitancy-inducing narrative, the science-supporting message had significant positive effects for all our dependent measures (rows 3, 4, and 5 in Table 1). Specifically, presenting (vs. not presenting) the science-supporting message led to lower MMR vaccine risk perceptions, stronger pro-vaccine policy views, and stronger intentions to encourage others to vaccinate their children and send a letter to a representative. Of note, however, this main effect was mostly driven by the expert science-supporting message, which remained significantly different from both the control condition and hesitancy-inducing narrative in the contrasts in Fig 1. In contrast, the narrative science-supporting message did not differ from the control message.

We also examined the interaction between the hesitancy-inducing narrative and the science-supporting factor and conducted specific contrasts. We found that the hesitancy-inducing narrative and the science-supporting factor interacted significantly for vaccine risk perceptions, vaccine policy views, and encouraging others to vaccinate their children (the second coding strategy; see Table 1). These interactions can be decomposed by inspecting the contrasts in Fig 1 and also the planned interactions between each science-supporting message and the hesitancy-inducing narrative. As shown by the contrasts in Fig 1, when no hesitancy-inducing narrative was presented, the expert science-supporting message alone led to a weaker perception that the vaccine is risky than did either not presenting any science-supporting message (i.e., the control condition) or presenting the science-consistent narrative. However, when a hesitancy-inducing narrative was presented, vaccine risk perceptions did not differ across the science-supporting message, the science-consistent narrative, and the control conditions. As seen in Fig 1, this same pattern was present for the other outcome variables, although applying the conservative Bonferroni correction for 15 possible pairwise comparisons rendered those comparisons not significant. With respect to the more focused interactions, the combined effect of the hesitancy-inducing narrative x science-supporting expert message was significant only for the second coding strategy of encouraging others to vaccinate their children (Table 1). Overall then, because this interactive pattern was weak and not always significant, we recommend replicating this finding. Finally, the more focused interaction between the hesitancy-inducing message and the science-consistent narrative was significant for policy views and sending a letter, although the contrasts with Bonferroni corrections were not significant. We also note that all effect sizes were small as shown by the partial eta-squares and odds ratios in Table 1.

Our results lead us to the following conclusions about our hypotheses and research questions. First, we had hypothesized that the hesitancy-inducing narrative could reduce pro-vaccine views and intentions. Our null findings do not lend support to H1. The second hypothesis posited that presentation of science-supporting message factor would increase pro-vaccine views and intentions. Our findings were consistent with this hypothesis but only for the science-supporting message in which expert information was relayed. H2 is, as a result, partially supported. Third, we had hypothesized that the science-supporting message might counteract the potentially negative influence of hesitancy-inducing narrative when presented together. Presenting the science-supporting messages along with the hesitancy-inducing narrative did not result in stronger pro-vaccine views or intentions relative to the sole presentation of the hesitancy-inducing narrative; thus, H3 is not supported. Finally, for RQ1, the obtained differences between the science-supporting messages with and without the hesitancy-inducing narrative were mostly inconclusive. We found suggestive evidence (i.e., one significant finding. and several trends) that the hesitancy-inducing narrative could limit the pro-vaccine effect of the expert science-supporting message.

Discussion

Understanding the effects of news coverage of measles and the MMR vaccine is crucial, especially in light of recent public health crises such as the U.S. measles outbreak as well as the potential of MMR misinformation to elicit hesitancy toward other known and novel vaccines [35]. This study offers a first empirical test of the potential influence of a personalized story of parents’ response to a child who suffered a vaccine side effect. Our study detected no significant effect of this message when presented alone: Except for one outcome variable (i.e., policy views), the vaccine-hesitant narrative by itself did not induce hesitancy or reduce pro-vaccine positions in a statistically significant way. This result is generally consistent with research suggesting that exposure to single messages rarely produces an impact [36] and stands in contrast to a study that reported increased vaccine risk perceptions due to exposure to a vaccine-critical website. However, the prior study is very different from ours in terms of topic, context, and methods [12]. Still, we should exercise caution when interpreting this non-significant finding as conclusive evidence given our mainly null effects. More research should be conducted in the future to test our hypothesis.

Our results also showed that when presented alone, the statistical information provided by an expert (i.e., the science-supporting message) about vaccine safety and effectiveness produced stronger pro-vaccine beliefs than did the control message, although the effect sizes were relatively small [31, 37]. However, for vaccine risk perceptions, the science-supporting expert message mostly ceased to offer a benefit when it followed the hesitancy-inducing narrative. Additionally, although when combined with the hesitancy-inducing narrative the expert science-supporting message did not differ from the other four conditions for any of our outcome measures, it differed from the other conditions significantly when it was presented alone. We interpret this particular finding and the trend in other outcome measures as potential evidence that the hesitancy-inducing narrative may weaken the pro-vaccine influence of the expert statistical message [37]. Future work should further examine this process to determine if it occurs in other contexts. We note that effects might differ, and we suspect might be more pronounced, if respondents were exposed to more tightly circumscribed, experimenter-created messages rather than the real-world ones studied here. By contrast, our data provide the ecological validity of using real news clips in the context of an actual outbreak.

We also found that the pro-vaccine narrative message did not have a significant effect either alone or in combination with the hesitancy narrative, showing its relative ineffectiveness compared to the statistical message about the MMR vaccine [19]. This finding also stands in contrast to the literature showing the effectiveness of narrative corrections [5]. More research is needed to understand how to increase the effectiveness of the appeals we tested in this experiment.

Our findings offer empirical evidence on a variety of vaccine-related beliefs and intentions. Investigating these diverse outcomes is important in real-world contexts and during disease outbreaks. For example, experimental research on policy support and behaviors is scarce [34, 38]. In light of ongoing legislative discussion and action concerning medical, religious, and personal belief exemptions at the level of the states [39, 40], understanding the impact of news stories on support for pro-vaccine and anti-vaccine policies is important. We showed that pro-vaccine scientific messages boost both pro-vaccine policy views and the willingness to send a pro-vaccine letter, although the small effect sizes should be kept in mind. Still, the empirical evidence of effects during an outbreak is important and should be replicated for other outbreaks and pandemics such as COVID-19. Although our effect sizes were small, such effects could have significance at the societal level during pandemics.

In addition to theoretical contributions to our understanding of persuasion effects, this study adds to our understanding of vaccine-related messages in ways that are important methodologically. Most experimental research on vaccine attitudes and misinformation has relied on small convenience samples [12, 14, 23, 41], online convenience samples such as Amazon Mechanical Turk [42, 43], other online non-probability panels [37], or in few cases, probability samples of specialized populations (i.e., only parents, which would be not be an ideal sample when we focus on outcomes such as policy views) [19]. Work on narrative bias and vaccine side effects that relies on fictitious diseases and hypothetical vaccines makes it difficult to render a judgment about real vaccines [1315]. Prior studies also focused mostly on print text, and occasionally still photographs [19], but not on actual broadcast news segments. Importantly, industry statistics show that most individuals prefer to get their news on television (e.g., United States) [44].

Limitations

Due to heightened media attention during the measles outbreak, participants may have been exposed to multiple news stories and a great deal of social media content about measles and the MMR vaccine, which may have affected their responsiveness to our experimental materials. Because of randomization and media exposure as a covariate, we would expect these effects to be comparable across conditions and not to contaminate our results.

Second, our findings are limited to vaccination against MMR, and may therefore operate differently for other vaccines, such as those against the flu, HPV (human papillomavirus) or COVID-19. Different vaccines also elicit different considerations. For example, the HPV vaccine elicits religious and parental concerns because of the assumption of some that it signals acceptability of sexual activity for teenagers. These differences could lead to different patterns of message processing concerning descriptions of side effects. Relatedly, future research should examine how other theoretical variables such as religiosity might alter reactions to such experimental messages.

Third, although our manipulations sought to maximize external validity by using edited versions of real news videos that were aired in the middle of a national measles outbreak, future research should replicate these effects in different media (e.g., news articles) and come up with designs to disentangle some of the effects we observed. One potential problem with less-controlled real news clips is that we cannot know, for example, whether beliefs shift because of the expert authority acting as a source cue, the nature of statistical information provided, or both. This aspect may also contribute to the study’s small effect sizes, which we note as another limitation. We also are unsure of whether participants processed the source and message information separately [45]. Similarly, we do not currently know whether the effect of the hesitancy-inducing narrative was produced by the credibility of the mother or the evocative visual image of the rash-ridden child [46,47]. Finally, our video content may not be fully representative of the media environment, which is both diverse and fast changing. These issues should be addressed in future research.

Implications for public health campaigns: A transparency dilemma?

Our results have important implications for public health campaigns and decisions in newsrooms as journalists tackle vaccine hesitancy-inducing or -reinforcing narratives. First, although the statistical message had more favorable and more consistently positive effects, public health campaigns and news coverage that aim to promote vaccination may benefit from using both the scientific messages and pro-vaccine narratives that amplify the stories of individuals and that may better address the emotional aspect of the misinformation [5]. Since we do not find negative effects of pro-vaccine parental stories, more research should be conducted to judge their effectiveness in different formats and contexts. Second, giving voice to vaccine-hesitant parents as a way of explaining the increase in measles cases can still have detrimental effects [21, 48, 49]. Moreover, the effect of a hesitancy-inducing narrative may increase if repeated exposure occurs or if the stories circulate within social media where repetition and like–minded interpersonal interaction prevail [17, 50]. More research is needed to determine how to communicate the relatively low risk involved in vaccination compared to the higher risks of vaccine-preventable infectious diseases.

At a broader level, these findings also speak to the importance of having the scientific community and news media acknowledge and carefully communicate the rare but real side effects of vaccines. As science writer Melinda Wenner Boyer noted, “Scientists are so terrified of the public’s vaccine hesitancy that they are censoring themselves, playing down undesirable findings and perhaps even avoiding undertaking studies that could show unwanted effects” [51]. This anecdotal evidence by a leading science writer at The New York Times should be taken as an early warning about the complexity and challenges in the public discourse on vaccine safety. Failure to be transparent about the side effects of vaccines and to communicate them effectively may enhance the credibility of anti-vaccination rhetoric. The reluctance of scientists to talk about side effects leaves the media with little guidance on how to cover them.

Acknowledgments

We thank the 2018–2019 Annenberg Science Knowledge Group at the Annenberg Public Policy Center for its collective contributions to the survey design and implementation. The 2018–2019 ASK Group included Ozan Kuru, Ph.D., Dominik Stecuła, Ph.D., Hang Lu, Ph.D., Yotam Ophir, Ph.D., Sally Chan, Ph.D., Ken Winneg, Ph.D., Kathleen Hall Jamieson, Ph.D., and Dolores Albarracín, Ph.D., Annenberg Public Policy Center, University of Pennsylvania. We thank Gary Gehman and Karen Riley from the Annenberg Public Policy Center for editing the experimental videos and copy-editing the manuscript respectively. Corresponding authors are Ozan Kuru, ozan.kuru@nus.edu.sg, Kathleen Hall Jamieson, kathleen.jamieson@asc.upenn.edu, and Dolores Albarracín, dalbarra@illinois.edu.

References

  1. 1. Centers for Disease Control. Measles Cases and Outbreaks | CDC [Internet]. 2019. [cited 2019 Jul 26]. Available from: https://www.cdc.gov/measles/cases-outbreaks.html
  2. 2. Belluz J. Washington measles outbreak: why it’s mostly affecting Russian speakers—Vox [Internet]. 2019 [cited 2019 Jun 9]. Available from: https://www.vox.com/2019/3/19/18263688/measles-outbreak-2019-clark-county
  3. 3. Centers for Disease Control. Measles | History of Measles | CDC [Internet]. [cited 2019 Sep 9]. Available from: https://www.cdc.gov/measles/about/history.html
  4. 4. Stecula DA, Kuru O, Hall Jamieson K. How Trust in Experts and Media Use Affect Acceptance of Common Anti-Vaccination Claims. Harvard Kennedy Sch Misinformation Rev. 2020 Jan 14;1(1).
  5. 5. Sangalang A, Ophir Y, Cappella JN. The Potential for Narrative Correctives to Combat Misinformation†. J Commun [Internet]. 2019 Jun 1 [cited 2019 Jul 29];69(3):298–319. Available from: https://academic.oup.com/joc/article/69/3/298/5481803 pmid:31205334
  6. 6. Centers for Disease Control. Vaccine Information Statement | MMR | Measles-Mumps-Rubella | VIS | CDC [Internet]. Vaccine Information Statements. 2019 [cited 2019 Sep 9]. Available from: https://www.cdc.gov/vaccines/hcp/vis/vis-statements/mmr.html
  7. 7. Tversky A, Kahneman D. Availability: A heuristic for judging frequency and probability. Cogn Psychol [Internet]. 1973 Sep 1 [cited 2019 Sep 5];5(2):207–32. Available from: https://www.sciencedirect.com/science/article/pii/0010028573900339
  8. 8. Dal Cin S, Zanna M, Fong G. Narrative Persuasion and Overcoming Resistance. In: Knowles E.S., Linn J.A., editors. Resistance and persuasion [Internet]. NJ: Lawrence Erlbaum Associates Publishers; 2004 [cited 2019 Sep 5]. p. 175–91. Available from: https://psycnet.apa.org/record/004-00227-009
  9. 9. Nabi RL, Green MC. The Role of a Narrative’s Emotional Flow in Promoting Persuasive Outcomes. Media Psychol [Internet]. 2015 Apr 3 [cited 2019 Sep 5];18(2):137–62. Available from: http://www.tandfonline.com/doi/full/10.1080/15213269.2014.912585
  10. 10. Wardle C. Fake news. It’s complicated. [Internet]. First Draft News. 2017 [cited 2020 Mar 9]. Available from: https://firstdraftnews.org/latest/fake-news-complicated/
  11. 11. Hanley SJB, Yoshioka E, Ito Y, Kishi R. HPV vaccination crisis in Japan. Vol. 385, The Lancet. Lancet Publishing Group; 2015. p. 2571. pmid:26122153
  12. 12. Betsch C, Renkewitz F, Betsch T, Ulshöfer C. The influence of vaccine-critical websites on perceiving vaccination risks. J Health Psychol [Internet]. 2010 Apr [cited 2020 Mar 10];15(3):446–55. Available from: http://www.ncbi.nlm.nih.gov/pubmed/20348365 pmid:20348365
  13. 13. Betsch C, Ulshöfer C, Renkewitz F, Betsch T. The influence of narrative v. statistical information on perceiving vaccination risks. Med Decis Mak [Internet]. 2011 Sep [cited 2020 Mar 10];31(5):742–53. Available from: http://www.ncbi.nlm.nih.gov/pubmed/21447730 pmid:21447730
  14. 14. Betsch C, Renkewitz F, Haase N. Effect of narrative reports about vaccine adverse events and bias-awareness disclaimers on vaccine decisions: A simulation of an online patient social Network. Med Decis Mak [Internet]. 2013 Jan [cited 2020 Mar 10];33(1):14–25. Available from: http://www.ncbi.nlm.nih.gov/pubmed/22875721 pmid:22875721
  15. 15. Betsch C, Haase N, Renkewitz F, Schmid P. The narrative bias revisited: What drives the biasing influence of narrative information on risk perceptions? Judgm Decis Mak. 2015;10(3):241–64.
  16. 16. Bode L, Vraga EK. In Related News, That Was Wrong: The Correction of Misinformation Through Related Stories Functionality in Social Media. J Commun [Internet]. 2015 Aug 1 [cited 2019 Oct 24];65(4):619–38. Available from: https://academic.oup.com/joc/article/65/4/619-638/4082315
  17. 17. Vraga EK, Bode L. Using Expert Sources to Correct Health Misinformation in Social Media. Sci Commun [Internet]. 2017 Oct 14 [cited 2019 Jul 26];39(5):621–45. Available from: http://journals.sagepub.com/doi/10.1177/1075547017731776
  18. 18. Dunwoody S, Kohl PA. Using Weight-of-Experts Messaging to Communicate Accurately About Contested Science. Sci Commun [Internet]. 2017 Jun 15 [cited 2019 Jul 26];39(3):338–57. Available from: http://journals.sagepub.com/doi/10.1177/1075547017707765
  19. 19. Nyhan B, Reifler J, Richey S, Freed GL. Effective messages in vaccine promotion: a randomized trial. Pediatrics [Internet]. 2014 Apr 1 [cited 2019 Jul 26];133(4):e835–42. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24590751 pmid:24590751
  20. 20. Nyhan B, Reifler J. When Corrections Fail: The Persistence of Political Misperceptions. Polit Behav [Internet]. 2010 Jun 30 [cited 2019 Jul 26];32(2):303–30. Available from: http://link.springer.com/10.1007/s11109-010-9112-2
  21. 21. Redlawsk DP. Hot Cognition or Cool Consideration? Testing the Effects of Motivated Reasoning on Political Decision Making. J Polit [Internet]. 2002 Nov 1 [cited 2019 Jul 26];64(4):1021–44. Available from: https://www.journals.uchicago.edu/doi/10.1111/1468-2508.00161
  22. 22. Chan MS, Jones CR, Jamieson KH, Albarracín D. Debunking: A Meta-Analysis of the Psychological Efficacy of Messages Countering Misinformation. Psychol Sci [Internet]. 2017 Nov 12 [cited 2019 Jul 29];28(11):1531–46. Available from: http://journals.sagepub.com/doi/10.1177/0956797617714579 pmid:28895452
  23. 23. Pluviano S, Watt C, Della Sala S. Misinformation lingers in memory: Failure of three pro-vaccination strategies. Moore AC, editor. PLoS One [Internet]. 2017 Jul 27 [cited 2019 Jul 26];12(7):e0181640. Available from: http://www.ncbi.nlm.nih.gov/pubmed/28749996 pmid:28749996
  24. 24. Kuru O, Pasek J, Traugott MW. When Pundits Weigh In: Do Expert and Partisan Critiques in News Reports Shape Ordinary Individuals’ Interpretations of Polls? Mass Commun Soc [Internet]. 2020 Sep 2 [cited 2021 Feb 14];23(5):628–55. Available from: https://www.tandfonline.com/doi/abs/10.1080/15205436.2020.1774780
  25. 25. Kohl PA, Kim SY, Peng Y, Akin H, Koh EJ, Howell A, et al. The influence of weight-of-evidence strategies on audience perceptions of (un)certainty when media cover contested science. Public Underst Sci [Internet]. 2016 Nov 3 [cited 2019 Jul 26];25(8):976–91. Available from: http://journals.sagepub.com/doi/10.1177/0963662515615087 pmid:26657318
  26. 26. Merkley E. Anti-Intellectualism, Populism, and Motivated Resistance to Expert Consensus. Public Opin Q. 2020;
  27. 27. Braddock K, Dillard JP. Meta-analytic evidence for the persuasive effect of narratives on beliefs, attitudes, intentions, and behaviors. Commun Monogr [Internet]. 2016 Oct 25 [cited 2019 Jul 29];83(4):446–67. Available from: https://www.tandfonline.com/doi/full/10.1080/03637751.2015.1128555
  28. 28. Nan X, Dahlstrom MF, Richards A, Rangarajan S. Influence of Evidence Type and Narrative Type on HPV Risk Perception and Intention to Obtain the HPV Vaccine. Health Commun [Internet]. 2015 Mar 4 [cited 2019 Jul 29];30(3):301–8. Available from: http://www.tandfonline.com/doi/abs/10.1080/10410236.2014.888629 pmid:25061715
  29. 29. Walter N, Murphy ST. How to unring the bell: A meta-analytic approach to correction of misinformation. Commun Monogr [Internet]. 2018 Jul 3 [cited 2019 Jul 29];85(3):423–41. Available from: https://www.tandfonline.com/doi/full/10.1080/03637751.2018.1467564
  30. 30. Thorson E. Belief Echoes: The Persistent Effects of Corrected Misinformation. Polit Commun [Internet]. 2016 Jul 2 [cited 2019 Aug 9];33(3):460–80. Available from: https://www.tandfonline.com/doi/full/10.1080/10584609.2015.1102187
  31. 31. Dixon GN, Clarke CE. Heightening Uncertainty Around Certain Science. Sci Commun [Internet]. 2013 Jun 13 [cited 2019 Jul 26];35(3):358–82. Available from: http://journals.sagepub.com/doi/10.1177/1075547012458290
  32. 32. Dixon G, Clarke C. The effect of falsely balanced reporting of the autism-vaccine controversy on vaccine safety perceptions and behavioral intentions. Health Educ Res [Internet]. 2013 Apr 1 [cited 2019 Jul 26];28(2):352–9. Available from: https://academic.oup.com/her/article-lookup/doi/10.1093/her/cys110 pmid:23193194
  33. 33. Liao Q, Wong WS, Fielding R. Comparison of Different Risk Perception Measures in Predicting Seasonal Influenza Vaccination among Healthy Chinese Adults in Hong Kong: A Prospective Longitudinal Study. PLoS One. 2013 Jul 19;8(7).
  34. 34. Carpiano RM, Fitz NS. Public attitudes toward child undervaccination: A randomized experiment on evaluations, stigmatizing orientations, and support for policies. Soc Sci Med [Internet]. 2017 Jul 1 [cited 2019 Nov 14];185:127–36. Available from: https://www.sciencedirect.com/science/article/pii/S0277953617303088 pmid:28578210
  35. 35. Ophir Y, Jamieson KH. Intentions to use a novel Zika vaccine: the effects of misbeliefs about the MMR vaccine and perceptions about Zika. J Public Health (Bangkok) [Internet]. 2018 [cited 2020 Jul 8];40(4):531–7. Available from: https://academic.oup.com/jpubhealth/article/40/4/e531/4925540 pmid:29554290
  36. 36. Bennett WL, Iyengar S. A New Era of Minimal Effects? The Changing Foundations of Political Communication. J Commun [Internet]. 2008 Dec 1 [cited 2019 Aug 9];58(4):707–31. Available from: https://academic.oup.com/joc/article/58/4/707-731/4098406
  37. 37. Cook J, Lewandowsky S, Ecker UKH. Neutralizing misinformation through inoculation: Exposing misleading argumentation techniques reduces their influence. Manalo E, editor. PLoS One [Internet]. 2017 May 5 [cited 2019 Jul 30];12(5):e0175799. Available from: http://dx.plos.org/10.1371/journal.pone.0175799 pmid:28475576
  38. 38. Saulsberry L, Fowler EF, Nagler RH, Gollust SE. Perceptions of politicization and HPV vaccine policy support. Vaccine [Internet]. 2019 Aug [cited 2019 Jul 29];37(35):5121–8. Available from: https://linkinghub.elsevier.com/retrieve/pii/S0264410X19306991 pmid:31296376
  39. 39. Sandstrom A. Amid measles outbreak, New York closes religious exemption for vaccinations–but most states retain it [Internet]. Pew Research Center. 2019 [cited 2019 Nov 14]. Available from: https://www.pewresearch.org/fact-tank/2019/06/28/nearly-all-states-allow-religious-exemptions-for-vaccinations/
  40. 40. Stecula DA, Kuru O, Albarracin D, Jamieson KH. Policy views and negative beliefs about vaccines in the United States, 2019. Am J Public Health [Internet]. 2020 Oct 1 [cited 2021 Feb 14];110(10):1561–3. Available from: http://ajph.aphapublications.org/ pmid:32816542
  41. 41. Jolley D, Douglas KM. The Effects of Anti-Vaccine Conspiracy Theories on Vaccination Intentions. Tripp R, editor. PLoS One [Internet]. 2014 Feb 20 [cited 2019 Jul 26];9(2):e89177. Available from: http://dx.plos.org/10.1371/journal.pone.0089177 pmid:24586574
  42. 42. van der Linden SL, Clarke CE, Maibach EW. Highlighting consensus among medical scientists increases public support for vaccines: evidence from a randomized experiment. BMC Public Health [Internet]. 2015 Dec 3 [cited 2019 Jul 26];15(1):1207. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26635296 pmid:26635296
  43. 43. Horne Z, Powell D, Hummel JE, Holyoak KJ. Countering antivaccination attitudes. Proc Natl Acad Sci U S A [Internet]. 2015 Aug 18 [cited 2019 Jul 26];112(33):10321–4. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26240325 pmid:26240325
  44. 44. Mitchell Amy. Americans Still Prefer Watching to Reading the News—and Mostly Still Through Television | Pew Research Center [Internet]. Pew Research Center. 2018 [cited 2019 Jul 29]. Available from: https://www.journalism.org/2018/12/03/americans-still-prefer-watching-to-reading-the-news-and-mostly-still-through-television/
  45. 45. Fagerlin A, Wang C, Ubel PA. Reducing the influence of anecdotal reasoning on people’s health care decisions: is a picture worth a thousand statistics? Med Decis Making [Internet]. 2005 Jul 5 [cited 2020 Mar 10];25(4):398–405. Available from: http://www.ncbi.nlm.nih.gov/pubmed/16061891 pmid:16061891
  46. 46. Obrecht NA, Chapman GB, Gelman R. An encounter frequency account of how experience affects likelihood estimation. Mem Cogn. 2009 Jul;37(5):632–43. pmid:19487755
  47. 47. Reyna VF, Brainerd CJ. Numeracy, ratio bias, and denominator neglect in judgments of risk and probability. Learn Individ Differ. 2008;18(1):89–107.
  48. 48. Betsch C, Sachse K. Debunking vaccination myths: Strong risk negations can increase perceived vaccination risks. Heal Psychol [Internet]. 2013 [cited 2019 Jul 26];32(2):146–55. Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/a0027387 pmid:22409264
  49. 49. Koehler DJ. Can journalistic “false balance” distort public perception of consensus in expert opinion? J Exp Psychol Appl [Internet]. 2016 Mar [cited 2019 Aug 9];22(1):24–38. Available from: http://www.ncbi.nlm.nih.gov/pubmed/26752513 pmid:26752513
  50. 50. Newley Purnell. WhatsApp Users Spread Antivaccine Rumors in India—WSJ [Internet]. The Wall Street Journal. 2019 [cited 2019 Jul 29]. Available from: https://www.wsj.com/articles/whatsapp-users-spread-antivaccine-rumors-in-india-11555153203
  51. 51. Moyer MW. Anti-Vaccine Activists Have Taken Vaccine Science Hostage. The New York Times [Internet]. 2018 [cited 2020 Mar 10]; Available from: https://www.nytimes.com/2018/08/04/opinion/sunday/anti-vaccine-activists-have-taken-vaccine-science-hostage.html?action=click&contentCollection=opinion&contentPlacement=9&module=package&pgtype=sectionfront&region=rank&rref=collection%252Fsectioncollection%252