Video development

The YouTube video was made by LSW in December 2020, shortly after the issuance of Emergency Use Authorizations by the United States Food and Drug Association for the Pfizer-BioNTech and Moderna vaccines for COVID-19, which are both mRNA vaccines. The programs Biorender and Vyond were used to create the video, which uses animations and cartoon characters to illustrate educational points about how the COVID-19 mRNA vaccines work. The full text of the video can be found in the Supplementary Information.

The Female-Narr-Video was the original version created in December 2020 for educational purposes (although the title slide and link on YouTube were changed for this study), and the Male-Narr-Video was created later as an additional experimental condition. In other words, the creation of the Female-Narr-Video was independent of the idea of testing whether narrator gender might associate with its effectiveness; only after receiving positive anecdotal feedback on the original video did we decide to study the efficacy of such variations. Given findings that audiences draw on negative stereotypes about women who communicate scientific findings, and rate women teaching about science lower than men, [22,23], we wondered whether reactions to the video could vary due to the gender of the narrator’s voice.

The video begins with an explanation of the flow of genetic information, then explains how weakened virus vaccines work before explaining what an mRNA vaccine means. The video then emphasizes that the Pfizer-BioNTech (referred to as Pfizer in the video) and Moderna vaccines have been tested for safety and efficacy, and ends with an explanation of how getting vaccinated can protect others in society.

Previous research on vaccine hesitancy interventions has identified common features for successful communication. This video incorporates many of these features, including: i) The video emphasizes the messages that approved vaccines are safe and go through evaluations [9], and that immunization also protects others, which has been found to activate positive emotions [2]. ii) The video does not focus on messages that have been reported to impede efforts to diminish vaccine hesitancy such as myth-busting, fear-based, or numerically focused messages [10]. iii) The video creates clear bottom-line (gist) messages through: explanations of concepts in simple language rather than scientific jargon; the use of animated illustrations to visually reinforce the concepts explained verbally; and the use of text on the screen to underscore take-home messages. Research in medical decision making has found that gist messages contribute to decision-making in a meaningful way relative to statistics and verbatim information [6,7,10], and previous research on vaccine communication has found that articles with gist were shared on social media more than those without [24].

At the time of making the video, there was not yet much literature on recommendations for interventions specific to the COVID-19 vaccines. In some respects, the unique characteristics of the COVID-19 vaccine may alleviate some of the attitudes that have contributed to parental vaccine hesitancy for childhood vaccinations. For instance, it has been hypothesized that the benefits of some vaccinations may not seem tangible due to unfamiliarity with the harm of the preventable disease [9,25], which may be less likely for COVID-19. However, there are new concerns particular to COVID-19 vaccines that have been reported [2,21], and the video does address some of these, including unease over the new mRNA vaccine platform that had not been used before, and concern over the speed of the vaccine development and approval (the video does not go into all of the details that contributed to the rapid vaccine development but does explain that the mRNA platform may allow quicker vaccine development than weakened virus platforms). Although previous research has shown that vaccine hesitancy is often more complex than a simple knowledge deficit [10,19], the overall tone of the video is an educational approach to influencing vaccination attitudes because in the case of the COVID-19 vaccines, the lack of knowledge about the new vaccine platform may contribute to hesitancy [26].

Previous research has indicated that the messaging matters in addition to the message—that is, it is not just what one says, but how it is said that influences the effectiveness of video communication [27,28]. This research has recommended that vaccine interventions use an enthusiastic tone [10] and establish the public health educator as an expert [7]. Although the video does not specify the credentials of the creator, the survey participants were told that the video was created by a college biochemistry professor. This study tests the effectiveness of a video that draws upon the best evidence in the vaccine hesitancy intervention literature for the message (information content) and messaging (style of communication) of vaccine communications, and additionally allows us to study whether there may be different patterns of effectiveness based on the messenger (the person who delivers the message).

Because the narrator is never depicted in the video, we saw the video as a platform to isolate and test the effects of a single variable related to the messenger: the gender of the narrator’s voice. Therefore, two versions were used in the study, identical but for the voice of the narrator: the Female-Narr-Video version had a female voice reading the transcript and the Male-Narr-Video version had a male voice. In both cases the timing of the narration was matched to the animations (which led to a 3 second difference in length of the videos). The videos used in the survey were posted on unlisted YouTube sites, and reposted on the following links for others beyond the study participants to see: Male-Narr-Video https://youtu.be/Fv5bs4SPiYE Female-Narr-Video https://youtu.be/j3hTeDyvgPs.

These videos have different title pages than the original video posted before the study was conceived.

Survey design

Participants completed a survey to measure COVID-19 vaccination intention, attitudes, and understanding. Many of the survey items were similar to validated items from the literature, with some adaptations since many of the previous scales of vaccine hesitancy have been used to measure parental attitudes towards childhood vaccinations [29]. Using the Qualtrics survey platform, participants were randomly equally distributed into one of 4 paths: In paths 1, 2, and 3 the consent information page was followed by a page where participants watched the Male-Narr-Video, the Female-Narr-Video, or read the text of the video transcript presented as a blog post. On each of these pages, the time the participants spent on the page was recorded and advancing to the next page was disabled for a few minutes (approximately for half of the length of the video and two minutes for the approximately 1,200 word text) to increase the likelihood that participants engaged the material. After the information was presented in the video or text format, the participants encountered a section with questions that first asked about their intent to get vaccinated and then asked about their attitudes concerning the COVID-19 vaccines. The participants in the 4^th path did not receive any information and went directly to the vaccine intention and attitudes section.

The vaccination intention item was based on the Imperial College London “Global attitudes towards a COVID-19 vaccine” survey [4] and asked “If a COVID-19 vaccine were made available to you this week, would you get it?” Response categories were: (1) Definitely no; (2) Probably no; (3) Undecided as of now; (4) Probably yes; (5) Definitely yes; and (6) I already got one or more dose of a COVID19 vaccine. This section also included questions on multiple domains related to vaccine hesitancy attitudes: a) perceptions of vaccine safety and efficacy, b) trust in medical, scientific, governmental and pharmaceutical authorities, c) common vaccine misconceptions and d) understanding of elements of vaccination (see Supplementary Information for full survey questions). The next section of the survey asked about characteristics of the video or blog post, which included an attention check question, questions about the enjoyment of the video/blog post, and how likely participants would be to share it on social media, and was only shown to paths 1–3. This section also asked participants to rate the quality of instruction by the narrator and on the narrator’s trustworthiness, comfort and knowledge. All paths of the survey then went to a block of demographic questions and ended with a code to enter for payment on Amazon Mechanical Turk (As the most direct measure of vaccine hesitancy is the vaccination intent survey item, the analyses in this paper focus on the responses to that question. Subsequent research may examine the other survey items in greater depth).

Experimental set up

The survey was administered to 1,632 participants on Amazon Mechanical Turk on February 25^th and 26^th, 2021 when the two approved COVID-19 vaccines in the United States were the Pfizer-BioNTech and Moderna mRNA vaccines. Guided by contemporary findings that around 40% of people would get a vaccine if offered one [4] and a desire to detect approximately a 10 percentage point difference at a power of 0.8 at a 95% confidence level, we selected a sample size of 400 for each group. Although not targeted for the analytical method we use, a z test for difference between two independent proportions with these parameters yields group sample sizes of 388, which is higher than the sample size suggested for similar parameters in logistic regression [30]. The only worker qualification required was that the Turk worker be located in the United States. Mechanical Turk is well-validated and considered a reliable source for survey data [31–33]. The pay was $5, based on a $15/h wage and a 20-minute estimate of the average completion time. The study was approved by the Macalester College Institutional Review Board (Approval #022103) and all research was performed in accordance with relevant guidelines/regulations of the Macalester Institutional Review Board. Informed consent was obtained by providing participants with written consent information at the beginning of the study, and participants indicated their consent by continuing forward with the survey. Signed consent forms were not collected to preserve the anonymity of the participants. The dataset used for this analysis is available in the S1 Appendix.

Unless otherwise noted, the exclusion criteria that were applied were: exclusion of participants who had indicated they had already been vaccinated, exclusion of participants who answered the attention check question incorrectly, and exclusion of participants who spent less than 7 min on the video page (for a 7 min 37 sec video) or who spent less than 2 min 5 sec on the blog post page. These criteria resulted in exclusion of 321 (28.6%) of respondents in one of the experimental conditions, consisting of 65 excluded for both reasons, 184 excluded for the attention check alone, and 62 excluded on the basis of time on task alone. This exclusion rate is consistent with estimates of untrustworthy MTurk workers [34]. The resulting sample consists of n = 270 for the Male-Narr-Video group, n = 255 for the Female-Narr-Video group, n = 282 for the text group, and n = 393 for the control group.

Data treatment and planned analysis

Since vaccination attitudes exist on a continuum with vaccine refusal on one end, active demand on the other, and uncertainty and hesitation in the middle [35,36], we considered vaccine intention as a continuum rather than a simple hesitant/intent dichotomy. Additionally, following previous research, we considered the possibility that the effects of an intervention may differ at distinct parts of the continuum [36]. This presumption of asymmetric effects is also consistent with the idea that vaccine intent could be conceived as a series of stages, separated by thresholds, as noted in health research [37]. Given this understanding, we modeled how exposure to each intervention associated with vaccine intention by using partial proportional odds models (generalized ordered logit) with the gologit2 program in Stata using the program’s autofit option and robust standard errors [38,39]. These models preserve the order across the categories of vaccine intent while also allowing for either a single coefficient for a variable across the different levels of intent or for separate coefficients for each level of intent. Thus, some variables may associate with a consistent change in moving from one level to the next level of vaccine intention across different levels of the continuum, while other variables may have asymmetric associations, such as varying levels of intensity or direction at different levels of vaccine intention. The findings discussed below were robust to a variety of alternative specifications, as described in the Supplementary Information.

Although participants were randomly assigned to treatment conditions, we included a series of control variables (respondents’ race/ethnicity, gender, political ideology, age, and education) in our models that may associate with vaccine intention. Mechanical Turk respondents are a diverse, but not nationally representative group [32] (see S1 Table in S1 Appendix for the frequency of these variables). We excluded variables for which there were no significant associations in the multivariate models for the sake of parsimony. The race/ethnicity variables are Black and Native American compared to all others who identify as neither. While some studies suggest that Hispanic/Latinx individuals may exhibit lower vaccine intent [40,41], and while Black, Indigenous, and Hispanic/Latinx populations have all experienced histories of medical abuse that may decrease current medical trust [40–42], we found no association between Hispanic/Latinx and vaccine intent in our analysis. (This finding of no association between Latinx/Hispanic respondents and vaccine intent may be due to within-group variability that both associates with intent and decreases the likelihood of performing Mechanical Turk work tasks [42]. Additionally, the included variables may account for some of the Latinx/Hispanic within-group variation, since 9% of in-sample respondents who identified at Latinx/Hispanic also identified as Black or Native American; previous research finds of visible markers of African and indigenous ancestry [43] can associate with health-related differences among Latinx/Hispanic people.)

In addition, we included separate variables for people who described themselves as politically conservative and people who described themselves as politically liberal, comparing them to those identified politically as neither conservative nor liberal (political moderates and those who identify as something else). While trust in science is politicized, studies find differences in how political identities relate to such trust [44,45]. Although political liberals may generally trust science more than others [45], conservatives may express greater distrust than others, linked more to cultural division and collective identity [46,47], suggesting that both groups should be examined in relation to moderates and others. Political conservatives in the US have been found to have lower vaccine propensity, linked in part to patterns of trust in expertise [48]. Specifically, conservatives often see science as needing to conform to beliefs about religious authority and common sense [47], which also provide similar bases for beliefs about hierarchical, traditional gender relations [49]. Given the persistence of beliefs about gender and scientific ability [22,50–52] and findings that violating traditional gender beliefs can induce negative reactions, particularly among political conservatives [53], we test the possibility that there may be conditioning of how political conservatives receive messages delivered by female experts by using an interaction term between these variables.

Perceptions of narrator quality

We might anticipate that the differences in the patterns in how the Female-Narr-Video and Male-Narr-Video associate with vaccination intent reflect a difference in how respondents perceived the quality of the two videos; however, we found that participants as a whole rated the quality of instruction in the two versions of the video equally highly. This similarity in rating runs contrary to expectations based on previous research that has found that when quality of instruction is measured on a 10-point scale, gender gaps in favor of men are wider, particularly in male-dominated fields [23]. In contrast, we find an insignificant difference in the other direction among respondents included in the analysis. In a bivariate model, respondents who saw the Female-Narr-Video rated the quality of instruction 0.08 points higher, on average, on a 10-point scale than those who saw the Male-Narr-Video (p = 0.406, see S2 Table in S1 Appendix). Although conservatives rated the video about a quarter point lower than other respondents, there is no evidence of conditioning of conservatives’ rating based on which version of the video they saw—the nonsignificant coefficient for the interaction term between conservative and the Female-Narr-Video condition has a positive sign. The survey items on likelihood to share the video on social media, and the trustworthiness, comfort, and knowledge of the narrator (S2-S3 Figs in S1 Appendix) likewise do not show evidence of explicit bias between the male and female-narrated versions of the video. Yet, the consistent persuasive power of the Male-Narr-Video as seen in the increased intention to vaccinate, particularly for political conservatives, may be evidence of bias against women as sources of scientific authority, and is worthy of continued study.

Limitations

Design and analysis decisions suggest a degree of caution in interpreting the results that we report here. In this section, we consider the implications of some of these design and analysis decisions.

Foremost, because we excluded some respondents from analysis on the basis of time devoted to experimental conditions or failing to pass attention checks, there is a possibility that this exclusion could introduce a post-treatment bias [55]. If some unmeasured characteristic both increased the likelihood of exclusion and was associated with decreased vaccination intent, the experimental condition groups would have a lower distribution of low vaccination intent respondents than the control group and any finding of increased vaccination intent could, therefore, be an artifact of the exclusion, rather than a result of any treatment condition.

We addressed this possibility by comparing models using ordinary least squares (OLS) regression between the analytic sample reported in the main text and the full sample (with no cases excluded). OLS models allow more direct comparison of effect sizes across models than generalized ordered logit models. (We discuss these analyses in greater detail in the Supplemental Information, while summarizing key insights here.)

These analyses show that the effect associated with the Male-Narr Video treatment condition is robust across both the limited and full sample. Although there is a small change (.025 on a 5 point scale, or about 10% of the coefficient magnitude), the effect still remains statistically significant. Furthermore, both the Female-Narr Video and text treatment coefficients are statistically insignificant in both the limited and full samples and neither coefficient shows a similar change as the coefficient for Male-Narr Video when moving from the limited to full samples. This approach to eliminating the possibility of post-treatment bias based on exclusion comes with two trade-offs. First, it alters the nature of the explanatory variable from treatment to an intent to treat with the condition, which is not a direct test of the main hypothesis that the video treatment can alter vaccination intent [55]. Second, this approach increases the risk that the full sample data misestimate treatment effects, a particularly salient concern when using MTurk data. Recent analyses of MTurk data estimate that around 25% of responses could be untrustworthy, which could attenuate treatment effects by around 10% [30]. We note that our exclusion rate and the change in magnitude of the Male-Narr Video from the analytic and full sample approximate these two estimates. Finally, because our models include control variables that could correlate both with the likelihood of exclusion and change in vaccination intent, some of the risk of post-treatment bias may be controlled for in the models. We see the balance of evidence reviewed here favoring the analytic sample. That is, the available evidence is less consistent with post-treatment bias than with the full sample containing nontreatment associated noncompliance that could affect estimates. Nonetheless, we cannot rule out the possibility that some unmeasured characteristic introduced some degree of post-treatment bias. Yet, even if there were some post-treatment bias, our main finding about the robust effect of the Male-Narr Video still receives support in the full sample analysis.

The conditioning of the Female-Narr Video among politically conservative respondents does not appear to be as robust to the inclusion in the full sample when examining the OLS models. While this outcome suggests that some caution is warranted in interpreting the specific finding, it does reinforce the general conclusion that the Female-Narr Video condition has greater heterogeneity in association with vaccination intent. That is, if the finding in the models based on the analytic sample is the result of post-treatment bias, it would be a post-treatment bias unique to just the Female-Narr Video treatment. Future research could address concerns about both post-treatment bias and data quality by instituting some of the recommendations for ensuring MTurk, such as using IP addresses as a screening tool [30].

In addition, our analysis is limited in identifying the precise mechanisms that might account for variation and possible mediators of the interventions that we tested. Although we provided both video and text interventions and altered one characteristic of the video (the narrator’s gender), we have little information about the precise qualities of the video that may have been effective and how those qualities might relate to gendered expectations, suggesting many avenues for future research. For example, the use of an animated video does not provide visual cues about the narrator, which could accentuate, attenuate, or interact with vocal characteristics. Similarly, other vocal qualities, such as pitch, accent, or volume could affect how viewers receive messages. These vocal qualities could vary across or within genders, and their association with message effects might vary. For instance, the differences between lower and higher pitches could vary for male and female narrators. Since our study used just one male and one female narrator voice, follow up work can interrogate the role of these other factors on perceived scientific authority and legitimacy.

Finally, our study has focused on communication of scientific information in relation to vaccine intent. While it is suggestive of ways that members of the public may react to scientific authority, there may be contextual influences that shape responses to the video in ways that limit generalizability to vaccination intent specifically and scientific authority more generally. The widespread reach of COVID-related hospitalization, death, and economic and social disruption may shape peoples’ openness to scientific information, particularly when vaccines were first introduced and could be seen as the most likely way to end the pandemic. More generally, it could be that people react to scientific authority differently based on the salience of issues and the extent to which they are politicized. Future research on such communication could examine both more and less political and more and less personally experienced scientific topics. Such research could give more insight about whether gender, political ideology, and scientific authority interrelate across topics. Increasing the visibility of women’s roles in scientific advances may contribute to longer-term change in gendered reactions to scientific information, although as others have argued, addressing questions of inclusion more broadly may also be necessary [22].