Evidence of disorientation towards immunization on online social media after contrasting political communication on vaccines. Results from an analysis of Twitter data in Italy

Samantha Ajovalasit; Veronica Maria Dorgali; Angelo Mazza; Alberto d’Onofrio; Piero Manfredi

doi:10.1371/journal.pone.0253569

Abstract

Background

In Italy, in recent years, vaccination coverage for key immunizations as MMR has been declining to worryingly low levels, with large measles outbreaks. As a response in 2017, the Italian government expanded the number of mandatory immunizations introducing penalties to unvaccinated children’s families. During the 2018 general elections campaign, immunization policy entered the political debate with the government in-charge blaming oppositions for fuelling vaccine scepticism. A new government (formerly in the opposition) established in 2018 temporarily relaxed penalties and announced the introduction of forms of flexibility.

Objectives and methods

First, we supplied a definition of disorientation, as the “lack of well-established and resilient opinions among individuals, therefore causing them to change their positions as a consequence of sufficient external perturbations”. Second, procedures for testing for the presence of both short and longer-term collective disorientation in Twitter signals were proposed. Third, a sentiment analysis on tweets posted in Italian during 2018 on immunization topics, and related polarity evaluations, were used to investigate whether the contrasting announcements at the highest political level might have originated disorientation amongst the Italian public.

Results

Vaccine-relevant tweeters’ interactions peaked in response to main political events. Out of retained tweets, 70.0% resulted favourable to vaccination, 16.4% unfavourable, and 13.6% undecided, respectively. The smoothed time series of polarity proportions exhibit frequent large changes in the favourable proportion, superimposed to a clear up-and-down trend synchronized with the switch between governments in Spring 2018, suggesting evidence of disorientation among the public.

Conclusions

The reported evidence of disorientation for opinions expressed in online social media shows that critical health topics, such as vaccination, should never be used to achieve political consensus. This is worsened by the lack of a strong Italian institutional presence on Twitter, calling for efforts to contrast misinformation and the ensuing spread of hesitancy. It remains to be seen how this disorientation will impact future parents’ vaccination decisions.

Citation: Ajovalasit S, Dorgali VM, Mazza A, d’Onofrio A, Manfredi P (2021) Evidence of disorientation towards immunization on online social media after contrasting political communication on vaccines. Results from an analysis of Twitter data in Italy. PLoS ONE 16(7): e0253569. https://doi.org/10.1371/journal.pone.0253569

Editor: Alexandre Bovet, University of Oxford, UNITED KINGDOM

Received: January 15, 2020; Accepted: June 8, 2021; Published: July 9, 2021

Copyright: © 2021 Ajovalasit et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Due to restrictions in the Twitter terms of service (https://twitter.com/en/tos) and the Twitter developer policy (https://developer.twitter.com/en/developer-terms/agreement-and-policy.html) we cannot provide the full text of tweets used in this study. However, for replication purposes, we provide the ID of every tweet used, also containing the manually labeled training set adopted; details on how to fetch tweets given their IDs are provided in https://developer.twitter.com/en/docs/tweets/post-and-engage/api-reference/get-statuses-lookup. The dataset of the Tweets IDs corresponds to the data available up to January 7 2019. Furthermore, note that any sample of tweets containing the same set of keywords listed in the manuscript and posted over the same time period is likely to yield study findings similar to those reported in the article. We extracted data available at that time from the Twitter public web interface; this data can also be purchased from Twitter via their Historical PowerTrack API http://support.gnip.com/apis/historical_api2.0/. The authors provide set of minumum requirements in form of Tweets IDs.

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

Introduction

The dramatic success of immunization programs in industrialized countries, with decades of high vaccine uptake and ensuing herd immunity, is suffering a drawback, namely the generalized fall of perceived risks arising from vaccine-preventable infectious diseases. This promotes the spread of resistance, or reluctance, to vaccination. This phenomenon, nowadays identified as “vaccine hesitancy” [1–3] is currently considered one of the top threats to global health because of its pervasive and complex nature [4]. Ensuring vaccination programs’ resilience to the hesitancy threat is a significant task of current Public Health systems.

In Italy, the MMR (measles, mumps, and rubella) vaccination coverage at 24 months, which was in the region of 91% in 2010, fell at 85.3% in 2015 and remained low after that. Parallel to this, large measles outbreaks, with 844 cases in 2016, 4,991 in 2017 (with four deaths), and 2,029 cases in the first six months of 2018 [5–7] were observed.

As a response, in the Italian National immunization plan for 2017–2019, the Italian government acted to increase the number of mandatory immunizations [8–11] by introducing penalties for non-vaccinators in the form of fines and restrictions to admittance to kindergarten and school. The decree’s ethical implications, mainly the introduction of sanctions, have been strongly challenged, especially in online social media (OSM). With the 2018 general elections, the vaccination policy flooded the political debate, with the government accusing the opposition of fuelling scepticism around vaccination. The new government, established in June 2018 and composed by a coalition between an anti-establishment movement and a far-right party, allowed, after several contrasting announcements, unvaccinated children to be admitted to school.

Over the past fifteen years, OSM emerged as a major popular source of information, including health topics [12–14]. However, within OSM, anyone can express her/his own opinion, regardless of her/his expertise in the particular topic considered. As a result, parents’ immunization decisions could be influenced by misconceptions and false information [15–17]. The massive misinformation pervading the OSM environment has been defined by the World Economic Forum as one of the main threats to current societies [15–19], in particular, because of the emergence of echo chambers, i.e., “polarised groups of like-minded people who keep framing and reinforcing a shared narrative” [16].

Although opposition to vaccination, favoured by equally misinformation, existed since the very introduction of the smallpox vaccine [20], recently, because of the increase in Internet access and the birth of the new communication platforms, misinformation is spreading at unprecedented rates [16, 21].

We focused our analysis on Twitter, a microblogging service, which is considered, as well as Facebook, a public square where anyone can express and share opinions and participate in discussions. On Twitter, user A may see user Bs’ messages without being involved in a direct relationship (“follow”). Twitter thus represents a social network and an information network at the same time. This makes Twitter different from Facebook because of its structure. For example, Facebook allows easier identification of echo chambers and homophily [22, 23].

For epidemiological purposes, Twitter data have been used for surveillance and descriptive studies, e.g., the spread of seasonal flu, the 2009 H1N1 pandemic outbreak, and the 2014 Western Africa Ebola outbreak. In all these examples, a clear correlation between the temporal spread of infections and social media interactions emerged [24].

Supported by the steadily increasing internet access, Twitter is currently one of the primary tools used by political leaders to communicate with their public [25–27]. However, this implies that when political leaders intervene on scientific subjects, such as immunization, they exert tremendous pressure on public opinion [28]. When health-related topics are the subject of political disputes, with contrasting information being massively delivered by not formally qualified persons, some individuals may be induced to change their opinions compulsively, originating a condition of disorientation. Properly defining disorientation and testing for its presence in Twitter signals is a main task of this article. A preliminary literature search on the subject suggested that the issue of “disorientation”, though ubiquitously present in many disciplines such as medical and cognitive sciences, spatial and information sciences, and social science [29–31], does not seem to have received systematic attention in the literature on information, opinions and online social media. Generally speaking, “disorientation” can be simply a consequence of the lack of adequate information, of the over-exposition to information, including misinformation, and more generally, of information disorder [32]. All these factors can make it difficult for people to filter the masses of available information properly. To simplify things and develop simple tests for the presence of disorientation in data, we assumed that disorientation (towards vaccines) could be coarsely identified as the lack of well-established and resilient opinions among individuals, therefore causing individuals to change their opinions as a consequence of sufficient external perturbations. The question then shifts on which the perturbations might be “sufficient”. Clearly, some perturbations–typically those arising as direct resp onses of the public to media news—can be very short-lasting. In relation to this, we define a concept of “short-term disorientation” as a state in which people keep changing suddenly (and often) their opinion on the debated subject because of the overwhelming impact of multiple contrasting information. However, other perturbations, such as those following from non-scientific arguments persistently promoted or supported at the highest political level, e.g., a political party, or even a government, might generate longer-term effects that we term here as “long-term disorientation”.

Consistently, in this article, we used sentiment analysis to describe the trend in communication about vaccines on Twitter in Italy throughout 2018 and to evaluate polarity in the opinions about immunization as preliminary steps to bring evidence–by appropriate statistical tests—that the prolonged phase of contrasting political announcements on a sensitive topic such as mass immunization might have originated a condition of disorientation among the Italian public.

Materials and methods

Twitter is an online social media and micro-blogging service born in 2006. Users (“tweeters”) write texts (“tweets”) of 280 characters maximum length, which are publicly visible by default until users decide to protect their tweets. According to statista.com (accessed on March 13^th, 2021), in 2021, Twitter has 340 million (estimated) active users worldwide.

Data extraction, transformation, and cleaning

We collected tweets in Italian containing at least one of a set of keywords related to vaccination behaviour and vaccine-preventable infectious diseases posted in 2018, using the Twitter Advanced Search Tool. In total, we retrieved 443,167 tweets. Keywords were chosen from a review of previous literature, and they were appropriately expanded for our purposes. Subsequently, we applied supervised classification techniques to screen out irrelevant tweets and analyze the polarity proportions of the retained ones. Consistently, we deliberately chose a broader set of keywords in order to retrieve the largest possible set of tweets and then apply finer tools to identify and leave-out noise.

Data cleaning was performed using the Python programming language. A probabilistic approach was used to re-filter tweets written in Italian; then, possible duplications were removed using the Tweets ID field with 318,371 tweets retained for the analysis. For each post, we tracked subsequent interactions by counting the number of re-tweets and likes.

Tweets classification, sentiment analysis, and training set

Sentiment analysis deals with the computational treatment of opinions, sentiments, and subjectivity within texts [33, 34]. Here, we used sentiment analysis methods for classifying tweets. In our analysis, we identified four categories: (i) favourable (F), if the tweet unambiguously showed a convinced pro-vaccine position, (ii) contrary (C), if the tweet unambiguously showed a position contrary to vaccination, (iii) undecided (U), if the tweet was neither favourable nor unfavourable, (iv)out of context (OOC), if the tweet was unrelated to immunization or if it did not fit any of the preceding categories (e.g., if it was merely spreading news or linking to another source, without expressing an opinion or a clear position). Tweets from the latter category were removed from subsequent analyses. Throughout the rest of the article, we will generically label the observed proportions of the three categories (F, C, U) (computed over any time period) as the “polarity” proportions. The sum of the contrary and undecided proportions can be taken as an estimate of the hesitant proportion in the overall Twitter population during the period considered. Notably, this is a wide population, potentially including people not involved in vaccination decisions, neither currently nor in the future, and therefore not necessarily relevant for the future vaccine coverage. Nonetheless, they represent a large population participating in a hot public debate and, therefore, relevant to opinion formation.

A supervised classification procedure [35, 36] was used to classify tweets into the four categories previously defined. First, a training set was created by manually tagging a random sample of 15,000 tweets out of the 318,371 retained for the analysis. Manual labelling was done by 15 trained university students. In particular, 15% of these 15,000 tweets were intentionally duplicated to measure the mutual (dis)agreement among annotators. The resulting accuracy was 0.6298 (CI 0.6034–0.6557), with a ‘Fleiss’ Kappa of 0.410, resulting in a fair agreement.

Next, we manually reviewed the duplicated tweets and those that showed invalid content (such as hashtag only or URL only tweets). Tweets labelled during previous explorative analysis were added. Eventually, we obtained a set of 14306 unique tweets that made the training set. In the classification process, we used unigram and bigram; we kept the hashtag (#vaccino) and removed the mentions (e.g. @screenname).

Eventually, the training set was used to compare five alternative classification models based on the following algorithms: Classification Tree, Random Forest, Naive Bayes, Support Vector Machine (SVM), and K-Nearest Neighbors.

Seeking evidence of disorientation in Twitter data.

Consistently with the proposed definition of disorientation, in what follows, we propose a few procedures aimed to test for the presence of disorientation about vaccination amongst tweeters in Italy. We distinguish between short- and long-term disorientation. The former deals with a condition in which people keep changing suddenly (and often) their opinion on the debated subject because of the overwhelming impact of short-term information disorder. The latter deals with longer-term opinion perturbations, as it can be the case when the highest political actors, e.g., a political party, or even a government, persistently promote or support non-scientific arguments, including forms of denialism, thereby generating longer-term disorientation effects, including disorientation waves.

Short-term disorientation.

To seek short-term disorientation symptoms in the data, we applied a number of tests relying on the size of the deviations (measured through the variance) from appropriately defined average opinions. The tests were conducted considering all the tweets retained, assuming they represented a random sample of an appropriate underlying super population. In particular, we proposed three different tests.

A basic multinomial test of daily tweeting trends

We applied a simple multinomial test to identify those changes in the polarity proportions resulting from randomness and separate them from those that did not. Our null hypothesis is that the (true) proportions of categories (F, C, U) were the ones observed throughout the entire year. In practice, we computed, for every day, the probability value (p-value) that the observed vector of opinion proportions is a (random) sample drawn from a multinomial population whose parameter vector is given by the overall yearly mean of the polarity proportions.

A “running” multinomial test for fast-changing opinions.

To further understand the short term changes in opinions, we tested whether each observed daily vector of polarity proportions represented a random sample drawn from a “running” multinomial population whose parameter vector is given by the average polarity proportions observed over the preceding 15 days. The figure of 15 days, representing our null hypothesis, was selected somewhat arbitrarily as a minimal duration representing a “stable” opinion (or “average preferences persistence”) in the short term. However, a sensitivity analysis was conducted to check the robustness of this choice.

A running-variance test.

Furthermore, all along the observed period, we computed a running 15-days variance of the proportion favourable to vaccination and tested (by the standard Chi-square) the null hypothesis that the 15-days variance is equal to the overall variance throughout the entire year.

Longer-term disorientation.

As for long-term perturbations, we applied a polynomial fit to the smoothed polarity proportions trend over the entire year to look for possible evidence of long-term disorientation amongst the public. Smoothing was carried out using a discrete beta-kernel based procedure proposed by [37]; the use of beta kernels allows overcoming the problem of boundary bias, commonly arising from the use of symmetric kernels. The finite support of the beta kernel function can match our time interval so that, when smoothing is made near the time interval boundaries, no weight is allocated outside the support. The smoothing bandwidth parameter has been chosen using cross-validation.

Results

Automatic data classification and polarity proportions

Among the five classification algorithms tested, the Support Vector Machine (SVM) performed best (details in the online appendix), and it was consequently adopted. Main summary results based on standard measures [38] are reported in Table 1. These measures are consistent with the classification provided by human annotation (S5 Table reported in the S1 File). As mentioned in the previous section, by selecting a broad set of keywords, we chose to retrieve a larger set of tweets and left to supervised classification algorithms the task of identifying noise. Consistently, 57.8% of the total tweets were classified as out-of-context and discarded. Of the remaining tweets, the overall proportions of classified as favorable, contrary and undecided were: F = 70.0% (CI: 61.5–74.0), C = 16.4% (CI: 12.7–25.2), U = 13.6% (CI: 8.06–20.5), respectively.

Download:

Table 1. Results of the support vector classifier (the classifier eventually selected) for the four categories considered in this work (favourable, contrary, undecided and out of context).

https://doi.org/10.1371/journal.pone.0253569.t001

Hesitancy.

The proportion of hesitant individuals in our overall Twitter population, given by the sum of the contrary and undecided proportions, resulted in 30,1%.

Institutional presence on Twitter.

The Italian Ministry of Health use of Twitter is relegated to press communications and the16.4 publication of statistics. Between 2013 and September 18^th, 2019, the Italian Ministry of Health tweeted 2,454 times (of which 172 included the word vaccin*), i.e., 25% the figure observed in France from the Ministère des Solidarités et de la Santé. Essentially the same holds for the Italian National Institute of Health.

Temporal trends.

The daily levels of Twitter interaction (including original tweets and subsequent likes or re-tweets) for 2018 (see Fig 1) show three prominent peaks, each accounting for hundreds of thousands of interactions. These three peaks represent users’ responses to well-identified triggering events. The first peak, recorded on June 22^nd, 2018, is the second-highest; it follows an Italian Minister of Interior public speech that defined the number of mandatory immunizations in the National Immunization Plan as “intolerably excessive”. Polarity proportions observed on this day were F = 71.7%, C = 15.8%, and U = 12.5%, respectively. The second peak, recorded on August 4^th, 2018, is the highest; it follows a government decree that suspends sanctions, such as non-admission to school, imposed by the previous government on unvaccinated children. Notably, whereas the number of tweets on this day exhibited a dramatic increase compared to the previous days, the underlying polarity proportions (F = 77.7%, C = 11.8%, and U = 10.5%) showed only moderate variations. The third peak (September 5^th, 2018) follows the government’s change of position on easing the sanctions on unvaccinated children (F = 73.6%, C = 14.4%, U = 12.0%). The graph in Fig 1shows a number of further lower peaks, still attributable to interventions in the political debate, over a long-term background of low-level activity.

Download:

Fig 1. Tweeting about vaccines in Italy during 2018: Time series of total daily interaction counts (tweets, likes, and re-tweets) and exact dates at main triggering political events or speeches.

https://doi.org/10.1371/journal.pone.0253569.g001

Testing the presence of disorientation

With the caveats reported above, the proportion of people “not favourable” to immunization–around 30%—was a worrying symptom of the complicated state of opinions about vaccination in Italy. The results of the various procedures proposed to investigate disorientation are reported below.

Short-term disorientation.

Using as null hypothesis the polarity proportions observed for the whole year (F = 70.0%, C = 16.4%, U = 13.6%), the basic multinomial test (Fig 2) is significant in 132 days (α = 5%), against an expectation of approximately 18 days (5% of 365).

Download:

Fig 2. Results of the basic multinomial test.

Blue circles, green squares, and purple diamonds denote the days when the null hypothesis was rejected at the significance levels of 10%, 5%, and 1%, respectively. For readability, we showed the smoothed polarity proportions. In the online appendix, we have reported in S1 Fig the real (raw) proportions used in the multinomial tests.

https://doi.org/10.1371/journal.pone.0253569.g002

The running multinomial test (Fig 3) is significant in 101 days (α = 5%), providing further evidence of instability in polarity proportions.

Download:

Fig 3. Results of the running multinomial test at 15 days.

Blue circles, green squares, and purple diamonds denote the days when the null hypothesis was rejected at the significance levels of 10%, 5%, and 1%, respectively.

https://doi.org/10.1371/journal.pone.0253569.g003

Last, the running-variance test (Fig 4) is significant in 80 days (α = 5%). In particular, significantly high variances appeared in February and March 2018, at the end of the electoral campaign, and around the voting days. In contrast, significantly low variances appeared after the new government took office and before schools opening, suggesting a possible stabilization of opinions after the transition from one government to the next.

Download:

Fig 4. 15-days running variance of the proportion favourable to vaccination (black line).

Blue circles, green squares, and purple diamonds denote the days when the null hypothesis was rejected at the significance levels of 10%, 5%, and 1%, respectively.

https://doi.org/10.1371/journal.pone.0253569.g004

Overall, the three tests performed agree in bringing statistical evidence towards a rapid shift in vaccination opinions, denoting the presence of short-term disorientation according to the first definition provided.

Smoothing and longer-term disorientation.

The smoothed time series show that many of the sudden changes of the daily polarity proportions originate from a rather small number of more stable and longer-lasting fluctuations (Fig 5). For the proportion favourable to immunization, the amplitude of these more stable oscillations is remarkable (from 60% to 76%), suggesting a substantial size of the “non-resilient” component of the population favourable to vaccination.

Download:

Fig 5. Kernel smoothing of daily polarity proportion jointly with the corresponding linear and quadratic interpolations.

Panels (a),(b),(c) report the favourable, contrary and undecided proportions, respectively.

https://doi.org/10.1371/journal.pone.0253569.g005

A stepwise polynomial fit of the smoothed polarity proportions (Fig 5) selected the parabolic function as the best one, allowing for a dramatic increase in the determination coefficient compared to the linear case, whereas higher-order functions increased R² only negligibly (R² values are reported in Fig 5).

Between January and May, the parabolic trend exhibits a clear increase of the favourable proportion (and a parallel decline in the proportions undecided and contrary), possibly reflecting the “tail” of the positive impact of the “vaccine decree” issued by the previous government, and a marked decline thereafter, when the new government had taken office, losing more than 7% by the end of the year. While we are not able to provide a direct causal link between the government change and the variation in polarity proportions, the association remains of concern in light of its political context.

Discussion

The contribution’s main objective was to investigate whether the 2018 series of contrasting announcements on immunization policy at the highest Italian political level originated disorientation amongst the Italian public. We carried out a sentiment analysis on tweets posted in Italian during 2018 containing vaccine-related keywords.

Our results are as follows. A polarity analysis showed that the proportion of tweets favourable to vaccination was about 70%, the unfavourable one about 16%, while the “undecided” accounted for 13%, in line with similar studies [39–41], yielding an estimate of the hesitant proportion in the range of 30%. As for the temporal trends of tweets, relevant interactions showed clear peaks in correspondence with vaccine-related news and political speeches, indicating that this OSM “is used as an agora for matters of public interest” [42]. Finally, as for the key category of “disorientation”, we proposed in this paper a twofold definition namely, short-term disorientation, characterised by unstable, fast-changing, opinions about vaccination, vs long-term disorientation. Our results documented the presence of short-term disorientation by a range of alternative tests. Additionally, a clear yearly trend emerged, showing that the proportion favourable to vaccination increased up to when the previous government–strongly supporting immunization over the media–was in charge (May 2018), and it started declining as soon as the new government, fostering a more ambiguous position, had taken office. We felt hard to believe that this association was unrelated to the new government’s continued and ambiguous series of announcements.

Compared to similar studies on vaccination opinions in online social media, we believe that the attempt to define and measure the concept of disorientation and document it is a major strength of the present work. We remark that available data only allowed us to test for disorientation at the “collective” (or aggregate) level. A different research design, tracking users over time (which would require data at the individual level) would in principle allow to investigate disorientation at the individual (or micro-) level.

The reported evidence of disorientation on vaccination is suggestive of the potentially harmful role played by the use of critical health topics for purposes of political consensus. Sadly, these happenings are not new, think, e.g., to the dramatic impact of the denialism of the HIV promoted by a former president of South-Africa in a critical phase of the HIV epidemic [43]. However, these aspects can become especially important due to online social media’s increasing role as a source of information (mainly misinformation) [44], which might yield social pressures eventually harmful to vaccine uptake. Said otherwise, persistent disorientation can be inflated by online misinformation, finally drifting into hesitancy. From this viewpoint, we believe that the category of disorientation will deserve future inquiry in more focused studies.

In the Italian case, the effect of disorientation might have been worsened by the almost lack, till the end of 2018, of a stable institutional presence on Twitter by Italian Public Health institutions. This fact, which appears in continuity with the traditional lack of communication between Italian public health institutions and citizens long before the digital era [14], calls for rapid public efforts in terms of an active presence on online social media, aimed to detect and contrast the spread of misinformation and the possible further spread of vaccine hesitancy [45, 46]. Clearly, in the current Italian context, with the ongoing COVID-19 third wave triggered by the new virus variants, the emergency situation has forced a temporary improvement. However, it cannot be disregarded the fact that at the end of the first pandemic wave (June-July 2020), an amazing large (41%) proportion of Italian adults declared themselves contrary to Covid-19-vaccination (https://www.cattolicanews.it/vaccino-anti-covid-italiani-poco-propensi).

Though not designed for this purpose, this analysis might provide valuable suggestions for vaccine decision-makers. Indeed, the large proportion of hesitant (in the region of 30%) should be carefully considered, if not for their potential impact on current coverage, at least for the social pressure they might enact within online social media, which might eventually feedback negatively on future coverage, as previously pinpointed.

About the limitations of the present analysis, it must be acknowledged that the definitions adopted for the concept of disorientation and its empirical investigation were given in an ad-hoc manner for the present analyses, due to the lack of literature support. From this viewpoint, the category of disorientation will deserve future inquiry in more focused studies, both conceptual and applied. We also have to mention the sub-optimal accuracy of the adopted classification, whose scores are only slightly above those that might result from a random classifier. This was mostly due to the agreement between human annotators, which was somewhat lower than expected. Nonetheless, we feel that these types of issues can often arise when dealing with controversial topics as the one covered by this work, which could trouble also well-trained human annotators.

More in general, the intrinsic limitations of Twitter data (e.g., the maximum length of texts; use of slang, abbreviations, and irony; the tendency to overcome the maximum length by subdividing a single thread into multiple tweets) have largely been acknowledged in the OSM and the related social sciences and public health literature [32, 40].

A further possible limitation lies in the fact that a small proportion of users is highly active and therefore responsible for a substantial proportion of tweets. This can introduce a bias towards the most active users.

From a broader perspective, it must be recalled that the spread of vaccine hesitancy pairs with the widespread diffusion of the so-called “Post Trust Society” [47] and of the “Post Truth Era” [48]. The present investigation can help public health policymakers better orient vaccine-related communication to mitigate the impact of vaccine hesitancy and refusal. This is, however, only a part of the story. Indeed, it is fundamental for public health systems to be able to develop real-time tools to identify fake news as well as tweets hostile to immunization—that might have the largest impact—and appropriately reply to them. This would require that public health communication agencies and institutions are also active in the real-time analysis of online media data, not just in the production of regular communication. On top of this, given the sensible role of the immunization topic, it is surely urgent to develop a moral code preventing the use of such topics for purposes of political consensus and ensuring avoidance of contradictions and ambiguities amongst government members.

A number of previous points might be worth considering in future research, by comparing the language used by tweeters (regardless of their position towards vaccination) and the language of the tweets posted by public health institutions, which represent an important aspect in the communication with agents, particularly with respect to “undecided” individuals, in order to enhance their vaccine confidence. A further point deals with the frequency of fake news spreader users. In this work, we took users as they were, without further control over their profiles. However, this is a key issue deserving careful investigation in future work. Also, the quantitative importance of the followers, which could represent a vehicle for misinformation spreading, possibly distinguished by polarity, as well as that of highly active tweeters, as it emerged in this study, is worth considering in future work on the subject.

Supporting information

S1 Fig. The figure represents the real (raw) polarity proportions of favourable (blue), contrary (purple), and undecided (green) with respect to vaccination topic.

https://doi.org/10.1371/journal.pone.0253569.s001

(TIF)

S1 File. This file contains all the classification reports for all the classifier tested (S1–S5 Tables in S1 File), the keywords adopted to retrieve the tweets (S6 Table in S1 File).

https://doi.org/10.1371/journal.pone.0253569.s002

(DOCX)

S2 File. This file contains the tweets’ Ids containing the Ids and the class of the tweet.

https://doi.org/10.1371/journal.pone.0253569.s003

(ZIP)

Acknowledgments

We warmly thank three anonymous referees and an Editor of the Journal whose valuable comments allowed us to greatly improve the quality of the manuscript. We also thank Emanuele Del Fava and Alessia Melegaro for their valuable comments on a previous draft of this work.

References

1. MacDonald NE, Eskola J, Liang X, Chaudhuri M, Dube E, Gellin B, et al. Vaccine hesitancy: Definition, scope and determinants. Vaccine. 2015;33(34):4161–4. pmid:25896383
- View Article
- PubMed/NCBI
- Google Scholar
2. Bedford H, Attwell K, Danchin M, Marshall H, Corben P, Leask J. Vaccine hesitancy, refusal and access barriers: The need for clarity in terminology. Vaccine. 2018 Oct 22;36(44):6556–8. pmid:28830694
- View Article
- PubMed/NCBI
- Google Scholar
3. MacDonald N, Dubé E, Butler R. Vaccine hesitancy terminology: A response to Bedford et al. Vaccine [Internet]. 2019;37(30):3947–8. Available from: pmid:29180029
- View Article
- PubMed/NCBI
- Google Scholar
4. https://www.who.int/emergencies/ten-threats-to-global-health-in-2019.
5. Pezzotti P, Bellino S, Prestinaci F, Iacchini S, Lucaroni F, Camoni L, et al. The impact of immunization programs on 10 vaccine preventable diseases in Italy: 1900–2015. Vaccine [Internet]. 2018;36(11):1435–43. Available from: pmid:29428176
- View Article
- PubMed/NCBI
- Google Scholar
6. Siani A. Measles outbreaks in Italy: A paradigm of the re-emergence of vaccine-preventable diseases in developed countries. Prev Med (Baltim) [Internet]. 2019;121(September 2018):99–104. Available from: pmid:30763627
- View Article
- PubMed/NCBI
- Google Scholar
7. Istituto superiore di Sanità: Morbillo & Rosolia News Rapporto N 34/2017, N 37/2018, N 43/2018.
8. Gualano MR, Bert F, Voglino G, Buttinelli E, D’Errico MM, De Waure C, et al. Attitudes towards compulsory vaccination in Italy: Results from the NAVIDAD multicentre study. Vaccine. 2018;36(23):3368–74. pmid:29729995
- View Article
- PubMed/NCBI
- Google Scholar
9. Italian Ministry of Health (2017) Piano Nazionale Prevenzione Vaccinale (PNPV) 2017–2019.
10. Italian Ministry of Health (2017) The decree on vaccine-based prevention.
11. http://www.salute.gov.it/portale/documentazione/p6_2_8_3_1.jsp?lingua=italiano&id=20
12. Kata A. Anti-vaccine activists, Web 2.0, and the postmodern paradigm—An overview of tactics and tropes used online by the anti-vaccination movement. Vaccine [Internet]. 2012;30(25):3778–89. Available from: http://dx.doi.org/10.1016/j.vaccine.2011.11.112 pmid:22172504
- View Article
- PubMed/NCBI
- Google Scholar
13. Aquino F, Donzelli G, De Franco E, Privitera G, Lopalco PL, Carducci A. The web and public confidence in MMR vaccination in Italy. Vaccine [Internet]. 2017;35(35):4494–8. Available from: http://dx.doi.org/10.1016/j.vaccine.2017.07.029 pmid:28736200
- View Article
- PubMed/NCBI
- Google Scholar
14. Benelli E. The role of the media in steering public opinion on healthcare issues. Health Policy (New York). 2003;63(2):179–86. pmid:12543530
- View Article
- PubMed/NCBI
- Google Scholar
15. Kata A. A postmodern Pandora’s box: Anti-vaccination misinformation on the Internet. Vaccine. 2010;28(7):1709–16. pmid:20045099
- View Article
- PubMed/NCBI
- Google Scholar
16. Schmidt AL, Zollo F, Scala A, Betsch C, Quattrociocchi W. Polarization of the vaccination debate on Facebook. Vaccine. 2018;36(25):3606–12. pmid:29773322
- View Article
- PubMed/NCBI
- Google Scholar
17. Zollo F, Novak PK, Del Vicario M, Bessi A, Mozetič I, Scala A, et al. Emotional dynamics in the age of misinformation. PLoS One. 2015;10(9):1–22. pmid:26422473
- View Article
- PubMed/NCBI
- Google Scholar
18. Howell L., Digital wildfires in a hyperconnected world. WEF Report 2013
19. Vicario M Del, Bessi A, Zollo F, Petroni F, Scala A, Caldarelli G, et al. The spreading of misinformation online. Proc Natl Acad Sci U S A. 2016; pmid:26729863
- View Article
- PubMed/NCBI
- Google Scholar
20. Salmon DA, Teret SP, MacIntyre CR, Salisbury D, Burgess MA, Halsey NA. Compulsory vaccination and conscientious or philosophical exemptions: Past, present, and future. Lancet. 2006 Feb 4;367(9508):436–42. pmid:16458770
- View Article
- PubMed/NCBI
- Google Scholar
21. Bessi A, Coletto M, Davidescu GA, Scala A, Caldarelli G, Quattrociocchi W. Science vs conspiracy: Collective narratives in the age of misinformation. PLoS One. 2015;10(2):1–17. pmid:25706981
- View Article
- PubMed/NCBI
- Google Scholar
22. Gottfried J, Shearer E. News Use Across Social Media Platforms 2016. 2016; http://assets.pewresearch.org/wp-content/uploads/sites/13/2016/05/PJ_2016.05.26_social-media-and-news_FINAL-1.pdf
23. Colleoni E, Rozza A, Arvidsson A. Echo Chamber or Public Sphere? Predicting Political Orientation and Measuring Political Homophily in Twitter Using Big Data. J Commun. 2014;64(2):317–32.
- View Article
- Google Scholar
24. Paul MJ, Dredze M, Broniatowski D. Twitter Improves Influenza Forecasting. PLoS Curr. 2014;1–12. pmid:25642377
- View Article
- PubMed/NCBI
- Google Scholar
25. Becatti C, Caldarelli G, Lambiotte R, Saracco F. Extracting significant signal of news consumption from social networks: the case of Twitter in Italian political elections. Palgrave Commun. 2019;5(1):1–16.
- View Article
- Google Scholar
26. Campanale M, Caldarola EG. Revealing political sentiment with Twitter: The case study of the 2016 Italian constitutional referendum. Proc 2018 IEEE/ACM Int Conf Adv Soc Networks Anal Mining, ASONAM 2018. 2018;861–8.
27. Ju A, Jeong SH, Chyi HI. Will Social Media Save Newspapers? Journal Pract [Internet]. 2014;8(1):1–17. Available from: http://dx.doi.org/10.1080/17512786.2013.794022
- View Article
- Google Scholar
28. Zhang E. J., Chughtai A. A., Heywood A., and MacIntyre C. R., (2019) Influence of political and medical leaders on parental perception of vaccination: A cross-sectional survey in Australia. BMJ Open, vol. 9, no. 3, p. e025866. pmid:30914408
- View Article
- PubMed/NCBI
- Google Scholar
29. Acemoǧlu D., Como G., Fagnani F., Ozdaglar A. (2013). “Opinion fluctuations and disagreement in social networks. Mathematics of Operations Research, 38(1), 1–27.
- View Article
- Google Scholar
30. Keller A. M., Taylor H. A., Brunyé T. T. (2020). Uncertainty promotes information-seeking actions, but what information? In Cognitive Research: Principles and Implications Vol. 5, Issue 1, pp. 1–17. pmid:32894402
- View Article
- PubMed/NCBI
- Google Scholar
31. Shi G., Proutiere A., Johansson M., Baras J. S., Johansson K. H. (2016). The Evolution of beliefs over signed social networks. Operations Research, 64(3), 585–604.
- View Article
- Google Scholar
32. Wardle C., Derakhshan H., (2017) Information Disorder Toward an interdisciplinary framework for research and policymaking, Council of Europe, http://tverezo.info/wp-content/uploads/2017/11/PREMS-162317-GBR-2018-Report-desinformation-A4-BAT.pdf
33. Sebastiani F. Machine Learning in Automated Text Categorization [Internet]. [cited 2020 Jan 14]. www.ira.uka.de/bibliography/Ai/automated.text.
34. Pang B, Lee L., Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2008 Vol.2.
- View Article
- Google Scholar
35. Tan PN, Steinbach M, Kumar V. Introduction to Data Mining. Addison Wesley, ISBN 0-321-32136-7; 2006.
36. Gokulakrishnan B., Priyanthan P., Ragavan T., Prasath N., & Perera A. (2012). Opinion mining and sentiment analysis on a Twitter data stream. International Conference on Advances in ICT for Emerging Regions, ICTer 2012—Conference Proceedings, 182–188.
37. Mazza A, Punzo A. DBKGrad: An R Package for Mortality Rates Graduation by Fixed and Adaptive Discrete Beta Kernel Techniques. 2012; Available from http://arxiv.org/abs/1211.1184
- View Article
- Google Scholar
38. Pedregosa et al. (2011) Scikit-learn: Machine Learning in Python. JMLR 12, pp. 2825–2830.
- View Article
- Google Scholar
39. Giambi C, Fabiani M, D’Ancona F, Ferrara L, Fiacchini D, Gallo T, et al. Parental vaccine hesitancy in Italy–Results from a national survey. Vaccine. 2018;36(6):779–87. pmid:29325822
- View Article
- PubMed/NCBI
- Google Scholar
40. Salathé M, Khandelwal S. Assessing vaccination sentiments with online social media: Implications for infectious disease dynamics and control. PLoS Comput Biol. 2011;7(10). pmid:22022249
- View Article
- PubMed/NCBI
- Google Scholar
41. Larson HJ, Smith DMD, Paterson P, Cumming M, Eckersberger E, Freifeld CC, et al. Measuring vaccine confidence: Analysis of data obtained by a media surveillance system used to analyze public concerns about vaccines. Lancet Infect Dis [Internet]. 2013;13(7):606–13. Available from: http://dx.doi.org/10.1016/S1473-3099(13)70108-7 pmid:23676442
- View Article
- PubMed/NCBI
- Google Scholar
42. Garimella K., De Francisci Morales G., Gionis A., Mathioudakis M. (2017). The Effect of Collective Attention on Controversial Debates on Social Media.
- View Article
- Google Scholar
43. Wang J. (2008). AIDS denialism and “The humanization of the African.” Race & Class, 49(3), 1–18.
- View Article
- Google Scholar
44. Lachlan K. A., Spence P. R., Edwards A., Reno K. M., & Edwards C. (2014). If you are quick enough, i will think about it: Information speed and trust in public health organizations. Computers in Human Behavior, 33, 377–380.
- View Article
- Google Scholar
45. Bello-Orgaz G, Hernandez-Castro J, Camacho D. Detecting discussion communities on vaccination in Twitter. Futur Gener Comput Syst. 2017;66:125–36.
- View Article
- Google Scholar
46. Karafillakis E, Larson HJ. The benefit of the doubt or doubts over benefits? A systematic literature review of perceived risks of vaccines in European populations. Vaccine [Internet]. 2017;35(37):4840–50. Available from: http://dx.doi.org/10.1016/j.vaccine.2017.07.061 pmid:28760616
- View Article
- PubMed/NCBI
- Google Scholar
47. Löfstedt RE. Risk management in post-trust societies. Risk Manag Post-Trust Soc. 2005;1–165.
- View Article
- Google Scholar
48. Keyes Ralph. The post-truth era. Dishonesty and deception in contemporary life. Macmillan 2004.
- View Article
- Google Scholar

[ref1] 1. MacDonald NE, Eskola J, Liang X, Chaudhuri M, Dube E, Gellin B, et al. Vaccine hesitancy: Definition, scope and determinants. Vaccine. 2015;33(34):4161–4. pmid:25896383
View Article
PubMed/NCBI
Google Scholar

[2] View Article

[3] PubMed/NCBI

[4] Google Scholar

[ref2] 2. Bedford H, Attwell K, Danchin M, Marshall H, Corben P, Leask J. Vaccine hesitancy, refusal and access barriers: The need for clarity in terminology. Vaccine. 2018 Oct 22;36(44):6556–8. pmid:28830694
View Article
PubMed/NCBI
Google Scholar

[6] View Article

[7] PubMed/NCBI

[8] Google Scholar

[ref3] 3. MacDonald N, Dubé E, Butler R. Vaccine hesitancy terminology: A response to Bedford et al. Vaccine [Internet]. 2019;37(30):3947–8. Available from: pmid:29180029
View Article
PubMed/NCBI
Google Scholar

[10] View Article

[11] PubMed/NCBI

[12] Google Scholar

[ref4] 4. https://www.who.int/emergencies/ten-threats-to-global-health-in-2019.

[ref5] 5. Pezzotti P, Bellino S, Prestinaci F, Iacchini S, Lucaroni F, Camoni L, et al. The impact of immunization programs on 10 vaccine preventable diseases in Italy: 1900–2015. Vaccine [Internet]. 2018;36(11):1435–43. Available from: pmid:29428176
View Article
PubMed/NCBI
Google Scholar

[15] View Article

[16] PubMed/NCBI

[17] Google Scholar

[ref6] 6. Siani A. Measles outbreaks in Italy: A paradigm of the re-emergence of vaccine-preventable diseases in developed countries. Prev Med (Baltim) [Internet]. 2019;121(September 2018):99–104. Available from: pmid:30763627
View Article
PubMed/NCBI
Google Scholar

[19] View Article

[20] PubMed/NCBI

[21] Google Scholar

[ref7] 7. Istituto superiore di Sanità: Morbillo & Rosolia News Rapporto N 34/2017, N 37/2018, N 43/2018.

[ref8] 8. Gualano MR, Bert F, Voglino G, Buttinelli E, D’Errico MM, De Waure C, et al. Attitudes towards compulsory vaccination in Italy: Results from the NAVIDAD multicentre study. Vaccine. 2018;36(23):3368–74. pmid:29729995
View Article
PubMed/NCBI
Google Scholar

[24] View Article

[25] PubMed/NCBI

[26] Google Scholar

[ref9] 9. Italian Ministry of Health (2017) Piano Nazionale Prevenzione Vaccinale (PNPV) 2017–2019.

[ref10] 10. Italian Ministry of Health (2017) The decree on vaccine-based prevention.

[ref11] 11. http://www.salute.gov.it/portale/documentazione/p6_2_8_3_1.jsp?lingua=italiano&id=20

[ref12] 12. Kata A. Anti-vaccine activists, Web 2.0, and the postmodern paradigm—An overview of tactics and tropes used online by the anti-vaccination movement. Vaccine [Internet]. 2012;30(25):3778–89. Available from: http://dx.doi.org/10.1016/j.vaccine.2011.11.112 pmid:22172504
View Article
PubMed/NCBI
Google Scholar

[31] View Article

[32] PubMed/NCBI

[33] Google Scholar

[ref13] 13. Aquino F, Donzelli G, De Franco E, Privitera G, Lopalco PL, Carducci A. The web and public confidence in MMR vaccination in Italy. Vaccine [Internet]. 2017;35(35):4494–8. Available from: http://dx.doi.org/10.1016/j.vaccine.2017.07.029 pmid:28736200
View Article
PubMed/NCBI
Google Scholar

[35] View Article

[36] PubMed/NCBI

[37] Google Scholar

[ref14] 14. Benelli E. The role of the media in steering public opinion on healthcare issues. Health Policy (New York). 2003;63(2):179–86. pmid:12543530
View Article
PubMed/NCBI
Google Scholar

[39] View Article

[40] PubMed/NCBI

[41] Google Scholar

[ref15] 15. Kata A. A postmodern Pandora’s box: Anti-vaccination misinformation on the Internet. Vaccine. 2010;28(7):1709–16. pmid:20045099
View Article
PubMed/NCBI
Google Scholar

[43] View Article

[44] PubMed/NCBI

[45] Google Scholar

[ref16] 16. Schmidt AL, Zollo F, Scala A, Betsch C, Quattrociocchi W. Polarization of the vaccination debate on Facebook. Vaccine. 2018;36(25):3606–12. pmid:29773322
View Article
PubMed/NCBI
Google Scholar

[47] View Article

[48] PubMed/NCBI

[49] Google Scholar

[ref17] 17. Zollo F, Novak PK, Del Vicario M, Bessi A, Mozetič I, Scala A, et al. Emotional dynamics in the age of misinformation. PLoS One. 2015;10(9):1–22. pmid:26422473
View Article
PubMed/NCBI
Google Scholar

[51] View Article

[52] PubMed/NCBI

[53] Google Scholar

[ref18] 18. Howell L., Digital wildfires in a hyperconnected world. WEF Report 2013

[ref19] 19. Vicario M Del, Bessi A, Zollo F, Petroni F, Scala A, Caldarelli G, et al. The spreading of misinformation online. Proc Natl Acad Sci U S A. 2016; pmid:26729863
View Article
PubMed/NCBI
Google Scholar

[56] View Article

[57] PubMed/NCBI

[58] Google Scholar

[ref20] 20. Salmon DA, Teret SP, MacIntyre CR, Salisbury D, Burgess MA, Halsey NA. Compulsory vaccination and conscientious or philosophical exemptions: Past, present, and future. Lancet. 2006 Feb 4;367(9508):436–42. pmid:16458770
View Article
PubMed/NCBI
Google Scholar

[60] View Article

[61] PubMed/NCBI

[62] Google Scholar

[ref21] 21. Bessi A, Coletto M, Davidescu GA, Scala A, Caldarelli G, Quattrociocchi W. Science vs conspiracy: Collective narratives in the age of misinformation. PLoS One. 2015;10(2):1–17. pmid:25706981
View Article
PubMed/NCBI
Google Scholar

[64] View Article

[65] PubMed/NCBI

[66] Google Scholar

[ref22] 22. Gottfried J, Shearer E. News Use Across Social Media Platforms 2016. 2016; http://assets.pewresearch.org/wp-content/uploads/sites/13/2016/05/PJ_2016.05.26_social-media-and-news_FINAL-1.pdf

[ref23] 23. Colleoni E, Rozza A, Arvidsson A. Echo Chamber or Public Sphere? Predicting Political Orientation and Measuring Political Homophily in Twitter Using Big Data. J Commun. 2014;64(2):317–32.
View Article
Google Scholar

[69] View Article

[70] Google Scholar

[ref24] 24. Paul MJ, Dredze M, Broniatowski D. Twitter Improves Influenza Forecasting. PLoS Curr. 2014;1–12. pmid:25642377
View Article
PubMed/NCBI
Google Scholar

[72] View Article

[73] PubMed/NCBI

[74] Google Scholar

[ref25] 25. Becatti C, Caldarelli G, Lambiotte R, Saracco F. Extracting significant signal of news consumption from social networks: the case of Twitter in Italian political elections. Palgrave Commun. 2019;5(1):1–16.
View Article
Google Scholar

[76] View Article

[77] Google Scholar

[ref26] 26. Campanale M, Caldarola EG. Revealing political sentiment with Twitter: The case study of the 2016 Italian constitutional referendum. Proc 2018 IEEE/ACM Int Conf Adv Soc Networks Anal Mining, ASONAM 2018. 2018;861–8.

[ref27] 27. Ju A, Jeong SH, Chyi HI. Will Social Media Save Newspapers? Journal Pract [Internet]. 2014;8(1):1–17. Available from: http://dx.doi.org/10.1080/17512786.2013.794022
View Article
Google Scholar

[80] View Article

[81] Google Scholar

[ref28] 28. Zhang E. J., Chughtai A. A., Heywood A., and MacIntyre C. R., (2019) Influence of political and medical leaders on parental perception of vaccination: A cross-sectional survey in Australia. BMJ Open, vol. 9, no. 3, p. e025866. pmid:30914408
View Article
PubMed/NCBI
Google Scholar

[83] View Article

[84] PubMed/NCBI

[85] Google Scholar

[ref29] 29. Acemoǧlu D., Como G., Fagnani F., Ozdaglar A. (2013). “Opinion fluctuations and disagreement in social networks. Mathematics of Operations Research, 38(1), 1–27.
View Article
Google Scholar

[87] View Article

[88] Google Scholar

[ref30] 30. Keller A. M., Taylor H. A., Brunyé T. T. (2020). Uncertainty promotes information-seeking actions, but what information? In Cognitive Research: Principles and Implications Vol. 5, Issue 1, pp. 1–17. pmid:32894402
View Article
PubMed/NCBI
Google Scholar

[90] View Article

[91] PubMed/NCBI

[92] Google Scholar

[ref31] 31. Shi G., Proutiere A., Johansson M., Baras J. S., Johansson K. H. (2016). The Evolution of beliefs over signed social networks. Operations Research, 64(3), 585–604.
View Article
Google Scholar

[94] View Article

[95] Google Scholar

[ref32] 32. Wardle C., Derakhshan H., (2017) Information Disorder Toward an interdisciplinary framework for research and policymaking, Council of Europe, http://tverezo.info/wp-content/uploads/2017/11/PREMS-162317-GBR-2018-Report-desinformation-A4-BAT.pdf

[ref33] 33. Sebastiani F. Machine Learning in Automated Text Categorization [Internet]. [cited 2020 Jan 14]. www.ira.uka.de/bibliography/Ai/automated.text.

[ref34] 34. Pang B, Lee L., Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2008 Vol.2.
View Article
Google Scholar

[99] View Article

[100] Google Scholar

[ref35] 35. Tan PN, Steinbach M, Kumar V. Introduction to Data Mining. Addison Wesley, ISBN 0-321-32136-7; 2006.

[ref36] 36. Gokulakrishnan B., Priyanthan P., Ragavan T., Prasath N., & Perera A. (2012). Opinion mining and sentiment analysis on a Twitter data stream. International Conference on Advances in ICT for Emerging Regions, ICTer 2012—Conference Proceedings, 182–188.

[ref37] 37. Mazza A, Punzo A. DBKGrad: An R Package for Mortality Rates Graduation by Fixed and Adaptive Discrete Beta Kernel Techniques. 2012; Available from http://arxiv.org/abs/1211.1184
View Article
Google Scholar

[104] View Article

[105] Google Scholar

[ref38] 38. Pedregosa et al. (2011) Scikit-learn: Machine Learning in Python. JMLR 12, pp. 2825–2830.
View Article
Google Scholar

[107] View Article

[108] Google Scholar

[ref39] 39. Giambi C, Fabiani M, D’Ancona F, Ferrara L, Fiacchini D, Gallo T, et al. Parental vaccine hesitancy in Italy–Results from a national survey. Vaccine. 2018;36(6):779–87. pmid:29325822
View Article
PubMed/NCBI
Google Scholar

[110] View Article

[111] PubMed/NCBI

[112] Google Scholar

[ref40] 40. Salathé M, Khandelwal S. Assessing vaccination sentiments with online social media: Implications for infectious disease dynamics and control. PLoS Comput Biol. 2011;7(10). pmid:22022249
View Article
PubMed/NCBI
Google Scholar

[114] View Article

[115] PubMed/NCBI

[116] Google Scholar

[ref41] 41. Larson HJ, Smith DMD, Paterson P, Cumming M, Eckersberger E, Freifeld CC, et al. Measuring vaccine confidence: Analysis of data obtained by a media surveillance system used to analyze public concerns about vaccines. Lancet Infect Dis [Internet]. 2013;13(7):606–13. Available from: http://dx.doi.org/10.1016/S1473-3099(13)70108-7 pmid:23676442
View Article
PubMed/NCBI
Google Scholar

[118] View Article

[119] PubMed/NCBI

[120] Google Scholar

[ref42] 42. Garimella K., De Francisci Morales G., Gionis A., Mathioudakis M. (2017). The Effect of Collective Attention on Controversial Debates on Social Media.
View Article
Google Scholar

[122] View Article

[123] Google Scholar

[ref43] 43. Wang J. (2008). AIDS denialism and “The humanization of the African.” Race & Class, 49(3), 1–18.
View Article
Google Scholar

[125] View Article

[126] Google Scholar

[ref44] 44. Lachlan K. A., Spence P. R., Edwards A., Reno K. M., & Edwards C. (2014). If you are quick enough, i will think about it: Information speed and trust in public health organizations. Computers in Human Behavior, 33, 377–380.
View Article
Google Scholar

[128] View Article

[129] Google Scholar

[ref45] 45. Bello-Orgaz G, Hernandez-Castro J, Camacho D. Detecting discussion communities on vaccination in Twitter. Futur Gener Comput Syst. 2017;66:125–36.
View Article
Google Scholar

[131] View Article

[132] Google Scholar

[ref46] 46. Karafillakis E, Larson HJ. The benefit of the doubt or doubts over benefits? A systematic literature review of perceived risks of vaccines in European populations. Vaccine [Internet]. 2017;35(37):4840–50. Available from: http://dx.doi.org/10.1016/j.vaccine.2017.07.061 pmid:28760616
View Article
PubMed/NCBI
Google Scholar

[134] View Article

[135] PubMed/NCBI

[136] Google Scholar

[ref47] 47. Löfstedt RE. Risk management in post-trust societies. Risk Manag Post-Trust Soc. 2005;1–165.
View Article
Google Scholar

[138] View Article

[139] Google Scholar

[ref48] 48. Keyes Ralph. The post-truth era. Dishonesty and deception in contemporary life. Macmillan 2004.
View Article
Google Scholar

[141] View Article

[142] Google Scholar

Figures

Abstract

Background

Objectives and methods

Results

Conclusions

Introduction

Materials and methods

Data extraction, transformation, and cleaning

Tweets classification, sentiment analysis, and training set

Seeking evidence of disorientation in Twitter data.

Short-term disorientation.

A basic multinomial test of daily tweeting trends

A “running” multinomial test for fast-changing opinions.

A running-variance test.

Longer-term disorientation.

Results

Automatic data classification and polarity proportions

Hesitancy.

Institutional presence on Twitter.

Temporal trends.

Testing the presence of disorientation

Short-term disorientation.

Smoothing and longer-term disorientation.

Discussion

Supporting information

S1 Fig. The figure represents the real (raw) polarity proportions of favourable (blue), contrary (purple), and undecided (green) with respect to vaccination topic.

S1 File. This file contains all the classification reports for all the classifier tested (S1–S5 Tables in S1 File), the keywords adopted to retrieve the tweets (S6 Table in S1 File).

S2 File. This file contains the tweets’ Ids containing the Ids and the class of the tweet.

Acknowledgments

References