Climate opinion dataset.

Survey research is a time-intensive and expensive method for gathering information on public opinion. To combat this challenge, in 2015 Howe et al. [1] developed a model to predict survey responses on a variety of climate related issues. The topics covered in the survey range from risk perceptions to policy preferences. The model provides information at the state, congressional district, metropolitan, and county levels, using a small set of demographic and geographic variables [1]. Based on validation on several independently conducted surveys, the model is reported to be accurate to within seven percentage points: slightly less accurate than the typical three percentage points expected of a traditionally-conducted survey [1]. While lower in accuracy, it provides the advantage of access to highly granular and spatially expansive data. For this analysis, the predicted survey responses are used as the response variable, as the granularity of the data allows for mapping the high resolution geo-located social media data to climate opinions. With this data source, climate opinion information can be mapped at the county rather than state level and ultimately provides more data for training and validation of the proposed predictive model.

Twitter data.

To capture the topics, themes, and events associated with regional climate opinions, we extract data from Twitter user accounts. Twitter offers two well-documented Application Program Interfaces (APIs) through which data can be accessed. The Standard API allows virtually unrestricted access to tweets produced in the last seven days, with a limit on the number of requests per hour. The Premium API allows users to access the full Twitter archive, but has much more restrictive rate limits. This API is designed for users and organizations who are interested in a paid subscription.

In the interest of data accessibility, this analysis relies on the Standard API. By using tweets collected over a week long period, we ensure that all of the information required to conduct the study can be collected without a premium API subscription. One shortcoming of the Standard API is that it is not guaranteed to return every tweet in a given period of time. However, by taking advantage of different parameters in Twitter’s search function (namely specifying the ID of the most recent tweet to retrieve), the majority of tweets within a specific window can be retrieved. In two validation tests conducted over two separate three hour windows, we found that the Standard API returned at least 98% of the tweets retrieved by the Premium API.

We utilize two data subsets in training the proposed predictive model: a large corpus to train the topic model (later referred to as the topic corpus) and a smaller, geo-located corpus to derive regional topic features (referred to as the regional corpus). Because of Twitter’s privacy policy, not all tweets can be geo-located, which leads to the reduction in size from the topic to regional corpus. However, as tweets are generally short, it is important to maximize the sample size used to develop the topic model, hence the two distinct corpora.

The topic corpus consists of every tweet that matches the keywords “carbon” (excluding “carbon monoxide”), “climate”, or “global warming” (and excludes “RT”, Twitter’s representation of a retweet) in the region of interest (US, in this case) over a seven day period. In this study, the date range is April 18^th through April 25^th, 2019—notably encompassing Earth Day and the Extinction Rebellion in London. The following is an example query to illustrate how data was collected, modified to enhance readability:

where query is the specific keywords being targeted, excluding specifies words to exclude from the results, count identifies the number of tweets to return in the query (100 is the maximum), tweet_mode describes whether or not data beyond the tweet itself should be returned, and max_id specifies the most recent tweet ID to be used in the query (allowing us to exclude tweets that have already been collected, as tweet IDs are issued chronologically.

After data acquisition, the final topic corpus includes roughly 350,000 tweets. The regional corpus consists of geo-located tweets that match the aforementioned criteria. The most precise form of geo-tagging is a tweet that includes the latitude and longitude where the tweet originated, but less than 1% of tweets include this information. Another form of geo-tagging is a tweet that is associated with a specific location or a tweet that originates from a user associated with a specific location (e.g. Chicago, Illinois). For these tweets to be returned by the query, the entire region as specified by Twitter must be encompassed by the search radius. To perform geo-tagging, the search criteria is updated to include geographic coordinates and a search radius as follows:

where latitude and longitude specify the geographic center of a county and radius is the radius from the center in which to look for tweets. For the tweet to be returned by the query, it must be geo-tagged. The final regional corpus includes 190,000 tweets.

Latent Dirichlet Allocation (LDA).

To extract the key topics in the corpora of Twitter data, we utilize an unsupervised clustering technique called Latent Dirichlet Allocation (LDA). Latent Dirichlet Allocation (LDA) has been used extensively to detect topics on Twitter [20]. LDA is a probabalistic topic model, a class of Bayesian latent variable models. At a high level, it represents documents as a mixture of topics which are in turn represented by a set of words [21]. Given a corpus of documents, an LDA model learns the topic representation of each document and the words associated with each topic. Once a model is trained, given a document, the model will produce a topic likelihood distribution which identifies the relevant topic(s) in a document. For our analysis, we use a popular Natural Language Processing (NLP) Python package, Gensim [22] with the Machine Learning for Language Toolkit (MALLET) implementation [23].

We note there are a significant number of different methods for extracting topics from large text corpora, each with their own advantages and shortcomings. In the case of LDA, the algorithm assumes that each document contains multiple topics. However, in the case of tweets, which are likely to contain a limited number of topics due to character restrictions, this may not be a reasonable assumption. Previous research has proposed a modified LDA algorithm, namely Twitter-LDA, which assumes that each tweet contains exactly one topic [20]. In a comparison between LDA and Twitter-LDA on unfiltered Twitter data, where the number of topics was set at 110, the Twitter-LDA outperformed standard LDA [20]. However, the rationale for using standard LDA in this study is two-fold: (1) the goal of the LDA analysis is to derive topic distributions for distinct geographic regions; as this is an aggregate measure, by assuming that each tweet contained exactly one topic, we would limit the components of the topic distribution, potentially reducing the accuracy of the overall distribution, and (2) because the data is filtered to only pertain to climate change, the topics identified are likely to have more overlap than in a general tweet corpus and thus it was not reasonable to assume that each tweet would only contain a single topic. Here we utilize LDA as a topic model, but we emphasize that the approach presented in this study is not dependent on that specific method. In summary, the choice of the algorithm should be guided by the research question and available data.

Data preprocessing.

Latent Dirichlet Allocation (LDA) requires preprocessing each of the incoming documents. We begin this process by removing punctuation, URLs, and mentions (i.e. users referencing other users) from tweets. Though this information can be important for certain applications with Twitter data, it offers little value in the development of a topic model. After removing these features, we convert the remaining text to lowercase. Next, we remove any stopwords from the document. Stopwords (e.g., “the”, “and”, “are”, etc.) are the most common words in the English language and have little bearing on the overall meaning of a document [24]. Additionally, we remove stopwords specific to Twitter discussion. Examples include ‘&’ (the Twitter rendering of the ampersand sign) and ‘RT’ (Twitter code indicating that a tweet is a retweet, or copy, of another user’s tweet). We also remove the words “climate” and “carbon” as well as the phrase “global warming” as those were the keywords contained in the query used to construct the dataset and would otherwise be over-represented in every topic. Finally, we reduce each word to its lemma, or root. This ensures that the algorithm does not incorrectly identify different tenses or forms of a word as separate words. To perform these tasks, we use the the NLTK package in Python [25].

Following this preliminary preprocessing, we take further steps are to increase the efficiency at which the model is developed. Notably, we remove words that occur very frequently and those that occur infrequently from the set of words (i.e., dictionary) which is used to construct the topic model. Reducing the size of the dictionary, significantly improves the run-time of the algorithm, allowing for rapid testing and iteration. To reduce the size of the dictionary, we first only consider words greater than two characters. We then eliminate words that appeared in greater than 50% of tweets (e.g., “the”) as they would provide little information about the relevant topic. Next, we drop words that appeared in fewer than 100 (less than 0.03% of the dataset) of the tweets.

Finally, we leverage a randomized subset of the tweets used in preliminary model training for identification of the optimal number of topics. The size of the subset depends on the phase of model development and is specified in the following section.

Final topic model.

Topic modeling requires user input for determining the optimal number of topics. In a well-performing model, the topics are distinct and intuitive. There are a variety of metrics that can assess different aspects of model quality but there are no formalized methods to make a holistic assessment. One popular metric is coherence, which rewards similarity within a topic and contrast between topics [26]. There are several ways to compute coherence and one of the most widely used metrics is C_v [26]. This metric combines several measurements, specifically, indirect cosine measure, boolean sliding window, and normalized pointwise mutual information, and has been shown to accurately indicate the degree to which a topic can be easily interpreted [26].

To determine the ideal number of topics, we iteratively construct models over a large range of topics with a step of two and computed coherence with the C_v metric. Based on these results, we select the range of topic numbers where coherence was highest and repeat the process with a smaller range and step size of 1.

Many topic models constructed with Twitter data have over one hundred topics in the final model [20]. However, as the tweets used in this analysis already have a prioi filtering related to climate change, we use lower ranges of topics as the training corpus is unlikely to contain as many topics as an unfiltered tweet stream. In the first round of model development, we test models with 10, 12, 14, …, 50 topics. Based on these results (depicted in S2 Fig in S1 File), we then test models with 14, 15, 16, …, 22 topics.

Based on the coherence score analysis, we identify seventeen as the appropriate number of topics in the model. This set largely encompasses the breadth of Twitter discussion on climate change while minimizing overlap between topics. The final topic model makes use of the entire 350,000 tweet corpus to ensure that the training data accurately represents the extent and variety of discourse on Twitter. A concise summary of the final topics is presented in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Categories from LDA topic analysis and a brief summary of their content.

https://doi.org/10.1371/journal.pone.0245319.t001

It should be noted that the topic names and descriptions (Table 1) are subjective and represent our best effort at concisely describing the complexity of each topic category. A key challenge in LDA analysis is discerning the distinct characteristics of each topic set. The state-of-the art involves referring to the top n-grams and most representative tweets for each category (see S3 Table and S4 Fig in S1 File)—an inherently subjective task. While this analysis is based on quantitative representations of these topics, it is critical to acknowledge the subjectivity inherent in unsupervised topic models and its implications for the study’s inferences.

Response variable—first principal component, CCO.

The Yale Climate Opinion Dataset [1] contains a variety of responses on various issues related to climate change, ranging from trust in scientists to opinions on renewable energy policies. Rather than developing separate models for each response, we leverage dimensionality reduction techniques and consolidate all of the survey responses into a single variable using Principal Component Analysis (S3 Fig in S1 File).

The first principal component accounts for 89% of the variance in the data, indicating that the survey responses are highly correlated. This is not surprising since the survey questions speak to people’s beliefs related to climate change; those that believe climate change is a pressing issue are also likely to believe that policy addressing these issues is important. Because of the extremely high correlation between variables, it is reasonable to consolidate them into a single index which we coin as “Consolidate Climate Opinion (CCO)”.

We posit that CCO represents the general degree of concern for the environment and climate change related issues. Fig 1a displays tweet frequency across the nation based on climate-relevant Twitter activity collected over this period. Fig 1b shows geographic variation in climate attitudes. Based on the data used from the Yale Climate Opinion Survey (S2 Table in S1 File), regions with higher CCO values have residents who are more worried about climate change and its impacts as well as more likely to support pro-environmental regulations and policies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Distribution of Consolidated Climate Opinion (CCO) and Twitter users in the United States.

(a) County-level CCO, calculated as the principal component reduction of 14 survey questions collected in the Yale Climate Opinion Dataset [1]. (b) County-level distribution of Tweet Density (tweets / person, log scale) in the United States based on climate-related tweets collected using the Twitter API from April 18^th through April 25^th, 2019.

https://doi.org/10.1371/journal.pone.0245319.g001

Predictors—topic distributions.

To develop topic distributions for each county, we process tweets from the regional corpus using the topic model developed from the full corpus. For each county in the U.S., we filter the dataset for tweets associated with that county. We then preprocess each tweet in the filtered subset is then preprocessed in the manner described earlier. The output of an LDA topic model is a set of numbers which represent the probability that a given word from each tweet belongs in each topic. We then sum this set of numbers is then summed across all of the words to derive a so called “topic distribution” for the tweet.

To create the final county distribution, we sum topic distributions for all tweets in a county. We normalize each county distribution by dividing by its sum such that the sum of the final county distribution is one, allowing for comparison between counties. The analysis excludes counties with four or fewer tweets as the distributions generated from such a small number of tweets are not accurate representations of the county’s beliefs.

Model development & assessment.

We use a random forest model [27] to map CCO on topic distributions. Random forest is a tree-based, non-parametric model. To make predictions, the random forest algorithm builds b decision trees and averages the estimates of all of the trees to create a final estimate of the response variable (CCO) as a function of the topic distribution observed in the county of interest. This is mathematically expressed as: where T_b is an individual regression tree and x represents the input data. The random forest algorithm is effective in modeling data that exhibits complex, nonlinear relationships due to its flexibility and lack of assumptions about the underlying structure of the data [27].

The focus of this analysis is characterize the regional differences in the relationships between tweet topic distribution and survey response. A significant amount of prior literature has demonstrated that beliefs in climate change vary significantly throughout the country [1, 28] and a natural extension of previous results is to develop models for each region of the continental US. If beliefs in climate change vary by region, it follows that the relationship between social media topic portfolios and CCO would also vary regionally. Furthermore, the development of regional models can help extract nuanced relationships that may otherwise be masked in a larger, national model. Specifically, we develop a model for the Northeast, Midwest, Southeast, Southwest, Pacific, and Rocky Mountain regions.

A key component of building each of the final six regional models is variable selection, as including too many variables in a model can lead to overfitting or inaccurate inferences on the relationships between predictors and response [29]. Here, we employed the Variable Selection Using Random Forests (VSURF) algorithm [30]. This is a data-driven algorithm that ranks the importance of every candidate variable in making a prediction. VSURF defines importance as the mean reduction in mean square error observed by including a variable in the model. A variable which leads to a greater reduction in error is ranked as more important. To determine the ranking, VSURF repeatedly builds models and averages the variable importance from each iteration. Beginning with the most important variables, models are built by step-wise addition of variables in order of decreasing importance. Once the improvement in model accuracybecomes negligible, the process ends and the most important variables are returned. We developed final models for all regions using the key variables identified by the VSURF algorithm.

In addition to reducing overfitting, variable selection also improves model interpretability [29]. Though random forest models are non-parametric, there are a wide variety of tools available to characterize the relationship between predictors and response. To characterize the relationship between predictors and response for each model, we use partial dependence plots. These plots show the effect a predictor variable on the response while the effects of the other variables in the model are accounted for [31]. Mathematically, the relationship of the predictor on the response is given as: (1) where represents the trained model, n is the number of observations in the training set, and x_−j is all variables other than x_j in the training set. In order to evaluate the generalization performance of each regional model}, we conduct randomized partitioning of the data into training and test sets such that 80% of the data is used for training, and 20% for testing. The regional models are then built using the training set and validated by making predictions using the test set. Model performance is evaluated using normalized root mean square error. Model performance results are presented in the Table 2 below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Model performance based on Normalized Root Mean Square Error (NRMSE) and R² for each regional model.

https://doi.org/10.1371/journal.pone.0245319.t002

These results show that the models perform better on in-sample data as compared to the out-of-sample data, which is to be expected as the models were trained on the in-sample data. In this study, we are primarily interested in in-sample performance as we are seeking to explain the available data rather than generalize to new use cases. However, including the out-of-sample performance is important as it demonstrates that the models don’t overfit the data and possess reasonable predictive power. Additionally, each model was compared to the null (mean-only) model as a benchmark for the performance of statistical regression models. The null model does not rely on leveraging a statistical model (based on supervised learning) to estimate the response variable as a function of the independent variables; instead it assumes that the best predictions can be achieved by simply calculating the historical average value of the response variable. Table 2 shows that in sample, the models outperform the null model by an average of 59.5% and outperform the null model by an average of 27.4%, indicating that they are successfully identifying relationships in the data.