Potential risk of over-reporting significant framing effects

Our first goal is to systematically map existing emphasis framing experiments on climate and environmental issues. Fig 1 provides an overview of our mapping (for further details, see Methods and Table B in S1 Text). Approximately 92% (n = 111) of the framing studies we reviewed report significant main framing effects. Only 7% (n = 9) report non-significant main effects, and 1% (n = 1) do not report any main effects. Around 20% (n = 24) of all studies do not report and discuss any heterogeneous treatment effects (e.g., interactions between participants’ characteristics, such as party ideology, and framing treatments). In contrast, 70% (n = 85) of all reviewed studies identify at least one significant subgroup effect, while 10% (n = 12) report no significant subgroup effects at all. In other words, 88% (n = 85) of all studies that report on heterogeneous treatment effects (n = 97) find some significant subgroup effects.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of 121 emphasis framing experimental studies in the field of climate and environmental politics, economics and psychology published between 2007 and 6/2020.

Note: Panel designs (i.e., repeated measurements for the same study participants at two or more points in time) are used to study whether framing effects vary over time (e.g. how long the effect of a one-time exposure lasts). Competing frames are used to emphasize competing arguments in a debate (e.g., pro and contra climate mitigation messages).

https://doi.org/10.1371/journal.pclm.0000297.g001

Besides the overview provided in Fig 1, we also observe some temporal trends in the data (for further details, see Table B in S1 Text and Fig B-F in S1 Text). First, the total number of published emphasis-framing experimental studies per year has substantially increased since 2007. While in 2007 there was only one published framing experimental study, in 2019 we identified 18 framing experimental studies in our mapping. Over the review period, the share of studies that employ large-n (n>1000), non-convenience and population-representative samples has increased. However, while the size and representativeness of study samples has grown over time, studies have rarely reported statistical power calculations, especially for estimation of sub-group effects. Thus, while an increasing number of studies report significant subgroup effects (see Fig E in S1 Text), there is a risk that the estimation of these effects is underpowered. Moreover, the vast majority (120 out of 121) reviewed studies use classical linear or logistic regression models to estimate main and sub-group effects. Only one of the reviewed articles has used more advanced statistical methods, such as Bayesian sparse regression methods, to control for potential omitted interaction biases and double-check the robustness of estimated heterogenous effects. Below, we discuss the purpose and application of these methods in more depth (see “Critical appraisal and re-analysis of framing studies”).

Survey-based experiments have experienced the largest growth rate over the review period–from 0 studies using this study design type to 16 studies using survey experiments in 2019. In contrast, field- and lab-based experiments or the combination of field and lab-based experiments with survey experiments stays at a very low level (around 0–1 studies per year). While US-focused studies have the largest overall share of all experiments, they also experienced the strongest average growth rate per year between 2007 and 2020. Yet, over time also the number of (internationally) comparative framing experiments increased–but at a lower average growth rate per year. Most of these comparative framing experiments compared the US with another country, often from Europe. Even though in recent years some framing experiments have been conducted in developing and emerging economies, such as Brazil, China, or India, we still see a substantial lack of framing experiments in the developing country context.

Moreover, we can identify some trends in terms of the framing types being studied (see Table A in S1 Text and Fig B in S1 Text). In the first period from 2008 to 2013, psychological distance frames belong to the most widely studied framing types. For example, some of these earlier framing studies [50, 51] varied the spatial, social, and temporal distance of climate and environmental impacts to assess whether people support ambitious mitigation more when they perceive climate and environmental change as a proximate problem. From 2013 onwards, issue and solution frames were the most widely studied frames, with a peak of up to 10 of these experimental studies published in 2016. Issue and solution frames often emphasize environmental risks and co-benefits of environmental protection or climate mitigation. For example, some studies [10, 29] in this category highlight that emphasizing co-benefits of climate mitigation (such as technological innovation, green jobs, community building, or health improvements) could foster public support for ambitious mitigation policies. The second most widely studied framing type in our sample of reviewed studies, especially since 2013, are value- and normative-based frames that emphasize values and social norms, and attribute responsibility for environmental problems and solutions. In recent years, also source-cue frames, varying the sender of a message, and re-labeling frames, changing specific words (e.g., global warming vs. climate change) in a message, have been studied more widely. In contrast, research interest in frames varying the degree of consensus or uncertainty about climate change existence and impacts has decreased over time.

Finally, the number of articles including published replication data has increased since 2007 but none of these studies were preregistered. In the last three years, the majority of published framing experiments makes replication data publicly available. The number of articles that reported non-significant main or heterogeneous treatment effects has also increased, especially since 2013. For example, before 2013 none of the reviewed studies reported any non-significant heterogeneous framing effects, while in 2019 at least four studies did so. However, the number of reported non-significant effects has increased only very slowly, and most published studies still report only significant framing effects.

Author survey suggests that bundling non-significant and significant effects eases publication

One concern that arises in view of the large proportion of studies finding statistically significant framing effects is that there may be a file-drawer problem, where mainly significant effects are published, and non-significant ones not [22]. We implemented an online survey to assess how the authors of published framing experiments experienced the publishing process and dealt with non-significant framing effects they encountered (for further details, see Section VI in S1 Text). We contacted all 173 authors of the 121 publications via email and received a total of 63 responses (a response rate of 36% of all authors and around 52% of all reviewed studies; most often the lead author of each study responded to our survey). We find that around 80% (n = 50) of all respondents also identified non-significant effects in their framing experiments. Around 76% (n = 38) of this subset of authors tried to publish their results, including non-significant effects in peer-reviewed journals. And again, only 63% (n = 24) of this sub-subset of authors who tried to publish non-significant effects (or 48% of survey respondents that identified them) were able to successfully publish studies with non-significant effects. However, according to these authors, in most cases, publishing their findings was only possible when non-significant results were bundled together with other significant effects (for further details, see Methods). Therefore, the observed gap between the small number of published non-significant framing effects (see Fig 1 above) and the substantially larger number of identified non-significant framing effects reported by the surveyed authors strongly suggests a potential publication bias towards significant treatment results.

Lack of publicly available data makes it difficult to formally assess file-drawer problem

Formally assessing the potential existence and magnitude of a ‘file-drawer problem’ would require public access to the data and re-analyses of the original study results as part of a meta-analysis. However, only 23% (n = 28) of the 121 articles we reviewed made their data publicly available. In addition, out of those 93 reviewed articles whose data was not published, we obtained data for 29 studies by contacting authors via email (i.e., overall, we could not get access to the data of more than 53% (n = 64) of all reviewed studies). The large number of experiments that report significant framing effects without publishing data or making replication data available on request thus raises significant barriers for researchers attempting to assess the robustness of published results. For example, extra and often unsuccessful efforts to obtain access to data increase the costs of systematically re-analyzing existing studies, assessing their results’ robustness, and estimating the size of the potential file-drawer problem.

Finally, the lack of publicly available data and large variance in experimental designs prevents meaningful meta-analyses on the distribution and magnitude of average framing effects. For example, while some experimental designs use a (placebo) control group, others only compare effects for different framing groups. Also, the manifold larger and smaller variations in treatment wording and design make a proper comparison of effect sizes in a meta-analysis very difficult. Besides systematic mapping of the work in this area, we thus focus our illustrative critical appraisal on one key area of interest to many researchers: heterogeneous framing effects.

Making inferences about framing effects by sub-group

As mentioned above, climate and environmental communication researchers are often interested in how framing effects vary across population subgroups. Druckman and McGrath [2], for instance, note that “rather than continually testing the impact of one frame after another, the literature would benefit from […] investigating which types of messages resonate in light of motivations and particular prior beliefs, values and identities.” For example, in view of the possibility of directional-motivated reasoning, one prominent argument in the climate and environmental communication literature is that frames aligning with peoples’ prior beliefs reduce cognitive dissonance [24, 25] and are thus more effective at shifting public opinion about climate change.

Researchers, therefore, typically split their sample into groups based upon respondent characteristics and then re-estimate their statistical models to assess, for example, whether the framing effect is more (or less) significant for Democrats or Republicans in the United States. While this is a valid approach for generating descriptive insights regarding variation in treatment effects, researchers occasionally slip into the use of causal language. For instance, some of the reviewed studies state that “issue frames can lead Republicans and those on the political right to view climate change policy as less important” [52] or highlight that “Republicans […] increase their support if Republican politicians take leadership roles in supporting proposed bills, and if air quality benefits are emphasized” [53].

Crucially, however, random assignment of treatment does not guarantee the identification of heterogeneous causal effects [54]. In this section, we therefore outline descriptive and causal inference in the exploration of sub-group effects and the fundamental challenges to inference through the example of omitted interaction bias. We also examine how incorporating additional interaction effects in a manner suggested by previous research [38, 44, 45, 55] affects the estimates of previously published subgroup effects. Finally, we reflect on these challenges in the pursuit of causal inference for sub-group effects, while also highlighting recent work that allows for more principled descriptive examination of sub-group effects.

One example of a threat to causal inference from considering sub-group effects in isolation is omitted interaction bias [38, 44, 45, 55]. Omitted interaction bias occurs where differences between the sub-groups on other characteristics, such as age, education, and income, also result in heterogeneous treatment effects that are left unmodelled, which are absorbed by the included interaction effect. While such sub-group analysis is valid for generating descriptive inferences about how treatment effects vary across these individuals, the typical approach of researchers is to interact the treatment with the sub-group of interest, which is vulnerable to omitted interaction bias. The lack of randomized experimental manipulation and/or adjustment for other potential heterogeneous treatment effects diminish the ability of researchers to draw causal inferences about how treatment effects vary across subgroups.

In the existing literature, however, researchers often draw causal inferences from the results of sub-group analyses, where the sub-group membership is not randomized and/or other sub-group effects are left unmodelled. For example, some studies conduct sub-group analyses where they estimate treatment effects separately for Democrats and Republicans. However, party identification is not a trait that can be randomly assigned. Thus, not accounting for other characteristics that may also moderate the effect of treatments associated with party identification (e.g., age, education, gender) could lead to biased estimates. However, instead of making a descriptive statement about significant differences in framing effects between Democrats and Republicans, many studies make causal inferences and policy recommendations about which frames most successfully shift support across different partisan groups [53].

As many sub-group characteristics of interest, such as partisan identification, are difficult if not impossible to manipulate experimentally, researchers likely need to adjust for other heterogeneous framing effects in order to make causal claims. Yet, several studies [37–39, 56–58] have shown that standard specification choices and statistical methods (e.g., ordinary least squares [OLS] regressions) can run the risk of producing non-robust and noisy heterogeneous framing results because of overfitted models, even in perfectly randomized experiments [37, 38, 54]. If researchers wish to exclude this potential risk to their causal inferences, then they need to assess how sensitive published heterogeneous framing effects are to model misspecification. Note however, that these may not be all potential threats to causal inference, see for instance Bansak [54] for a generalized framework for estimating causal moderation effects and possible methods for conducting sensitivity analysis in this context.

Recent research points towards the use of machine learning in estimating such heterogeneous framing effects across population sub-group effects and prevent omitted interaction bias [38, 44, 45, 55] (for further details, see Methods). With uncertainty about the true data generation process, the number of relevant heterogeneous treatment effects and conditional effects amongst relevant covariates results in large numbers of parameters to be estimated. In such circumstances, OLS estimation faces problems of statistical efficiency. Machine learning methods for variable selection, such as Lasso, overcome this problem by setting parameters with little predictive power to zero. This improves statistical efficiency by excluding non-meaningful heterogeneous effects, thereby reducing the set of heterogeneous effects estimated. As our systematic mapping (see above) shows, the vast majority (120 out of 121) reviewed studies used classical linear or logistic regression models to estimate subgroup effects and does not make use of more advanced methods to assess or mitigate omitted interaction bias, such as LASSOplus [38].

To re-assess published framing effects along these lines, we employed one such method, LASSOplus [38]. This estimator was chosen as it is explicitly tailored to estimating heterogeneous treatment effects, as is often the goal of framing studies, with substantial simulation evidence showing its improvement upon OLS for this task. LASSOplus has also been shown to be a superior approach compared to other sparse regression techniques that seek to overcome overfitting and omitted interaction bias [38]. LASSOplus allows for simultaneous estimation of sub-group effects for all included pretreatment covariates (e.g., age, education, income, and gender) and regularizing insignificant effects to avoid overfitting (for further details, see Methods). Many studies covered by our mapping and review focus on the politically polarized country case of the United States (see Fig 1) and examine how framing effects vary by respondents’ partisanship. Hence, we decided to illustrate the sensitivity of heterogeneous framing effects by assessing the robustness of partisan subgroup effects for studies with publicly available data. From the 28 studies with publicly available data, 10 studies focus on partisan subgroup effects in the US context. We thus concentrated our re-analysis of partisan subgroup effects on these 10 available studies.

Sub-group effect estimates are sensitive to the exclusion of other sub-group effects

Fig 2 displays re-estimated subgroup effects for Democrats, Independents, and Republicans for the ten studies with publicly available data. We compare effects estimated using both classical OLS and LASSOplus [38].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Partisan Sub-Group effects are not robust.

Points indicate estimated treatment effects for sub-group framing effects by partisanship. The y-axis displays the estimated sub-group treatment effects estimated using LASSOplus that allows for all possible covariate interactions. The x-axis displays the estimated sub-group effects using OLS and not allowing for covariate interactions, equivalent to a difference-in-means test. The solid black line displays the 45 degree line, with points falling on this indicating identical estimates for the different methods of estimating sub-group effects. As many studies have multiple treatments and outcomes the number of points displayed is greater than the number of studies re-analyzed (for further details, see Methods).

https://doi.org/10.1371/journal.pclm.0000297.g002

While all of the original studies report significant partisan subgroup effects when using OLS (see x-axis of Fig 2), we find that for the vast majority of re-analyzed studies (9 out of 10) partisan sub-group effects are not statistically distinguishable from zero when using LASSOplus (see y-axis of Fig 2 and for further details, see Methods section on the exact treatment design of each study). In addition to assessing the robustness of published subgroup effects by partisanship, we also explored other potential subgroup effects (e.g., by age, education, income, gender) that were not the focus of the original studies. However, also in this explorative analysis of heterogeneous framing effects, we do not find support for robust variation in framing effects across different subgroups (see Table C-L in S1 Text).

Overall, our illustrative re-analysis of heterogeneous framing effects highlights the sensitivity of sub-group effects to the inclusion of other relevant sub-group effects. What implications do the findings of this critical appraisal of published framing effects across partisan subgroups have in the real world?

First, in general, our re-analysis suggests that other variables that correlate with partisanship also interact with framing treatments and that the significant partisan subgroup effects detected in the original studies may be absorbing other key interactions, such as between frames and age. For instance, the re-analysed studies by Saunders (id73) and Hardisty et al. (id74) both indicate significant interaction effects between their respective framing treatments and partisanship. In our critical appraisal with LASSOplus, we could, however, not replicate this result but instead found that age significantly interacted with the frames. While LASSOplus penalizes very small effect sizes to prevent overfitting and reduce omitted interaction bias, it does so less often for larger effect sizes. This implies that, for instance, Hardisty et al.’s finding that cost framing changed preferences among Republicans but not Democrats might still be existent but substantially smaller than originally expected. In this case, the strong subgroup effect for age in the reanalysis suggests that observed partisan effect could in fact be attributed to underlying differences in age among partisan groups.

Second, our analysis and discussion encourage researchers to think exactly about what type of inference they wish to make when estimating sub-group effects. Do they wish to simply describe how a treatment varies across different groups or do they seek to claim that respondents’ characteristics cause them to respond differently to the treatment. If the latter, then researchers need to reflect upon and discuss the inferential challenges for doing so and use statistical approaches such as those discussed here to assess the sensitivity of our results. If the former, then researchers should be upfront and clear with this decision, as well as use other explorative and descriptive approaches toward treatment effect heterogeneity [59, 60] to understand their findings.

Third, our analysis warrants us to think more carefully about the robustness of hypothesized subgroup effects, but also about the substantial meaning of effect sizes. To illustrate this point, we discuss the results of our re-analysis for two re-labeling framing experimental studies published by Schuldt et al.–for one [61] partisan sub-group effects were still significant and for the other [62] not. In Schuldt, Konrath and Schwarz (id86) [61], where reframing has a significant effect amongst Republicans, after adjusting for additional heterogenous treatment effects. The original study and our re-analysis indicate that Republicans were more likely to agree that the phenomenon is real when it was referred to as climate change rather than global warming. However, Democrats were not affected by the specific question-wording. In contrast, we did not observe such significant differences with the other re-labeling framing experiment results by Schuldt, Enns and Cavaliere [62]. The original study [62] makes a causal inference about subgroup effects by stating that ’the US public is more likely to doubt the existence of global warming than climate change—and that Republicans are driving the effect”. However, in this case the LASSOplus re-analysis does not support such causal inference. Why would only one of the two re-labeling experiments that varied the terms “climate change” and “global warming” lead to differences in partisan sub-group effects after adjusting for other heterogenous treatment effects? Here, it is key to consider the differences in outcome variables across the two studies, namely belief in the existence of climate change/global warming in Schuldt, Konrath and Schwarz [61] and concern about climate change/global warming in Schuldt, Enns and Cavaliere [62]. In the case of the first Schuldt study [61], effect sizes for Republicans (at least in the representative sample in 2011) were substantial. However, in the second study by Schuldt et al. [62] effect sizes of re-labeling on the concern outcome were not as large. This example thus illustrates that researchers should be careful when interpreting subgroup effects for different outcome variables in framing studies. The choice of outcome variables can affect the size of identified effects and their robustness. Additionally, given the demands of statistical power in reliably estimating sub-group effects, researchers should take care to engage in ex-ante power analysis to ensure sufficient power in the design stage of their studies or use ex-post power analysis to highlight potential challenges when exploring sub-group effects after data has been collected.

Fourth, our re-analysis can also help us to more carefully interpret the robustness of subgroup results from different study designs and timings. For example, Soutter and Mottus [63] as well as Schuldt et al. [64] point out that differences in results between their replicated re-labeling experiments were likely to stem from differences in sample composition and the large time difference between implementation of the original and replicated experiments. Given that LASSOplus penalizes very small effect sizes to prevent overfitting, it also cautions researchers to over-interpret smaller effect sizes across subgroups (e.g., partisanship) and more carefully reflect how the specific research design (e.g., self-reported or behavioral outcomes, question wording, survey or field experiment etc.) and timing (e.g., framing salience in public discourse at the time of the study) affect the experimental results.

This discussion highlights the importance of paying more attention to the size of estimated framing effects. As we show, larger effect sizes reduce the risk of drawing false causal inferences due to omitted interaction bias (e.g., falsely attributing a detected subgroup completely to differences across partisan groups rather than also age, education, or gender differences). Ultimately, policymakers and climate communicators care about the substantive effect sizes of different framing options when planning their communication strategies.

A systematic mapping of framing studies

In line with the “Preferred Reporting Items for Systematic reviews and Meta-Analyses” (PRISMA) [46] and the best-practice guidelines for systematic mapping outlined by James et al. [40] and the Collaboration for Environmental Evidence [47], we developed a review protocol (see Section III in S1 Text) and systematically reviewed framing studies in the field of environmental politics, economics and psychology according to the following three steps. We also provide a critical appraisal and reanalysis of a subset of published framing experiments that focus on heterogeneous framing effects across partisan subgroups (see below).

First, we conducted a scoping analysis of environment-related framing experiments published in a peer-reviewed scientific journal in Google Scholar, Web of Science, and personal databases (from Dr. Fesenfeld) using the following search string: (("emphasis fram*" OR "issue fram*" OR "policy fram*" OR "refram*" OR "fram* experiment" OR “information treatment” OR “communication” OR “message” OR “priming” OR “persuasive information” OR “argument”) AND (("survey" AND "experiment") OR ("field" AND "experiment") OR ("lab*" AND "experiment")) AND (“climate change” OR “environment”))

In addition, we used a forward and backward snowball technique to identify relevant framing experiments using citations and the reference lists of the reviewed articles. We limited the scope to studies that were published before or in 6/2020. We only identified relevant studies published between 2007 and 6/2020.

Second, during our scoping analysis, and in line with the PRISMA standard [46], we identified 121 peer-reviewed articles in 46 social science journals that we classified as framing experimental studies in the field of environmental politics, economics and psychology (see PRISMA scheme in Fig 3 below). The PRISMA standard aims to report systematic mappings transparently and comprises an evidence-based minimum set of reporting items. We defined the so-called Population, Intervention, Outcome and Study Design (PIOS) criteria for the inclusion and exclusion of articles in our systematic mapping (see Table 1 below) [3]. We only included studies that randomly varied (emphasis) framing treatments and assessed their effects on an individual’s climate change and environmental beliefs, attitudes, or behaviors. Our main systematic review analysis included studies that varied the information’s connotation but excluded so-called equivalence framing experiments. We however included studies that used both emphasis framing and equivalence framing experiments but then only reviewed the results from the emphasis framing experiment of these studies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Prisma flow diagram.

Third, we systematically analyzed those 121 articles by coding each of the articles according to the following criteria: a) Significant main treatment effect: To what extent did the experiment report any type of significant main treatment effect? If the study included multiple treatment groups, and at least one of these had a significant effect on the outcome variable, we coded the study as reporting significant main effects. If no main effect was reported, we marked this category as not applicable. b) Significant heterogeneous treatment effect: To what extent did the experiment report any type of significant heterogeneous treatment effect? If the study included multiple treatment groups, and at least one had a significant heterogeneous effect for population subgroups, we coded the study as reporting significant heterogeneous effects. If no heterogeneous effect was reported, we marked this category as not applicable. c) Comparative research design: To what extent did the experiment use a comparative research design? If the study focused on more than one country case, we coded the study as a comparative research design. d) Case: In which countries were the experiment(s) conducted?. e) Panel research design: To what extent did the experiment use a panel research design? We coded the study as panel research design if the study was conducted at multiple points in time (at least two data collection waves). f) Experimental design setting: What type of experimental design did the experiment use? We coded whether the study used a field-, survey- or lab-experimental design or a combination of those experimental design types. g) Competing frames: To what extent did the study use different, competing frames? We coded studies as using competing frames if they used one-sided messages and employed frames that emphasize competing arguments and subsets of information. h) Method used: To what extent did the study use an advanced statistical method to check for the results’ robustness? We coded studies using an advanced computational method if they employed LASSOplus, LASSO, Ridge Regression, or Kernel regularized least squares. i) Sample type: To what extent did the study use a convenience/population non-representative sample not or a non-convenience/population representative sample? We coded studies as convenience/non-representative sample if they study did not use a probability-based, stratified or controlled quota sampling methods to aim at representing the target population. Most of the time convenience samples in our review had a sample size of below 500 and were based on student samples. Population representative/non-convenience samples were mostly larger (n >1000). j) Published data: To what extent did the study make the data publicly available? We only coded studies as publicly available material if the authors had deposited the data in a public repository, such as Harvard Dataverse.

https://doi.org/10.1371/journal.pclm.0000297.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Population, intervention, outcome and study design (PIOS) criteria for the inclusion and exclusion of articles.

https://doi.org/10.1371/journal.pclm.0000297.t001

In contrast to emphasis framing, equivalence framing uses different, but logically equivalent phrases to label and describe an issue. An example for an equivalence frame would be to state that a person has a 10% risk of dying or a 90% chance of surviving due to climate-induced extreme weather. The rationale for focusing our review on emphasis frames is that equivalence frames are a less prominent strategy in climate and environmental communication (research) and policymaking [34, 48, 76–78]. Policymakers typically vary the emphasis on a specific subset of relevant arguments in a policy debate, rather than using logically equivalent phrases to alter public opinion. As mentioned, equivalence framing could be a powerful communication technique to alter people’s climate change and environmental attitudes and policy preferences [2, 48]. Yet, in the context of real-world climate and environmental communication and respective research is appears to be a less prominent framing strategy. Our search results support this assumption. For example, in the Web of Science and Google Scholar we only identified four experimental studies related to equivalence framing in climate and environmental communication [50, 79–81] (using the following search string: (("equivalence fram*") AND (“climate change” OR “environment”) AND (“experiment”)). Overall, there seems to exist a clear lack of experimental research on equivalence framing in climate and environmental communication and we thus focus our review here on experimental emphasis framing studies. Moreover, studies that did not use a survey-, lab- or field-experimental design or did not focus on the environmental domain were not included in the review. For example, some studies use discourse narrative analysis or observational survey analysis to detect framing effects. These studies were not included because they do not use an experimental design. Finally, while our search string was successful at identifying most relevant framing experimental studies in the field of climate and environmental communication that fulfilled our Population, Intervention, Outcome and Study Design (PIOS) criteria, with our search string we also identified a large number of irrelevant studies from other fields of research, such as engineering, computer science, or education research, which we excluded.

We trained three research assistants as coders. In addition, three of the authors also coded articles and double-checked the coding results. In the case of coding-related uncertainty, we asked coders to make comments. The authors then independently looked at these comments and came to an individual decision. Subsequently, the authors discussed these pending cases to make a final decision. Overall, we estimated an inter-coder reliability of 0.7933, implying a substantial agreement between coders.

We also qualitatively analyzed the sample articles and inductively created six framing-type groups, as presented in Fig 1 of the paper. Namely, these are “Issue/Solution Frames”, “Value/Norm/Attribution Frames”, “Re-Labelling Frames”, “Psychological Distance Frames”, “Consensus/Uncertainty Frames”, and “Source Cue Frames”. The definition of each category and relevant examples are listed in Table A in S1 Text. To clarify, the objective of making this typology was to identify the central focus of the treatment conditions in each framing experiment. For those studies that contained two types of manipulations, we coded 0.5 for each category.

Critical appraisal and re-analysis of framing studies

As described in the main body, most framing experiments study heterogeneous treatment effects across population subgroups. We thus conducted a critical appraisal to test the robustness and substantial relevance of reported heterogeneous framing effects by re-analyzing ten typical and widely cited studies that reported heterogeneous framing effects across partisanship subgroups and made replication material publicly available. These re-analyses are not representative of all published framing experiments. Unfortunately, however, the quantity of publicly available data material is very limited, so we could not fully assess existing results’ robustness. In this sense, the re-analysis’s primary goal was to investigate empirically and potentially verify our suspicion of potential bias against the reporting of non-significant effects. We also intended that the re-analysis process could familiarize applied researchers using advanced statistical methods that they may use in future communication research to check the robustness and relevance of effects.

In line with the original studies, we first used classical ordinary least squares (OLS) regressions in our re-analysis to replicate the original study results. To check for the robustness and substantive relevance of framing effects, we went beyond using these standard linear regression methods and employed a recently developed Bayesian method for variable selection in high-dimensional settings, LASSOplus [38]. Here, our premise is that robust framing effects should be detectable through different statistical methods. In essence, robust framing effects should be detectable using classical linear regressions and more advanced computational sparse regression techniques that regularize weak and noisy effects [37, 38, 82]. LASSOplus belongs to a family of advanced computational sparse regression methods developed to test the robustness and substantial relevance of (heterogeneous) experimental effects [37, 38, 82]. Such methods constitute variable selection, through the use of regularization (i.e. setting parameter estimates to zero that do not significantly contribute to predicting the outcome of interest). It thereby reduces the risk of over-fitting the estimation model. This is necessary given the large potential number of sub-group effects that could be explored when assessing the heterogeneity of treatment effects. In other words, LASSOplus penalizes weak and noisy effects to increase efficiency and thereby lessens the risk of false positives. We use LASSOplus as it is designed explicitly for the estimation of heterogeneous treatment effects. LASSOplus allows for the estimation and selection of multiple effects simultaneously, without engaging in potentially arbitrary sub-setting of data. This approach allows the researcher to include many interaction effects to avoid potential omitted interaction bias while simultaneously preventing overfitting the model [38, 44, 45]. Such approaches are supported by many Monte Carlo experiments demonstrating their performance compared to naive regression approaches [37, 38, 82].

In sum, compared to classical linear regressions, this method provides more conservative and robust estimates with credible intervals. It also permits the estimation of interaction effects that can be interpreted independently of their lower-order terms (for further details about LASSOplus–e.g., its Bayesian prior structure and regularization parameters–refer to the original methodological paper [38]). It is important to note that these more advanced computational methods are no ‘manna from heaven’ to draw causal inference. However, they can complement existing statistical methods (e.g., OLS regressions) to identify the most robust framing effects by preventing omitted interaction bias.

In our implementation of LASSOplus for each model we collected 1000 samples of the posterior distribution, using a burn-in period of 1000 samples and thinning with 10 samples. The posterior chains were inspected by the researchers to assess and ensure convergence.

In the following, we summarize the ten studies we re-analyzed in Fig 2. The re-analyzed studies include analyzed data by Stokes and Warshaw (id17); Christenson, Goldfarb and Kriner (id35); Singh and Swanson (id41); id57: Schuldt, Enns and Cavaliere (id57); Bolsen, Leeper and Shapiro (id71); Saunders (id73); Hardisty, Johnson and Weber (id74); Bolsen and Druckmann (id83); Schuldt, Konrath and Schwarz (id86); and DeGolia, Hiroyasu and Anderson (id115). Please refer to the original studies for further details about the theoretical expectations and experimental design and the supplementary information for full regression tables (see Tables C-L in S1 Text).

Stokes and Warshaw (id17) use a survey experiment to study effects on public support for different renewable portfolio standards bills by varying information about the bill’s residential electricity costs, jobs and pollution effects, as well as climate change framing and source cues. Their results indicate that all of these factors are important drivers of public support. Focusing on partisan differences in source cue framing effects, the authors indicate that Democratic (Republican) respondents are more likely to support Democratic (Republican) state legislators’ bills. Using LASSOplus, we, however, cannot find any significant subgroup effects among partisan differences.

Christenson, Goldfarb and Kriners’ (id35) study use a US nationally representative sample to conduct a survey experiment testing how information about economic and environmental costs and benefits affects fracking support. Their results provide limited evidence of motivated partisan reasoning as framing effects are most considerable for respondents with conflicting partisanship and climate change beliefs. Using LASSOplus, we can confirm that the effects only show limited evidence of motivated partisan reasoning as we could not fully reproduce these significant effects.

Singh and Swanson (id41) study the effect of the different issue- and source-cue frames on US citizens’ perceived importance of climate change policy. While the original study shows that the framing conditions did not affect climate policy’s perceived importance, it reports several significant subgroup effects for different ideological groups of the sample. Using LASSOplus, we can confirm the main treatment effects’ null findings but cannot reproduce any of the original heterogonous treatment effects.

The study by Schuldt, Enns and Cavaliere (id57) uses data from a probability-based survey experiment conducted among 1461 US adults in 2016 to test the prediction that Demoracts and Republicans respond differently whether global warming vs climate change exists. Their results show that, in the United States, Republicans are more concerned about the term "climate change" than the term "global warming". However, our re-analysis using LASSOplus did not confirm these significant heterogeneous treatment effects reported by the original study.

The study by Bolsen, Leeper and Shapiro (id71) uses a framing experiment to test whether messages highlighting social norms or mentioning science in communication affect respondents’ willingness to take action against and beliefs about global warming. Their results show that the norm- and science-based interventions strongly affect attitudes about global warming, support for policies that would reduce carbon emissions, and behavioral intentions to take voluntary action. These effects partly differ depending on party preference. However, running LASSOplus regressions does not confirm these marginally significant differences in effects between ideological groups for the confidence outcome reported by the original study.

Saunders (id73) studies Anthropogenic Global Warming conspiracy beliefs by testing both phrasing (global warming vs climate change) and motivated partisan reasoning. Results indicate that, in line with the theoretical expectations, for the case of climate change, trust moderates hoax beliefs among Republicans. However, this is not the case for global warming, where trust does not moderate conspiracy endorsement among Republicans. Re-analyzing data using LASSOplus does not confirm any of the significant main or heterogeneous treatment effects reported by the original study.

Hardisty, Johnson and Weber (id74) conducted a framing experiment among 889 Americans to assess how labelling charges for environmental costs as either an earmarked tax or an offset included as a surcharge for emitted carbon dioxide affects consumer choices. Their results indicate that cost framing changed preferences for both respondents self-identifying as Republicans and Independents. Democrats’ preferences were not significantly affected by these frames. Conversely, the LASSOplus results did not confirm any significant main or heterogeneous treatment effects reported by the original study.

The study by Bolson and Druckmann (id83) investigates the role of partisan group identity and the politicization of science in weakening the impact of a scientific-consensus-based message about human-induced climate change in the United States. Based on OLS regressions, the original study found that partisan identity and politicized messages can alter the effects of messages about the scientific consensus regarding the negative impacts of climate change. However, the LASSOplus results did not confirm any significant main or heterogeneous treatment effects reported by the original study, which employed standard linear regression techniques.

The study by Schuldt, Konrath and Schwarz (id86) uses a survey experiment with 2267 US respondents to test how different wording of global climate change (global warming vs climate change) affects whether individuals perceive the phenomenon to be real or not. Their results indicate that, as expected, Republicans were more likely to endorse that the phenomenon is real when it was referred to as climate change rather than global warming. In contrast, Democrats were not affected by the specific question-wording. This study deems an exception amongst all the re-analyzed studies as it is the only study where reframing has a significant effect amongst Republicans when using LASSOplus.

DeGolia, Hiroyasu and Anderson (id115) use a survey experiment with a two (economic, ecological) by two (gain, loss) factorial design to evaluate how different types of benefit and loss attribute frames in environmental communication affect public support. Among other subgroup analyses, their results indicate that ecological and economic frames differed based on individuals’ political ideologies and environmentalism, as conservatives were most responsive to economic messaging. At the same time, liberals were most responsive to ecological framing. For this study, we also did not find any significant subgroup effects when using LASSOplus.

Author survey

To assess how the authors of published framing experiments experienced the publishing process and dealt with non-significant framing effects they encountered, we implemented an online survey using the survey platform Qualtrics. We contacted all 173 authors of the 121 publications via email and received a total of 63 responses (a response rate of 36% of all authors and around 52% of all reviewed studies; please note that most of the time the lead authors of each study responded to our survey).

In addition to a number of closed-ended questions about the publishing experience of experimental framing studies in the field of climate and environmental communication (see Section VI in S1 Text), we also included an open-ended question in the survey to understand under which circumstances authors were able to publish non-significant results. In total, we have received 22 open-ended answers and several of these answers explicitly highlighted the importance of bundling non-significant and significant results together to increase the publication chance. For example, one authors stated “It helped that although our positive frames had no effect, our negative frames did”. Other authors stated the following: “Non-significant effects are publishable if the experiment is not the only object of interest in the study. Put another way, of course it’s hard to publish null effects if that is the only hypothesis”. Also, one author mentions: “As we had included a couple of other interesting analysis and additional minor experiments into the study design, it was easy to report the null effect together with other very interesting findings from the data.”. Also the following author states: “There were multiple analyses in the paper—some null, some significant.”. Another author stated: “While writing up and reporting both results, I think it helped in the publishing process that there was a significant effect found in addition to the null effect.”. Another author mentioned: “We had other effects that were significant, though.” and yet one other author highlighted: “It helped that although our positive frames had no effect, our negative frames did.”

We interpreted these qualitative statements as clear evidence that bundling non-significant and significant results together could yield higher publication chances. Moreover, some authors also pointed out that publishing non-significant results would be easier if these results are counter-intuitive. For example, one author stated “So that’s how we "sold" the non-significant effect: as going against the conventional wisdom.”