COVID-19 as a post-normal science (communication) context

We conceptualize journalists’ preprint use during the pandemic as a response to post-normal science (PNS) [21, 22], defined by four features, all of which apply to the COVID-19 context: i) high levels of scientific uncertainty, ii) science policy considerations that involve values (not just evidence), iii) high relevance to society, and iv) an urgent need for political decision-making. Such contexts challenge the norms of science, pushing researchers to consider and engage with an “extended peer community” that includes policy makers, journalists, and members of the public. Journalists and other science communicators must also adapt their norms and practices in PNS contexts; as Brüggemann and colleagues write, “These post-normal situations, combined with the changing media environment and a polarized society, shape and challenge the professional roles and norms that underlie their communication practices.” [20]

Scholars have documented several practices that science journalists “normally” use to find, verify, and communicate research, summarized in Table 1. These studies suggest that journalists apply a mix of passive and active strategies to find research studies; rely on triangulation, quality assessments, and the opinions of outside experts to verify them; use quotes from scientists to add context to their coverage; and strive to communicate research simply and objectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Normal and post-normal science journalism practices described in previous research.

https://doi.org/10.1371/journal.pone.0277769.t001

Scholarship on PNS journalism is more limited and has focused primarily on communication practices rather than on how journalists find or verify research. This body of literature (summarized in Table 1) proposes that journalists should strive to bring reflexivity into their reporting, communicate tensions or uncertainties, and highlight (or even advocate for) potential solutions rather than simply reporting on problems. It also suggests that journalists should contextualize new research, describe the process of science, and actively engage audiences in dialogue. However, scholarship on PNS journalism described above is largely prescriptive rather than descriptive, outlining how journalists should communicate about PNS and not how they actually do so. It has also focused primarily on climate science, a limitation given that the norms that emerge from PNS situations may differ across contexts [20]. This research contributes to filling both of these gaps by extending the PNS framework to a novel context (COVID-19 preprints) and by documenting journalists’ practices for covering this post-normal research.

Research questions and objectives

The practices, roles, and norms that emerge during PNS situations can either complement or replace existing ones [20]. This can be seen in the solutions scholars and journalists have proposed to mitigate the potential risks associated with preprint media coverage, which include consulting unaffiliated experts [35], assessing study quality with a critical eye [36], and “emphasizing the preliminary nature of conclusions” [18]. Rigorous fact checking, working closely with study authors, and using independent sources to validate research findings have also been identified as important protective measures [37–39], as has building awareness among journalists and their audiences about the nature of preprints [3]. Although many of these recommended practices, such as fact checking or consulting unaffiliated experts, are simply “basic science journalism principles” [40], others, such as labeling papers as unreviewed or helping audiences understand the process of scholarly publishing, mark a departure from traditional journalistic practice. That is, the proposed solutions for reporting on preprints represent a combination of “normal” and “post normal” activities.

Perhaps-because some of these activities are post normal, their uptake among journalists has been uneven. Studies find that media stories mentioning COVID-19-related preprints early in the pandemic inconsistently described these studies as preliminary, unreviewed, in need of verification, or a “preprint” [13, 17, 41]. Those stories that do make the preprint status of the research clear tend to offer only a brief explanation (or none at all) of what the term preprint means or how it relates to the larger academic publishing system [14, 17]. However, while these results shed some light on what audiences may encounter in preprint news coverage, they fail to capture what might be going on behind the scenes. That is, it remains unknown whether and how journalists apply other, less visible recommended practices for covering preprints, such as critical evaluation, consultation with outside experts, or use of outside sources to verify results. It is also unclear whether journalists’ coverage of COVID-19 preprints during the pandemic is an artifact of the crisis or evidence of a larger shift in journalism practice. This research aims to help fill these gaps by addressing the following research questions:

RQ1. What benefits and risks do journalists consider in deciding whether to cover preprints?

RQ2. What practices do journalists use to find, verify, and communicate the preprints they cover?

RQ3. How has the COVID-19 pandemic affected journalists’ use of preprints?

Sample

All interview participants (described below) worked for one of the following outlets: The Guardian (science section), HealthDay, IFL Science, MedPage Today, News Medical, New York Times (science section), Popular Science, and Wired. These publications were selected for their focus on science and health news, as well as their reach and popularity with readers in Canada, Europe, and the US. These outlets also represent the changing media landscape [44, 45], as they include the science sections of traditional, legacy news organizations (i.e., The Guardian, New York Times) and historically print-only science magazines (Popular Science, Wired) as well as digital native health sites (News Medical, MedPage Today) and science and health blogs (HealthDay, IFLScience).

Participants

The 19 health and science journalists who participated in this study reported on research for one or more of the previously mentioned eight outlets. We identified journalists from these publications first by collecting all the stories available through the outlet’s RSS feed or, if a feed was unavailable, through the Twitter timeline of the official account that posted a link to every story. Using these two methods, we identified stories published in the corresponding sections between March 1 and April 30, 2021. We then read each story for mentions or links to research (both preprints and peer reviewed) and saved the accompanying bylines. Scripts used to identify and save stories are openly available [46].

Recruitment

LLM randomly ordered the sampled stories in Google Sheets and recruited from bylined authors, top to bottom, from the ordered list; bylines appearing to be from organizations (e.g., American Heart Association News) and politicians were excluded. LLM gathered contact information from publicly available sources (e.g., outlet masthead or contact listing, personal website). KF emailed potential participants up to three times. 19 journalists from seven of the eight publications agreed to be interviewed (see Table 2 for participant characteristics). Recruitment and interviews occurred between July and November 2021.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of journalists who participated in interviews (n = 19) about use of preprints.

https://doi.org/10.1371/journal.pone.0277769.t002

Interviews

KF conducted semi-structured interviews of journalists via Zoom. Participants were asked about their professional experience with reporting on preprint and peer-reviewed research and how the pandemic had affected that experience and their views on the use of preprints. The interview guide is available online [47]. Interviews lasted between 10 and 47 minutes, with most averaging about 35 minutes. All interviews were recorded and then transcribed by a third-party company; transcripts were de-identified prior to analysis.

Data analysis

Our analysis was guided by Brüggemann et al.’s framework for analyzing and understanding post-normal science communication [20]. This framework comprises five analytical steps, which we address as follows:

1. Classify whether the situation has post-normal characteristics (Literature Review);
2. Document how actors (e.g. journalists) are reacting to the situation (Method, Results);
3. Compare these reactions to what would be expected in a “normal” context (Literature Review, Discussion);
4. Explain what might be causing the divergences (Discussion); and
5. Consider the societal implications of these emerging norms (Discussion).

We selected thematic analysis as our method for documenting how actors are reacting to the situation due to its overall flexibility and for its utility in identifying experiences, perspectives, and behaviors across a data set [48, 49]. Interview transcripts were de-identified, then inductively analyzed using Braun and Clarke’s steps of thematic analysis [50, 51], which allowed us to identify, examine, and report patterns in how journalists viewed and used preprints in their work. This process began with three researchers (AF, LAM, LLM) independently undertaking a close line-by-line reading of the first 12 transcripts to familiarize themselves with the data. Next, the authors independently identified initial codes, example quotes, and working definitions of the codes relevant to the research questions and informed by the literature on “normal” science communication, discussed above. All code data were managed in Google Sheets and shared amongst the research team during several collaborative video conference discussions. Guided by these discussions, and informed by the research questions and theoretical framework, a single researcher (AF) reviewed each author’s codes, identified patterns and areas of overlap, and synthesized the most relevant and common themes into a series of tables comprising working theme labels and exemplar quotes. At this point, the team reviewed these thematic tables, added comments and suggestions, then met again to discuss the findings and whether sufficiency (i.e., the point at which the collected data from participants enables researchers to answer the research question) [52, 53] had been met.

The team agreed that data collection should continue, so an additional 7 interviews were conducted, transcribed, de-identified, and coded. Based on the collective transcripts, the team agreed that data and analytical sufficiency had been achieved.

RQ1. What benefits and risks do journalists consider when deciding to cover preprints?

Across the interviews, most journalists described the decision to cover preprints as a careful risk-benefit analysis. This was true both in general (i.e., when deciding whether to cover preprints at all), as well as on a case-by-case basis (i.e., when deciding whether to cover or cite a specific preprint). At the heart of this decision was a consideration of audience needs, where preprints were described as something that should only be reported on if the potential benefits for readers outweighed possible risks of early coverage. This sentiment is captured by statements such as, “The calculation is: ‘Do we think that the audience needs to hear the story now or can they wait six to eight weeks… for the story to be peer reviewed?’” [J1].

The timely nature of preprint findings, in comparison to peer-reviewed research, was a key benefit journalists mentioned as influencing this calculation. This was particularly true during the pandemic, as many of the journalists felt that COVID-19 research with relevance to public health should be made available as soon as possible. However, some journalists saw timeliness as a benefit that extended beyond the post-normal COVID-19 context, such as J9, who reported that “preprints feel like science in motion and in creation… they’re a place to find the kind of dynamics and flux of science.” The timely nature of preprint research was also seen as valuable because it offered journalists a competitive “edge” over colleagues who relied on only peer-reviewed research. Yet, timeliness also acted as a barrier, with several journalists noting that tight deadlines prevented them from verifying the results or general empirical integrity of preprints (see below), or in some cases, from covering them at all.

Accessibility was also described as a key advantage of preprints, which are freely available, while many peer-reviewed papers are not. Journalists described accessibility as a personal benefit, as it allowed them to find and use research more easily, but also as a societal one: “That’s knowledge that it’s not, I think, ethical to be only available to rich people. Especially if it was produced in part with tax dollars, then it’s unconscionable that only somebody with a lot of money can get to it” [J16]. One journalist said they appreciated that authors of preprints were typically more accessible for interviews than authors of peer reviewed papers, presumably because they were more excited to discuss their work while the “study is fresh” [J7]. However, this was not a perspective mentioned by other participants.

Journalists considered these anticipated benefits alongside several potential risks. Chief among these was the potential for preprints to misinform, a risk many journalists noted had become particularly relevant during the pandemic. For example, J4 described how the conversation around COVID-19 and schools had become “extremely muddied by preprints,” while J16 recalled the challenge of reporting on a vaccine-related preprint in the context of “anti-vax folks” who might misuse the evidence to promote their own agendas.

This potential to misinform was closely linked to the challenge of verification, which many of the journalists noted was a barrier to their use of preprints. Journalists described that they did not possess the expertise needed to assess the reliability or accuracy of the research—a challenge shared by both journalists with an advanced degree in a Science, Technology, Engineering, or Math (STEM) field and those without. Journalists noted that preprints could change considerably between coverage and publication in an academic journal, that some may never be published at all, and that it was difficult to tell the difference between which would pass the scrutiny of peer review. Although journalists attempted to mitigate this risk with a number of verification strategies (outlined in the next section), the challenge of verification remained front of mind: “we all want to believe that we can tell what a good preprint is, from what a bad preprint is, and I don’t always think that that is true” [J4].

Perhaps as a result of their different risk-benefit calculations, journalists varied widely in the degree to which they supported using preprint research. Some were generally apprehensive, reporting that they felt it was “best to not use them” [J15]. Others were positive, noting that “preprints are now a reality. Everybody can access it. Everybody shares them” [J12]. Still others landed somewhere in the middle, stating that coverage of preprints was acceptable but only “when handled with caution” [J19]. We discuss some of these “cautious” reporting practices in the following section.

RQ2. What practices do journalists use to find, verify, and communicate the preprints they cover?

Equally varied as the risks and benefits influencing the decision to cover preprints were the strategies journalists used to find, verify, and communicate the research in their stories.

Some journalists relied on active strategies to find preprints, such as J3, who reported “us[ing] preprint repositories like arXiv and medRxiv” during the pandemic. In some cases, these active strategies were contrasted with journalists’ “normal” sourcing practices, which were often more passive (e.g. receiving press releases). Still, some journalists discovered preprints in similarly passive ways, such as through press releases, authors who mentioned them during interviews, or through services such as the UK’s Science Media Centre [54], which releases round ups of expert commentary on new research studies, including preprints. Although journalists sometimes welcomed these passive strategies for finding preprints, they were more often treated with skepticism, “because they can get a little bit promotional” [J7].

When it came to verification, almost all of the journalists said that an additional level of scrutiny was required to vet preprints than to vet peer-reviewed journal publications. This belief was closely tied to the perceived value of the peer-review process, which journalists viewed as a “safety net, basically, that the research has gone through” [J8]. Interestingly, this extra level of skepticism seldom required the use of new, post-normal verification practices, but was instead described as adherence to standard science journalism best practices. As J16 summarized:

We’re using the same toolkit, our toolkit for looking at a paper and evaluating its newsworthiness. We’re like, ‘Okay. Well, is this a good paper? Is the science good? Do the statistics make sense to us? Do the results actually answer the question that it says it answers, and what’s left out?” We ask those things of formally-published—you know, if it comes out in Science. We ask those questions, too, because sometimes the answer is ‘No’ and sometimes the answer is like, ‘Actually, this does seem dicey.’

However, some journalists suggested that they took shortcuts when using non-preprints and deadlines loomed, essentially allowing the peer-review process to replace some of the best practices used to verify information. In addition, a minority of journalists did not explicitly mention treating preprints with extra caution. One such journalist ended up questioning their lack of skepticism during the interview process, reflecting:

I’ve never really thought of [preprints] as a bad thing. And when you’re investigating it in a way that you are, it makes me wonder whether there is a lot of manipulation of the facts going on. So hopefully your study when you report it will give me a few extra clues of what I should be watching for. [J7]

The practices journalists described using to verify preprints included critically reading the methods and results, triangulating findings with those from other, ideally peer-reviewed studies, and relying on outside expertise in a process that resembled scholarly peer review:

Sometimes you need help from other people, say, and you gotta take the study, email it to some experts, and say, ‘Okay, I’m gonna do my own peer review with some peers, and we’re gonna review it.’ And it may be faster, it may not take six months, but we’re gonna take a day or two and point out some good and bad things on this study [J5].

Still, despite these multiple and varied formal strategies, intuition also played a role in journalists’ verification practices. As J4 explained, “I think that, sort of, it does come down to gut feeling—how much you trust the gut feeling of the people you’re reading.” This reliance on intuition may be linked to the time-sensitive nature of journalistic work, which, as discussed above, sometimes prevented journalists from applying best practices when verifying preprints. While some journalists were critical of colleagues and peers who relied on gut feeling, others saw intuition as an important journalistic tool that could be used to identify studies warranting additional verification. As J8 explained, if a finding “sounds a little too good to be true…it might be.”

Finally, journalists applied a range of communication practices when covering preprints. Being transparent about the unreviewed nature of the research was chief among these practices, either because of journalists’ own beliefs or because doing so was mandated by their organization. For example, journalists made comments such as, “This is something that should also be highlighted in the article, and I try to highlight it: ‘It’s a preprint. It’s not peer reviewed yet,’” (J12). Other journalists went further, adding that any disclosure of preprint status should be accompanied by an explanation of what the term meant: “if I was going to write, ‘This is not peer reviewed,’ I’d then have to—would spend at least a sentence—explaining why” [J13]. This sentiment was often closely tied to beliefs about the audiences’ level of scientific literacy:

saying whether something is or isn’t peer-reviewed—specifically for the kind of outlets that I write for which is, like national media, women’s magazines, that kind of thing—is that people probably don’t understand, like, what the significance of that is. So, if you want to be kind of legit you just have to kind of really spell out what actually happened. [J13]

In addition to this novel journalistic practice, many journalists also emphasized the importance of applying more standard best practices for science reporting, such as providing context and highlighting study limitations. Some journalists went so far as to say that they covered preprints “in the exact same way as published papers” [J12], although this perspective was uncommon.

RQ3. How has the COVID-19 pandemic affected journalists’ use of preprints?

Most journalists reported that COVID-19 had changed their use of preprints, although there was variability in the extent of these perceived changes. Some felt that the pandemic had created “a complete paradigm shift” [J12], both in their own work and in that of their peers. These journalists reported that they were not using preprints before COVID-19, but that the pandemic context “made us all just feel like it was normal and okay to be, you know, skeptically reporting on them and, and paying them a lot of attention” [J4]. Others were more moderate in their views, reflecting that they used preprints occasionally but still “don’t report on them much” [J18]. A small number of journalists said that their use of preprints had been unchanged by the pandemic, but this was not a typical perspective.

At the heart of this perceived shift were considerations of the audience’s needs and of the urgency of the crisis, with journalists offering explanations such as “…there was so little information on COVID… we need[ed] to stay really on top of this and cover things, just give the people the information that they need right now” [J3]. Yet, the shift was also described as being tied to a parallel shift within science itself. Specifically, journalists felt that “COVID revealed flaws in the publishing system” [J8] and that the pandemic had normalized preprint use among scholars as well. Other journalists believed that the pandemic had changed the quality of preprints themselves:

I would say prior to COVID…when I would come across preprints or a writer would pitch me a preprint, it was a kind of, “We’ve got this thing early, but it’s almost definitely going to be peer reviewed, it’s going to be released in ‘X’ journal three months later”… they were journal articles in waiting, really… I don’t think we were saying, ‘and this might get thrown out entirely, who knows?’ [J9]

Across these varied perspectives, journalists seemed to agree that preprint coverage had been a “net positive” [J8] in the context of the pandemic. However, they differed in the degree to which they believed, or hoped, this change would persist in the long term. Some had started covering preprints on topics other than COVID-19, or noted that even major legacy publications had started using them. Others reported that they were moving away from preprints, and that in “normal times, I probably wouldn’t go to preprints” [J19]. Still others were unsure whether preprints would remain important within journalism outside of the urgent pandemic context.

Limitations and future directions

Our own backgrounds, as with all qualitative description, shaped this analysis—both as a limitation and a strength. A former journalist (KF) led the interviews, which may have influenced participants’ responses. We conducted research at a time of relative stability during the pandemic—the initial vaccine rollout had been completed and boosters were being administered in the US, Canada, and the UK, where most of the journalists were based. It is likely that the views in this paper would differ from those of journalists interviewed at the very beginning of the pandemic. However, the timing allows for us to link the changing practices and norms of journalists to the changing (i.e., post-normal) communication context. Still, the pandemic remained very much a concern during the time of data collection and publication, with variants of COVID-19 spreading (i.e., Delta, Omicron) and creating uncertainty.

In terms of our sample, all publications in the data set were text-based (not multimedia), English only, science and health news-focused, and based in the Global North. Future research could expand outside these categories, especially given that journalistic preprint use appears to differ across geographic regions [19]. Additionally, although we included niche publishing models (i.e., HealthDay and News Medical), these models remained underrepresented in our sample. HealthDay, for instance, specialized in producing what it called “evidence-based health content” [71] to license to media companies (e.g., CNBC, U.S. News & World Reports, WebMD), hospitals, managed care organizations, publishers, nonprofits, and government agencies. However, email requests for interviews to 8 journalists with the organization went unanswered. While we cannot know why our requests were ignored, HealthDay editors and reporters may have their own norms and practices that are different from those of the journalists we interviewed. With the changing media landscape and broadening definition of “journalist” [44], more research is needed to understand differences and shared norms and practices of journalists at diverse types of outlets, including those at the “margins” of more traditional, legacy journalism [72, 73].