Main text

Less than a decade ago, before the “resolution revolution,” cryo-electron microscopy (cryo-EM) was indulgently called “blobology” [1–4]. Whereas seminal work of cryo-EM experts resulted in high-resolution 3D maps and atomic models of ordered assemblies such as 2D crystals, helical arrays, and icosahedral viruses [5–11], commonly obtained 3D maps of less regular or asymmetric objects could be interpreted only in terms of global 3D architecture, domain organization, and—at most—secondary structure elements. Atomic model building was the privilege and expertise of crystallographers, requiring careful consideration of structural details such as bond geometry, steric clashes, and hydrogen bonds. Now, however, thanks to spectacular progress in both hardware and software, cryo-EM scientists suddenly face the necessity of building atomic models into near-atomic resolution maps. This unanticipated promotion from “blobologists” to “structure solvers” [12] is not as straightforward as it may seem, because model building and refinement are labor-intensive and require expertise in macromolecular structure. Spurred on by the improved resolution of newly obtained maps, the growing cryo-EM community has generated hundreds of excellent—but also some error-containing and energetically unfavorable—atomic models [1,13,14]. Such errors not only jeopardize the cryo-EM field itself but also misguide downstream research that relies on accurate molecular models, such as mutational analysis and structure-based drug design.

Although rigorous structure and model validation tools tailored for cryo-EM are currently under intense development [14,15], improving the quality of cryo-EM model building remains an important area of research. The recent introduction of computational model-building tools geared toward cryo-EM offer the possibility of automated model building [16–19]. However, building accurate models into near-atomic resolution cryo-EM maps remains a substantial challenge, because atom positions at this resolution are not unambiguous and must be inferred with aid from molecular mechanics models.

Citizen scientists have been able to contribute to challenging problems in fields such as RNA design [20], neuroscience [21], sequence alignment [22], and quantum physics [23]. Thus, one possible model-building option is Foldit (https://fold.it), a citizen science computer game that challenges players to solve complex biochemistry puzzles [24]. Recent improvements to Foldit enable players to build protein structures into crystallographic, high-resolution maps more accurately than expert crystallographers or automated model-building algorithms [25]. Unlike crystallographic maps, which often rely on phase data inferred from model coordinates, cryo-EM maps are more suitable targets for Foldit because averaged EM data are directly interpretable and are independent of the model. Here, we show that crowd-powered model building by Foldit players can indeed substantially help cryo-EM scientists.

To assess the usefulness of Foldit for cryo-EM, the players were provided with cryo-EM densities corresponding to 4 segmented subunits of the antefeeding prophage (AFP, from a soil bacterium Serratia entomophila)—Afp1, Afp5, Afp7, and Afp9 [34]. For ease of comparison, the maps were filtered to 3.2-Å resolution to avoid local quality variation and contain information up to the same resolution of 3.2 Å, which is currently considered fairly high by the cryo-EM community but still arduously low for fully automated model-building algorithms. The players tried to achieve the highest possible Foldit score, which combines the Rosetta force field with map fitting [26,27]. The structures generated by players were compared with those produced by a cryo-EM expert who created models using the manual model-building and real-space refinement software Coot [28], followed by additional real-space refinement in Phenix [29]. Structures generated by the state-of-the-art automated model-building algorithms Rosetta “denovo_density,” Phenix Map-to-Model, ARP/wARP, and Buccaneer [16–19] were included in the comparison. Standard EM validation tools and crystallographic statistics were used to evaluate the 4 approaches.

Table 1 compares the results of the various methods, using multiple criteria to evaluate both the model fit to the map and physical plausibility. Rosetta, Phenix, and Buccaneer struggled to correctly place certain chains in the appropriate density (Fig 1, S1–S11 Figs). This difficulty likely stems from errors in side-chain assignment, because the map resolution is often too poor for unambiguous side-chain identification, and these approaches fit regions of the map with incorrect sequences (Fig 1). All 4 automated methods had difficulty generating plausible geometry (Table 1, S1 Text). The Foldit structures and those generated by the microscopist produced accurate structures that were geometrically plausible and fit the maps well. Examining the models more closely shows that in most cases, the Foldit players placed slightly greater importance on bond geometry and steric clashes than the microscopist, who sacrificed these aspects for better fitting to the map (Table 1, Fig 1, S1 Text, S1 Table, and S1–S3 Figs). Although close in quality, at this resolution, we suggest it is appropriate to prioritize model geometry over map fit (Fig 1D–1G). Indeed, above 3-Å resolution, outliers are unlikely to be sufficiently supported by experimental data [14].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Comparison of model building for Afp7 in (A) an overall view and (B and C) views to compare side-chain fitting.

The Foldit structure is rendered in green, the microscopist structure in gray, the Phenix model in magenta, and Rosetta model in yellow. Because of the large deviations from the other structures, the Rosetta model is omitted in the zoomed-in views in parts B and C. The electron potential map is contoured at 2 σ. (D, E, and F) Comparison of key geometric and map fit parameters for each of the cases displayed here. (D) Comparison of Ramachandran outlier and allowed backbone conformations. (E) Comparison of Molprobity Clashscore—in both cases, lower is better. (F) Comparison of 3 different map-to-model correlation coefficients, in which higher values are better. More complete statistical analysis can be found in Table 1 and S8 Fig, with the underlying data provided in S1 Data. (G) Map-to-model FSC curves for Microscopist (gray), Foldit (green), Phenix (pink), ARP w/ARP (orange), and Buccaneer (blue) models. CC_mask, CC_box, CC_peaks, and CC_volume are correlation coefficients calculated between the model and the map. The differences between these correlation coefficients arise from whether the entire map is used (CC_box), only the map around the atomic centers (CC_mask), the molecular envelope defined by the model (CC_volume), or the strongest peaks in the model and map (CC_peaks) [14]. Afp7, antefeeding prophage 7; FSC, Fourier shell correlation; RMSD, root mean square deviation.

https://doi.org/10.1371/journal.pbio.3000472.g001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Validation scores for all models.

https://doi.org/10.1371/journal.pbio.3000472.t001

The Foldit score function appears to correctly reflect model quality. As expected, because of the absence of the phase problem, and unlike previous Foldit collaborations with crystallographic data [25,30], the structures from Foldit players that were the best in each puzzle as determined by Phenix validation [14] were also the best according to the Foldit score function, which is based on the Rosetta score function [31], with terms that model properties such as electrostatics, hydrogen bonds, solvation, and torsion angles, with an additional parameter that accounts for electron potential map fit. This observation suggests that by collaborating with Foldit, only minimal work will be required by microscopists to obtain an accurate, high-quality model.

Building models of large molecules into low-resolution data can be a time-consuming process for microscopists building structures by hand. However, in the 4 datasets presented here, Foldit players had arrived at finished structures in less than 48 hours (S16 Fig). Examining the workflow of Foldit players revealed that different players used distinct strategies in their model building. In the case of Afp9, the winning players chose to prioritize map fitting first and waited until the end to optimize the geometry of the structure (S1 Movie). Alternatively, in the case of Afp5, the winning players instead performed geometry optimization intermittently over the course of map fitting (S2 Movie). The general consensus—among the winning players who generated these 4 Foldit solutions—was to fold the protein “by hand” in the early stages of the puzzle and then run “recipes” (in-game algorithms written by the players) toward the end of the puzzle. Detailed accounts from all of the Foldit players who produced these 4 models are described in the “Foldit Player Testimonials” section in S2 Text.

These results indicate that there are multiple routes toward cryo-EM model building and that Foldit players could greatly speed the arduous model-building process for many cryo-EM projects. Although collaborating with Foldit players currently requires contacting the Foldit developers, future developments will include the ability for cryo-EM researchers the ability to communicate with Foldit players easily.

The strategy described here takes advantage of the collective ability of nonprofessional citizen scientists; however, the Foldit modeling tools are also available for individuals. Foldit Standalone runs offline on a single workstation and can be used by researchers to build and refine their structures with the Foldit scoring function [32]. Alternatively, Foldit Custom Contests can now be administered by researchers to allow online, collaborative model building and refinement among a research group or department or even a class of students [33]. Although we anticipate that for best results, researchers should draw on the collective expertise of the Foldit players, these other options may be attractive in the very competitive cryo-EM field.

To conclude, with the rapid improvement of cryo-EM map quality, it is now paramount for our building and refinement skills and tools to improve commensurably. Enlisting the help of citizen scientists, such as Foldit players, is one option to do so.

Materials and methods

To generate puzzles for Foldit players, the cryo-EM map (EMD-4782) sharpened with an overall b-factor of 105 Ų [34] was segmented around each fitted monomer of Afp1, Afp5, Afp7, and Afp9 (PDB 6rao, [34]), with a radius of 3 Šaround fitted atoms. For a detailed description of the Foldit puzzle setup and order, please see S1 Text. To calculate the FSC between models and map, a single version of an unfiltered, unsharpened segmented map was generated for each target by keeping a zone enclosing all fitted models (Microscopist, Rosetta, Phenix, Foldit) with a radius of 3 Šaround fitted atoms. The FSC was then calculated between the segmented map and a simulated map (up to Nyquist resolution with same pixel spacing) from each fitted model.

Supporting information

Acknowledgments

The authors would like to thank T. Terwilliger for assistance with map shifting, F. Dimaio and R. Moretti for useful conversations, Andriy Kryshtafovych for facilitating communication between experimentalists for the CASP13 targets and the Foldit developers, and all the Foldit players.