Overview of our methodology

An overview of our methodology is depicted in Fig. 1. As can be seen, there are four major stages: dataset construction, feature extraction, feature selection and zinc-binding prediction. At the first stage of dataset construction, all the four major types of zinc-binding residues, i.e. CHEDs are selected from the target proteins in the curated benchmark dataset. In the second stage, these CHEDs are further encoded into feature vectors which are represented based on a local window of residues centered at each selected CHED by making use of four generic categories of features: sequence, 3D structure, network and other features. In the third stage, we perform extensive feature selection to characterize the most important and contributive features that are relevant to the prediction by recursively training RF classifiers and examining their prediction accuracy. The classifiers are trained based on diverse residue- and protein-based features. In the final stage, we optimize the RF-based classifiers using the selected important features and apply the trained classifiers to predict zinc-binding residues and compare the performance of our method with the other two methods.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Overview of our methodology.

There are four major stages: dataset construction, feature extraction, feature selection and zinc binding prediction.

https://doi.org/10.1371/journal.pone.0049716.g001

Dataset

The benchmark dataset used in this study was derived from a protein dataset that was previously prepared by Passerini et al. [22]. This dataset contains 2,727 protein chains with zero HSSP, i.e. no pair of proteins had HSSP value>0 by running UniqueProt [23]. The HSSP value measures the pairwise sequence similarity between the two sequences by taking into account both sequence identity and sequence length [23]. We extracted proteins with a resolution better than 2.5 Å, which resulted in a final reduced dataset of 1,103 chains.

We finally selected a total of 49,631 CHEDs from the 1,103 chains. Among them, 144 chains bind to at least one zinc ion (See Table 1 for statistics), 79% of which are Zn3 or Zn4 binding chains (defined as “Znx”, where “x” denotes the number of amino acid residues bound to zinc ions). The total number of residues binding to zinc is 484. Among these residues, more than 86% of the selected Cys, His, Glu and Asp residues bind to Zn3 or Zn4. Residues are considered as binding to zinc if the distance between the atoms of the residue and zinc [24] is within a threshold of 3 Å. There are totally 223 zinc atoms binding to the selected 144 chains, as shown in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Statistics of four zinc-binding residue types (CHED) grouped based on the type of zinc ions.

https://doi.org/10.1371/journal.pone.0049716.t001

Zinc ions of zinc metalloproteins are significant for their catalytic and structural functions. Zinc ions that bind to four residues (defined as “Zn4”) are considered as structural zinc, which maintain the stability of the protein but are not involved in any biochemical reaction. Zinc ions bind to three residues (defined as “Zn3”) are generally catalytic zinc, which bind to molecules and may be involved in a reaction. Except for Zn3 and Zn4, other zinc ions require cooperation with other metal ions in order to fulfill the function, usually linking with side chain atoms or water molecules. These zinc ions are called co-catalytic zinc (Co-catalytic Zn) [12]. Apart from the above zinc types, some zinc ions only bind to one or two residues; they are located on the surface of the zinc metalloproteins and lack the biological function. In this study, we only focused on the functional zinc ions (i.e. Zn3, Zn4 and Co-catalytic Zn which have catalytic or structural roles [25], [26]). In summary, CHEDs binding to these zinc ions are positive samples (484 residues, 130 C, 220 H, 80 E and 54 D) (Table 1), the remaining selected CHEDs are negative samples (49,147 residues).

We also generated an independent test set. We randomly divided the 484 positive samples into six parts, one of which was used as the independent test set while the remaining five parts were merged together and used as the benchmark dataset which was used for feature selection and 5-fold cross-validation test purposes. The negative samples were randomly selected with the negative to positive ratio of 6∶1 for both benchmark training and independent test datasets. Note that the independent test dataset was never used during the model training and feature selection process; the prediction model was trained based on the benchmark dataset and then applied to the independent test dataset for its performance evaluation.

Moreover, an apo protein structure dataset was produced according to the holo structures included in the current datasets, using the same strategy as described by Ebert and Altman [4]. More specifically, we first searched the PDB database by detecting the apo protein structures that do not contain zinc and share 95% or more sequence identity with the holo structures. We then identified the residues in the apo structures that correspond to those binding zinc in the holo form. Finally, we obtained a set of 31 apo/holo structures containing 96 zinc-binding residues. For evaluating the prediction performance on the apo protein structure set, the ratio of negative to positive was also set as 6∶1. The curated benchmark dataset, independent test dataset and apo protein dataset are available at http://protein.cau.edu.cn/zincidentifier/.

Feature extraction

We select and extract a variety of features that describe different sequence, structure and network microenvironments which might be of importance for zinc-binding site prediction and which are used in combination as input to the RF-based classifiers. Features used in our framework comprise of four generic types: sequence, structure, network and other features (listed in Table 2). In particular, although solvent exposure features are generally considered as structural properties, this particular feature type is important for improving the prediction performance of zinc-binding residues. Also, we find several solvent exposure features useful for the prediction which were mostly understudied in previous work, which is discussed in the following sections. Therefore, we consider the solvent exposure feature as an additional feature type and would like to investigate its ability to improve upon the prediction versus other structure features.

1. Sequence features. A wide range of sequence features are used which make a significant improvement for the prediction. These include: (i) position-specific scoring matrices (PSSMs) from PSI-BLAST [27] using a default E-value cutoff with three iterations; (ii) native disorder predicted by DISOPRED2 [28]; (iii) protein aggregation properties generated by TANGO [29]; (iv) sequence length; (v) CHED residue percentage in the protein chain; (vi) conservation score directly derived from the PSSM. The conservation score is defined as:where p_i,j is the frequency of amino acid j at position i. It was extracted from the PSSM generated by PSI-BLAST.
   A combination of these sequence or sequence-derived features has been shown to be able to further improve the prediction performance in our recent work [30]–[33].
2. Structure features. DSSP [34] is used to extract a series of structure features, including the secondary structure, solvent-accessible surface area (SASA), hydrogen bonds and protein backbone torsion angles (PHI and PSI angles). HBPLUS [35] is used to calculate the number of hydrogen bonds. NACCESS is also used to generate the respective solvent accessibility of the side chain, main chain, non-polar and polar region [36].
3. Network features. Graph-theoretic approach derived from the residue-residue contact graphs is becoming a useful method for unraveling the modular network organization of protein structure and predicting the protein function given the structure information [37]–[39]. If the spatial coordinate distance of the corresponding Cα atoms of two residues is within a cutoff distance threshold, the two residues are defined as being in contact. Residue-residue contact graphs were constructed using the spatial coordinates of protein structures, with a cutoff distance of 6.5 Å between the Cα atoms of two residues in the structure. We use the JUNG library (available at http://jung.sourceforge.net/) to calculate the network properties that represent the local microenvironment of residues, including clustering coefficient, degree, density, eccentricity, betweenness, closeness and status.
4. Other features. New solvent exposure features are also used in our study. They provide an important description of the local spatial environments of the residue of interest, providing useful information that is complementary to the traditional solvent exposure measures, such as solvent accessibility and accessible surface area [40]–[42]. We find that these features are particularly important for the prediction and thus discuss their relevance by considering them as a separate feature type from other structure features. These include the number of C_β atoms in the lower Half-Sphere (HSEBD), residue depth (RD), the coordination number (CN), the number of C_α atoms in the upper Half-Sphere (HSEAU), the number of C_β atoms in the upper Half-Sphere (HSEBU), the number of C_α atoms in the lower Half-Sphere (HSEAD) and atom depth of a residue's C_α atom (RDa). The hsexpo program in the Biopython package is used to calculate these solvent exposure features [43].

We use a sliding window approach with a size of 9 residues to extract all sorts of different features and take as input to train the RF models. Each residue is represented by Vn (n = 1, 2, 3, …, 9, denoting the position of the residue in such window; the centered residue is denoted as V5). Similarly, the elements in the PSSM (with a dimension of 9×20 = 180) are denoted as V1, V2, …, V180, respectively.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Four generic feature types used in this study, including sequence, structure, network and other features.

https://doi.org/10.1371/journal.pone.0049716.t002

Feature selection

Due to the nature of having a large number of heterogeneous and possibly redundant features in many data mining tasks, feature selection techniques play an increasingly important role in bioinformatics applications [44]. Thus, a great deal of attention is being paid to application of these techniques that can potentially improve the prediction accuracy of machine learning classifiers, for example, in the prediction of protein folds [45], catalytic residues [46], protein crystallization [47], helix-helix interaction in membrane proteins [48] and disulfide connectivity [49]. In our study, we used a two-step feature selection method based on RF [50] to identify most informative and contributive features for the prediction of the zinc-binding sites. In the first step, the mean decrease Gini index (MDGI) was used to generate the optimal feature candidates (OFCs), which was calculated by the RF package implemented in R [51]. MDGI describes the importance of an individual vector element for correctly classifying a residue as zinc-binding or non-zinc binding. The mean MDGI Z-Score of each vector element is calculated by the following equation:where x_i is the mean MDGI of the i-th feature, is the mean of all MDGI values of all vector elements of the feature x, and σ is the standard deviation (SD). We obtained the optimal feature candidates (OFCs) by selecting the features from the top 70 MDGI Z-Scores features.

The second step is a stepwise feature selection based on the RF classifier. To evaluate the prediction performance of the classifiers, 5-fold cross-validation tests were performed, where the whole benchmark dataset was randomly split into five subsets. In each validation step, one subset was left out as the testing set, while the remaining four subsets formed the reduced training set. The RF model was trained on this reduced training set and subsequently validated by testing it on the left out subset. This procedure was repeated five times, each time leaving out a different subset of the samples such that in the end each subset was in turn left out exactly once. The final performance was calculated by averaging over the five subsets. In the first round of the stepwise feature selection, we trained the RF model with the 70 optimal selected feature candidates. Then in the next round, one feature would be removed from the set of the optimal feature candidates. If the resulting RF classifier achieved a higher score (score = AUC*recall*precision, which can be considered as a balanced score incorporating the three measures AUC, recall and precision), this feature would be removed. In particular, this score was averaged over 100 times to obtain a stable value, hence the order in which features were removed in each round was almost certain. This combined score was used to evaluate the performance of the RF classifier because it is useful for comprehensively monitoring the changes of all of the three measures. The stepwise feature selection procedure continued until this score no longer increased. This procedure allows the most important and informative features to be systematically identified.

Random forest-based classifier

In this study, we formulate the prediction task of zinc-binding sites as a binary classification problem and solve it using a machine learning approach-random forest (RF), where the zinc-binding residue is labeled with 1 and non-zinc binding residue is labeled with −1. RF is an ensemble tree-structured classifier [51]. A typical RF extends many classification trees; each tree generates a classification, and the tree “votes” for one of the two classes (positive or negative). To classify and predict whether a residue is zinc-binding or not, the physicochemical properties and the microenvironment of the selected residue are represented by a feature vector and encoded into the RF-based predictor to classify this vector as either positive (zinc-binding) or negative (non-zinc binding). The random forest classifier selects the classification as the final prediction that has the the largest number of votes (among all the trees in the forest). An advantage of RF is that it is particularly suitable for dealing with a large dataset with high-dimensional and noisy input features. In addition, as RF does not require time-consuming optimization process, the classifier training and prediction by RF are generally much faster than many other algorithms such as SVM. Due to these advantages, RF has been extensively applied in many classification and regression tasks and demonstrated satisfactory performance, for example, in predicting residue-residue contacts [52], helix-helix interaction in membrane proteins [48], DNA-binding residues [53] and protein interaction sites [54]. We use the randomForest R package [51] for the implementation of the RF algorithm.

Performance Evaluation

We use four performance measures, namely, Recall (REC), Precision (PRE), the Area Under the Curve (AUC) and the Area Under the Recall-Precision Curve (AURPC) to evaluate the prediction performance of the methods. In our study, the dataset is heavily imbalanced with the negative to positive ratio of 100∶1. For this highly imbalanced dataset, use of the Accuracy measure, i.e. the proportion of true prediction results (true positives+true negatives) in the dataset, is not appropriate, since all-positive or all-negative classifiers can also achieve a very good classification rate. Therefore, we consider using alternative metrics such as AUC and AURPC for a comprehensive evaluation of the models' performance. AUC is the area under the receiver-operating characteristic (ROC) curve, which is a plot of true positive rate (TPR) against false positive rate (FPR). AURPC is also used in our work to evaluate the performance, which is regarded as a good alternative to AUC if there is a large skew in the class distribution [55].

The Precision is defined as:The Recall (TPR) is defined as:The Specificity is defined as:The FPR is defined as:where TP is the number of true positives, TN is the number of true negatives, FP is the number of false positives and FN is the number of false negatives, respectively.

Two-step feature selection by random forest

By means of reducing the high dimensionality of feature vectors, feature selection can be effectively used to select the most informative features and generate a more succinct feature subset from the initial feature set with a large number of noisy and redundant features, which can allow us to significantly improve the prediction performance of machine learning-based classifiers [44]. To give insights into the feature characteristics and select the most contributive features for the prediction, we performed a two-step feature selection. The first step was to select the optimal feature candidates, while the second step was a stepwise feature selection to generate the final list of optimal features which would be used to train the best model.

In the first step of feature selection, we used the MDGI score to generate the initial OFCs based on a RF classifier. As a result, the top 70 features ranked were selected as the initial OFCs (Fig. 2). As can be seen, PSSM had the highest mean MDGI Z-Score, solvent accessibility feature calculated by NACCESS and solvent exposure feature also had larger Z-Scores. Meanwhile, some important features such as CHED_residue_percentage, chain length and network_closen_cent were ranked with a lower score. The percentage of each of the four types of features, as shown in the pie chart in Fig. 2, indicates that most of the contributive features in our method were predominantly derived from sequence information.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. Ranking of the optimal feature candidates (OFCs) based on the MDGI Z-Scores.

The bar represents the MDGI Z-Scores of the corresponding feature group. We group the features into four types: sequence, structure, network and other features, which are represented by purple, red, blue and green bars, respectively. The percentage of each of the four types of features is shown in the pie chart. The labeled features in the graph are the 14 final features selected by the two-step feature selection method.

https://doi.org/10.1371/journal.pone.0049716.g002

At the second step, a stepwise feature selection method was used (see the “Feature Selection” section). If the removal of a feature led to a higher prediction score of AUC*Recall*Precision, such feature would be deleted from the feature set. By iteratively removing the redundant and less informative features from the initial optimal feature candidate set, it is expected that the prediction performance could be gradually improved during this process. This two-step feature selection that combined random forest MDGI Z-Score ranking and a stepwise selection, provides a realistic approach for selecting an optimal subset of more important features within a reasonable computational time compared with other intensive feature selections [56]. In this work, this feature selection procedure assessed a total of 878 features in 9 h and 70 features in 290 h (56 rounds), respectively, on a 2*4 Xeon Intel X5650 processor Linux server (Ubuntu 10.10). Consequently, we obtained a more compact and useful feature subset that improved prediction of the RF classifiers with the application of this two-step feature selection method.

Feature importance and contribution

Finally, 14 features were selected by our two-step feature selection method. The nomenclature of the selected features is specified as follows: PSSM_5_H and PSSM_5_C denote the PSSM feature in the respective positions of 9 (the “H” column in the PSSM) and 5 (the “C” column in the PSSM) of the centered residue (i.e. 5th position in the local window); Conservation_score_V5 describes the conservation score of the centered residue (Conservation feature corresponds to a 9-dimensional vector with 9 residues in the local window); All_polar_abs_V5 describes the absolute solvent accessibility of the all-polar side chain of the centered residue, which was calculated by NACCESS; ChainCHED_H_(P) denotes the percentage of His (H) in the CHEDs of the protein chain; ChainCHED_D_(P) and ChainCHED_E_(P) have similar meanings as ChainCHED_H_(P); Ex_CN_V5 and Hbplus_V5 belong to the new solvent exposure feature type and describe the coordination number (CN) and the number of H-bonds of the centered residue, respectively; Chain_E_sum and Chain_CHED_sum denote the numbers of Glu (E) and CHED in the chain, respectively; chain_length represents the number of all residues in the chain; Net_closen_cent_V9 denotes the closeness centrality of the 9th residue in the local window. In particular, inclusion of Net_closen_cent_V9 in the final feature set means that the network closeness property of neighboring residues of the centered residue in the residue-contact graph also plays an important role for prediction of zinc binding sites. The boxplots for the 14 final selected features are given in Fig. 3. It can be seen that while some feature values of the positives contrast sharply to their negative counterparts including All_polar_abs_V5, Chain_CHED_sum, Chain_E_sum, chain_length, Ex_CN_V5, other feature values did not show any obvious distinction between the positives and negatives (including chain_residue_H(P), Hbplus_V5, Net_closen_cent_V9, PSSM_5_C and PSSM_5_H), yet they collectively made a significant contribution to the performance of the final best RF model.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Boxplot of the 14 final features selected by a two-step feature selection that collectively made a significant contribution to zinc-binding residue prediction.

Green and red boxes denote zinc-binding and non-zinc binding residues, respectively. The horizontal line in each box indicates the mean value of a corresponding feature. For each feature name in the x axis, po: positive samples (zinc-binding residues); ne: negative samples (non-binding residues).

https://doi.org/10.1371/journal.pone.0049716.g003

It is interesting to note that almost a majority of selected features were derived from sequence information rather than structure. Among the final 14 features selected by our two-step feature selection method, indeed 10 of them were derived from sequence information. In contrast, there were only two selected structure features. The possible reasons might be: (1) It has been well established that certain sequence features are important for predicting zinc-binding sites, such as the evolutionary information contained in the PSSM matrix and conservation score, which are well known features making an important contribution to the prediction performance; (2) Most initial features used in our study are derived from sequence information. For example, 70 features were selected and retained after the first step feature selection according to Z-Score, 47% of which are sequence features and most of which have relatively high Z-Scores (see Fig. 2). Therefore, it is expected that a large portion of the selected features after the second step feature selection would be derived from sequences; (3) The selected 10 sequence features have a good complementarity with other selected feature types, such as the structure and network features, thereby making a collective contribution to improved prediction of zinc-binding sites.

We further went on to elucidate the importance and contribution of the 14 final selected features at both residue and protein levels. We examined the influence of each of the 14 features on the prediction performance by iteratively removing each from the final feature set and consequently calculating the corresponding REC, PRE, AUC and AURPC of the resultant RF classifiers. The results are presented in Table 3. PSSMs and conservation score have been found useful for the prediction in previous studies [12]. In this study, they also made a great contribution to the prediction performance, especially the PSSM_5_C feature. The importance of this feature is reflected by the fact that there would be a large decrease in 7 out of 8 performance measures when we used the rest of 13 features to train the RF model. In the absence of the PSSM_5_C feature, the performance of the trained models considerably deteriorated, as reflected by the REC, AUC and AURPC at both residue and protein levels (at residue level: REC decreased from 0.804 to 0.760, AUC from 0.960 to 0.954 and AURPC from 0.832 to 0.815; at protein level: REC decreased from 0.748 to 0.703, AUC from 0.966 to 0.960, AURPC from 0.825 to 0.808). These correspond to the first and second largest performance decrease, which indicates the importance of the inclusion of this feature for zinc-binding site prediction of our method.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Influence of the removal of 14 individual optimal features on the prediction performance, as evaluated by REC, PRE, AUC and AURPC.

https://doi.org/10.1371/journal.pone.0049716.t003

Moreover, the percentage of Glu (E) and Asp (D) in CHEDs of the protein chain also made a considerable contribution to the performance improvement, which is reflected by the changes of AURPC at both residue and protein levels by comparing the resulting performances in the presence and absence of the given features. About 96% of zinc binding residues were CHEDs in our dataset. Compared to Glu and Asp, Cys and His had higher percentages of residues that bind zinc. The percentage of Glu and Asp were only 16.5% and 11.2% in the positive CHEDs, whereas in the negative CHEDs, 41.2% and 35.7% were Glu and Asp, respectively. In other words, a protein chain is more likely to bind zinc ions if it has a lower percentage of EDs and a higher percentage of CHs. Accordingly, it is expected that the percentage of Glu and Asp in CHEDs of the chain is a useful feature for zinc-binding site prediction.

As can be seen from Table 3, there are two structure features selected in the final feature set. They are All_polar_abs_V5 (calculated by NACCESS) and Hbplus_V5 (calculated by HBPLUS), which define the absolute solvent accessibility of the all-polar side chain and the number of hydrogen bonds of the centered residue, respectively. After removal of the two features from the feature set, PRE of the resulting RF classifiers decreased from 0.750 to 0.714 in the case of Hbplus_V5 and to 0.715 in the case of All_polar_abs_V5, respectively. These correspond to the first and second largest decrease of PRE, demonstrating the importance of these two features for the prediction. In addition, there are two other features Ex_CN_V5 and Net_Closen_CentV9 in the final feature set. The former is a descriptor of solvent exposure of the centered residue in terms of the residue coordination number (CN), while the latter is a network feature which describes the mean geodesic distance from the centered residue to the neighboring node within the residue contact graph of the protein. We can see from Table 3 that Ex_CN_V5 makes a good contribution to AURPC and PRE at both residue and protein levels. However, there is not obvious decrease after the removal of the selected network feature Net_closen_cent_V9. It is well known that closeness centrality indicates a degree of residue centrality in the protein 3D structure. This feature has been previously found useful for predicting functionally important residues, including enzyme catalytic sites which are generally centrally placed in the structure [57]. However, in this study, we find that closeness centrality is not an informative feature for predicting zinc-binding sites. As zinc ions in the zinc-binding proteins are often located at the side of the structure and play structural roles (for example, zinc in zinc finger proteins helps stabilize and coordinate the alpha-helix and two beta-strands necessary for the stabilization of the structure and interaction with DNA, RNA and other molecules), the closeness centrality is not an effective feature for predicting zinc-binding sites. Yet, this feature may be useful for balancing the performance and contribute to the prediction when used in combination with other important features, which is the case when classifying the positives at other thresholds (data not shown).

Comparison with other methods

We further compared our method with other previously developed methods. The performance of each of the compared methods was summarized using the AURPC, AUC, REC and PRE measures at both residue and protein levels (A true positive at the residue level was defined as a zinc-binding residue that was correctly predicted, while a true positive at the protein level was defined as a zinc-binding protein for which at least one zinc-binding residue was correctly predicted). In particular, for the sake of the comparison, we compared different methods at the fixed PRE. In addition we plotted the ROC and Recall-Precision curves to show the performance of each method (Fig. 4). ROC curves were plotted using the “ROCR” package [58] at the two levels. Recall-Precision curve is a measure of the overall quality of the prediction and is often used as an alternative complement to ROC curves especially when analyzing datasets with a large imbalance in the class distribution [59] by incorporating both the Recall and Precision measures. The AURPC was calculated using the method of Davis and Goadrich [55]. Although most zinc-binding site prediction studies predominantly evaluate the performance using AURPC, we employed both AUC and AURPC to comprehensively evaluate the performance of different methods in this study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Recall-Precision and ROC curves displaying the performance of the three methods at both residue (A and B) and protein levels (C and D): our method (red line), SitePredict (black line) and zincfinder (green line) when applied to the 5-fold cross-validation benchmark dataset in this study.

https://doi.org/10.1371/journal.pone.0049716.g004

There exist several methods that have demonstrated good performance for predicting zinc-binding residues [7], [11], [12]. However, due to the unavailability of webserver or standalone software, some methods could not be directly compared to our method. Moreover certain methods can only be applied to predict zinc-binding proteins [21] and are not capable of predicting zinc-binding residues, while others predict the spatial position of zinc-binding or metal-binding sites [19] rather than zinc-binding residues. In addition, as our benchmark dataset was derived from the previous dataset on which Shu et al.'s method was built, we cannot compare with that method either. Therefore, we compared the predictive performance of our method with another two state-of-the-art methods that focused on prediction of zinc-binding residues, namely, SitePredict [7] and zincfinder [11]. The prediction performance of the three methods is shown in Table 4.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Performance comparison between our method, zincfinder and SitePredict on the 5-fold cross-validation benchmark dataset.

https://doi.org/10.1371/journal.pone.0049716.t004

SitePredict is a structure-based method that also uses the RF algorithm to predict zinc-binding sites [7]. The RF classifiers of SitePredict were trained on diverse residue-based site properties comprising of spatial clustering of residue types and evolutionary conservation. We tested the performance of SitePredict by submitting our 5-fold cross-validation (benchmark) dataset to its online webserver. At the residue level, SitePredict generated 252 TP, 84 FP and 151 FN predictions, which corresponds to a REC of 0.625 at the fixed PRE of 0.750. In contrast, our method produced 324, 108 and 79 for TP, FP and FN predictions in the 5-fold cross-validation tests, corresponding to a REC of 0.804 and a PRE of 0.750, respectively. Although our method predicted more FPs than SitePredict, the REC of our method is significantly better than that of SitePredict (0.804 versus 0.625), which means that our method is more accurate in predicting the positive samples (zinc-binding residues) than SitePredict. Meanwhile, AUC was also improved from 0.923 to 0.960, implying that our method outperformed SitePredict. We also compared our method with SitePredict using AURPC. From Fig. 4A, we can see that the Recall-Precision curve of our method is consistently higher than that of SitePredict, with AURPC increased from 0.738 to 0.832. At the protein level, we also observe a similar tendency that the performance of our method is better than that of SitePredict, with REC, AUC and AURPC increased from 0.649, 0.923 and 0.726 to 0.748, 0.966 and 0.825, respectively, suggesting that our method provides a better classification performance for recognizing zinc-binding sites than SitePredict.

Zincfinder is a sequence-based tool for zinc-binding site prediction using SVMs [11]. Zincfinder was applied to our benchmark set and generated 288 TP, 96 FP and 108 FN predictions, respectively at the residue level, with a REC of 0.728 at the equal PRE of 0.750. Compared to zincfinder, REC of our method was increased to 0.804. Zincfinder achieved AUC and AURPC which were respectively worse than those of our method at the residue level, but it attained a higher AURPC at the protein level (0.825 versus 0.841, Table 4). While our method was outperformed by zincfinder at the protein level (reflected by the REC and AURPC), it achieved a predictive power comparable to, if not better than, zincfinder with lower REC and AURPC at the protein level, but with higher REC and AURPC at the residue level as well as higher AUC at both levels.

We also compared our method with these two methods on the independent test set. The performance is shown in Table 5. We can see that the REC, AURPC and AUC of our method are respectively better than those of the other two methods at both residue and protein levels. In addition we plotted the Recall-Precision and ROC curves at the two levels in Fig. 5. The Recall-Precision and ROC curves have a similar tendency as those on the benchmark set at the two levels. Overall, these results suggest that our method has outperformed SitePredict and zincfinder on this independent test set.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Recall-Precision and ROC curves displaying the performance of the three methods at both residue (A and B) and protein levels (C and D): our method (red line), SitePredict (black line) and zincfinder (green line) when applied to the independent test set in this study.

https://doi.org/10.1371/journal.pone.0049716.g005

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Performance comparison between our method, zincfinder and SitePredict on the independent test dataset.

https://doi.org/10.1371/journal.pone.0049716.t005

Considering that zincfinder used only sequence features and our method also had a majority of selected features derived from sequence information, we also trained our model using the 10 selected sequence features only, applied it to make the prediction, and compared the results with zincfinder (See Table 4 and Table 5, respectively). It can be seen that our method is able to provide a competitive and in some cases better performance than zincfinder in terms of Recall, AUC and AURPC at both residue and protein levels based on the benchmark and independent datasets. This suggests that our method that only used the sequence features has also outperformed zincfinder. Overall, the results indicate that zincfinder has a better ability to predict more TPs at the protein level while our method is more accurate and displays a greater predictive power at the residue level. In addition, it is noteworthy that there is an overlap of more than 60% of protein chains in our training set and the benchmark dataset of zincfinder, which may explain why the sequence-based tool zincfinder was better than the other structure-based tool SitePredict and why our method was outperformed by zincfinder at the protein level. Altogether, our method provides a competitive performance compared to the other two state-of-the-art methods on the 5-fold cross-validation benchmark dataset.

Predictions for the apo protein structures

Next, we examined how well our method performs on the apo protein structures, the prediction of which is more relevant for the actual use of our approach. We thus evaluated the performance of our approach based on the generated apo protein structure dataset (See Dataset section). The performance is shown in Table 5. As can be seen, our method achieved the Recall of 0.719 and 0.742 at a corresponding Precision of 0.750 at the respective residue and protein levels on this apo structure dataset. The AUC is 0.953 and 0.970, AURPC is 0.713 and 0.762 at the two levels. Despite the limited size of the apo structure dataset, these results indicates that our method is less sensitive to the structural rearrangements due to the binding of zinc ion, and that our method can be used to identify the zinc-binding sites in the apo protein structures.

Predictions for the four types of zinc-binding residues

As discussed above, there are four major types of residues that bind to zinc- Cys, His, Glu and Asp (CHED). Table 6 shows a comparison of the predictive performances of the three different tools for predicting these types on the training set. Cys and His were predicted with higher accuracy than Glu and Asp. For example, the REC values of Cys and His are 0.909 and 0.891 at an equal PRE of 0.750, respectively, which are much higher than those of Glu and Asp, which only had REC values of 0.319 and 0.656, respectively. A similar tendency was observed for AURPC. On the other hand, we note that AUCs of Glu and Asp are higher than those of Cys and His. This is mainly because Cys and His have a higher percentage in positive samples (zinc-binding residues) while Glu and Asp have a higher percentage in negative samples (non-binding residues). The proportions of zinc binding residues of CHED are C: 26.9%, H: 45.5%, E: 16.5% and D: 11.2%. In contrast, the proportions of all CHED samples are C: 10.9%, H: 19.7%, E: 36.1% and D: 33.3%. Hence, the larger percentage of Glu and Asp in negative examples resulted in much more FP predictions than those of Cys and His, leading to a lower PRE. It is also why AURPC was decreased (Fig. 6A) with the decrease of REC. REC indicates the prediction performance of positive samples, which does not necessarily reflect the predictive performance of negative samples. From this perspective, AUC can be considered as a balanced measure of the overall quality of the prediction as it incorporates both SEN (the same as REC) and SPE (which is TN/(TN+FP)) measures. A higher ROC curve of Glu and Asp indicates a better accuracy of negative samples (Fig. 6B). Therefore, AUC is also an important performance measure of our method, playing a crucial rule that is complementary to AURPC. For the sake of comparison, we also list the predictive performance of CHED by another two methods SitePredict and zincfinder in Table 6. We can see that there are straight lines in some of the precision-recall and ROC curves for zincfinder, for example, the ROC curves of ASP, this is mainly because that for 83.3% of negative samples of Asp (D) residues, the predicted value of zincfinder is 0. Similarly, 89.7% of negative samples of Glu (E) residues were predicted as 0, thus there were no data for zincfinder with FPR>0.103 and as a result there was a straight line in its ROC curve.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 6. Recall-Precision and ROC curves displaying the performance of the three methods for predicting four types of zinc-binding residues CHED when applied to the 5-fold cross-validation benchmark dataset.

(A) and (B): Recall-Precision and ROC curves for our method; (C) and (D): Recall-Precision and ROC curves for SitePredict; (E) and (F): Recall-Precision and ROC curves for zincfinder, respectively. Cys (C), black line; His (H), red line; E (Glu), green line; D (Asp), blue line.

https://doi.org/10.1371/journal.pone.0049716.g006

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Predictive performance of the four types of zinc-binding residues (CHED) of our method, SitePredict and zincfinder on the 5-fold cross-validation benchmark dataset.

https://doi.org/10.1371/journal.pone.0049716.t006