Problem formulation

In our system, we consider three categories of words related to genes, diseases, and methylation. We refer to these three categories as ‘concepts’. In order to extract the associations between methylated genes and diseases, we only consider the cases where the three concepts appear in the same sentence. We call the order in which the three concepts appear in sentences a ‘pattern order’. Table 1 shows all possible (six) different pattern orders. We call an instance of a pattern order a 'pattern'. Below we provide an example of a sentence from a PubMed (http://www.ncbi.nlm.nih.gov/pubmed) abstract (PubMed: 21693594) that includes a pattern (marked as italic and underlined):

The CPG island in the FILIP1L <Gene> promoter was heavily methylated <Methylation Word> in ovarian cancer <Disease> cells.

In this example, the pattern is (FILIP1L, methylated, ovarian cancer), while the pattern order is (<Gene>, <Methylation word>, <Disease>).

We represent the association extraction task as a binary classification problem. Patterns that express associations between methylated genes and diseases are named ‘positive patterns’, while those that do not express such associations are named ‘negative patterns’. It is possible that a sentence contains more than one pattern, and a sentence may contain both negative and positive patterns as well. Below is an example of a sentence from a PubMed abstract that contains negative and positive patterns:

We found that methylation <Methylation Word> of the CRY1 <Gene> promoter was detectable in Parkinson's disease <Disease>, but absent in PER1 <Gene> promoter.

The pattern (methylation, CRY1, Parkinson's disease) appears as a positive pattern in the above sentence, while the pattern (methylation, Parkinson's dis-ease, PER1) is a negative pattern. Accordingly, we developed a methodology to identify negative and positive patterns even if they appear in the same sentence.

Text pre-processing

Figure 1 depicts the structure of DEMGD. The text pre-processing module of DEMGD implements six steps. Firstly, sentence boundary determination is performed by the set of rules from Weiss et al. [24] to determine the end of sentences. The second step is tokenization that breaks sentences into words, and we used the following rules to determine the Boundary Of Words (BOW):

Newline, tab, space, !, and ? are always BOW

1. • Period followed by whitespace is BOW
2. • If the word to which “, ‘, (, ), [, ], <, or >, is attached at the end or the beginning does not include the matching punctuation in the middle of the word, it is BOW (for example, the parenthesis are considered part of the name of P14(ARF) gene, so the parenthesis in this case cannot be used as a boundary of a word)
3. • Otherwise, it is not BOW.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. DEMGD system architecture.

The input to the system is the Input Text, and the output is Summary Tables and Full Reports. The system consists of four modules: Text Pre-processing, Structured Data Representation, Classification and Associations Extraction.

https://doi.org/10.1371/journal.pone.0077848.g001

The third step is named entity recognition that aims to identify genes, diseases and methylation words. We used three manually-compiled dictionaries (for genes, diseases and methylation words). These dictionaries include all various ways an entity can be expressed. For example, the diseases dictionary includes ‘Type 1 Diabetes’, ‘Diabetes Type-1’, ‘Diabetes Type 1’, ‘Diabetes Type1’, ‘Type1 Diabetes’ and ‘Type-1 Diabetes’. This is done in order to maximize the recall rate of named entity recognition step. We used dictionary-based longest matching technique to extract multi-word entities (e.g., the disease ‘Diabetes Type 1’ consists of three words). One of the main requirements of a NER system is determining the boundary of multi-word entities. If a sentence includes ‘Diabetes Type 1’, it is inaccurate to extract only ‘Diabetes’ instead of the whole entity ‘Diabetes Type 1’. Therefore, the following steps describe the logic of dictionary-based longest matching technique:

1. 1. Take the first word in the sentence ‘w₁’.
2. 2. Check if ‘w₁’ exists in the diseases dictionary.
3. 3. If w₁ exists in the dictionary, then take the following word ‘w₂’, and check if the sequence of words ‘w₁ w₂’ exists in the dictionary.
4. 4. Repeat step 3 until we add a word ‘w_n’ such that the sequence of words ‘w₁ w₂ … w_n-1 w_n’ does not exist in the dictionary.
5. 5. The sequence ‘w₁ w₂ … w_n-1’ is determined to be a named entity.
6. 6. Repeat the steps 1 to 5 using the genes and the methylation words dictionary.
7. 7. Repeat the steps 1 to 6 starting from word ‘w_n’.

We evaluated the performance of NER step on our test set T used in the original manuscript. The precision/recall/F-score are 94.48/90.13/92.26, 98.61/100.00/99.30 and 84.69/69.17/76.15% for genes, methylation words and diseases, respectively.

The forth step is stop-words elimination to remove common words that may not contribute to discrimination between classes, and this step is performed using a stop-words list. The fifth step is stemming in which all suffixes and prefixes are removed from words, and this step is performed using Porter Stemmer [25] (MATLAB version) (http://tartarus.org/martin/PorterStemmer/). The last step is keywords selection. Keywords are determined by using information gain that estimates the information gained when predicting a class based on the presence (or absence) of a specific word in a sentence. The information gain is determined as in [26] (see Information S1).

Structured data representation

This DEMGD module converts free, unstructured text into a structured representation. DEMGD implements two types of representations where each representation provides a different aspect of statistical information. The first type implements DTM representation, whereas the second type implements PWM representation. The following two subsections explain implementation of DTM and PWM approaches separately. Then Hybrid Approach subsection, explains how the two approaches are combined in DEMGD.

DTM

In DTM, each column corresponds to a keyword, and rows correspond to sentences. The elements are represented using (i) binary, (ii) frequency, and (iii) TF-IDF values. The first mechanism is based on using binary values where 1 represents words that appear in sentences, and 0 represents words that do not appear in sentences. The second mechanism uses frequency values that represent the number of times a word appears in a sentence. The last mechanism uses term-frequency inverse-document-frequency (TF-IDF) with z-score normalization. TF-IDF is defined as in [27], and Z-score normalization is defined as in [28] (see Information S1). These are used in connection with different machine learning algorithms. We used frequency values representation of DTM described previously for Naïve Bayes, binary representation of DTM for rule generation algorithms, and TF-IDF with z-score normalization for decision trees, KNN, and SVM (see Classification Module subsection).

PWMs

So far, PWMs [29] have been used to solve different problems in bioinformatics such as motif discovery [30], binding site identification [31], etc. Here we introduce a novel application of PWMs for text mining.

The fundamental idea underlying this approach is to align sentences along several key concepts so as to segment sentences to different parts. In our case we consider the three concepts: genes, diseases and methylation words. Because there are six different pattern orders of these three concepts (as illustrated in Table 1), we generated six different PWMs, and each matrix represents a specific pattern order (see Figure 2). Each row in a PWM represents a specific word. The number of columns of each PWM is four, because for each pattern order, we can distinguish four segments in each sentence. For example, let us assume that the pattern order is <gene> <methylation word> <disease>. We will consider four segments (<segment 1>, <segment 2>, <segment 3>, <segment 4>) of the sentence to generate a PWM for this pattern order:

<Segment 1> <gene> <segment 2> <methylation word> <segment 3> <disease> <segment 4>

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 2. The structure of PWMs.

We can generate six PWMs, and each matrix corresponds to a pattern order. For example, the first PWM to the left corresponds to the pattern order (<Disease>, <Gene>, <Methylation Word>). Each row corresponds to a word, and each column corresponds to a segment, and cells of the matrix represent the frequency of words in each segment.

https://doi.org/10.1371/journal.pone.0077848.g002

The first column of a PWM represents the frequency of words in the first segment. Similarly, the second, third and fourth columns represent the frequency of words in the second, third and fourth segments, respectively. We generated 12 different PWMs: six PWMs for the negative class (called ‘negative PWMs’), and six PWMs for the positive class (called ‘positive PWMs’).

PWMs Generation from text.

To generate PWMs from a collection of positive sentences or negative sentences, we first identify the pattern that appears in the sentence. Then we identify words in each segment. After that, we update the PWM that corresponds to the corresponding pattern order. In the following example, the first segment is before the gene concept, which includes the words (‘CPG’ and ‘island’):

The CPG island in the FILIP1L <Gene> promoter was heavily methylated <Methylation Word> in ovarian cancer <Disease> cells (PubMed: 21693594).

So the matrix elements intersecting with the rows that correspond to ‘CPG’ and ‘island’, and the first column will be incremented by one. Similarly, the second, third and fourth columns are updated. Figure 3 shows how the PWM, which represents this specific pattern order, is updated. The next step after generating the PWMs is to normalize the matrices by dividing frequency in each cell in a column by the total sum along the column.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. PWM generation.

The PWM summarizes frequency of words in each segment. For example, the words ‘CPG’ and ‘island’ appear in the first segment of the sentence, so the rows that correspond to these words and the first column is incremented by one. Similarly, the same step is applied to words in the remaining three segments. The same matrix is updated using other sentences with the same pattern order.

https://doi.org/10.1371/journal.pone.0077848.g003

Computing matching scores for sentences.

The main application of PWMs is to match sentences that contain a specific pattern order with the corresponding PWMs to compute the matching scores for the pattern in the sentence. We use the following example to explain how to compute these scores. Consider a sentence that contains one pattern (MIR203, methylated, MM) in two pattern orders ((<Gene>, <Methylation Word>, <Disease>) and (<Gene>, <Disease>, <Methylation Word>)):

Promoter of MIR203 <Gene> was found methylated <Methylation Word> in approximately 25% multiple myeloma <Disease> cell lines but not methylated <Methylation Word> in normal controls

The pattern in the sentence in this example will be given four scores. Two scores will be given using the positive and negative PWMs that correspond to the first pattern order (<Gene>, <Methylation Word>, <Disease>), and two scores will be given using the positive and negative PWMs that correspond to the second pattern order (<Gene>, <Disease>, <Methylation Word>). As an example, Figure 4 shows how to compute the score for the pattern in the sentence for this example using a positive PWM that corresponds to the first pattern order (<Gene>, <Methylation Word>, <Disease>). To compute the scores for the sentence in this example, the following steps are performed:

1. 1. Identify the pattern that appears in the sentence (MIR203, methylated, MM). These concepts partition the sentence to segments.
2. 2. Identify words in each segment. The words ‘promoter’, ‘found’ and ‘approximately’ appear in the first, second and third segments, respectively, and the words ‘cell’, ‘controls’, ‘lines’, ‘methylated’ and ‘normal’ appear in the fourth segment.
3. 3. Determine the weight of each word from each segment. In the first, second and third segments, we find (‘promoter’, 0.2336), (‘found’, 0.619), and (‘approximately’, 0.1724), respectively.
4. 4. If there are several words in one segment, we chose the maximum weight in the segment. In the fourth segment, we find (‘cell’, 0.0603), (‘controls’,0), (‘lines’,0.0822), (‘methylated’, 0.1224) and (‘normal’, 0.1315), so the maximum weight is 0.1315 for the word ‘normal’ and this one is chosen.
5. 5. Sum the four weights of the four segments to get the final score of the pattern in the sentence. For this example we get 0.2336 + 0.619 + 0.1724 + 0.1315 = 1.1565.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Computing the scores.

The figure shows an example of a normalized PWM. To compute the score, we sum the weights of one word from each column. For example, the word ‘promoter’ appears in the first segment, so we take its weight from the first column in the PWM. The same step is applied to the second, and the third segments. However, five words appear in the last segment, so we take maximum weight. The score of the pattern is 0.2336+0.1619+0.1724+0.1315=0.5994.

https://doi.org/10.1371/journal.pone.0077848.g004

We should note that there are different ways to compute the score in case of several words appearing in the same segment, such as summing all weights or multiplying the respective probabilities of occurrence of the words, etc.. However, the method we described above is significantly different from those customary for sequence analysis and it achieved on our data the best results, so we implemented it for the system. The analogous steps are followed to compute the score for the sentence using the negative PWM.

Pattern representation

We represent each pattern in a sentence with twelve features based on the scores and a class. The twelve features consist of six scores from the six positive PWMs and six scores from the six negative PWMs. Figure 5 shows the features used to represent the dataset based on the generated scores.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Dataset representation using PWMs.

Each pattern in a sentence is represented with twelve features and a class label. The first six features correspond to the scores generated from the positive PWMs, and the following six features correspond to the scores generated from the negative PWMs.

https://doi.org/10.1371/journal.pone.0077848.g005

The main advantage of PWMs approach is its ability to score patterns in a sentence independently from each other so that each pattern may get different scores. For example, there are positive and negative patterns in the following sentence:

We found that methylation <Methylation Word> of the CRY1 <Gene> promoter was detectable in Parkinson's disease <Disease>, but absent in PER1 <Gene> promoter.

The pattern P1 (methylation, CRY1, Parkinson's disease) is a positive pattern, but the pattern P2 (methylation, Parkinson's disease, PER1) is a negative pattern. Table 2 shows the features of the sentence with respect to each pattern. This way the features of the patterns have different values even though the patterns appear in the same sentence. This approach allows machine learning algorithms to distinguish between positive and negative patterns even if they appear in the same sentence. Interestingly, DTM approach does not distinguish between positive and negative patterns within the same sentence.

Hybrid approach

This approach combines the previous two approaches (DTM and PWM). The hybrid approach requires developing two classification models. The first classification model is generated by applying the DTM approach to sentences with one pattern only, and the second classification model is generated by applying the PWM approach to sentences with several patterns. For each sentence in the testing set we first determined the number of patterns it contains. Sentences that do not contain any pattern are discarded. The first model trained using features obtained via DTM approach is applied if a sentence has only one pattern. The second model trained using features obtained via PWM approach is applied if the sentence contains several patterns.

Classification module

Using the structured representation of patterns in each sentence, patterns will be classified as either positive or negative by the Classification Module. We compared the performance of several different classifiers on our data in the search for the most efficient machine learning model including rule-based ones (FOIL [32], CPAR [33], CMAR [34], CBA [35], PRM [33], and TFPC [36]), K-nearest neighbor (KNN) [37], support vector machines (SVM) [38], decision trees (C4.5 [39], random forest [40], and random tree [41]) and Naïve Bayes [42]. We used WEKA [43] implementation (http://www.cs.waikato.ac.nz/ml/weka/downloading.html) of Naïve Bayes and decision trees, LUCS-KDD implementation of rule generation algorithms (http://cgi.csc.liv.ac.uk/~frans/KDD/Software/), LIBSVM [44] for SVM implementation (MATLAB version) (http://www.csie.ntu.edu.tw/~cjlin/libsvm/#matlab), and MATLAB (2011b version) implementation of KNN. We implemented the classification module based on the classification algorithm that performed the best in comparison, which in our case appears to be the random forest models.

Association extraction module

Finally, the patterns that were predicted by the classification model to be positive are considered to represent potential associations. The Association Extraction Module organizes the associations into a summary table and a full report. The summary table lists for each sentence: a/ genes, diseases and methylation words that appear in the sentence, b/ the corresponding sentence, and c/ the abstract PMID where the sentence appears (in case PubMed abstracts were submitted to the systems). The full report lists for each full article or abstract: a/ genes, diseases and methylation words that appear in the sentence, b/ the corresponding color-tagged sentence, along with colored tagging of all genes, diseases and methylation words that appear in a full article or abstract.

Data acquisition

The approaches discussed here required an initial creation of a dataset for developing and testing machine learning association identification models. 1,124 abstracts were extracted from PubMed database, from which we extracted 2,049 sentences where each contained at least one pattern. Because some sentences contain more than one pattern, a total of 4,663 different patterns were obtained from these sentences.

Through hand-curation, we classified the patterns into negative or positive. 49% of patterns were negative (2302 negative patterns), while the remaining 51% of patterns were positive (2361 positive patterns). We used 30% of the sentences to generate PWMs, and we call that set of sentences set P. The remaining 70% of sentences we call set C and it is used for 10-fold cross-validation of machine learning algorithms. In addition, we generated another set, set T, which contains 75 sentences extracted from 42 abstracts (separate from 1,124 abstracts described previously). These produced 200 manually-classified patterns (100 negative and 100 positive).

Classification performance measures

For performance evaluation, we computed the following performance measures presented in Table 3. There, TP (true positive) indicates that a positive pattern is predicted as positive, while FN (false negative) indicates that the positive pattern was predicted as negative. On the other hand, TN (true negative) indicates that a negative pattern is predicted as negative, while FP (false positive) indicates that a negative pattern was predicted as positive. TP, FP, TN and FN were calculated for the cases when entities are identified.

Classification performance using the PWMs approach

In this approach we used the features generated by PWMs to train machine learning models. The models must classify each pattern as positive or negative even if multiple patterns appear in the same sentence. PWMs were generated from set P. 10-fold cross-validation on set C is used to evaluate all algorithms. We tested range of parameters for each algorithm (Table S1 in Supporting Information) and recorded the best performance with the corresponding parameters. We began by evaluating the performance of algorithms when applied on sentences with multiple patterns from set C. The best accuracy (details about used performance measures are available in Table S1 in Supporting Information) achieved was 85.5% by a random forest model. However, when we applied the algorithms on sentences from set C that contain only one pattern, we noticed a decrease in performance. The best accuracy achieved was 69.2%, again by a random forest model. Then we applied the algorithms on the entire set C that contains sentences with one pattern and sentences with multiple patterns. The random forest model outperformed the other algorithms and achieved 81.5% accuracy. Table 4 summarizes the best performance of all algorithms using PWMs approach.

Classification performance using the DTM approach

Here we used the features generated by DTM approach to train machine learning models. Considering that DTM approach cannot be used to classify several patterns if they are present in the same sentence, we applied DTM to classify sentences instead of patterns. If a sentence includes only one pattern, we label the sentence as negative if the pattern is negative; otherwise, the sentence is labeled as positive if the pattern is positive. However, if a sentence has several patterns, the sentence is labeled positive in case there is at least one positive pattern; otherwise, the sentence is labeled as negative (all patterns are negative). When we applied 10-fold cross-validation using all algorithms to the entire set C, a random forest model achieved the best performance with 80% accuracy. We noticed a slight decrease in performance when the algorithms were applied on sentences from set C with one pattern only. The best accuracy achieved was 77% by a random forest model. Table 5 summarizes the best performance of all algorithms using DTM approach.

Classification performance using the hybrid approach

Due to the fact that PWM approach worked best for sentences with multiple patterns, while DTM worked best for sentences with single patterns, we implemented a hybrid approach in which we trained two classification models. We evaluated the performance of five different algorithms. Table 6 shows the accuracy of several machine learning algorithms after applying the hybrid approach with 10-fold cross-validation on set C. The best performance achieved was with the random forest models and the achieved performance of the hybrid approach was 83.5% and 84.7% for accuracy and F-score, respectively.

Classification performance on an independent testing set

From the previous analysis, we determined that the random forest models with the hybrid approach achieved the best performance in 10-fold cross-validation. A separate testing set T is used to evaluate performance of the hybrid method (see Table 7).

Comparison with other systems

There are no publicly available tools that allow for extraction of methylated genes in different diseases, based on submitted text, thus it is not possible to make the comparison of our results to such methods. Other computational methodologies for extracting associations between methylated genes and specifically cancer [17,18] have been used in compiling MeInfoText [17] and its successor, MeInfoText 2.0 [18] databases. The difference between these systems and DEMGD is that these systems allow users to retrieve associations between methylated genes and diseases from databases, but our system allows users to submit text (abstracts or full articles) and extract associations between methylated genes and diseases from the submitted text. MeInfoText databases provide users with information about the associations of methylated genes and different cancers. For the text mining system used to compile MeInfoText, the performance has been evaluated using 75 associations, and the reported precision and recall are 99% and 93%, respectively. We note that conclusions about the performance derived using only 75 associations is rather inaccurate and not comparable to performance of our method assessed on significantly larger dataset. In MeInfoText 2.0, the associations were extracted automatically by a text mining system that implements two models, and the reported precision/recall are 94.7% / 90.1% and 91.8% / 90%, respectively for the two models. However, the systems/software that is used in [17,18] to extract the associations and the utilized datasets to train and test the systems were not available during the time of our study. Therefore, we could not perform an independent comparison between the text mining tools used for creating MeInfoText and MeInfoText 2.0 and our system, using our datasets or using data employed in MeInfoText 2.0. Thus, we only report their published results. We want, however, to indicate that performance of our system is assessed based on a much larger manually-classified datasets (we provide to public the annotated PubMed abstracts we used), and we also provide publicly accessible text-mining tool that extracts such information.

Several methods have been developed to extract associations based on similar association structure. Hakenberg et al. [45] developed seven methods to extract twelve different types of associations between different biomedical entities including gene-disease, gene-drug, drug-diseases, mutation-disease, etc. These methods depend on co-occurrence of pairs of named entities and aim to rank the associations based on the confidence. Chun et al. [46] developed a method based on filtering falsely identified named entities to extract associations between genes and diseases. However, methods that are based on co-occurrence of pairs of named entities generate a large number of false positive relations. In our case, the goal of the machine learning method we developed is to determine if a given co-occurrence of named entities constitutes an association. Coulet et al. [47] developed a method based on syntactic parsing to extract associations between pairs of genes, drugs and phenotypes. Unlike other methods that depend on simple co-occurrence, their method depends on analyzing the syntactic structure of sentences to identify the type of associations between named entities such as 'inhibits', 'induces', 'causes', etc., and it is more flexible than rule-based approaches. However, the main drawback of syntactic parsing methods is the low recall, and it requires a large corpus so that there are several opportunities to identify the associations [47].