Distributed word vectors for a corpus based query suggestion

The design goals for LAILAPS-QSM are to provide a service for query suggestion that is (i) search corpus based, (ii) open, and (iii) extensible. Search corpus based means there is no dependency to query logs, controlled vocabulary etc. that need an elaborately maintenance. Only the text corpus itself is required that is anyway immanently available to build the text index. A “open” service implies that it should be seamlessly embeddable into existing information retrieval Web frontends or other query engines. Furthermore, this implies open access, open source, open input/output formats, and that the software should run on many platforms. Extensibility implies a design that is modular and easy to make additions to the implementation or to customize the trained query suggestion model. All of these design goals are made under consideration of an efficient implementation to enable responsive query frontends.

Word count vectors

The LAILAPS-QSM query suggestion algorithm makes use of the word distributional hypothesis whereupon words that occur in similar contexts tend to have similar semantic meanings [12], which can be expressed by word count vectors [13]. The basic idea behind training such vectors is that the meaning of a word is characterized by its context, i.e. neighbouring words. Thus, words and their contexts are considered to be positive training samples. Because of the large data volume in life sciences, the vocabulary size and number of documents result in very large binary word distribution matrices W of dimension m × n, whereas m is the number of unique words and n is the number of documents. Taking the PubMed abstracts from last 10 years as example, m is 1.3 * 10⁹ and n is about 8.3 * 10⁶.

Distributed word vectors

To find concepts that use distributed word vectors but meet the requirements of low memory and CPU consumption, we used the neural network language model (NNLM) [14]. A feedforward neural network with a linear projection layer and a non-linear hidden layer was used to jointly learn the word vector representation and a statistical language model. But the runtime complexity of such networks for large text corpora is still too high. Mikolov [11] published the Word2vec algorithm. It uses a log-linear neuronal network to quickly train vector representations of words from large text corpora. As illustrated by the example in Table 1, the idea is to compute for each word of a given vocabulary its influence for all possible semantic relationship that are expressed in a given text corpus. The number of expected semantic relationships is the dimension of the word vectors and need to be estimated beforehand.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Example for word vector representation computed by a feedforward neural network.

The word vector representations estimate the influence of a word in the context of the semantic relationship expressed by the particular word vector. This matrix is a 2 × 4 matrix, representing a vocabulary size of 4 and vector dimensions (number of expected relationships) of 2. The word vector w₁ could represent the relationships “yield” and w₂ “lipid source” respectively.

https://doi.org/10.1371/journal.pcbi.1006058.t001

The first suggested neural network architecture computes these word vectors as the bag-of-words model (CBOW). Here the order of words in the history does not influence the word vector. In other words, the CBOW model predicts the current word based on the context. The second neural network architecture, the Skip-gram model, tries to maximize classification of a word based on a different word in the same sentence. More precisely, this model uses the current word as an input and predict words within a certain range before and after the current word.

In both the CBOW and the Skip-gram models, the hierarchical softmax [15] function was used as an activation function in the output layer. In the traditional softmax function, we have to compute the conditional probabilities of all vocabulary words w in the context of words history, the time complexity of this computation is O(|V|). In the hierarchical softmax function, the vocabulary is represented as a Huffman binary tree based on word frequencies and the time complexity is O(log₂|V|).

Implementation

The implementation consists of three parts, (i) the preparation of the text corpora, (ii) the training of the query suggestion model, and (iii) the query suggestion service. To ensure platform independence, enable a high reusability and profit from year-long in-house experiences JAVA has been chosen for the implementation. To ensure a platform independent access of the query suggestion service, it was implemented as a RESTful web service.

Text corpus building

In order to provide a ready trained web service, we trained a general purpose model with 13,930,050 documents (downloaded in June 2017) from PubMed articles, and gene and protein function descriptions from UniProt, Plant Ontology and Gene Ontology. Tokens are significant words or other units in a document. Usually whitespace characters are used as delimiters between two words. We used Apache Lucene a common text indexing system, split words based on rules from the Unicode Text Segmentation algorithm [16] and applied database name pattern from [17] to avoid the splitting of database-identifiers word pairs.

After analyzing query logs of the LAILAPS search engine [18] and The Arabidopsis Information Resource (TAIR) [19], we have found that more than 61.5% queries are composed by more than one term. The property of phrases is that their semantic meaning is beyond single word’s meanings. Therefore, the assumption is that phrases are formed by words that appear frequently together. For example, “heading date”, “salt stress” etc.

To improve the specificity of the text corpus, we add such frequent word pairs as phrases to the text corpus. To do so, we followed the approach proposed in the Word2vec algorithm and compute the correlation coefficient for counts of single words ω_i, ω_j and counts of word pairs as

The parameter δ is used as a discounting coefficient and prevents phrases composed of very infrequent words to be formed. A suitable value for δ is determined by testing values between 1 and 20 and compare the resulting sets of extracted phrases with these later used in the results section for benchmarking the query suggestion quality. The phrases extracted for δ > 5 do not miss test set queries and a δ < 3 results in a high number but less specific phrases. As a compromise between phrase specificity and corpus size we use a δ = 5.

To get a reasonable threshold estimation for score, we computed the phrase overlap to Gene Ontology (GO), the UniProt taxonomy section, Chemical Entities of Biological Interest (ChEBI), and the Trait Ontology (TO). Iteratively, we computed the maximum overlap with ontology terms which was achieved by δ = 5 and score = 9*10⁻⁸. The resulting corpus size is 3,402,729 tokens including 1,465,152 phrases, whereas 20,200 phrases match to terms in the mentioned ontologies.

Model training

Tools for text corpus tokenization, phrase extraction, and the training of the Word2Vec model [20] are bundled into one JAVA package. Beside the source code, an already compiled version is available as JAR package for an instant use in command line:

https://bitbucket.org/ipk_bit_team/bioescorte-suggestion

The final trained model is stored as a binary encoded associative list of word to word vector pairs. The size of the general purpose model [21] is 2.35GB and is loaded by the query suggestion module, usually from locally mounted storage.

The RESTful web service

To provide the query suggestion module as a remote API to the community, a RESTful web service was implemented. It features to send a user typed query and retrieve a JSON based document with the top five alternative query suggestions. The web service documentation is available at:

http://lailaps.ipk-gatersleben.de/client/suggestionapi.html

The endpoint URL for IPK hosted RESTful service that apply the general purpose pre-trained model is:

http://webapps.ipk-gatersleben.de/lailapssuggestion/api/v1/semanticsuggestion/<query>

Furthermore, on-site installations are possible too, by downloading a ready build web archive (WAR) file [21] and deploy it to an in-house servlet container, i.e. Apache Tomcat or Jetty. The RESTful services implement the HTTP GET verb that enables a seamless integration in any programming or script language. The URL parameter “<query>” is the original query for which you want to get suggestions. The result is returned in JSON format:

{

      “suggestions”:

      [

      <suggestions>

      ]

}

An example API call for alternatives for “heading date” is

http://webapps.ipk-gatersleben.de/lailapssuggestion/api/v1/semanticsuggestion/heading%20date

The JSON result is

{

      “suggestions”:

      [

      ”ghd7”,

      ”panicle”,

      ”tillering”,

      ”hd3a”,

      ”qtls controlling”

      ]

}

Evaluation of semantic relationship

The very essential prerequisite is to suggest only semantically closely related queries. Therefore, we evaluate the semantic similarity of the suggested queries to the original one. Doing so, our approach was to use plant science domain knowledge that was curated and published in ontologies. We follow the approach of Resnik’s [22] to compute the information content of the shared parent of two terms T₁ and T₂ as where T_s is the set of terms that subsume T₁ and T₂. P(T_s) is the relative frequency of T_s in the text corpus. In other words, the similarity rate metric is the minimal relative frequency over all ontology terms that subsume both terms. A low relative frequency over a give text corpus means a special, not that common term. In contrast, a term with high frequency is commonly found in the text corpus and is used to express a broad semantic concept. If both, the original and suggested term, are successor of such a common term, they are allowed to express semantically quite different concepts. For example, if the only common parent term for “Triticum” and “Zea” would be “cellular organism” they need not to have that high semantic similarity. Because, “cellular organism” is found frequently in the training text corpus (1,456,281 times) and describes a general taxonomic classification. In contrast, if the commonly shared parent term would be “Poaceae”, both must be more similar to each other, because “Poaceae” is less commonly found (8,131 times) and describes a plant family.

This metric is efficiently to compute and has a correlation of 0.79 to human judgment. Whereas past study [23] has shown that human judgment of semantic term similarity has a correlation of 0.90. It is observed that Resnik’s metric is a compromise of reasonable high accuracy [24].

Because the value of Sim_Resnik is less than the information content of T₁ or T₁, a uniform semantic similarity (in a 0–1 scale) can be computed that follows the approach of Lin [25] as

Evaluation results

For the evaluation, we selected 15 queries that were selected based on TAIR query logs. We requested a snapshot of the log files and got an anonymized excerpt of 571,113 queries between January 2012 and April 2013 [26]. As shown in Table 2, the queries were classified into trait, biological entity, taxonomy, and metabolic function.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Test set for query suggestion benchmarking. Based on query log analysis five major classes of query can be identified: Trait, biological entity, taxonomy metabolic function.

In the second column, related to each class most frequent query objectives where randomly selected from query log.

https://doi.org/10.1371/journal.pcbi.1006058.t002

To compute for each query the relative frequency in the SimLin matrix, we counted the individual token frequency in PubMed abstracts. To get the common subsuming term, we used the Gene Ontology (GO) [27], the UniProt taxonomy section [28], Chemical Entities of Biological Interest (ChEBI) [29] and the Trait Ontology (TO) [30].

All 15 queries were used as input to the LAILAPS-QSM RESTful web service. For each query, we can get top 10 suggested queries, then we will try to match these suggested queries in domain ontologies, the matched results will be selected to compute the Sim_Lin metric. The minimal common ancestor phrases were retrieved by i) manual selection of the most similar concept for the original and suggested query and ii) the direct parent node for them. The evaluation results are compiled in S1 File. The average similarity is 0.70, whereas 34% have a score above 0.80. The minimal similarity score is 0.16, which is an outlier because “medicago sativa” is in fact not an alternative query for “zea mays”. In fact, it was suggested because some PubMed abstracts express a relationship of a different type.

In comparison to the beforehand discussed value of 0.90 for human similarity judgment, the evaluation results let us conclude that LAILAPS-QSM enhances the quality and sensitivity of database searches, particularly more relevant documents can be retrieved. The clear benefit is the capability to infer semantic similar terms directly from the source. This ensures the inclusion of recent discoveries of facts and their relation to existing knowledge. This goes beyond syntactic or rule based approaches and enables researcher to keep track with stored knowledge and avoid to miss data because of less awareness of alternative relationships to further facts.