SciCrunch Registry Overview and Analysis

The NIF Registry (RRID:nlx_144509) provides the entry point of all research resources searched via NIF. A subset of the databases in the Registry are more deeply integrated via the NIF Data Federation, which provides deep query of the contents of these databases [2, 7, 8]. As described in the introduction, the Registry is a human curated, catalog of research resources, annotated with the NIFSTD ontologies [9, 10] and curated according to curation guidelines (http://confluence.crbs.ucsd.edu/display/NIF/Resources+and+Curation) developed by NIF.

As described in our previous paper ([3]), each entry in the Registry includes a resource identifier, name, description, URL and resource type, which represents a set of metadata common across several major cataloging efforts in biomedicine including Biomedical Resource Ontology (BRO) [11] (RRID:nlx_143813), BioSiteMaps, and Eagle-i (RRID:nlx_144312) (See Fig 1). Indeed there are eight major types of resources: Data, Software, Training, Service, Material, Funding, People and Jobs. These eight major resource classes have 302 granular subclasses largely aligned to BRO and ERO and newer classes such as online course or mobile app that have not yet been reconciled. Most resources also include a subset of 30 additional attributes including parent organizations, supporting agencies, grant numbers, related diseases, access restrictions, related organisms, alternative or old URLs. All data in the registry is available not only as a searchable list, but also as a linked data graph. Each resource, is available as RDF served as part of the NeuroLex semantic media wiki. We include in S1 Text a copy of the registry, which is also available via a SPARQL end point (http://neurolex.org/wiki/NeuroLex_SPARQL_endpoint).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Overview of the resource registry.

Left Panel: Figure depicts the registry entry for the ImageJ resource. Fields entered by curators include: Name, Keywords, Parent Organization, Supporting Agency, Reference, and URL / Alternative URL (not shown). The RDW generated fields include: Website Status and Mentioned in Literature aka the list of literature citations discussed in Section titled “Literature Management: Tracking resources mentioned in papers via automated tools’s web content over time / SPAM detector” of this paper. Top Right Panel: Figure depicts the growth of the registry over time, starting with 300 resources in 2008, and ending with 12,568 resources in 2014. Bottom Right Panel: The Figure shows the percentage of resources, that have a website updated date on the home page with a particular year. These dates were extracted in July 2014. The latest data from the NIF registry can be downloaded from https://www.neuinfo.org/mynif/search.php?t=registry. (Download of registry is available via a CSV file).

https://doi.org/10.1371/journal.pone.0146300.g001

NIF also includes information on when the resource was last updated and if the resource has gone off-line. The graph in Fig 1 (lower right) shows the “last updated” data for the current contents of the NIF Registry. This information is extracted by curators and also through some of the automated pipelines described below. Of our current contents, relatively few resources have disappeared altogether (198 as of 10/2014). However, as shown in the graph in Fig 1(lower right), a significant number of resources are no longer actively maintained, as determined by their “last updated” or copyright date.

As shown in Fig 1 (upper right), the content of the NIF Registry has grown significantly since the initial launch of the pilot project in 2006 and first production release in 2008 [8]. An overview of the resource graph of the NIF Registry as of October 2014 is shown in Fig 2. Connections between resources represent resources that state a relation with another resource, e.g., the Protein Databank and the European Protein Databank. Resources are color coded according to the year they were entered. As can be seen, population of the NIF Registry was not uniform, with different types of resources being added at different times. This pattern largely reflects the ingestion into NIF of listings of research resources maintained by other organizations. For example, in 2013, a large number of software tools were added from the Genetic Analysis Software site (http://www.jurgott.org/linkage/ListSoftware.html). This comprehensive listing was maintained by a single individual that was largely neglected while the curator switched universities. The ingestion into the NIF enhanced coverage of genetic analysis software and served as a stable repository for this type of boutique collection.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Resource types and year added to the registry.

Research resources are each tagged with one or more resource types, the most common are represented in this graph (for all data see http://neurolex.org/wiki/Resource_Type_Hierarchy). The year that a resource was added to the registry is denoted by the color, note that 2009 and earlier data are lumped into 2010. Edges between points denote resources tagged with both, e.g. database and software. The SciCrunch registry data is available in S1 Text.

https://doi.org/10.1371/journal.pone.0146300.g002

Resource Disambiguator for the Web (RDW) System Architecture

The RDW (RRID:nlx_144525) tools were developed in order to provide curators with a set of automated and semi-automated tools to perform four major functions:

1. Determine whether the resource has moved
2. Ensure that the content is up to date
3. Track mentions of the resource within the biomedical literature.
4. Identify new resources from the literature: The creation of a new research resource is typically described in a publication that accompanies the release.

The web app code is available at http://dx.doi.org/10.5281/zenodo.20557 and the literature batch processing code at http://dx.doi.org/10.5281/zenodo.20560. Fig 3 shows an overview of the system architecture. Briefly, the Batch Services in the middle panel of Fig 3 represent a set of automated tools that extract likely resources from the literature, download the Registry, and run a set of tasks on each registry item. The Registry batch services include fetching the current registry data and a set of basic operations such as testing of the up/down status of registry URLs, testing the last updated date on the web site and the registry spam detector, which asks whether the content of a web site has significantly shifted. These services recapitulate and enhance the previously described curation pipeline developed using Textpresso [12].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Resource Disambiguation System Architecture.

The left panel represents literature sources including full text sources such as four Elsevier journals and the PubMed Central Open Access subset (∼1 million documents), and a set of services that grants access to PubMed abstracts (∼24 million documents), Springer and Nature content. The middle panel represents the set of automated tasks where data from the publications and the registry itself are processed. The right panel represents the set of curator facing dashboards that allow the curators to gain information from the batch services and give feedback to the algorithms represented in the middle panel. Finally, a Postgres database is the back-end data store and management system created to track the ongoing processes, compare and store all data. A set of web services is fed out that provides the output of pipelines and curation effort to the public. DISCO [7] is a data aggregation platform that feeds this data to the Neuroscience Information Framework project.

https://doi.org/10.1371/journal.pone.0146300.g003

The left side represents literature sources currently searched, which include the full set of abstracts from PubMed (RRID:nlx_82958, http://www.ncbi.nlm.nih.gov/pubmed), and full text sources such as the PubMed Central (RRID:nlx_18862, http://www.ncbi.nlm.nih.gov/pmc/) open access subset and a small number of journals from the Elsevier corpus. In total about one million full text articles are accessible. The full text literature sources are amenable to a thorough analysis of each sentence and were used to create algorithms that can detect resources in text. Unfortunately, the remainder of the literature remains somewhat opaque. Portions of the remaining 23 million biomedical research articles can be accessed via various programming interfaces such as PubMed (via NIF), Springer, or Nature application programming interfaces (APIs).

For full text papers, URLs and names of research resources (via named entity recognition, NER) are extracted from papers allowing us to ask about the resources used in the paper that may match an existing resource registry entry while non-matching mentions may represent a new candidate for the resource registry. The literature API batch operations are limited to asking each API for the set of names, synonyms and abbreviations for each research resource currently part of the Registry because, as mentioned above these APIs do not accept URLs as a query string.

The right side of the architecture diagram depicts the web interfaces where users interact with the batch operations and learning algorithms, including a dashboard overview of the performance of the entire system per resource. The interface is described in more detail in the next section. The system is underpinned by a Postgres database and interacts with NIF and SciCrunch public user interfaces via a set of REST web services accessed in most cases using the DISCO framework [7].

RDW Interface Overview.

The RDW web interface is built in GRAILS (https://grails.org/) and allows curators to interact with the system (Fig 4). Integral to the development of the learning algorithms is the ability for curators to provide input in the form of Good/Bad decisions for training each function. A good decision means that RDW performed correctly on a particular task while a bad decision means that it performed incorrectly. The curators render this decision by clicking on an icon (Fig 4B). In the example shown in Fig 4, which is asking the curator about a mention of ImageJ in the literature, a good decision means that ImageJ was correctly identified and a bad decision means that the resource identified was not ImageJ. The RDW system collects these decisions into the database. The interface also allows curators to visualize algorithm output and their decisions via several dashboards. As the curators are asked to provide feedback on several tasks, the interface is divided into several sections. However, in order to simplify the curator’s learning curve, all screens follow the same layout and require the same type of feedback.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Screen shot of RDW user interface components.

A. URL Mentions: Registry mentions interface component showing an exert of a paper containing the ImageJ URL. B. NER, Named Entity Recognition, Mentions: Registry mentions matches based on resource name, also for ImageJ. Curators toggled on the thumbs up icon for some of these mentions. C. Dashboard for individual resource: Graph of ImageJ resource mentions over time resulting from the all URL matches from A and curator annotations from B. ImageJ was first detected in a 2001 paper. Note that the same search in PubMed yields 2002 as the earliest mention. The main paper describing the tool is in 2004. Until 2008 there are only few papers detected for this tool.

https://doi.org/10.1371/journal.pone.0146300.g004

The interface contains sections devoted to the following tasks. The details of the algorithms used in each task are provided in the next sections; here we provide an overview of each curator interface and any action required.

1. Determining whether a URL is functional:—Good/Bad curator decisions are captured to determine whether the site is actually down (“good”) or if it is not down (“bad”). The latter case happens, for example, when a website does not allow automated crawlers to view its content. A “bad” decision by the curator removes the down flag from the resource and removes the resource form subsequent web crawls.
2. Determining whether a resource has moved:—Good/Bad curator decisions define whether the algorithm properly redirects from an old or existing URL to a new one. This information is stored in the database for other functions including URL detection, where URLs found in papers can be properly aggregated to a single resource, e.g.,during its existence, NIF moved from nif.nih.gov to neuinfo.org. In the Registry database, both of these URLs are registered to NIF.
3. Determining Registry updates and spam detection:—This interface provides a list of resource entries that show significant change in their content as determined by three similarity measures (Jaccard Coefficient, Semantic Similarity and Cosine Similarity). We have found that this function is very reliable in detecting websites that are no longer functional and have been taken over by spam. If a resource is determined to be spam, curators manually update the Registry entry.
4. Finding mentions of a resource in the literature by URL:—Good/Bad curator decisions mean that the algorithm detected a good or bad match with a registry resource. The interface allows curators to view, sort and filter resource mentions. For each detected resource mention the surrounding context and paper information is also provided to the curator (Fig 4A). Curators can provide additional information about why a match is bad through the “Notes” field.
5. Finding mentions of a resource in the literature by name:—Good/Bad curator decisions mean that the algorithm detected a good or bad match with a registry resource name, and synonyms stored as part of each registry record (for listing of all registry fields see Fig 1 left panel and S1 Text), see Fig 4B for an example match of RDW and ImageJ.
6. Finding mentions of a resource name via the literature APIs:—Good/Bad curator decisions mean that the API produced a good or bad match with a registry resource. The interface allows curators to view, sort and filter resource mentions, but as content is closed source curators must go to their library to find the paper by identifier and attempt to determine where the mention occurs.
7. Finding new resource candidates mentioned in the literature:—This screen (See S1 Fig) provides curators with likely resources as determined by text mining that do not currently match any Registry entry. The Good/Bad decisions are captured per Type of resource for the most common types including database, software tool and tissue bank. Curators are presented with the possible resource and the surrounding text for context. If curators find a suitable candidate they can create an entry in the Registry. If curators find that the URL is an alternate to an existing resource, a structured note is made and the URL is added to the Registry entry.
8. Visualizing results:—RDW contains three different visualization dashboards that provide an overview of current resources. They can display system status as it pertains to the most common tools detected in the literature via different routes, the resource mentions per year for individual resources (Fig 4C), the overall up/down statistics (Fig 5 Inset) as well as the co-occurrence of registry entities in papers (See S2 Fig).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Analysis of URL mentions of research resources in papers.

We show the top 30 resources sorted by the number of times URLs match a resource in a paper. Multiple mentions of a resource in a single paper are not counted as additional data points. For this analysis there were 100K of 500K detected URLs that matched one of the following: the exact URL present in the resource registry, an alternative or old URL noted in the resource registry, a URL that redirects to a resource registry item, or a simplification of the URL construct to the core resource, for example http://clinicaltrials.gov/NCI12345 would be counted as a reference to ClinicalTrials.gov. 500K URLs were detected in 700K papers (open access subset of PubMed Central, accessed 9/2014), the 400K that did not match registry items were not included in this analysis, but are included in other analysis of the NER system. Inset: URL status of main web page as assessed the week of 8/15/2014. An automated tool checks each web site in the registry to determine if the site is up or down each week and results are accumulated for curators. Curators are alerted when the site is down for 3 consecutive weeks.

https://doi.org/10.1371/journal.pone.0146300.g005

RDW also has administrative functions for user management and management of training of classifiers from the curator provided good/bad decisions and classification of the identified resource mentions.

RDW provides REST web services for data harvest via DISCO [7], including the literature identifier for all high confidence resource mentions per registered resource and registered resource site up/down information. These data are displayed as part of the NIF / SciCrunch Registry user facing record where they are also available for download. High confidence resource mentions are defined as any mentions either directly made by curators or made by the RDW incorporated filter classifiers having a score (SVM classifier output value) greater or equal to 0.5 on the resource mention. The threshold is selected by changing the score threshold and checking a sample of the classifier selected mentions for errors. As one may note on Fig 4B, where the thumbs up is marked for some mentions of ImageJ, curators do not have time to test each new resource mention. However, we hypothesize that resource owners will be interested in all papers citing their resource and will be willing to provide good/bad information to train/refine RDW algorithms. To take advantage of this source of data, the registry will be moving to an extensible interface (SciCrunch) where several RDW tools will be exposed to resource owners not only curators.

In the following sections, we provide more detail on the algorithms used to perform the above functions.

Registry Management: Tracking the up/down status of resources in the registry

For each resource in the registry, RDW checks the accessibility of the resource’s web site each week and keeps a history of the web site’s up/down status (See Fig 4). Because web sites can be intermittently down for maintenance or inaccessible due to temporary heavy load or network problems, a site is considered inactive only after three or more consecutive weeks of inaccessibility. Because a few sites reject access to non-browser based, automatic web agents there is possibility of false positives (tagging a resource as down even though it was up). To remedy this, RDW has a screen for curators to check if the latest inactive resources (down for three or more weeks) automatically detected by RDW are valid or not and provide feedback to the system. The system will then exclude those false positives from the inactive resources list. RDW automatically adds the current up/down and inactive status to the Registry via web services; this information is displayed through the Registry in SciCrunch and NIF user communities. If a resource is determined to be dead, the tag “THIS RESOURCE IS NO LONGER IN SERVICE” is added prominently to the registry entry. We never remove a resource entry from the Registry, as these resources may have been referenced in various papers. Often times, our registry landing page is the only record remaining of a discontinued resource.

For each resource URL, RDW also attempts to provide information on the latest updates to determine whether a resource is actively maintained and kept up to date. To do this, RDW extracts all available text from the “home” or “about” page and looks for dates using regular expressions. For example text such as “Copyright © 2003–2014” on the following software tool page (http://www.bioconductor.org/packages/2.13/bioc/html/CSSP.html) produces a result of 2014 as the date noted by the system. Data from this service are included in S1 Text and show that out of a total of 13,000 records, 1597 include the year that the web site was last updated. Of those, 516 were updated within the last year, but 78 resources report dates before 2004 suggesting that a significant subset of web sites are essentially dormant. We note, however, that this data is not complete because many web sites either lack dates or the dates are not accessible to our crawler.

Registry Management: Tracking the movements of a resource

Over years, resources may move to new web locations usually, but not always, leaving redirect information on the original location to the new site. Curators track movements of resources manually by noting alternate URLs and old URLs in the Registry. As with dead links, when URLs no longer contain the scientific content and no redirect can be found then the tag “THIS RESOURCE IS NO LONGER IN SERVICE” is added. In this case, curators keep the URL in the ‘oldURL’ field for purposes of tracking the resource in the literature. The number of resources with the out of service tag is relatively small, 198 (about 1.5%), but the number of resources that move, as measured by the oldURLs, is 733, again beyond the number that can easily be managed through manual curation.

The RDW tracks the movement of resources in a semi-automated fashion. RDW downloads the text content of home and about pages, strips html tags, and uses text analysis on this content to determine if a web site has moved. Using the content of the main pages of registered resources, classifiers are trained to detect if the resource redirects to a different web site. The problem is cast as a binary text categorization problem where if the content indicates a redirection it is assigned the label ‘good’ and all the others in the training data set are assigned a ‘bad’ label. We generated our training data set by accessing, downloading and searching the text of each registry resource web site for the phrase ‘moved to’ resulting in 178 entries of which 42 were labeled as ‘good’ (See S2 Text).

We tested both supervised and semi-supervised approaches. For the supervised approach, we used a linear kernel SVM. As the semi-supervised approach, we used transductive SVM [13]. The semi-supervised learning approach tries to leverage co-occurrence patterns between the labeled training data and unlabeled data from the same distribution to improve generalization performance. As learning features, we used the length of the document in number of tokens and the existence/non-existence of unique words in the training set (bag of words representation). Unique words are defined as the lower cased unique terms in the training portion of the data set. Any new words introduced in the testing set are regarded as nonexistent during the classification. Since the bag of words approach ignores the local dependency of words in human languages and treats each word independent of other words in the context, we also tested short phrases of two contiguous tokens (bigrams) as features. Usually ‘site moved’ documents are relatively short, therefore we used the length of the document as a feature to take this into account.

Registry Management: Tracking the change of a resource’s web content over time / SPAM detector

In addition to resources changing URLs we also need to detect changes in content. Wholesale changes in content usually indicate that the URL now points to a SPAM site. However more subtle changes to resources constantly occur, for example updates of grant information, or even the scope and nature of a resource. In the Registry, curators keep a running log of changes and updates. Indeed the number of edits performed every year, except 2014, surpasses the total number of resources (see Table 1). Although these metrics are gross, i.e., they do not track whether the curator made substantial changes or just minor edits or whether the same resource was edited multiple times, they do tell us that each year every resource was potentially subject to some update. Given the number of resources and the rate of growth the ability of a human curator to examine each record is beginning to hit a ceiling. Automated tracking and detection of these subtle changes is important if the Registry is to be kept up to date.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Curation edits made to Registry resources over time.

https://doi.org/10.1371/journal.pone.0146300.t001

The RDW has compiled a set of algorithms to help curators prioritize resources for re-curation, such that their time is spent examining resources that have changed as opposed to those that did not. Therefore RDW downloads resource web pages, strips HTML tags, same as above, and compares the text over time. The comparison uses three different similarity measures to periodically check for changes on each registered resources web site. These measures are;

• Jaccard similarity and containment using w-shingling [14].
• Cosine similarity using TF-IDF weighted document vectors.
• Jensen-Shannon divergence between current and original Latent Dirichlet Allocation (LDA) topic distributions of the web content of the resource.

For Jaccard similarity, a shingle is defined as a contiguous subsequence of tokens in a document D[14]. W-shingling S(D, w) for a document D is defined as the set of all unique shingles of size w in that document. Jaccard index, a common metric for resemblance of sets, between two documents A and B can be defined in terms of w-shingling as follows Since web sites frequently add new content, the following containment index [14] is also calculated for the current and original content of each resource; Since the containment index is not symmetric, i.e. B can contain A (addition of new content to the original) or A can contain B (removal of content from the original), the maximum of c(A, B)_w and c(B, A)_w is used as the containment index for each resource. A shingle size of five is used which is arbitrarily selected with the goal of a phrase of length 5 distinctive enough to detect movement within a document and the number of resulting shingles are not too much to preserve memory. The w-shingling approach allows for location invariance of phrases of size w such that reshuffling of content has less impact on the similarity of the content.

For cosine similarity, a standard information retrieval (IR) similarity measure for vector space representation of documents, of the current and original content of a resource’s web site after stripping away the web markup tags, we used augmented TF-IDF to prevent bias towards longer documents. For the terms t in the documents (d ∈ D) augmented TF-IDF scheme is defined as (1) where f_{t, d} is the frequency of the term t in the document d, fm_d is the maximum frequency of any term in the document d and N is the total number of documents. The site content (current and original) is tokenized, lower-cased, stripped of stop words before TF-IDF calculations. Cosine similarity between an original site content document vector (x_o) and its current version (x_c) is defined as

Since TF-IDF de-emphasizes terms that are used throughout the document set and emphasizes terms unique to each document, the cosine similarity will focus more on salient content words than functional words.

Latent Dirichlet Allocation (LDA) [15] is a generative probabilistic model for discrete data such as text documents. LDA models a document collection as a finite mixture over an underlying set of topics [15]. Given a number of topics, LDA estimates the mixture proportions over latent topics of each document, where each topic is represented by a distribution over words also estimated simultaneously from the training set of documents using maximum likelihood. How much the current content of a resource differs from the original content can also be modeled as the difference between their topic distributions. The standard measure for the difference between two probability distributions P and Q is the Kullback-Leibler divergence If the two distributions are same, the Kullback-Leibler divergence is zero. Since Kullback-Leibler is unbounded otherwise and non-symmetric, Jensen-Shannon divergence, defined as can be used, which is bounded between 0 and 1 and symmetric. Thus the document similarity can be defined as 1 − D_JS. Since the words making up a topic is estimated from the whole training set of documents, one can filter out certain topics from the topic distribution of a document thus only focusing on the topics of interest for semantic similarity of content changes over time.

This tool has not yet been tested as to which method is optimal for detecting changes in web content, so currently curators are presented results from all three and update the registry entries accordingly.

Literature Management: URL Extractor

Referencing a resource via URL is one of the major ways that research resources are mentioned in the literature [3]. PubMed Central open access journals are provided in the Journal Archiving and Interchange XML format with an explicit tag for URLs facilitating URL extraction. We excluded references sections of the papers while extracting URLs. To disambiguate URLs detected in full text papers, we first group them by URL host. An example of a host name is clinicaltrials.gov, in which case there are many individual trials that may be listed in papers such as https://clinicaltrials.gov/show/NCT00385866 and https://clinicaltrials.gov/show/NCT00335426. In most cases where individual data sets within a database are referenced, the host name represents the core research resource. The cumulative distribution of URL mentions in papers grouped by hosts is shown in Fig 6. As seen from this distribution the host versus URL frequency follows a Zipfian distribution, that is, a few of the hosts e.g. the Protein Data Bank and Wormbase, account for a significant portion of the mentions. (See Fig 5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Cumulative distribution of mentions(y) by resource (host) (x).

The dotted line separates hosts with 100 or more mentions (left side of the dotted line) from hosts less than 100 mentions. 327 hosts were responsible for half of the mentions. See S3 Text for a list of common hosts vs total number of papers. The near linear log-log plot of rank ordered hosts versus URL mentions indicates a Zipfian distribution.

https://doi.org/10.1371/journal.pone.0146300.g006

We use different approaches to analyze the cumulative URL mentions by host distribution from the exponential portion of the curve (many publications mentioning a resource) versus the long tail (few papers mentioning a resource). Resource candidates with 100+ mentions account for most of the unmatched URLs. Therefore, we chose a cutoff value of 100+ mentions. The 100+ mention resource candidates are aligned with resources via clustering and the remaining long tail is largely used to identify new resources using machine learning based filtering of new resource candidates as explained in the following sections.

Literature Named Entity Recognition (NER): Resource Identification by Named Entities

In addition to searching for URLs, we also employ Named Entity Recognition (NER) to recognize resource mentions in the text by name. Although for some fairly unique names, e.g. ImageJ, Freesurfer, entity recognition is fairly straightforward, for others, particularly as acronyms are fairly common, challenges arise. For example Gemma is a common first name and a resource (http://www.chibi.ubc.ca/Gemma). So matches by simple lookup without taking account of the context in which the name is mentioned are poor. However, identification of resource mentions in biomedical texts can be cast as a sequence learning task. The Conditional random fields (CRF) [18] framework is one of the most popular probabilistic sequence learning approaches and has been applied to many named entity recognition tasks including tagging biological entities such as genes, proteins, cell lines [19, 20]. Like these, we detect names of resources in text and describe the pipeline below for detecting existing resource mentions.

The CRF is provided with sentences where each token is labeled to demarcate the named entities of interest for training. Once trained, the CRF model will provide labels for each token of sentences extracted from biomedical texts. A label on a token indicates if the token is a part of a named entity (e.g. research resource). For labeling, the IOB format is commonly used. Below is an example sentence with a resource mention; The|O:{DT} same|O:{JJ} method|{NN} was|O:{VBD} used|O:{VBN} in|O:{IN} Delorme|O:{NNP} et|O:{CC} al.|O:{NNP} (|O:{(} 2007|O:{CD})|O:{)} and|O:{CC} is|O:{VBZ} implemented|O:{VBN} in|O:{IN} the|O:{DT} EEGLAB|B-RESOURCE:{NNP} software|O:{NN}.|O:{.}

In IOB format the prefix B denotes the beginning of a named entity. The rest of terms in the NE are denoted by the prefix I and terms not belonging to any named entity by O. We extended the IOB format to encode additional information, namely, part-of-speech tags delimited by curly brackets for each token.

Training Features.

The features used for NER are summarized in Table 2 accompanied by example features for the word ‘EEGLAB’ from the above example sentence. The majority of the features used are binary orthographic features commonly seen in other NER systems and approaches. These features include the identity of the word at index t in the sentence, if a word is all in uppercase or not or prefixes/suffixes of the word of length two through four letters. Word class is first introduced in ABNER [19]. It replaces contiguous upper case letters with an ‘A’, contiguous lower case letters with an ‘a’, contiguous numbers with a ‘0’ and rest of the symbols with a ‘x’ in a word. Part-of-speech tag of each word in the sentence is the only syntactic feature used. The only domain specific feature is membership of a word/phrase to the resource names, acronyms and NIF-ID lookup tables generated from the NIF/SciCrunch registry. The same set of features are extracted also for the word before and after the current word in the sentence taking surrounding context into account.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Features used for Resource Identification.

https://doi.org/10.1371/journal.pone.0146300.t002

Literature URL Extractor: Resource Candidate Selection by Machine Learning Results

The average precision and F₁ curves for most confident learning scheme and active learning experiments is summarized in Figs 7 and 8. Based on the paired t-test on the area under the curve (AUC) values for both learning schemes over ten experiments, the active learning scheme is better than the most confident prediction selection scheme (p-value = 2.2e-6 for precision and p-value = 0.0013 for F₁ case which are statistically significant). The average accuracy of the active learning based learning scheme after 40 iterations (42 curator labeled training set) was 77.6% with a standard deviation of 1.6% which has 22% higher accuracy than the Textpresso score reducing the error rate about 50%. The average is created from the ten training/test splits after 40 iterations of active learning. Therefore this is the method that is utilized in the RDW as the preferred method for finding resource candidates.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The average precision for active learning versus most confident (MC) prediction selection.

https://doi.org/10.1371/journal.pone.0146300.g007

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. The average F₁ for active learning versus most confident (MC) prediction selection.

https://doi.org/10.1371/journal.pone.0146300.g008

Experiments for automatic detection of resource site movements

For training / classification of support vector machines, we used SVM^light package [16]. We generated ten 70%/30% random stratified splits and used them for our experiments. The average results for unigram/bigram features and supervised and semi-supervised (TSVM) machine learning approaches are shown in Table 3.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Redirect Candidate Classification Results.

https://doi.org/10.1371/journal.pone.0146300.t003

For supervised learning, bigram features seem to increase the recall at the expense of the precision. We conjecture that this is due to the introduced dependency between terms using bigrams more discriminative features can be represented, such as the bigram ‘moved to’ as opposed to the single terms ‘moved’ and ‘to’ occurring anywhere in the text.

Semi-supervised learning has good recall and somewhat lower precision than the supervised learning. The approach with the maximum F₁ was incorporated into the RDW system to provide curators a list of resource redirect candidates. Furthermore, the URLs identified by the algorithm have been added to the registry and are tracked and used in all subsequent tasks (e.g. URL extraction).

Resource Identification by Named Entity Recognition

We created ten training/test sets by randomly selecting 30% of the articles as the test set and the rest as the training set for the NER system. We trained ten NER systems on the 70% training data and tested each on their corresponding test set. On average the NER system had 89.2% (SD: 4.0%) precision and 88.8% (SD:4.2%) recall (F₁: 89.0%, SD: 4.1%).

Literature Resource Identification: Active Learning based Filtering of Springer, Nature and NIF Literature Searches for Resource Mentions

The above approaches for resource mentions were applied exclusively to the full text of articles from the PubMed Central. However, the vast majority of the literature could not be accessed by these methods. For the remainder of the literature we relied on two types of search APIs: PubMed abstracts via NIF Literature (http://nif-services.neuinfo.org/servicesv1/resource_LiteratureService.html#path__literature_search.html) and publisher-provided APIs that search full text from Nature (http://www.nature.com/developers/documentation/api-references/opensearch-api/) and Springer (https://dev.springer.com/). This set of APIs yield by far the largest resource mention candidates (close to two million resource mention candidates currently), since they access collectively close to 23 million articles. However due to their closed source nature, we have no control over the search algorithms and how much context (if any) around the search match is returned.

To test the accuracy of these results we selected a resource (ModelDB (http://senselab.med.yale.edu/ModelDB/)) and annotated each of the 443 results returned from the publisher search API queries as being a valid mention or not. The dataset is available as a supplementary document (See S6 Text). Out of 443 candidates, 179 were validated by curators as ModelDB mentions with an accuracy of 40%. We therefore consider these results as “low confidence” and expose them only to the curators until we can be certain that these are accurate. Because they are subject to many errors, we currently, do not include any of these matches in the registry output.

To simulate an active learning scheme for search result candidate filtering, we randomly split the annotated 443 resource mention candidates into 70%/30% training and testing sets, respectively. We repeated this procedure ten times to generate ten sets of 70%/30% random splits. Given one such random split, we randomly selected one good and one bad labeled resource mention candidate as the seed set to train a linear SVM classifier to start active learning iterations. The remainder of the training data is set aside as the pool set. As training features we used the title of the paper, the journal in which the paper is published, the description/context returned with the search result (if any) and MESH headings (for NIF Literature search results only). We combined the text from these search result fields into one document and using a bag of words representation generated a binary feature vector of occurrence of non-stop words for each training/test resource mention candidate. We also used the resource’s name as an additional feature to learn resource specific words for discrimination.

At each iteration of the active learning scheme, we train the SVM classifier on the current seed set and use the trained classifier to classify the pool set and select the pool instance with the least confidence (the absolute minimum score) to add to the seed set and remove from the pool set. To see the classification performance at each active learning iteration, we classify the 30% test set using the SVM classifier trained on the current seed set. We repeated this procedure ten times on each random split. The average F₁ curve for 100 active learning iterations is shown in Fig 9. The average accuracy of the active learning scheme after 30 iterations (32 curator labeled resource mention candidates) was 97.3% with standard deviation of 0.4% compared to 40% accuracy of search results before filtering.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. The average F₁ for active learning to filter publisher search results.

https://doi.org/10.1371/journal.pone.0146300.g009

Therefore, once sufficient curation feedback is given, the algorithm can detect with reasonable accuracy correct mentions of a resource even for this low confidence match set. All data with sufficient training is released as part of the SciCrunch registry comprising data for the literature citation field. However, it should be noted that training for one resource does not necessarily enhance accuracy across resources. We would therefore require training data from curators for each source, a time consuming task for curators. We are considering crowd-sourcing this task to the resource owners.

Test of Coverage of RDW

To test the coverage of our three approaches to identifying resource mentions, we used the list of manually annotated papers that mention ModelDB, downloaded from http://senselab.med.yale.edu/modeldb/ptrm.asp (June 2014) as the gold standard (248 entries with PubMed IDs). Fig 10 shows coverage of the three resource mention extraction approaches. The three approaches combined found 471 papers that mention ModelDB out of which 201 were valid, 72 of which were in common with the gold standard set. Of the 471 mentions, URLs were detected in 28 cases, six of which were in the common set. Therefore RDW found 22 high confidence mentions of ModelDB missed by the ModelDB staff. Out of the 443 search results (471–28 URL detections) ModelDB mention candidates, 113 were valid ModelDB mentions not present in the human curated data gold set indicative of increased coverage by the introduced method. These 113 new papers were of high interest to the principal investigators of ModelDB, who spend substantial effort pulling the information together from manual search efforts (personal communication T. Carnevale 2014). However, the system did not identify 176 papers that were in the gold set. We analyzed each missed ModelDB mention to identify the reason for the failure. The majority of the uncovered portion of the gold set (153 out of 176) was either in closed source journals (from other publishers such as Elsevier, Wiley, IEEE) or not included in PubMed Central open access data set (45 out of 151). Twelve misses were due to the Springer API not returning them in search results, another four were missing from Nature searches. To focus on resource usage, reference sections of papers are excluded from RDW URL mention extractions. Because of this, three mentions were missed. Another three were missed by NIF Literature search. Out of 176 uncovered mentions, 172 cannot be detected by RDW due to not having access to them. Out of the remaining four, one was missed by CRF NER and the other three by URL detector.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. RDW literature coverage on ModelDB literature mentions test data set.

The RDW pipeline results are shown along side of a Gold Set of manually annotated papers that mention ModelDB, downloaded from http://senselab.med.yale.edu/modeldb/ptrm.asp (June 2014). The RDW data set includes mentions by name and URL from journals that are the open access subset of PubMed Central, as well as Springer and Nature APIs. RDW found 201 valid papers that mention ModelDB, and 72 of those were in common with the Gold Set. Of the 201 mentions, URLs were detected in 22 cases, and of those 6 were in the common set. Please see S7 Text for the complete data set.

https://doi.org/10.1371/journal.pone.0146300.g010