Figures
Abstract
Background
Dozens of omics based cancer classification systems have been introduced with prognostic, diagnostic, and predictive capabilities. However, they often employ complex algorithms and are only applicable on whole cohorts of patients, making them difficult to apply in a personalized clinical setting.
Results
This prompted us to create hemaClass.org, an online web application providing an easy interface to one-by-one RMA normalization of microarrays and subsequent risk classifications of diffuse large B-cell lymphoma (DLBCL) into cell-of-origin and chemotherapeutic sensitivity classes. Classification results for one-by-one array pre-processing with and without a laboratory specific RMA reference dataset were compared to cohort based classifiers in 4 publicly available datasets. Classifications showed high agreement between one-by-one and whole cohort pre-processsed data when a laboratory specific reference set was supplied. The website is essentially the R-package hemaClass accompanied by a Shiny web application. The well-documented package can be used to run the website locally or to use the developed methods programmatically.
Conclusions
The website and R-package is relevant for biological and clinical lymphoma researchers using affymetrix U-133 Plus 2 arrays, as it provides reliable and swift methods for calculation of disease subclasses. The proposed one-by-one pre-processing method is relevant for all researchers using microarrays.
Citation: Falgreen S, Ellern Bilgrau A, Brøndum RF, Hjort Jakobsen L, Have J, Lindblad Nielsen K, et al. (2016) hemaClass.org: Online One-By-One Microarray Normalization and Classification of Hematological Cancers for Precision Medicine. PLoS ONE 11(10): e0163711. https://doi.org/10.1371/journal.pone.0163711
Editor: Kristy L. Richards, Cornell University, UNITED STATES
Received: February 3, 2016; Accepted: September 13, 2016; Published: October 4, 2016
Copyright: © 2016 Falgreen et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All datasets are available from the Gene Expression Omnibus (GEO) website (http://www.ncbi.nlm.nih.gov/geo/). The GEO numbers of the datasets are: GSE10846 GSE56315 GSE56315 GSE31312 GSE10846 GSE34171.
Funding: This work was supported by Karen Elise Jensen Fonden (http://www.kejfond.dk/): MB; Mobility Stipend, Faculty of Health Sciences, Aarhus University: SF; MSCNET, EU FP 6: HEJ; CHEPRE, Danish Agency for Science, Technology, and Innovation: HEJ; and Next Experimental Therapy Partnership, Innovation Fund Denmark (https://nextpartnership.dk/): HEJ. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
In addition to current clinically used risk factor scoring systems, several independent gene expression profile (GEP) based risk stratifications have been proposed, with biological and clinical significance in hematological cancers. Although drug targetable genes, which are only expressed in subtypes of e.g. DLBCL tumours have been identified, they are not readily applicable in clinical research and routine settings due to a lack of available routine diagnostic tests [1, 2].
[3] developed an important example of a biological sub-classification of lymphoma. On the basis of GEP analyses, DLBCL cases were classified as activated B-cell phenotype (ABC) or germinal center B-cell phenotype (GCB) with different clinical outcomes. The validity of this classification and its prognostic importance have been confirmed in a number of later studies [4–8]. Recently, we have refined the ABC/GCB subclassification of DLBCL to include a B-cell Associated Gene Signature (BAGS) classifier capable of classifying DLBCL samples into 5 different B-cell subtypes: Naive (N), Centrocyte (CC), Centroblast (CB), Memory (M), and Plasmablasts (PB) [9]. The BAGS classifier stratifies the GCB phenotype into CC and CB subtypes, with superior survival in the CC subtype. Thus, different treatment regimes could be considered in subsets of the GCB class of patients. In another study we developed classification based resistance gene signatures (REGS) for the most prominent drugs used in the treatment of DLBCL patients: Cyclophosphamide (C), Doxorubicin (H), and Vincristine (O) [10]. However, these and most existing algorithms are only applicable in the presence of whole cohorts of patients, making them difficult to apply in a routine clinical setting.
The traditional lymphoma staging and risk classification systems are based on the Ann Arbor classification for staging of lymphoma (extent of disease and extranodal involvement) and simple prognostic tools such as the international prognostic index (IPI, [11]) for large cell lymphoma and the Follicular Lymphoma International Prognostic Index (FLIPI, [12]), both derived from patient age, performance status, easy available blood tests, and disease stage. Due to the simplicity of these clinical risk stratification algorithms they are still the most widely used risk scoring systems today. Risk stratification according to these algorithms has been systematized and made easily accessible for desktop, online, and even smart-phone use. Easily accessible molecular classification methods are, however, lagging behind, thereby delaying the translation of new molecular findings into clinical practice. A few methods exist for cancer types other than lymphoma, including acute myeloid leukemia (AML) [13], and for lymphoma [14] has developed an ABC/GCB classifier, which is stable across microarray technologies and trial centres. This ABC/GCB classifier is, however, potentially biased towards classes which differentiate the prognosis instead of biological classes, since the ensemble of classifiers were chosen based on their ability to separate survival.
In clinical settings, the methods need to be applicable for a single patient sample and straightforward to use. This prompted us to develop a user-friendly and flexible web-based tool for ABC/GCB, BAGS, and REGS classification using our recently developed classifiers for microarray data based on the Affymetrix’s Human Genome U-133 Plus 2 array. The classifications made by the web-based tool hemaClass.org are compared to the existing state-of-the-art and approved ABC/GCB classifications of DLBCL. We believe that hemaClass.org will provide a novel and user-friendly concept for bringing complex molecular classification of diseases more swiftly into daily clinical practice.
Methods
Classification workflow
The workflow architecture of hemaClass.org is illustrated in Fig 1. The user is presented with a graphical interface for uploading data and adjusting settings. The user data is RMA pre-processed by the server (with an optional user one-by-one RMA reference), and subsequently processed by the classification algorithms. The results are then returned for download and inspection via the user interface.
Software availability and technical details
The interactive web application available at hemaclass.org was created using the statistical programming language R [15], the software package shiny [16], and the accompanying Linux server software. All hemaClass.org functionality, including the RMA normalization and classification procedures, are available through the accompanying package hemaClass based on a number of packages from the Comprehensive R Archive Network [15] and the Bioconductor environment [17]. The Shiny server handles the interaction between the front end web application and the back end R processing. The back end is essentially the well-documented hemaClass package which can be utilized as a programmatical interface to the functionality of the website. However, the package also allows users to run a local instance of the website if one wishes to avoid uploading large files to our server. The development and latest version of hemaClass is open source and freely available at https://github.com/oncoclass/hemaclass for sharing, modification, and redistribution. All bug-reports, suggestions, and comments on the website or package are welcome and should be posted to the github page following the link above. The regular RMA pre-processing is carried out with the Bioconductor package affy [18]. Core functions for the one-by-one RMA pre-processing are written in C++ and imported to R using Rcpp and RcppArmadillo [19–22].
Data overview
The seven gene expression datasets used in this paper are summarized in Table 1. All GEP data are from the Affymetrix GeneChip HG-U133 Plus 2.0 array and available at the Gene Expression Omnibus (GEO) [23] website (http://www.ncbi.nlm.nih.gov/geo/). To establish the classifiers the following datasets are used:
- 1. Gene expressions from 181 CHOP treated DLBCL patients are used to establish the ABC/GCB classifier. This cohort will be referred to as the LLMPP CHOP (Lymphoma/Leukemia Molecular Profiling Project CHOP) cohort [7]. The cohort is also used as a default reference set throughout the paper for one-by-one RMA normalization of arrays.
- 2. The BAGS classifier is based on gene expression data from eight human tonsils sorted in five B-cell subsets. This dataset is also used for scaling of gene expression data for BAGS classification, and will be referred to as the Tonsil dataset [9].
- 3. The REGS classifiers are based on a panel of 12 Multiple Myeloma (MM) and 14 DLBCL cell lines. This panel will be referred to as BCELL26. The DLBCL part of the cell line panel is used for scaling of patient data and will be referred to as DLBCL14 [10].
For validation the following four DLBCL cohorts are used:
- 4. The Aalborg OCT cohort (CHEPRETRO) of 89 Danish DLBCL patients undergoing first-line treatment at Aalborg University Hospital [9].
- 5. The International DLBCL Rituximab-CHOP Consortium MD Anderson (IDRC) cohort of 470 DLBCL patients treated with R-CHOP first-line therapy [24]. Note, that these samples are formalin-fixed, paraffin-embedded (FFPE).
- 6. The Lymphoma/Leukemia Molecular Profiling Project R-CHOP (LLMPP R-CHOP) cohort of 233 DLBCL patients treated with R-CHOP first-line therapy [7].
- 7. The Mayo-Dana-Farber Cancer Institute (MDFCI) cohort of 90 DLBCL patients treated with R-CHOP first-line therapy [8].
The GEO datasets were downloaded using the R-package DLBCLdata [25].
One-by-one RMA normalization
Robust multichip average (RMA) pre-processing consists of three steps in the order: (1) Background correction, (2) quantile normalization, and (3) summarization of probes to probe-sets [26, 27]. Confer [28] for a comprehensive account on RMA. Both the quantile normalization and summarization procedures of RMA are cohort based and hence need to be altered to facilitate a one-by-one RMA pre-processing scheme. Previous approaches similar to the one-by-one normalization approach used by hemaClass.org have been described by [29] and [30] As quantile normalizer, the empirical cumulative distribution function (ECDF) of the mean of the sample quantiles of an RMA background corrected reference dataset is used in place of the usually applied ECDF of the mean of the sample quantiles of the user supplied data [31]. To mimic the summarization procedure of RMA [27] the probe effects estimated by median polish for the same reference data is subtracted all probes of the user data. The RMA pre-processed expression value for each probe-set is then estimated as the median of the associated probes. For more detail on our one-by-one normalization approach see S1 Text section S4. Finally, before classification the median of each probe-set in the RMA reference dataset is subtracted from the corresponding probe-set in the user data, since the classifiers were trained on median centered data.
The one-by-one normalization has the implicit assumption that the samples and reference follow the same distribution. This assumption might be violated by batch effects arising from differences in laboratory specific sample preparations and can cause severe bias in the normalization; to accomodate this hemaClass.org allows users to upload their own RMA reference dataset prepared under similar conditions. In the this paper a laboratory specific RMA reference is referred to as an InLab reference. InLab references were simulated by selecting a random subset of 30 samples from each cohort. Samples were also one-by-one RMA normalized using the LLMPP CHOP dataset as an external reference; this is referred to as ExLab reference normalization.
Classification methods
Elastic nets.
Logistic and multinomial regression were used in all classification methods available at hemaClass.org. However, in GEP experiments, the number of probe-sets present on the microarray always outnumbers the sample size. Collinearity present among the features further aggravates the problem of identifying genes responsible for the underlying biological mechanism. Regression under these ill-posed circumstances is typically handled by so-called regularization. Here the elastic net penalty [32, 33], which is a combination of the Lasso [34] and ridge regression [35], was used. Similar to the Lasso, this penalty ensures simultaneous variable selection and model estimation by forcing small coefficients to be zero, yielding sparse solutions, but contrary to the Lasso the elastic net penalty is capable of selecting more variables than samples.
The elastic net penalty contains two parameters α and λ. The parameter α interpolates the elastic net penalty between the ridge and the Lasso penalty which corresponds to values of 0 and 1, respectively. The parameter λ determines the amount of shrinkage of the coefficients with larger values inducing more shrinkage until no variables are contained in the model. Regularized logistic and multinomial regression were performed with the R-package glmnet [32].
ABC/GCB classification.
The ABC/GCB classifier was established using logistic regression with an elastic net penalty on the LLMPP CHOP cohort. Of the 181 patients 74 were ABC, 76 were GCB, and 31 were non-classified. Using the 150 patients classified as either ABC or GCB, a dichotomous classifier capable of assigning each sample an estimate of the probability of being ABC was established.
To avoid over-fitting and limit the number of noise contributing genes, the elastic net parameters α and λ were chosen through 10 fold cross-validation. The parameter α was varied between 0.1 and 1 with step size 0.025 and log(λ) was varied between −10 and 2 with step size 0.06. The optimal combination of the parameters, and thereby the number of probe-sets, was found at the values minimizing the deviance. The results of the cross validations are shown in S1 Fig. The minimum deviance of 0.13 was attained at α = 0.15 and log(λ) = −7.29. This resulted in a gene expression classifier consisting of 381 probe-sets corresponding to 273 Ensembl Gene IDs.
When a tumour sample was classified according to the ABC/GCB classifier using cohort or InLab one-by-one normalized data the associated gene expressions are rescaled probe-set wise by the standard deviation of the LLMPP CHOP data divided by the standard deviation of the cohort or InLab reference data. For ExLab one-by-one normalization the data was used directly, since the training data for the ABC/GCB classifier was the same as the ExLab reference in the current study.
Wright’s ABC/GCB Classification.
Standard GEP ABC/GCB classification is done using Wright’s naive Bayes classifier. This method is not included on hemaClass.org, but is used in the current study for comparison of results from our elastic net classifier.
Bayesian compound covariate classification [36] with probeset list, weights and prior probabilities as described by [7] was used to perform ABC/GCB classification (specific details obtained by personal communication with George Wright). In addition to this the probesets were brought to the same scale as Lenz et al.’s [7] probesets by a rescaling of the probeset-wise standard deviation.
REGS classification.
In the paper by [10] REGS classifiers were established for prediction of resistance to the drugs C, H, and O. The classifiers were established on BCELL26 using regularized logistic regression analogous to the procedure described for the aforementioned ABC/GCB classifier. The number of microarray probes and corresponding genes for each of the REGS classifiers is shown in S5 Table.
The probability of resistance to the combination therapy, pCHO, was estimated based on the probabilities of drug resistance toward each of the three drugs: PC, PH, and PO, respectively. This probability is calculated as the posterior probability of being resistant, given resistance towards each of the individual drugs under the assumption of conditional independence and uniform priors. The formula is also known as Graham’s formula:
Derivation of the formula is shown in S1 Text section S2. If a drug is left out in the combination therapy the drug is simply removed from the formula. This appoach to resistance to the combination therapy was used in [10].
When a tumour sample is classified according to the REGS classifiers the associated gene expressions are rescaled probe-set wise by the standard deviation of DLBCL14 divided by the standard deviation of the cohort, InLab, or ExLab RMA reference dataset.
Resistance classifiers for other chemotherapeutic drugs and diseases are also available on hemaClass.org, though established elsewhere [37–39]. The Rituximab sensitivity classifier of [39] and [40] uses an elastic net approach, as above, but with three classes. The Melphalan sensitivity classifier of [37] uses sparse partial least squares to classify samples as either “sensitive”, “intermediate” or “resistant”. This classifier was developed for multiple myeloma (MM) patients and was thus based on other data [37].
BAGS classification.
The BAGS classifier established by [9] was based on multinomial regression regularized by an elastic net penalty. The classifier was trained on the Tonsil dataset in a manner similar to the ABC/GCB classifier. The BAGS classifier uses 327 probes corresponding to 205 Ensembl Gene IDs.
When a tumour sample is classified according to the BAGS classifier the associated gene expressions are rescaled probe-set wise by the standard deviation of the Tonsil data divided by the standard deviation of the cohort, InLab, or ExLab one-by-one reference dataset. The rescaling is performed to make the data comparable to the Tonsil dataset.
Inter-method reproducibility assessments
To evaluate the reproducibility of the class probabilities obtained through cohort or reference based RMA normalization, Pearson’s correlation coefficient for the logit-transformed probabilities and 95% confidence interval (CI) were calculated for each classifier and dataset. The identity and total least square regression lines were compared to assess bias in the estimated probabilities [41]. Total least squares regression was used as errors are present in both classification probabilities.
For each classifier the associated categories were obtained by thresholding the estimated probabilities. The ABC/GCB classifier was thresholded by 0.1 and 0.9, i.e. a tumour sample was classified as ABC when the estimated probability exceeded 0.9, GCB when it was below 0.1, and unclassified otherwise. For the BAGS classifier a tumour was classified as the class N, CB, CC, M, or PB with the highest probablity, if the associated probability exceeded 0.5 and unclassified when this threshold was not met for any subtype. For the REGS classifiers, C, H, O, and CHO combined, the thresholds were the 33% and 66% percentile of the estimated probabilities. The classifiers were applied to datasets using cohort, InLab, and ExLab one-by-one RMA normalization. Confusion matrices tabulating classifications from cohort normalized data versus InLab or ExLab normalized data were created, and from these the Accuracy (percent with similar classification to cohort), Cohen’s weighted κ, and corresponding 95% CIs were computed to assess the agreement between the determined classes.
Results
Using hemaClass.org
The website is an easy-to-use, interactive interface for the hemaClass package with the desired RMA normalization and the classification methods selected by the user. The usage of the website is largely self-explanatory with context-dependent boxes aiding users with further information, warnings, or errors. A comprehensive tutorial and guide to both the website and package is provided on the website or by running vignette(“howto”) in R. Uploaded patient samples are normalized and classified depending on settings chosen by the user.
ABC/GCB classification
In order to classify patients as ABC/GCB based on the implemented one-by-one normalization method a classifier based on the regularised logistic regression was established. The classifications were evaluated in the four clinical cohorts CHEPRETRO, MDFCI, IDRC, and LLMPP R-CHOP, which have all been classified according to Wright’s naive Bayes classifier [7, 9, 36]. The rates of agreement between the two classifiers based on cohort normalized data are shown in Table 2. Note that unclassified samples were included in the estimation of this rate i.e. a patient classified as ABC by one classifier but unclassified by the other is considered an error. The table also includes the alternative measure of agreement, Cohen’s weighted κ, where misclassifications involving the unclassified group are weighted by 1/2. High agreement between the two classifiers are observed for CHEPRETRO and LLMPP R-CHOP, while the accuracy and Cohen’s weighted κ are lower for IDRC and MDFCI. The accompanying confusion matrices are shown in the first rows of S1 Table.
The second column shows the accuracy of the classifiers with 95% CI. The third column shows the Cohen’s weighted κ with 95% CI.
The logit probability of being ABC estimated using the established cohort-based classifier was compared to the corresponding estimate obtained through the ExLab one-by-one normalized classification scheme in Fig 2A for CHEPRETRO. The probabilities estimated through the two methods are very similar, but values from ExLab normalization are slightly uncalibrated (or biased) and skewed downwards, indicating that different cut points might be used for the classifications. For InLab one-by-one normalization this error and bias is greatly minimized as shown in Fig 2B.
The areas marked with green indicate patients with similar classification between cohort based normalization and ExLab one-by-one normalization (A, C, E, G, I), or InLab one-by-one normalization (B, D, F, H, J). The areas marked with red indicate complete misclassifications. For ABC/GCB and REGS the white areas indicate unclassified and intermediate sensitivity, respectively, in at least one of the classifiers. The dashed and solid line show the identity and total least squares line, respectively.
For both methods, patients are classified as ABC when the estimated probability exceeds 0.9 and GCB when it is below 0.1. In Table 3 the resulting classifications for the four validation datasets are compared in terms of accuracy and Cohen’s weighted κ for cohort, using either Wright’s Bayes classifier or the elastic net classifier, against InLab or ExLab RMA reference normalization. For ExLab normalization CHEPRETRO and LLMPP R-CHOP both show a high Cohen’s weighted κ and accuracy considering that misclassifications involving unclassified samples count as errors, while values are moderate for MDFCI and IDRC when comparing to cohort based classifications for both the elastic net and Wright’s classifier. The reduced rate of agreement and Cohen’s weighted κ using ExLab one-by-one normalization in IDRC may be due to the samples being FFPE although this seems to be remedied by InLab one-by-one normalization. Accuracy and Cohen’s weighted κ are very high when comparing InLab based classifications to cohort based classifications for the elastic net classifier, but values are still moderate for MDFCI when comparing against Wright classifications. The accompanying confusion matrices for the elastic net classifier are shown in the lower part of S1 Table. Note that changes in predicted classes are mainly due to shifts into NC from ABC or GCB. Direct disagreements between the classifiers are seemingly rare and only occurs in the IDRC dataset.
The classifications are compared in terms of accuracy, Cohen’s weighted κ, and Pearson’s correlation coefficient r all supplied with 95% CIs. The comparisons in the first and last three columns are based on the ExLab and InLab reference based normalization method, respectively. For ABC/GCB classification, results from InLab or Exlab classification with the elasitic net classifier is compared against ABC/GCB classes for cohort normalized data obtained using both Wrights Bayes classifier and the elastic net classifier.
REGS classification
The probability of sensitivity towards each of the three drugs C, H, and O was estimated using hemaClass.org for both InLab and ExLab one-by-one normalization. The logit probabilities of sensitivity are plotted against those obtained by cohort based normalization in Fig 2 for CHEPRETRO data. Panels C, E, G, and I show the plots based on ExLab normalization and panels D, F, H, and J show the plots for the InLab based normalization. The probabilities obtained by ExLab and cohort based normalization are comparable, but similar to the other classifiers ExLab normalization leads to slightly skewed and biased probabilities, indicating that different cut points should be considered. The probabilities obtained by the InLab one-by-one normalization resembles the cohort based to a great extent, indicating that similar well-calibrated probabilities are obtainable for different laboratories by supplying an InLab reference set.
Based on the estimated probabilities, the patients were categorised as sensitive, intermediate, or resistant based on the thresholds specified in Section. The classes obtained by hemaClass.org are compared to those obtained by the cohort based approach in Table 3 in terms of rate of agreement and Cohen’s weighted κ. Low to moderate rates of agreement are observed for the ExLab normalized data, and again the InLab one-by-one approach yielded higher agreement with classifications obtained from cohort based normalization. The associated confusion matrices for InLab and ExLab one-by-one normalization are shown in S3 and S4 Tables, respectively.
BAGS classification
The BAGS classifier was evaluated in a manner similar to the ABC/GCB classifier. The logit probability of a patient’s tumour originating from one of the five subpopulations was estimated by means of cohort, InLab, and ExLab one-by-one normalization. The logit probabilities estimated by the ExLab normalization are plotted against logit probabilities from cohort based normalization in Fig 3 panels A, C, E, G, and I for CHEPRETRO. The correlations between the logit probabilities are highly significant, but also skewed and biased. The logit probabilities estimated using InLab one-by-one normalization are plotted against logit probabilities for the cohort based normalization in Fig 3 panels B, D, F, H, and J for CHEPRETRO. The InLab one-by-one normalization removes much of the aforementioned bias.
The coloured regions in the figure correspond to a threshold probability of 0.5. The dashed and solid line show the identity and total least squares line, respectively.
Based on the probabilities estimated by means of each of the three normalization methods the patients of the four clinical cohorts are grouped into the BAGS. The rates of agreement between the cohort based, and InLab or ExLab based classifications are shown in Table 3 along with Cohen’s weighted κ. For the BAGS classifier low rates of agreement were also found when comparing results from ExLab normalization with cohort normalization, while InLab normalization again led to improved agreement. The associated confusion matrices are shown in S2 Table.
Discussion
Despite the enormous amount of resources spent on developing molecular based cancer classification systems, most of these are still not available in daily clinical practice. To allow for fast validation of our recent findings [9, 10], we have developed an easily accessible web application that permits other users to apply ABC/GCB, BAGS, and drug resistance classification to their own datasets. Since GEP classifiers rely on RMA normalized data, we also implemented a reference based RMA normalization that allows samples to be pre-processed one-by-one instead of in entire cohorts.
One-by-one normalization was done using both an external RMA reference (ExLab) and a mimicked laboratory specific reference (InLab). Classifications obtained through one-by-one pre-processing performed by hemaClass.org were then compared to those obtained using cohort based normalization in four clinical cohorts. The results showed that a one-by-one array analysis approach is feasible and performs comparably with the whole cohort based method when an InLab reference is used, while differences between ExLab and cohort normalized classifications were too large to be satisfactory. Users are thus encouraged to supply their own reference for RMA pre-processing. It seems that this approach allows for a realistic application of microarray based lymphoma classification for research projects, and after suitable standardisation and calibration, even for clinical use.
The poorer results for ExLab one-by-one normalization and classification is likely caused by not accounting for lab-specific batch effects in the normalization process. To check for incorrect normalization hemaClass.org calculates the inter-quartile range (IQR) of the relative log expression (RLE) [42] across probe-sets of one-by-one normalized arrays. Large values of the RLE IQR indicates low-quality arrays resulting from e.g. incorrect laboratory procedures for cohort normalized data, but can also indicate normalization against an improper reference for one-by-one normalized data, and can thus be used in cases with uncertainty of the validity of a user supplied reference. Based on the validation in S1 Text section S6 we recommend removing samples from the analysis if the RLE IQR exceeds a value of 0.6. Users are, however, still encouraged to take proper precautions when selecting an RMA reference. For labs using a fixed InLab reference, the RLE IQR could also be used as a means of statistical process control. If RLE IQR values start to increase over time, this could indicate changes in laboratory procedures that might have to be adressed.
The present treatment algorithms for DLBCL are based on disease stage and clinical risk stratification without accounting for the underlying tumour-biology [43] and does not routinely account for the enormous variations in tumor biology between patients. The CHOP combination therapy (cyclophosphamide, doxorubicin, vincristine, and prednisone) has been the backbone of DLBCL therapy for decades with the only significant improvement being the addition of monoclonal CD20 antibodies (Rituximab) [44]. Despite the addition of antibody therapy to conventional chemotherapy only 55% of patients with poor risk disease achieve durable remission [45]. Thus, the need for new therapeutic options in DLBCL is obvious. Currently a number of new drugs have shown promising activity in DLBCL, but their role outside clinical trials have not been defined. These drugs are different from conventional chemotherapeutic compounds by targeting specific deregulated cell-cycle pathways [46]. An important example is inhibition of the NF-κB pathway by proteasome inhibitors (i.e. bortezomib). Interestingly, the constitutive activation of the NF-κB pathway is characteristic for the ABC subtype of DLBCL which consequently enhances the effect of bortezomib in this subtype [47].
With the increasing number of new drugs likely to become available over the next years and the fact that their efficacy may vary between subsets of patients defined by gene expression profiles, the current treatment of patients based on disease stage and clinical information alone will not be sufficient. hemaClass.org provides an example of fast processing of complex molecular information in a way that is simple and readily at hand for clinicians.
From a practical perspective a couple of challenges have to be addressed. First, the prognostic potential of the classifiers in these types of analyses has been established on specific tissues, which means one has to trust that the right tissue has been extracted and handled correctly through all the steps in the laboratory ending up with a reference array data set of sufficient quality. Our reference data have for instance been controlled by looking at the frequency of ABC/GCBs and BAGS classes and their survival curves as well as tissue control by experienced pathologists. Second, one has to address the need for a reference dataset for one-by-one RMA pre-processing established under similar conditions as the samples one wishes to classify. However, calibration of laboratory equipment is a well-known issue for many experimental techniques used in molecular biology like qPCR, mass spectrometry, immunohistochemistry, and flow cytometry. An important part of the calibration is that samples should be calibrated towards a dataset consisting of a representative set of tissue samples. We suggest that these challenges could be addressed by establishing a central tissue bank with officially approved data, e.g. by an international medical consortium, for the specific disease, similar to The Gene Expression Barcode [48] where consensus data for many tissues can already be found.
Alternatively the frozen RMA (fRMA) approach suggested by [30] could be used. This approach allows samples to be RMA normalized one-by-one against a frozen reference established across many different tissues and laboratories, taking the variation across laboratories and tissues for single probes into account. The current implementation of fRMA does not center and scale the data, so it cannot be used with the classifiers implemented in hemaClass.org, but had the training data for the classifiers been normalized with fRMA this might eliminate the need for centering and scaling sample data.
Another limitation of the current web application is that hemaClass.org only works with Affymetrix HG-U133 Plus 2.0 microarrays. This can, however, be circumvented by either re-annotation to HUGO Gene Nomenclature Committee (HGNC) approved symbols as suggested by [14] or by re-annotated chip definition files as suggested by [49]. At the moment we are working on extending the web application to other array types along these lines. In the future, gene expressions will likely be measured using RNA-seq technology instead of microarrays. By summarizing the expression levels at gene-annotations rather than affymetrix probes and scaling the data it might also be possible to use microarray based classifiers with RNA-seq data. Alternatively the transcriptome, for the training data used for establishing the BAGS and REGS classifiers, would have to be measured with RNA-seq and the classifiers retrained.
Traditionally ABC/GCB classification has been achieved using the naive Bayes classifier of [36] which is based on cohorts, MAS5.0 normalized arrays, and a Bayesian approach assuming an equal amount of ABC and GCB patients. However, a classifier based on logistic regression regularized by an elastic net penalty was implemented to make the classification more adaptable to RMA normalization and one-by-one processing. This classifier proved to be quite comparable with the naive Bayes classifier over the four studied datasets confirming the strong and stable signal of the ABC/GBC subclasses of DLBCL.
Under the validation of the one-by-one method one should notice that the unclassified is treated as a class in its own right. This implies a lower accuracy compared to an approach where the unclassified are left out of the validations. The latter approach seems reasonable as changing classifications to unclassified is less serious than changing real classes. Despite the disputed properties of Cohen’s κ the conservative approach is retained and the issue is addressed using a Cohen’s weighted κ approach. Given that an idealised approach is problematic to formulate, readers are encouraged to consider the confusion matrices in the supplementary material to make an overall evaluation of the performance.
ABC/GCB, BAGS, and REGS are only a part of the GEP-based armamentarium of methods for stratifying lymphoma patients into risk groups [50–52] and it would be interesting to extend the tool to include other classification systems. For a comprehensive review see [53]. To our knowledge only a few other classification methods have been made easily accessible as either web or desktop applications. Hopefully, this research will inspire bioinformaticians and statisticians to make their cancer classification methods easily accessible for usage, speedy validation, critical reviews, and mutual inspiration.
Conclusion
Although high throughput technologies in molecular biology have been around for almost two decades, only a few of the numerous biomarkers identified have undergone extensive validation and made it into the clinic [54]. It is our hope that making our own findings publicly available in this way will speed up validation and testing of BAGS and REGS by other researchers without having to delve into extensive bioinformatics implementations. Although hemaClass.org is still separated from the clinic we believe that a web based tool and suggestion for a clinical reference sample will bring cancer classification closer to the clinic. Hopefully, this work can also spawn interesting discussions on the clinical requirements of GEP based diagnostic and prognostic tools.
All material for reproducing this paper and its results is found at https://github.com/oncoclass/hemaclass-paper. Comments, suggestions, bug reports, and other issues are warmly welcome at https://github.com/oncoclass/hemaclass/issues or by mail to the corresponding author.
Supporting Information
S1 Text. All supplementary figures and tables, derivation of Grahams formula, detailed information on regular and one-by-one RMA normalization, and calculation of the RLE value and it’s impact on classification accuracy.
https://doi.org/10.1371/journal.pone.0163711.s001
(PDF)
S1 Table. Confusion tables for the ABC/GCB classifiers.
The columns represent cohort based normalisztion using the ABC/GCB classifier based on elastic net. The first part of the table compares Wright’s method for ABC/GCB classification with the elastic net based. In the second and third part ExLab and InLab reference based normalization is compared to cohort based normalization using the ABC/GCB classifier based on elastic net.
https://doi.org/10.1371/journal.pone.0163711.s002
(PDF)
S2 Table. Confusion tables for the BAGS classifier.
ExLab and InLab reference based normalization are shown in the columns and cohort normalization in the rows.
https://doi.org/10.1371/journal.pone.0163711.s003
(PDF)
S3 Table. Confusion tables for the REGS classifiers.
ExLab normalization is shown in the rows and cohort normalization in the columns.
https://doi.org/10.1371/journal.pone.0163711.s004
(PDF)
S4 Table. Confusion tables for the REGS classifiers.
InLab normalization is shown in the rows and cohort normalization in the columns. Note, 30 samples were used as reference data and hence not present in this table.
https://doi.org/10.1371/journal.pone.0163711.s005
(PDF)
S5 Table. Number of probes used in the classifiers and the number of corresponding HGNC and Ensembl gene IDs.
https://doi.org/10.1371/journal.pone.0163711.s006
(PDF)
S1 Fig. Ten fold cross validation for the parameters α and λ in a logistic regression regularized by elastic net.
In panels A and B the deviance is plotted against the model parameter α and regularization parameter λ, respectively. In Panel C the regularization curves are shown. Black and grey curves represent selected and non-selected probe-sets, respectively. Positive and negative coefficients indicate that high expression values for the associated gene are related to ABC and GCB, respectively. The red line indicates the model chosen through 10 fold cross validation. The gene symbols for the 20 probe-sets associated with the largest absolute coefficients in the chosen gene expression predictors are displayed in Panel C.
https://doi.org/10.1371/journal.pone.0163711.s008
(TIFF)
S2 Fig. Absolute value of the median (A) and IQR (B) RLE values for different RMA normalizations of the CHEPRETRO dataset and ROC curves for using these values to separate between an InLab and Exlab RMA reference (C, D).
https://doi.org/10.1371/journal.pone.0163711.s009
(TIFF)
S3 Fig. Absolute value of the median (A) and IQR (B) RLE values for different RMA normalizations of the LLMPP R-CHOP dataset and ROC curves for using these values to separate between an InLab and Exlab RMA reference (C, D).
https://doi.org/10.1371/journal.pone.0163711.s010
(TIFF)
S4 Fig. Absolute value of the median (A) and IQR (B) RLE values for different RMA normalizations of the LLMPP CHOP dataset and ROC curves for using these values to separate between an InLab and Exlab RMA reference (C, D).
https://doi.org/10.1371/journal.pone.0163711.s011
(TIFF)
S5 Fig. Absolute value of the median (A) and IQR (B) RLE values for different RMA normalizations of the IDRC dataset and ROC curves for using these values to separate between an InLab and Exlab RMA reference (C, D).
https://doi.org/10.1371/journal.pone.0163711.s012
(TIFF)
S6 Fig. Absolute value of the median (A) and IQR (B) RLE values for different RMA normalizations of the MDFCI dataset and ROC curves for using these values to separate between an InLab and Exlab RMA reference (C, D).
https://doi.org/10.1371/journal.pone.0163711.s013
(TIFF)
S7 Fig. Proportion of samples retained (A, C, E), and accuracy (B, D, F) of BAGS classification (percent similar with cohort based) against increasing RLE IQR thresholds for different references in CHEPRETRO. The vertical line marks the suggested threshold of 0.6.
https://doi.org/10.1371/journal.pone.0163711.s014
(TIFF)
S8 Fig. Proportion of samples retained (A, C, E), and accuracy (B, D, F) of ABC/GCB classification (percent similar with cohort based) against increasing RLE IQR thresholds for different references in CHEPRETRO. The vertical line marks the suggested threshold of 0.6.
https://doi.org/10.1371/journal.pone.0163711.s015
(TIFF)
S9 Fig. Proportion of samples retained (A, C, E), and accuracy (B, D, F) of REGS(combined) classification (percent similar with cohort based) against increasing RLE IQR thresholds for different references in CHEPRETRO. The vertical line marks the suggested threshold of 0.6.
https://doi.org/10.1371/journal.pone.0163711.s016
(TIFF)
S10 Fig. Proportion of samples retained (A, C, E), and accuracy (B, D, F) of BAGS classification (percent similar with cohort based) against increasing RLE IQR thresholds for different references in LLMPP R-CHOP. The vertical line marks the suggested threshold of 0.6.
https://doi.org/10.1371/journal.pone.0163711.s017
(TIFF)
S11 Fig. Proportion of samples retained (A, C, E), and accuracy (B, D, F) of ABC/GCB classification (percent similar with cohort based) against increasing RLE IQR thresholds for different references in LLMPP R-CHOP. The vertical line marks the suggested threshold of 0.6.
https://doi.org/10.1371/journal.pone.0163711.s018
(TIFF)
S12 Fig. Proportion of samples retained (A, C, E), and accuracy (B, D, F) of REGS(combined) classification (percent similar with cohort based) against increasing RLE IQR thresholds for different references in LLMPP R-CHOP. The vertical line marks the suggested threshold of 0.6.
https://doi.org/10.1371/journal.pone.0163711.s019
(TIFF)
Acknowledgments
We thank Mads Boye and Bo Nygaard Bai at IT Services, Aalborg University, for their assistance on deploying the public server. SF is supported by a Mobility PhD fellowship at the Graduate School of Health, Faculty of Health Sciences, Aarhus University. The research is supported by MSCNET, a translational programme studying cancer stem cells in multiple myeloma supported by the EU FP6, CHEPRE, a programme studying chemo sensitivity in malignant lymphoma by genomic signatures supported by the Danish Agency for Science, Technology, and Innovation, and the National Experimental Therapy Partnership (NEXT), which is financed by a grant from Innovation Fund Denmark, as well as Karen Elise Jensen Fonden. The technical assistance from Ann-Maria Jensen, Louise Hvilshøj Madsen, and Helle Høholt is greatly appreciated.
Author Contributions
- Conceptualization: SF AEB RFB MB.
- Data curation: SF AEB RFB.
- Formal analysis: SF AEB RFB LHJ MB.
- Funding acquisition: TCE HEJ KD MB.
- Investigation: SF AEB RFB LHJ MB.
- Methodology: SF AEB RFB LHJ MB.
- Project administration: MB.
- Resources: JSB AS KHY HEJ KD.
- Software: SF AEB RFB LHJ JH MB.
- Supervision: MB.
- Validation: SF AEB RFB JSB MB.
- Visualization: SF AEB RFB LHJ.
- Writing – original draft: SF AEB RFB MB.
- Writing – review & editing: SF AEB RFB LHJ JH KLN TCE JSB AS KHY HEJ KD MB.
References
- 1. Jaffe ES. The 2008 WHO Classification of Lymphomas: Implications for Clinical Practice and Translational Research. Hematology. 2009 Jan;2009(1):523–531. pmid:20008237
- 2. Sehn LH, Gascoyne RD. Diffuse Large B-Cell Lymphoma: Optimizing Outcome in the Context of Clinical and Biologic Heterogeneity. Blood. 2014;125(1):22–32. pmid:25499448
- 3. Alizadeh AA, Eisen MB, Davis RE, Ma C, Lossos IS, Rosenwald A, et al. Distinct Types of Diffuse Large B-Cell Lymphoma Identified by Gene Expression Profiling. Nature. 2000 Feb;403(6769):503–511. pmid:10676951
- 4. Rosenwald A, Wright G, Chan WC, Connors JM, Campo E, Fisher RI, et al. The Use of Molecular Profiling to Predict Survival after Chemotherapy for Diffuse Large-B-Cell Lymphoma. New England Journal of Medicine. 2002 Jun;346(25):1937–1947. pmid:12075054
- 5. Hans CP, Weisenburger DD, Greiner TC, Gascoyne RD, Delabie J, Ott G, et al. Confirmation of the Molecular Classification of Diffuse Large B-Cell Lymphoma by Immunohistochemistry using a Tissue Microarray. Blood. 2004 Jan;103(1):275–282. pmid:14504078
- 6. Poulsen CB, Borup R, Nielsen FC, Borregaard N, Hansen M, Grønbæk K, et al. Microarray-based classification of diffuse large B-cell lymphoma. European Journal of Haematology. 2005;74(6):453–465. pmid:15876249
- 7. Lenz G, Wright G, Dave SS, Xiao W, Powell J, Zhao H, et al. Stromal Gene Signatures in Large-B-Cell Lymphomas. New England Journal of Medicine. 2008 Nov;359(22):2313–23. pmid:19038878
- 8. Monti S, Chapuy B, Takeyama K, Rodig SJ, Hao Y, Yeda KT, et al. Integrative Analysis Reveals an Outcome-Associated and Targetable Pattern of p53 and Cell Cycle Deregulation in Diffuse Large B-Cell Lymphoma. Cancer Cell. 2012 Sep;22(3):359–372. pmid:22975378
- 9. Dybkær K, Bøgsted M, Falgreen S, Bødker JS, Kjeldsen MK, Schmitz A, et al. Diffuse Large B-Cell Lymphoma Classification System That Associates Normal B-cell Subset Phenotypes with Prognosis. Journal Of Clinical Oncology. 2015;33(12):1379–1388. pmid:25800755
- 10. Falgreen S, Dybkær K, Young KH, Xu-Monette ZY, El-Galaly TC, Laursen MB, et al. Predicting Response to Multidrug Regimens in Cancer Patients using Cell Line Experiments and Regularised Regression Models. BMC Cancer. 2015;15(235):1–15. pmid:25881228
- 11. The International Non-Hodgkin’s Lymphoma Prognostic Factors Project. A Predictive Model for Aggressive Non-Hodgkin’s Lymphoma. New England Journal of Medicine. 1993;329(14):987–994. pmid:8141877
- 12. Solal-Céligny P, Roy P, Colombat P, White J, Armitage JO, Arranz-Saez R, et al. Follicular lymphoma international prognostic index. Blood. 2004;104(5):1258–1265. pmid:15126323
- 13. Huang LT. An Integrated Method for Cancer Classification and Rule Extraction from Microarray Data. Journal of Biomedical Science. 2009 Jan;16(1):1–25. pmid:19272192
- 14. Care MA, Barrans S, Worrillow L, Jack A, Westhead DR, Tooze RM. A Microarray Platform-Independent Classification Tool for Cell of Origin Class Allows Comparative Analysis of Gene Expression in Diffuse Large B-Cell Lymphoma. PLOS ONE. 2013 Jan;8(2):e55895. pmid:23424639
- 15.
R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria; 2015. Available from: http://www.R-project.org/.
- 16.
Chang W, Cheng J, Allaire J, Xie Y, McPherson J. shiny: Web Application Framework for R; 2015. R package version 0.12.2. Available from: http://CRAN.R-project.org/package=shiny.
- 17. Gentleman RC, Carey VJ, Bates DM, Bolstad B, Dettling M, Dudoit S, et al. Bioconductor: Open Software Development for Computational Biology and Bioinformatics. Genome Biology. 2004 Jan;5(10):R80. pmid:15461798
- 18. Gautier L, Cope L, Bolstad BM, Irizarry RA. affy–Analysis of Affymetrix GeneChip Data at the Probe Level. Bioinformatics. 2004;20(3):307–315. pmid:14960456
- 19.
Eddelbuettel D. Seamless R and C++ Integration with Rcpp. 1st ed. New York: Springer-Verlag; 2013. https://doi.org/10.1007/978-1-4614-6868-4
- 20. Eddelbuettel D, François R. Rcpp: Seamless R and C++ Integration. Journal of Statistical Software. 2011;40(8):1–18.
- 21. Eddelbuettel D, Sanderson C. RcppArmadillo: Accelerating R with high-performance C++ linear algebra. Computational Statistics and Data Analysis. 2014 March;71:1054–1063.
- 22.
Sanderson C. Armadillo: An Open Source C++ Linear Algebra Library for Fast Prototyping and Computationally Intensive Experiments.; 2010. Available from: http://arma.sourceforge.net.
- 23. Barrett T, Wilhite SE, Ledoux P, Evangelista C, Kim IF, Tomashevsky M, et al. NCBI GEO: Archive for functional genomics data sets—update. Nucleic Acids Research. 2013;41(D1):D991–D995. pmid:23193258
- 24. Visco C, Li Y, Xu-Monette ZY, Miranda RN, Green TM, Li Y, et al. Comprehensive Gene Expression Profiling and Immunohistochemical Studies Support Application of Immunophenotypic Algorithm for Molecular Subtype Classification in Diffuse Large B-Cell Lymphoma: A Report from the International DLBCL Rituximab-CHOP Consortium. Leukemia. 2012;26(9):2103–2113. pmid:22437443
- 25.
Bilgrau AE, Larsen SF. DLBCLdata: Automated and reproducible download and preprocessing of DLBCL data; 2015. R package version 1.0. Available from: http://github.com/AEBilgrau/DLBCLdata.
- 26. Irizarry RA, Hobbs B, Collin F, Beazer-Barclay YD, Antonellis KJ, Scherf U, et al. Exploration, Normalization, and Summaries of High Density Oligonucleotide Array Probe Level Data. Biostatistics. 2003;4(2):249–264. pmid:12925520
- 27. Irizarry RA, Bolstad BM, Collin F, Cope LM, Hobbs B, Speed TP. Summaries of Affymetrix GeneChip Probe Level Data. Nucleic Acids Research. 2003;31(4):e15. pmid:12582260
- 28.
Bolstad BM. Low-level Analysis of High-Density Oligonucleotide Array Data: Background, Normalization and Summarization. University of California, Berkeley; 2004.
- 29. Katz S, Irizarry RA, Lin X, Tripputi M, Porter MW A summarization approach for Affymetrix GeneChip data using a reference training set from a large, biologically diverse database BMC bioinformatics. 2006;7:464 pmid:17059591
- 30. McCall MN, Bolstad BM, Irizarry RA. Frozen robust multiarray analysis (fRMA) Biostatistics. 2010;11(2):242–253 pmid:20097884
- 31. Bolstad BM, Irizarry RA, Astrand M, Speed TP. A Comparison of Normalization Methods for High Density Oligonucleotide Array Data Based on Variance and Bias. Bioinformatics. 2003 Jan;19(2):185–193. pmid:12538238
- 32. Friedman J, Hastie T, Tibshirani R. Regularization Paths for Generalized Linear Models via Coordinate Descent. Journal Statistical Software. 2010;33(1):1–24.
- 33. Zou H, Hastie T. Regularization and Variable Selection via the Elastic Net. Journal of the Royal Statistical Society Series B. 2005 Apr;67(2):301–320.
- 34. Tibshirani R. Regression Shrinkage and Selection via the Lasso. Journal of the Royal Statistical Society Series B. 1996;58(1):267–288.
- 35. Hoerl AE, Kennard RW. Ridge Regression: Biased Estimation for Nonorthogonal Problems. Technometrics. 1970;12(1):55–67.
- 36. Wright G, Tan B, Rosenwald A, Hurt EH, Wiestner A, Staudt LM. A Gene Expression-Based Method to Diagnose Clinically Distinct Subgroups of Diffuse Large B Cell Lymphoma. PNAS. 2003 Aug;100(17):9991–9996. pmid:12900505
- 37. Bøgsted M, Holst JM, Fogd K, Falgreen S, Sørensen S, Schmitz A, et al. Generation of A Predictive Melphalan Resistance Index by Drug Screen of B-cell Cancer Cell Lines. PLOS ONE. 2011;6(4):e19322. pmid:21559449
- 38. Bøgsted M, Bilgrau AE, Wardell CP, Bertsch U, Schmitz A, Bødker JS, et al. Proof of the Concept to Use a Malignant B Cell Line Drug Screen Strategy for Identification and Weight of Melphalan Resistance Genes in Multiple Myeloma. PLOS ONE. 2013;8(12):e83252. pmid:24376673
- 39. Laursen MB, Falgreen S, Bødker JS, Schmitz A, Kjeldsen MK, Sørensen S, et al. Human B-cell Cancer Cell Lines as a Preclinical Model for Studies of Drug Effect in Diffuse Large B-Cell Lymphoma and Multiple Myeloma. Experimental Hematology. 2014;42(11):927–938. pmid:25072621
- 40.
Laursen MB, Sørensen S, Falgreen S, Reinholt LR, Schmitz A, Bødker JS, et al. Studies of Anti-CD20 Antibody Mediated Complement Dependent Cytotoxicity in a Preclinical Cell Line Model of Diffuse Large B-Cell Lymphoma. Submitted. 2015;.
- 41. Chen S, Billings SA, Luo W. Orthogonal Least Squares Methods and their Application to Non-Linear System Identification. International Journal of Control. 1989 Nov;50(5):1873–1896.
- 42. Experimental Design and Low-Level Analysis of Microarray Data Bolstad BM, Collin F, Simpson KM, Irizarry RA, Speed TP International Review of Neurobiology. 2004;60:25–58 pmid:15474586
- 43. Schmoll HJ, Van Cutsem E, Stein A, Valentini V, Glimelius B, Haustermans K, et al. ESMO Consensus Guidelines for Management of Patients with Colon and Rectal Cancer. A Personalized Approach to Clinical Decision Making. Annals of Oncology. 2012 Oct;23(10):2479–2516. pmid:23012255
- 44. Coiffier B, Lepage E, Briere J, Herbrecht R, Tilly H, Bouabdallah R, et al. CHOP Chemotherapy Plus Rituximab Compared with CHOP Alone in Elderly Patients with Diffuse Large-B-Cell Lymphoma. New England Journal of Medicine. 2002 Jan;346(4):235–242. pmid:11807147
- 45. Ziepert M, Hasenclever D, Kuhnt E, Glass B, Schmitz N, Pfreundschuh M, et al. Standard International Prognostic Index Remains a Valid Predictor of Outcome for Patients with Aggressive CD20+ B-Cell Lymphoma in the Rituximab Era. Journal of Clinical Oncology. 2010 May;28(14):2373–2380. pmid:20385988
- 46. Friedberg JW. New strategies in Diffuse Large B-Cell Lymphoma: Translating Findings from Gene Expression Analyses Into Clinical Practice. Clinical Cancer Research. 2011 Oct;17(19):6112–6117. pmid:21844008
- 47. Dunleavy K, Pittaluga S, Czuczman MS, Dave SS, Wright G, Grant N, et al. Differential Efficacy of Bortezomib Plus Chemotherapy Within Molecular Subtypes of Diffuse Large B-Cell Lymphoma. Blood. 2009;113(24):6069–6076. pmid:19380866
- 48. The Gene Expression Barcode 3.0: Improved data processing and mining tools McCall MN, Jaffee HA, Zelisko SJ, Sinha N, Hooiveld G, Irizarry RA, Zilliox MJ Nucleic Acids Research. 2014;42:938–943 pmid:24271388
- 49. Dai M, Wang P, Boyd AD, Kostov G, Athey B, Jones EG, et al. Evolving Gene/Transcript Definitions Significantly Alter the Interpretation of GeneChip Data. Nucleic Acids Research. 2005 Jan;33(20):e175. pmid:16284200
- 50. Shipp MA, Ross KN, Tamayo P, Weng AP, Kutok JL, Aguiar RCT, et al. Diffuse Large B-Cell Lymphoma Outcome Prediction by Gene-Expression Profiling and Supervised Machine Learning. Nature Medicine. 2002 Jan;8(1):68–74. pmid:11786909
- 51. Lossos IS, Czerwinski DK, Alizadeh AA, Wechser MA, Tibshirani R, Botstein D, et al. Prediction of Survival in Diffuse Large-B-Cell Lymphoma Based on the Expression of Six Genes. New England Journal of Medicine. 2004 Apr;350(18):1828–1837. pmid:15115829
- 52. Malumbres R, Chen J, Tibshirani R, Johnson NA, Sehn LH, Natkunam Y, et al. Paraffin-Based 6-Gene Model Predicts Outcome in Diffuse Large B-cell Lymphoma Patients Treated with R-CHOP. Blood. 2008 Jun;111(12):5509–5514. pmid:18445689
- 53. Coutinho R, Gribben J. Biomarkers of Diffuse Large B-Cell Lymphoma: Impact on Diagnosis, Treatment, and Prognosis. Current Biomarker Finding. 2013;3:17–34.
- 54. Chen X, Andersson R, Cho WC, Christiani D, Coico R, Drazen J, et al. The International Effort: Building the Bridge for Translational Medicine: Report of the 1st International Conference of Translational Medicine (ICTM). Clinical and Translational Medicine. 2012 Jan;1(1):1–15. pmid:23369397