DeepCellState network architecture for predicting cell type specific drug response

The network architecture of DeepCellState, featuring a common encoder and cell type specific decoders, is shown in Fig 1B. The encoder part uses two dense layers, with dropout applied to the input for increased generalization capabilities by forcing the model to denoise the input [19]. This is followed by a latent vector, which in turn is the input for the decoder, consisting of two layers with the same dimensions as the dense layers in the encoder, but in reverse order. The encoder side is trained using transcriptional drug response data from multiple cell types, whereas distinct decoders are trained for each cell type separately. Additional details about the network architecture and training strategy are provided in Methods.

Initial training and evaluation of model on two cell types

We initially developed the framework for two cell types, retrieving 12737 and 12031 transcriptional profiles from the LINCS database [6] measuring the drug perturbation response in MCF-7 and PC-3 cell lines respectively, to use for initial training and evaluation (Methods). The transcriptional response profiles, measured at 978 “landmark genes” on the L1000 platform, were averaged across treatment time and doses in order to decrease noise [3]. We only kept profiles of drugs that were available for both MCF-7 and PC-3 for the convenience of implementation, resulting in 1750 averaged profiles for each cell type.

A naïve baseline method for predicting the transcriptional response in a cell type after treatment with a specific drug is to predict that the response to that drug will be the same as in another cell type. After training our network and using 10-fold cross-validation, holding out 10% of drugs, we obtained an average Pearson correlation coefficient (PCC) of 0.60 between predicted and actual treatment response to drugs not seen by the network, a significant improvement (p<1e-300, t-test) compared to the baseline average correlation of 0.28 between the responses in the input and output cell types (Figs 2A, 2B, 2C, 2D and S2), corresponding to an average fold-increase of 2.14 over baseline.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Evaluation of the DeepCellState performance.

(A): 10-fold cross-validation results. DeepCellState performance compared to the baseline. (B): Distribution of fold-change in DeepCellState performance over baseline for predicting MCF-7 (blue) and PC-3 (red) response. (C): From top to bottom pentobarbital response in PC-3 (baseline), MCF-7 pentobarbital response predicted by DeepCellState based on the PC-3 response, and the observed MCF-7 response (ground truth). (D): Scatter plot of individual gene values predicted for pentobarbital response compared to the observed response in MCF-7 cells. (E): Comparison of DeepCellState and baseline performance for prediction of statin response from an external data set. (F): Comparison of predicted anticancer drug responses. HepG2 profiles are used as input for the model while the output profiles are for MCF-7 cells. DeepCellState* represents results obtained by applying transfer learning: performing additional training on 14 drugs and predicting one which is repeated 15 times each time leaving out one drug. (G): Comparison with alternative methods for paired data modeling, using PC-3 responses to predict MCF-7 samples.

https://doi.org/10.1371/journal.pcbi.1009465.g002

We wanted to explore how the results will change if the number of drugs was different for PC-3 and MCF-7 cell types. We repeated the 10-fold cross-validation 3 times: with 70% of PC-3 training profiles removed, with 70% of MCF-7 training profiles removed, and 70% of both PC-3 and MCF-7 training profiles removed. In each case, we measured MCF-7 profiles prediction accuracy using PC-3 profiles as input. As expected, in the latter case, the performance was the worst, dropping to 0.50 PCC. Interestingly, for the other two experiments similar performance was obtained. For the case when MCF-7 training data was reduced, 0.55 PCC was achieved, while for the case when 70% of PC-3 profiles are dropped, the performance decreased to 0.56. These results suggest that the output profile’s cell type data is slightly more important than the number of training profiles which are used for the model input.

To make sure that the training was not biased by being exposed to drugs with similar targets, we next revised the testing strategy to hold out entire drug families with the same modes of action [20], yielding similar results (S3A Fig and S1 Table and Methods).

Generalization of the method to multiple cell types

Proceeding to generalize the method to multiple cell types, we incrementally added transcription profiles from A375, HA1E, HT29, HELA, and YAPC cells (1796, 1796, 1750, 1570, and 1570 number of profiles respectively after averaging across treatment times and dosages), obtaining increased performance when the prediction was averaged over multiple input cell types compared to the two cell types case (PCC = 0.66, p = 1.88e-59, S3B Fig and S2 Table). The total time to train using this much larger data was only one hour and forty minutes, increasing less than two times compared to the two cell type model. We also tested the prediction capability using completely unseen cell types as input. We have used A375 profiles as input to predict MCF-7 profiles by a model which was trained using only MCF-7 and PC-3 profiles. This resulted in a lower performance compared to using PC-3 profiles as input, yet the results were still significantly better than baseline (S3C Fig, PCC = 0.49 compared to baseline PCC = 0.28, p<1e-300).

Evaluation of trained model on external data and application of transfer learning

To further validate the performance of DeepCellState, we evaluated its predictive capabilities on independently generated data sets on expression profiling platforms other than the L1000 platform used for the training data. In the first dataset we used, the expression profiles of four statins were measured in HepG2, MCF-7, and THP-1 cells [21] using CAGE technology. Using HepG2 profiles as input and predicting MCF-7 response, the DeepCellState predictions showed a correlation of 0.43 with the true response in MCF-7 cells (1.64-fold over baseline, p = 0.037, S3 Table and Fig 2E). Next, we tested our method using data from a study measuring the response to 15 anticancer agents [22] on Affymetrix Human Genome U133A 2.0 Array platform, achieving 0.41 PCC (Fig 2F) compared to 0.28 PCC for baseline (1.46-fold improvement, p = 0.026, S4 Table). The common drugs from the two external data sets were removed from the training and validation sets. More details on the data processing are in Methods.

Performance on the anticancer external dataset could be further improved through transfer learning, performing additional training of our model on a subset of the external data set. To this end, we used the model trained on the LINCS data and additionally trained it on 14 drugs from the anticancer set to predict the effects of treatment with the 15th drug. This was repeated 15 times, each time leaving out one drug, resulting in an average performance of 0.56 PCC (p = 4.04e-06 compared to baseline). This result suggests that the transfer learning approach is useful to complete a small drug response data set which is not large enough to train the deep learning model by itself.

Prediction of transcriptional effects of loss of function

We further hypothesized that the DeepCellState approach may be useful for predicting the response to cellular perturbations other than drug treatment. To test this, we added transcriptional profiles from loss of function (LoF) experiments from the same database to the training set. Using the updated set, we obtained an average correlation of 0.56 (corresponding to an average fold-increase over baseline of 1.6, p<1e-300, S3D Fig) on the LoF experiments, the lower performance compared to drug perturbations likely due to that the number of available profiles for LoF was substantially lower than the number of drug treatment profiles (10230 LoF profiles in total compared to 24768 drug perturbation profiles for MCF-7 and PC-3 before averaging).

Comparison with alternative approaches

We benchmarked DeepCellState against other potential methods to predict the cell response given the response in another cell. For this purpose, we focused on paired MCF-7 and PC-3 samples, with 30% data used for the test set. Recently, variational auto-encoders (VAEs) were used to deal with the paired biological data [23]. Anchor loss was used in this approach to ensure that paired samples are encoded closely in the latent space. Generative adversarial networks (GANs) were also applied to paired and semi-paired biological data [24,25]. The authors of MAGAN [25] proposed a dual GAN setup with a special correspondence loss for the labeled data. In the dual GAN framework, each generator learns a mapping from one type of data to another in order to generate increasingly realistic output that cannot be distinguished from real data by a discriminator.

scGen [15] was proposed to predict single cell perturbation responses based on VAE architecture, where an encoder is used to learn the semantic embedding of both the normal cell state and the stimulated state. The difference between such embeddings could then be considered as the embedding of the drug stimulation. When applying the method onto new cell types, the authors proposed to perform latent space vector arithmetic, adding the stimulation embedding to the normal cell embedding, and to pass the resulting vector through the decoder, obtaining the predicted responses. Since we have multiple drugs and the amount of data for each drug is limited, we modified the method to learn the difference between different cell types (Methods).

Comparison of the tested methods on the benchmark dataset is shown in Fig 2G and S5 Table. We were unable to obtain a good performance using the dual GAN approach (PCC = 0.26), likely due to the training data being too small. The data for each cell type has very high variance, making it hard for a discriminator to learn the difference between simulated and real cell response. scGen achieved an average correlation of 0.22, the main reason being that scGen is designed specifically for one kind of stimulation. However, in the problem we are trying to solve, multiple drugs and multiple cells need to be handled simultaneously and scGen showed limited capacity in handling this multi-task problem. The VAE using anchor loss obtained significant improvement over the baseline (PCC = 0.48 compared to baseline PCC = 0.28), however, the profiles predicted by DeepCellState were considerably more accurate (PCC = 0.61, p<2.09e-40). After these experiments, we attempted to incorporate VAE as well as GAN into the DeepCellState approach, but the improvement was not significant enough to justify the added complexity.

We next performed comparison with methods from study by Hodos et al (DNPP and FaLRTC) [2] using all the cell types and performing 10-fold cross-validation. Since VAE and GAN methods are designed specifically for translating between two domains, we could not include them in this comparison. We have achieved 0.53 PCC for DNPP, 0.49 for FaLRTC, and 0.65 for our approach (S4 Fig). We repeated the comparison using the processed data from the original study which contains considerably more cell types (71 vs 30). Our method still outperformed DNPP and FaLRTC, achieving 0.56 PCC, while DNPP and FaLRTC both achieved 0.46 (S5 Fig). The details of this comparison are provided on our GitHub repository.

The tested deep learning approaches had similar running time to our method. Since FaLRTC does not require any training, it can be run on the CPU and produces results for all the data in less than fifteen minutes. Our method took about three hours for the same data, however after the training is complete, the inference time for a new drug profile is only a fraction of a second

Model performance analysis

We next investigated more deeply the relationship between the number of input profiles used in training and the prediction performance. Results showed that significant improvement over baseline was possible already for a small number of training examples, with higher performance as the number of training examples increased (Fig 3A). To study the impact of the latent layer size on the performance of DeepCellState, we tested five different layer sizes (8, 16, 32, 64 and 128 neurons) and trained our model 10 times for each size. Average results for the test set (shown in Fig 3B) showed that performance started to drop significantly at a latent layer size below 32.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Analysis of the developed method.

(A): Average PCC obtained for different size subsets of the training set. The y-axis indicates average correlation, and the x-axis indicates number of drugs used. (B): Dependency between the number of latent nodes and the average PCC. (C): t-SNE clustering of profiles in the input and the latent space. Profiles used are from test fold 1 of the two cell type model (MCF7, PC3) which achieved correlation above 0.40 PCC. (D): Matrix displaying latent vector response correlation. Drugs on each axis are sorted from high to low correlation in the input space. (E): Overlaps of top target genes of TFs based on chromatin immunoprecipitation data compared to target gene prediction based on DeepCellState trained model. (F): Heatmap of most important genes for MCF-7 and PC-3 decoders.

https://doi.org/10.1371/journal.pcbi.1009465.g003

To pinpoint the benefit of the skip connection and regularization, we trained a simple model without a dropout layer, L1 regularization, and skip connection. For the two cell type case, the drop in performance was modest (PCC = 0.58, p<3.29e-9). However, for the external datasets the drop in performance was much more significant. For the statin data, we got an average PCC decrease of 0.04, and for the anticancer data the decrease was even higher, about 0.09 on average across three runs. We believe there are two reasons for this. Firstly, skip connection helps the model to deal with situations when it cannot predict a gene value. In this case, the value from the input layer can be copied directly. Secondly, without the regularization the model captures the technical bias of the LINCS data, performing significantly worse when data generated by a different platform is used. Overall, these results confirm that skip connection and regularization are beneficial for our model. Finally, we tested the benefits of adding the regularization techniques one by one and the results have reinforced our previous conclusion (S6 Table).

Biological interpretation of trained model

We set out to analyze general and cell type specific regulatory properties captured by the trained model, expecting that the latent vector itself should capture a general response to the input perturbations, whereas the decoder parts should represent behavior related to functionality specific to each cell type. To this end, we computed the PCC between the output of the nodes in the latent layer for the drugs in the test set between MCF-7 and PC-3 cells. Indeed, although considerably different, the inputs from the same drug in the two cell types consistently yielded similar output in the latent layer as illustrated in a t-SNE plot (Fig 3C) in the input and the latent space, where drugs with profiles divergent for different cell types in the input space (Fig 3C, left panel) could be observed to be encoded using very similar latent representations (Fig 3C, right panel). Similarly, a matrix containing correlations between latent responses for MCF-7 and PC-3 cells, sorted by the magnitude of the correlation of the input profiles, showed a consistently strong signal on the diagonal (Fig 3D), independent of the correlation between the responses in the input space.

It has been shown previously that autoencoders are able to capture transcription factor (TF) gene regulatory relationships [14,26]. To test if this was the case also for the DeepCellState framework, we used TF target gene lists from ChIP-Atlas [27] which include MCF-7 specific targets. After filtering, we computed the overlap between the targets implied by our model (see Methods) and targets based on ChIP-seq binding data, obtaining average fold enrichment of 2.5 (Fig 3E), with higher enrichment for the MCF-7 targets of four out of six transcription factors, compared to the general set of targets according to ChIP-Atlas. Identification of the most important genes in the cell type specific decoder response (see Methods) revealed enrichment of biological processes [28] that matched known properties of each cell type (Fig 3F, S7 Table). For the top genes common to the decoders, the high ranking hits were enriched for terms including “response to abiotic stimulus” (p = 0.0144) and “cellular response to chemical stimulus” (p = 0.018), expected since the model was trained using drug profiles (S8 Table). Additionally, several significantly enriched terms (p = 1.12e-04, p = 2.32e-04) were related to positive regulation of cell death and apoptotic process, likely due to some doses being toxic as previously observed [29]. Conversely, top genes of the MCF-7 decoder included several genes involved in mammary gland development (SRC, STAT5B, EGF). Similarly, BMP4, GLI2, and FGFR2, which are known to be involved in prostate morphogenesis, were among the top important genes of the PC-3 decoder. Taken together, these results suggest that the trained network is able to capture known biological processes and regulatory relationships, and that additional, deeper analysis may reveal relationships previously unnoticed.

LINCS data processing

Drug response expression profiles from LINCS phase 2 dataset (GSE70138) were used for training and evaluation. The latest level 5 data (signatures from aggregating replicates and converted to z-scores) was used (downloaded May 27, 2020). There are usually several treatment times and doses for each drug treatment; we averaged the profiles for the same drug across different timepoints and doses to reduce noise. Finally, the profiles were normalized to the range from -1 to +1 based on the whole data matrix.

Model architecture and training

The architecture of the autoencoder used in our method is shown in Fig 1B. The encoder consists of two dense layers with 512 and 256 neurons respectively. Dropout is applied to the input to increase the generalization capabilities of the model. The dropout rate is 0.5 which means half of the input genes will be set to 0 during training. This makes our model act as a denoising autoencoder since all values in the output layer need to be predicted. The last layer of the encoder is flattened and fed into a dense layer with 128 nodes, which represents our latent vector. The decoder takes the latent vector as an input and consists of two layers with 256 and 512 neurons respectively. The output layer has a direct connection to the dropout layer. The motivation is that some of the input can be “leaked” into the output [31].

We use L1 regularization for the latent layer, which acts as a sparsity constraint on the activity of the latent representations. Activation function is leaky relu [32] with alpha equal to 0.2 for all layers except the output layer, which uses tanh activation. The optimizer used is ADAM [33] with a learning rate equal to 1e-4 and batch size is 128. Validation set is employed for early stopping with patience of 5 epochs.

To initialize our model, we first train a regular autoencoder using profiles of all cell types as both input and output. Then we create a separate decoder for each cell type by copying the weights from the autoencoder. These decoders are trained one by one together with the shared encoder, where the inputs are profiles from all cell types and the outputs are profiles with the decoder’s cell type only. In the final stage, when there has been no improvement in the validation set for 5 epochs, the encoder weights are frozen and only decoders are trained until there is no improvement. When making a prediction for a certain drug and cell combination, any cell profile can be used as an input. The cell specific decoder is used to make the final prediction.

DeepCellState is implemented in Python 3 using TensorFlow [34] library and the models were trained using NVIDIA Quadro RTX 6000 GPU. Training two cell decoders takes about one hour to complete, however new predictions can be made in less than a second.

External data processing

We trained an encoder and three decoders (HepG2, MCF-7 and PC-3) for external validation. The common drugs from the two external data sets were removed from the training and validation sets. The statin data set was generated by converting CAGE data to gene expression data by mapping to GENCODE version 34. CAGE tags in a 1000 base pair window around a landmark gene start were summed. We added a small value c = 2 to each gene value to avoid problems when calculating log fold change. Each generated profile was multiplied by a million and divided by the total sum of the profile. The resulting profiles for controls and treatments were averaged across replicates followed by a computation of log fold change after treatment for each gene. For the anticancer data we directly calculated log fold change between the average control and treatment values. In both statin and anticancer tests, we only used landmark genes, setting missing gene values to 0. Only the 24 h time point was used as it is the most common time point in LINCS phase 2 compound data (96% of the data).

Methods related to analysis/interpretation of model

Since it is hard to directly learn the inner-workings of the deep learning model, we study it by feeding in different inputs and analyzing the output. To calculate the top genes for each decoder, we iterated through all the data, for each profile using 100 random genes subsets of size 100 to input in the model. The gene sets which performed the best to make the prediction were considered to be important. For each profile we picked the top 10 subsets that performed the best. After doing this for all the profiles, we ranked the genes based on their frequency in these subsets.

For the TF analysis, we downloaded target genes for 22 TFs profiled in MCF-7 cells from ChIP-Atlas [27]. We filtered them by retaining TFs that were upregulated in at least one drug treatment profile (i.e. has value > 0.5), and that had at least 50 targets in ChIP-Atlas (with targets defined as genes having a binding score of at least 100). This resulted in 6 remaining TFs and we picked the top 50 target genes for each. To make a prediction of target genes using DeepCellState, we started with a random profile and performed gradient descent optimization to find the input profile that maximizes the TF. As the obtained profile will also have other upregulated genes, the top 50 with the highest values are the DeepCellState prediction for the TF targets. For each TF we calculated fold enrichment and p-values using binomial statistic for the overlap between the DeepCellState set and the ChIP-Atlas set, using a similar method as described on the PANTHER website [28].

Comparison with alternative approaches

scGen [15] was designed to predict single cell perturbation responses based on VAE architecture, where an encoder f is used to learn the semantic embedding, z, of both the normal cell state and the stimulated state. The difference between such embeddings could then be considered as the embedding of the drug stimulation. When applying the method onto new cell types, the authors proposed to perform latent space vector arithmetic, adding the stimulation embedding, δ, to the normal cell embedding, and to pass the resulting vector through the decoder, h, obtaining the predicted responses.

Mathematically, we have where δ_n2s is the stimulation embedding; z_stimulated is the embedding of the stimulated cell profile; z_normal is the embedding of the normal cell profile; x_normal is the input normal cell profile; is the predicted stimulated cell profile.

Regarding our task, since we have multiple drugs and the amount of data for each drug is limited, we modified the method to learn the difference between different cell types, instead of the difference between conditions. Consequently, we have

Both the encoder, f, and decoder, h, are trained using the standard VAE training protocol described in the original paper [15] with our data as discussed above.

MAGAN [25] was developed to tackle the problem of aligning corresponding sets of samples. MAGAN is composed of two GANs, each with a generator network G that takes as input X and outputs a target dataset X′. For the minibatch (x₁, x₂), the loss for generator G₁₂ is defined as sum of reconstruction loss, discriminator loss, and correspondence loss:

The loss for discriminator D₁ is defined as:

Since in our task all the data are paired, we used the following correspondence loss, as suggested in the original paper [25]:

We used the total loss to train two generators, G_pc3→mcf7 and G_mcf7→pc3, together with two discriminators, D_pc3 and D_mcf7.

To implement VAE with anchor loss, we have trained a standard VAE with two decoders and employed the following loss to ensure that all the m paired samples are encoded close to each other by encoder E in the latent space:

For DNPP and FaLRTC, we generated 10 test folds, selecting 10% of profiles randomly. When constructing the input for the methods, we set the test set profiles to “nan” values. The two methods then fill out the “nan” values and the new profiles are compared against the ground truth.