Learning curves for drug response prediction in cancer cell lines - PubMed (original) (raw)

doi: 10.1186/s12859-021-04163-y.

Thomas Brettin 3 4, Yvonne A Evrard 5, Yitan Zhu 6 3, Hyunseung Yoo 6 3, Fangfang Xia 6 3, Songhao Jiang 7, Austin Clyde 6 7, Maulik Shukla 6 3, Michael Fonstein 8, James H Doroshow 9, Rick L Stevens 4 7

Affiliations

Learning curves for drug response prediction in cancer cell lines

Alexander Partin et al. BMC Bioinformatics. 2021.

Abstract

Background: Motivated by the size and availability of cell line drug sensitivity data, researchers have been developing machine learning (ML) models for predicting drug response to advance cancer treatment. As drug sensitivity studies continue generating drug response data, a common question is whether the generalization performance of existing prediction models can be further improved with more training data.

Methods: We utilize empirical learning curves for evaluating and comparing the data scaling properties of two neural networks (NNs) and two gradient boosting decision tree (GBDT) models trained on four cell line drug screening datasets. The learning curves are accurately fitted to a power law model, providing a framework for assessing the data scaling behavior of these models.

Results: The curves demonstrate that no single model dominates in terms of prediction performance across all datasets and training sizes, thus suggesting that the actual shape of these curves depends on the unique pair of an ML model and a dataset. The multi-input NN (mNN), in which gene expressions of cancer cells and molecular drug descriptors are input into separate subnetworks, outperforms a single-input NN (sNN), where the cell and drug features are concatenated for the input layer. In contrast, a GBDT with hyperparameter tuning exhibits superior performance as compared with both NNs at the lower range of training set sizes for two of the tested datasets, whereas the mNN consistently performs better at the higher range of training sizes. Moreover, the trajectory of the curves suggests that increasing the sample size is expected to further improve prediction scores of both NNs. These observations demonstrate the benefit of using learning curves to evaluate prediction models, providing a broader perspective on the overall data scaling characteristics.

Conclusions: A fitted power law learning curve provides a forward-looking metric for analyzing prediction performance and can serve as a co-design tool to guide experimental biologists and computational scientists in the design of future experiments in prospective research studies.

Keywords: Cell line; Deep learning; Drug response prediction; Learning curve; Machine learning; Power law.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no competing interests.

Figures

Fig. 1

Fig. 1

Learning curve plotted on a linear scale in (a) and on a log scale in (b). The vertical axis is the generalization score in terms of the mean absolute error of model predictions. Each data point is the averaged prediction error, computed on a test set, of a gradient boosting decision tree (GBDT) that was trained on a subset of training samples of the GDSC1 dataset

Fig. 2

Fig. 2

Histograms of dose-independent drug response (AUC) values of the four datasets listed in Table 1

Fig. 3

Fig. 3

Two neural network architectures used in the analysis: a single-input network (sNN, 4.254 million trainable parameters) and b multi-input network (mNN, 4.250 million trainable parameters)

Fig. 4

Fig. 4

Workflow for generating learning curve data, LC, for a single split of a dataset. A single dataset split includes three sample sets: training T, validation V, and test E

Fig. 5

Fig. 5

Learning curves generated by using dGBDT for multiple data splits of each of the datasets in Table 1. a The entire set of learning curve scores, LCraw, where each data point is the mean absolute error of predictions computed on test set E as a function of the training set size mk. A subset of scores in which the sample size is above mkmin (dashed black line) was considered for curve fitting. b Three curves were generated to represent the fit: q0.1 (blue curve) and q0.9 (green curve) representing the variability of the fit, and y~ (black curve) representing the learning curve fit

Fig. 6

Fig. 6

Comparison of learning curves of hGBDT, sNN, and mNN, for each of the four drug response datasets in Table 1. For each combination of a drug response dataset and an ML model, the data points and the corresponding curve are, respectively, the computed y~ values and the power law fit. The shaded area represents the variability of the fit which is bounded by the 0.1th quantile, q0.1, and 0.9th quantile, q0.9

Similar articles

Cited by

References

    1. Sharma SV, Haber DA, Settleman J. Cell line-based platforms to evaluate the therapeutic efficacy of candidate anticancer agents. Nat Rev Cancer. 2010;10:241–253. doi: 10.1038/nrc2820. - DOI - PubMed
    1. Gillet J-P, Varma S, Gottesman MM. The clinical relevance of cancer cell lines. JNCI J Natl Cancer Inst. 2013;105(7):452–458. doi: 10.1093/jnci/djt007. - DOI - PMC - PubMed
    1. Ben-David U, et al. Genetic and transcriptional evolution alters cancer cell line drug response. Nature. 2018;560:325–330. doi: 10.1038/s41586-018-0409-3. - DOI - PMC - PubMed
    1. Yang W, et al. Genomics of drug sensitivity in cancer (GDSC): a resource for therapeutic biomarker discovery in cancer cells. Nucleic Acids Res. 2013;41:955–961. doi: 10.1093/nar/gks1111. - DOI - PMC - PubMed
    1. Seashore-Ludlow B, et al. Harnessing connectivity in a large-scale small-molecule sensitivity dataset. Cancer Discov. 2015;5(11):1210–1223. doi: 10.1158/2159-8290.CD-15-0235. - DOI - PMC - PubMed

MeSH terms

Substances

LinkOut - more resources