Improved protein structure prediction using potentials from deep learning (original) (raw)

Data availability

Our training, validation and test data splits (CATH domain codes) are available from https://github.com/deepmind/deepmind-research/tree/master/alphafold_casp13. The following versions of public datasets were used in this study: PDB 2018-03-15; CATH 2018-03-16; Uniclust30 2017-10; and PSI-BLAST nr dataset (as of 15 December 2017).

Code availability

Source code for the distogram, reference distogram and torsion prediction neural networks, together with the neural network weights and input data for the CASP13 targets are available for research and non-commercial use at https://github.com/deepmind/deepmind-research/tree/master/alphafold_casp13. We make use of several open-source libraries to conduct our experiments, particularly HHblits36, PSI-BLAST37 and the machine-learning framework TensorFlow (https://github.com/tensorflow/tensorflow) along with the TensorFlow library Sonnet (https://github.com/deepmind/sonnet), which provides implementations of individual model components50. We also used Rosetta9 under license.

References

  1. Dill, K. A., Ozkan, S. B., Shell, M. S. & Weikl, T. R. The protein folding problem. Annu. Rev. Biophys. 37, 289–316 (2008).
    ADS CAS PubMed PubMed Central Google Scholar
  2. Dill, K. A. & MacCallum, J. L. The protein-folding problem, 50 years on. Science 338, 1042–1046 (2012).
    CAS PubMed Google Scholar
  3. Schaarschmidt, J., Monastyrskyy, B., Kryshtafovych, A. & Bonvin, A. M. J. J. Assessment of contact predictions in CASP12: co-evolution and deep learning coming of age. Proteins 86, 51–66 (2018).
    CAS PubMed Google Scholar
  4. Kirkwood, J. Statistical mechanics of fluid mixtures. J. Chem. Phys. 3, 300–313 (1935).
    ADS CAS MATH Google Scholar
  5. Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K. & Moult, J. Critical assessment of methods of protein structure prediction (CASP)—Round XIII. Proteins 87, 1011–1020 (2019).
    CAS PubMed Google Scholar
  6. Zhang, Y. & Skolnick, J. Scoring function for automated assessment of protein structure template quality. Proteins 57, 702–710 (2004).
    CAS PubMed Google Scholar
  7. Zhang, Y. Protein structure prediction: when is it useful? Curr. Opin. Struct. Biol. 19, 145–155 (2009).
    CAS PubMed PubMed Central Google Scholar
  8. Senior, A. W. et al. Protein structure prediction using multiple deep neural networks in the 13th Critical Assessment of Protein Structure Prediction (CASP13). Proteins 87, 1141–1148 (2019).
    CAS PubMed Google Scholar
  9. Das, R. & Baker, D. Macromolecular modeling with Rosetta. Annu. Rev. Biochem. 77, 363–382 (2008).
    CAS PubMed Google Scholar
  10. Jones, D. T. Predicting novel protein folds by using FRAGFOLD. Proteins 45, 127–132 (2001).
    Google Scholar
  11. Zhang, C., Mortuza, S. M., He, B., Wang, Y. & Zhang, Y. Template-based and free modeling of I-TASSER and QUARK pipelines using predicted contact maps in CASP12. Proteins 86, 136–151 (2018).
    CAS PubMed Google Scholar
  12. Kirkpatrick, S., Gelatt, C. D. Jr & Vecchi, M. P. Optimization by simulated annealing. Science 220, 671–680 (1983).
    ADS MathSciNet CAS PubMed MATH Google Scholar
  13. Berman, H. M. et al. The Protein Data Bank. Nucleic Acids Res. 28, 235–242 (2000).
    ADS CAS PubMed PubMed Central Google Scholar
  14. Altschuh, D., Lesk, A. M., Bloomer, A. C. & Klug, A. Correlation of co-ordinated amino acid substitutions with function in viruses related to tobacco mosaic virus. J. Mol. Biol. 193, 693–707 (1987).
    CAS PubMed Google Scholar
  15. Ovchinnikov, S., Kamisetty, H. & Baker, D. Robust and accurate prediction of residue–residue interactions across protein interfaces using evolutionary information. eLife 3, e02030 (2014).
    PubMed PubMed Central Google Scholar
  16. Seemayer, S., Gruber, M. & Söding, J. CCMpred—fast and precise prediction of protein residue–residue contacts from correlated mutations. Bioinformatics 30, 3128–3130 (2014).
    CAS PubMed PubMed Central Google Scholar
  17. Morcos, F. et al. Direct-coupling analysis of residue coevolution captures native contacts across many protein families. Proc. Natl Acad. Sci. USA 108, E1293–E1301 (2011).
    CAS PubMed Google Scholar
  18. Jones, D. T., Buchan, D. W., Cozzetto, D. & Pontil, M. PSICOV: precise structural contact prediction using sparse inverse covariance estimation on large multiple sequence alignments. Bioinformatics 28, 184–190 (2012).
    CAS PubMed Google Scholar
  19. Skwark, M. J., Raimondi, D., Michel, M. & Elofsson, A. Improved contact predictions using the recognition of protein like contact patterns. PLOS Comput. Biol. 10, e1003889 (2014).
    ADS PubMed PubMed Central Google Scholar
  20. Jones, D. T., Singh, T., Kosciolek, T. & Tetchner, S. MetaPSICOV: combining coevolution methods for accurate prediction of contacts and long range hydrogen bonding in proteins. Bioinformatics 31, 999–1006 (2015).
    CAS PubMed Google Scholar
  21. Wang, S., Sun, S., Li, Z., Zhang, R. & Xu, J. Accurate de novo prediction of protein contact map by ultra-deep learning model. PLOS Comput. Biol. 13, e1005324 (2017).
    ADS PubMed PubMed Central Google Scholar
  22. Jones, D. T. & Kandathil, S. M. High precision in protein contact prediction using fully convolutional neural networks and minimal sequence features. Bioinformatics 34, 3308–3315 (2018).
    CAS PubMed PubMed Central Google Scholar
  23. Ovchinnikov, S. et al. Improved de novo structure prediction in CASP11 by incorporating coevolution information into Rosetta. Proteins 84, 67–75 (2016).
    PubMed PubMed Central Google Scholar
  24. Aszódi, A. & Taylor, W. R. Estimating polypeptide α-carbon distances from multiple sequence alignments. J. Math. Chem. 17, 167–184 (1995).
    MATH Google Scholar
  25. Zhao, F. & Xu, J. A position-specific distance-dependent statistical potential for protein structure and functional study. Structure 20, 1118–1126 (2012).
    CAS PubMed PubMed Central Google Scholar
  26. Xu, J. & Wang, S. Analysis of distance-based protein structure prediction by deep learning in CASP13. Proteins 87, 1069–1081 (2019).
    CAS PubMed Google Scholar
  27. Aszódi, A., Gradwell, M. J. & Taylor, W. R. Global fold determination from a small number of distance restraints. J. Mol. Biol. 251, 308–326 (1995).
    PubMed Google Scholar
  28. Kandathil, S. M., Greener, J. G. & Jones, D. T. Prediction of interresidue contacts with DeepMetaPSICOV in CASP13. Proteins 87, 1092–1099 (2019).
    CAS PubMed PubMed Central Google Scholar
  29. He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition 770–778 (2016).
  30. Simons, K. T., Kooperberg, C., Huang, E. & Baker, D. Assembly of protein tertiary structures from fragments with similar local sequences using simulated annealing and Bayesian scoring functions. J. Mol. Biol. 268, 209–225 (1997).
    CAS PubMed Google Scholar
  31. Liu, D. C. & Nocedal, J. On the limited memory BFGS method for large scale optimization. Math. Program. 45, 503–528 (1989).
    MathSciNet MATH Google Scholar
  32. Li, Y., Zhang, C., Bell, E. W., Yu, D.-J. & Zhang, Y. Ensembling multiple raw coevolutionary features with deep residual neural networks for contact-map prediction in CASP13. Proteins 87, 1082–1091 (2019).
    CAS PubMed Google Scholar
  33. Konagurthu, A. S., Lesk, A. M. & Allison, L. Minimum message length inference of secondary structure from protein coordinate data. Bioinformatics 28, i97–i105 (2012).
    CAS PubMed PubMed Central Google Scholar
  34. Dawson, N. L. et al. CATH: an expanded resource to predict protein function through structure and sequence. Nucleic Acids Res. 45, D289–D295 (2017).
    CAS PubMed Google Scholar
  35. Mirdita, M. et al. Uniclust databases of clustered and deeply annotated protein sequences and alignments. Nucleic Acids Res. 45, D170–D176 (2017).
    CAS PubMed Google Scholar
  36. Remmert, M., Biegert, A., Hauser, A. & Söding, J. HHblits: lightning-fast iterative protein sequence searching by HMM–HMM alignment. Nat. Methods 9, 173–175 (2012).
    CAS Google Scholar
  37. Altschul, S. F. et al. Gapped BLAST and PSI-BLAST: a new generation of protein database search programs. Nucleic Acids Res. 25, 3389–3402 (1997).
    CAS PubMed PubMed Central Google Scholar
  38. Yu, F. & Koltun, V. Multi-scale context aggregation by dilated convolutions. Preprint at arXiv https://arxiv.org/abs/1511.07122 (2015).
  39. Oord, A. d. et al. Wavenet: a generative model for raw audio. Preprint at arXiv https://arxiv.org/abs/1609.03499 (2016).
  40. Clevert, D.-A., Unterthiner, T. & Hochreiter, S. Fast and accurate deep network learning by exponential linear units (ELUs). Preprint at arXiv https://arxiv.org/abs/1511.07289 (2015).
  41. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014).
    MathSciNet MATH Google Scholar
  42. Kabsch, W. & Sander, C. Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded and geometrical features. Biopolymers 22, 2577–2637 (1983).
    CAS PubMed Google Scholar
  43. Yang, Y. et al. Sixty-five years of the long march in protein secondary structure prediction: the final stretch? Briefings Bioinf. 19, 482–494 (2018).
    CAS Google Scholar
  44. Zemla, A., Venclovas, C., Moult, J. & Fidelis, K. Processing and analysis of CASP3 protein structure predictions. Proteins 37, 22–29 (1999).
    Google Scholar
  45. Mariani, V., Biasini, M., Barbato, A. & Schwede, T. lDDT: a local superposition-free score for comparing protein structures and models using distance difference tests. Bioinformatics 29, 2722–2728 (2013).
    CAS PubMed PubMed Central Google Scholar
  46. Abriata, L. A., Tamo, G. E. & Dal Peraro, M. A further leap of improvement in tertiary structure prediction in CASP13 prompts new routes for future assessments. Proteins 87, 1100–1112 (2019).
    CAS PubMed Google Scholar
  47. Kayikci, M. et al. Visualization and analysis of non-covalent contacts using the Protein Contacts Atlas. Nat. Struct. Mol. Biol. 25, 185–194 (2018).
    CAS PubMed PubMed Central Google Scholar
  48. Croll, T. I. et al. Evaluation of template-based modeling in CASP13. Proteins 87, 1113–1127 (2019).
    CAS PubMed PubMed Central Google Scholar
  49. Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. In Proc. 34th International Conference on Machine Learning Vol. 70, 3319–3328 (2017).
  50. Abadi, M. et al. Tensorflow: a system for large-scale machine learning. In Proc. 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) 265–283 (2016).
  51. Söding, J., Biegert, A. & Lupas, A. N. The HHpred interactive server for protein homology detection and structure prediction. Nucleic Acids Res. 33, W244–W248 (2005).
    PubMed PubMed Central Google Scholar
  52. Cong, Q. et al. An automatic method for CASP9 free modeling structure prediction assessment. Bioinformatics 27, 3371–3378 (2011).
    CAS PubMed PubMed Central Google Scholar
  53. Zhang, Y. & Skolnick, J. TM-align: a protein structure alignment algorithm based on the TM-score. Nucleic Acids Res. 33, 2302–2309 (2005).
    CAS PubMed PubMed Central Google Scholar
  54. Tovchigrechko, A., Wells, C. A. & Vakser, I. A. Docking of protein models. Protein Sci. 11, 1888–1896 (2002).
    CAS PubMed PubMed Central Google Scholar
  55. Audet, M. et al. Crystal structure of misoprostol bound to the labor inducer prostaglandin E2 receptor. Nat. Chem. Biol. 15, 11–17 (2019).
    CAS PubMed Google Scholar

Download references

Acknowledgements

We thank C. Meyer for assistance in preparing the paper; B. Coppin, O. Vinyals, M. Barwinski, R. Sun, C. Elkin, P. Dolan, M. Lai and Y. Li for their contributions and support; O. Ronneberger for reading the paper; the rest of the DeepMind team for their support; the CASP13 organisers and the experimentalists whose structures enabled the assessment.

Author information

Author notes

  1. These authors contributed equally: Andrew W. Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre

Authors and Affiliations

  1. DeepMind, London, UK
    Andrew W. Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Žídek, Alexander W. R. Nelson, Alex Bridgland, Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David Silver, Koray Kavukcuoglu & Demis Hassabis
  2. The Francis Crick Institute, London, UK
    David T. Jones
  3. University College London, London, UK
    David T. Jones

Authors

  1. Andrew W. Senior
    You can also search for this author inPubMed Google Scholar
  2. Richard Evans
    You can also search for this author inPubMed Google Scholar
  3. John Jumper
    You can also search for this author inPubMed Google Scholar
  4. James Kirkpatrick
    You can also search for this author inPubMed Google Scholar
  5. Laurent Sifre
    You can also search for this author inPubMed Google Scholar
  6. Tim Green
    You can also search for this author inPubMed Google Scholar
  7. Chongli Qin
    You can also search for this author inPubMed Google Scholar
  8. Augustin Žídek
    You can also search for this author inPubMed Google Scholar
  9. Alexander W. R. Nelson
    You can also search for this author inPubMed Google Scholar
  10. Alex Bridgland
    You can also search for this author inPubMed Google Scholar
  11. Hugo Penedones
    You can also search for this author inPubMed Google Scholar
  12. Stig Petersen
    You can also search for this author inPubMed Google Scholar
  13. Karen Simonyan
    You can also search for this author inPubMed Google Scholar
  14. Steve Crossan
    You can also search for this author inPubMed Google Scholar
  15. Pushmeet Kohli
    You can also search for this author inPubMed Google Scholar
  16. David T. Jones
    You can also search for this author inPubMed Google Scholar
  17. David Silver
    You can also search for this author inPubMed Google Scholar
  18. Koray Kavukcuoglu
    You can also search for this author inPubMed Google Scholar
  19. Demis Hassabis
    You can also search for this author inPubMed Google Scholar

Contributions

R.E., J.J., J.K., L.S., A.W.S., C.Q., T.G., A.Ž., A.B., H.P. and K.S. designed and built the AlphaFold system with advice from D.S., K.K. and D.H. D.T.J. provided advice and guidance on protein structure prediction methodology. S.P. contributed to software engineering. S.C., A.W.R.N., K.K. and D.H. managed the project. J.K., A.W.S., T.G., A.Ž., A.B., R.E., P.K. and J.J. analysed the CASP results for the paper. A.W.S. and J.K. wrote the paper with contributions from J.J., R.E., L.S., T.G., A.B., A.Ž., D.T.J., P.K., K.K. and D.H. A.W.S. led the team.

Corresponding author

Correspondence toAndrew W. Senior.

Ethics declarations

Competing interests

A.W.S., J.K., T.G., J.J., L.S., R.E., H.P., C.Q., K.S., A.Ž. and A.B. have filed provisional patent applications relating to machine learning for predicting protein structures. The remaining authors declare no competing interests.

Additional information

Peer review information Nature thanks Mohammed AlQuraishi and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended Data Fig. 1 Schematics of the folding system and neural network.

a, The overall folding system. Feature extraction stages (constructing the MSA using sequence database search and computing MSA-based features) are shown in yellow; the structure-prediction neural network in green; potential construction in red; and structure realization in blue. b, The layers used in one block of the deep residual convolutional network. The dilated convolution is applied to activations of reduced dimension. The output of the block is added to the representation from the previous layer. The bypass connections of the residual network enable gradients to pass back through the network undiminished, permitting the training of very deep networks.

Extended Data Fig. 2 CASP13 contact precisions.

a, Precisions (as shown in Fig. 1c) for long-range contact prediction in CASP13 for the most probable L, L/2 or L/5 contacts, where L is the length of the domain. The distance distributions used by AlphaFold (AF) in CASP13, thresholded to contact predictions, are compared with submissions by the two best-ranked contact prediction methods in CASP13: 498 (RaptorX-Contact26) and 032 (TripletRes32), on ‘all groups’ targets, with updated domain definitions for T0953s2. b, c, True distances (b) and modes of the predicted distogram (c) for CASP13 target T0990. CASP divides this chain into three domains as shown (D3 is inserted in D2) for which there are 39, 36 and 42 HHblits alignments, respectively (from the CASP website).

Extended Data Fig. 3 Analysis of structure accuracies.

a, lDDT12 versus distogram lDDT12 (see Methods, ‘Accuracy’). The distogram accuracy predicts the lDDT of the realized structure well (particularly for medium- and long-range residue pairs, as well as the TM score as shown in Fig. 4a) for both CASP13 (n = 500: 5 decoys for domains excluding T0999) and test (n = 377) datasets. Data are shown with Pearson’s correlation coefficients. b, DLDDT12 against the effective number of sequences in the MSA (_N_eff) normalized by sequence length (n = 377). The number of effective sequences correlates with this measure of distogram accuracy (r = 0.634). c, Structure accuracy measures, computed on the test set (n = 377), for gradient descent optimization of different forms of the potential. Top, removing terms in the potential, and showing the effect of following optimization with Rosetta relax. ‘P’ shows the significance of the potential giving different results from ‘Full’, for a two-tailed paired data _t_-test. ‘Bins’ shows the number of bins fitted by the spline before extrapolation and the number in the full distribution. In CASP13, splines were fitted to the first 51 of 64 bins. Bottom, reducing the resolution of the distogram distributions. The original 64-bin distogram predictions are repeatedly downsampled by a factor of 2 by summing adjacent bins, in each case with constant extrapolation beyond 18 Å (the last quarter of the bins). The two-level potential in the final row, which was designed to compare with contact predictions, is constructed by summing the probability mass below 8 Å and between 8 and 14 Å, with constant extrapolation beyond 14 Å. The TM scores in this table are plotted in Fig. 4b.

Extended Data Fig. 4 TM score versus per-target computation time computed as an average over the test set.

Structure realization requires a modest computation budget, which can be parallelized over multiple machines. Full optimization with noisy restarts (orange) is compared with initialization from sampled torsions (blue). Computation is measured as the product of the number of (CPU-based) machines and time elapsed and can be largely parallelized. Longer targets take longer to optimize. Figure 2e shows how the TM score increases with the number of repeats of gradient descent. n = 377.

Extended Data Fig. 5 AlphaFold CASP13 results.

a, The TM score for each of the five AlphaFold CASP13 submissions are shown. Simulated annealing with fragment assembly entries are shown in blue. Gradient-descent entries are shown in yellow. Gradient descent was only used for targets T0975 and later, so to the left of the black line we also show the results for a single ‘back-fill’ run of gradient descent for each earlier target using the deployed system. T0999 (1,589 residues) was manually segmented based on HHpred51 homology matching. b, Average TM scores of the AlphaFold CASP13 submissions (n = 104 domains), comparing the first model submitted, the best-of-five model (submission with highest GDT_TS), a single run of full-chain gradient descent (a CASP13 run for T0975 and later, back-fill for earlier targets) and a single CASP13 run of fragment assembly with domain segmentation (using a gradient descent submission for T0999). c, The formula-standardized (z) scores of the assessors for GDT TS + QCS52, best-of-five for CASP FM (n = 31) and FM/TBM (n = 12) domains comparing AlphaFold with the closest competitor (group 322), coloured by domain category. AlphaFold performs better (P = 0.0032, one-tailed paired statistic _t_-test).

Extended Data Fig. 6 Correct fold identification by structural search in CATH.

Often protein function can be inferred by finding homologous proteins of known function. Here we show that the FM predictions of AlphaFold give greater accuracy in a structure-based search for homologous domains in the CATH database. For each of the FM or TBM/FM domains, the top-one submission and ground truth are compared to all 30,744 CATH S40 non-redundant domains with TM-align53. For the 36 domains for which there is a good ground-truth match (score > 0.5), we show the percentage of decoys for which a domain with the same CATH code (CATH in red, CA in green; CAT results are close to CATH results) as the top ground-truth match is in the top-k matches with score > 0.5. Curves are shown for AlphaFold and the next-best group (322). AlphaFold predictions determine the matching fold more accurately. Determination of the matching CATH domain can provide insights into the function of a new protein.

Extended Data Fig. 7 Accuracy of predictions for interfaces.

Protein–protein interaction is an important domain for understanding protein function that has hitherto largely been limited to template-based models because of the need for high-accuracy predictions, although there has been moderate success54 in docking with predicted structures up to 6 Å r.m.s.d. This figure shows that the predictions by AlphaFold improve accuracy in the interface regions of chains in hetero-dimer structures and are probably better candidates for docking, although docking did not form part of the AlphaFold system and all submissions were for isolated chains rather than complexes. For the five all-groups heterodimer CASP13 targets, the full-atom r.m.s.d. values of the interface residues (residues with a ground-truth inter-chain heavy-atom distance <10 Å) are computed for the chain submissions of all groups (green), relative to the target complex. Results >8 Å are not shown. AlphaFold (blue) achieves consistently high accuracy interface regions and, for 4 out of 5 targets, predicts interfaces below <5 Å for both chains.

Extended Data Fig. 8 Ligand pocket visualizations for T1011.

T1011 (PDB 6M9T) is the EP3 receptor bound to misoprostol-FA55. a, The native structure showing the ligand in a pocket. b, c, Submission 5 (78.0 GDT TS) by AlphaFold (b), made without knowledge of the ligand, shows a pocket more similar to the true pocket than that of the best other submission (322, model 3, 68.7 GDT TS) (c). Both submissions are aligned to the native protein using the same subset of residues from the helices close to the ligand pocket and visualized with the interior pocket together with the native ligand position.

Extended Data Fig. 9 Attribution map of distogram network.

The contact probability map of T0986s2, and the summed absolute value of the Integrated Gradient, ∑ c |S I,J i,j,c |, of the input two-dimensional features with respect to the expected distance between five different pairs of residues (I,J): (1) a helix self-contact, (2) a long-range strand–strand contact, (3) a medium-range strand–strand contact, (4) a non-contact and (5) a very long-range strand–strand contact. Each pair is shown as two red dots on the diagrams. Darker colours indicate a higher attribution weight.

Extended Data Fig. 10 Attribution shown on predicted structure.

For T0986s2 (TM score 0.8), the top 10 input pairs, including self-pairs, with the highest attribution weight for each of the five output pairs shown in Extended Data Fig. 9, are shown as lines (or spheres for self-pairs) coloured by sensitivity, lighter green colours indicate more sensitive, and the output pair is shown as a blue line.

Supplementary information

Rights and permissions

About this article

Cite this article

Senior, A.W., Evans, R., Jumper, J. et al. Improved protein structure prediction using potentials from deep learning.Nature 577, 706–710 (2020). https://doi.org/10.1038/s41586-019-1923-7

Download citation