How Many Bootstrap Replicates Are Necessary? (original) (raw)

Abstract

Phylogenetic Bootstrapping (BS) is a standard technique for inferring confidence values on phylogenetic trees that is based on reconstructing many trees from minor variations of the input data, trees called replicates. BS is used with all phylogenetic reconstruction approaches, but we focus here on the most popular, Maximum Likelihood (ML). Because ML inference is so computationally demanding, it has proved too expensive to date to assess the impact of the number of replicates used in BS on the quality of the support values. For the same reason, a rather small number (typically 100) of BS replicates are computed in real-world studies. Stamatakis et al. recently introduced a BS algorithm that is 1–2 orders of magnitude faster than previous techniques, while yielding qualitatively comparable support values, making an experimental study possible.

In this paper, we propose stopping criteria, that is, thresholds computed at runtime to determine when enough replicates have been generated, and report on the first large-scale experimental study to assess the effect of the number of replicates on the quality of support values, including the performance of our proposed criteria. We run our tests on 17 diverse real-world DNA, single-gene as well as multi-gene, datasets, that include between 125 and 2,554 sequences. We find that our stopping criteria typically stop computations after 100–500 replicates (although the most conservative criterion may continue for several thousand replicates) while producing support values that correlate at better than 99.5% with the reference values on the best ML trees. Significantly, we also find that the stopping criteria can recommend very different numbers of replicates for different datasets of comparable sizes.

Our results are thus two-fold: (i) they give the first experimental assessment of the effect of the number of BS replicates on the quality of support values returned through bootstrapping; and (ii) they validate our proposals for stopping criteria. Practitioners will no longer have to enter a guess nor worry about the quality of support values; moreover, with most counts of replicates in the 100–500 range, robust BS under ML inference becomes computationally practical for most datasets. The complete test suite is available at http://lcbb.epfl.ch/BS.tar.bz2 and BS with our stopping criteria is included in RAxML 7.1.0.

Preview

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Andrews, D.W.K., Buchinsky, M.: On the Number of Bootstrap Repetitions for Bootstrap Standard Errors, Confidence Intervals, and Tests. Cowles Foundation Paper 1141R (1997)
    Google Scholar
  2. Andrews, D.W.K., Buchinsky, M.: A Three-Step Method for Choosing the Number of Bootstrap Repetitions. Econometrica 68(1), 23–51 (2000)
    Article Google Scholar
  3. Andrews, D.W.K., Buchinsky, M.: Evaluation of a Three-step Method for Choosing the Number of Bootstrap Repetitions. J. of Econometrics 103(1-2), 345–386 (2001)
    Article Google Scholar
  4. Andrews, D.W.K., Buchinsky, M.: On The Number of Bootstrap Repetitions for BCa Confidence Intervals. Econometric Theory 18(4), 962–984 (2002)
    Article Google Scholar
  5. Brooks, S.P., Gelman, A.: General Methods for Monitoring Convergence of Iterative Simulations. J. of Computational and Graphical Statistics 7(4), 434–455 (1998)
    Google Scholar
  6. Davidson, A.C., Hinkley, D.V.: Bootstrap Methods and Their Application. Cambridge University Press, Cambridge (2003)
    Google Scholar
  7. Davidson, R., MacKinnon, J.G.: Bootstrap Tests: How Many Bootstraps? Econometric Reviews 19(1), 55–68 (2000)
    Article CAS Google Scholar
  8. Efron, B., Tibshirani, R.J.: An Introduction to the Bootstrap. Chapman and Hall, New York (1993)
    Book Google Scholar
  9. Felsenstein, J.: Confidence Limits on Phylogenies: An Approach Using the Bootstrap. Evolution 39(4), 783–791 (1985)
    Article PubMed Google Scholar
  10. Gelman, A., Rubin, D.B.: Inference from Iterative Simulation using Multiple Sequences. Stat. Sci. 7, 457–511 (1992)
    Article Google Scholar
  11. Grimm, G.W., Renner, S.S., Stamatakis, A., Hemleben, V.: A Nuclear Ribosomal DNA Phylogeny of acer Inferred with Maximum Likelihood, Splits Graphs, and Motif Analyses of 606 Sequences. Evolutionary Bioinformatics Online 2, 279–294 (2006)
    CAS Google Scholar
  12. Guindon, S., Gascuel, O.: A Simple, Fast, and Accurate Algorithm to Estimate Large Phylogenies by Maximum Likelihood. Sys. Biol. 52(5), 696–704 (2003)
    Article Google Scholar
  13. Guo, W., Peddada, S.: Adaptive Choice of the Number of Bootstrap Samples in Large Scale Multiple Testing. Stat. Appls. in Genetics and Mol. Biol. 7(1) (2008)
    Google Scholar
  14. Hall, P.: On the Number of Bootstrap Simulations Required to Construct a Confidence Interval. The Annals of Statistics 14(4), 1453–1462 (1986)
    Article Google Scholar
  15. Hedges, S.B.: The Number of Replications Needed for Accurate Estimation of the Bootstrap P Value in Phylogenetic Studies. Mol. Biol. Evol. 9(2), 366–369 (1992)
    CAS PubMed Google Scholar
  16. Hillis, D.M., Heath, T.A., John, K.S.: Analysis and Visualization of Tree Space. Sys. Biol. 54(3), 471–482 (2005)
    Article Google Scholar
  17. Holmes, S.: Bootstrapping Phylogenies Statistical Science, 18(2), 241–255
    Google Scholar
  18. Manly, B.F.J., et al.: Randomization, Bootstrap and Monte Carlo Methods in Biology. CRC Press, Boca Raton (1997)
    Google Scholar
  19. Moret, B.M.E.: Large-scale Phylogenetic Reconstruction. In: Brown, J.R. (ed.) Comparative Genomics: Basic and Applied Research, pp. 29–48. CRC Press/Taylor & Francis (2007)
    Google Scholar
  20. Mossel, E., Vigoda, E.: Limitations of Markov Chain Monte Carlo Algorithms for Bayesian Inference of Phylogeny. Ann. Appl. Probab. 16(4), 2215–2234 (2006)
    Article Google Scholar
  21. Nylander, J.A.A., Wilgenbusch, J.C., Warren, D.L., Swofford, D.L.: AWTY (are we there yet?): A System for Graphical Exploration of MCMC Convergence in Bayesian Phylogenetics. Bioinformatics (2007) (advance access, published August 30)
    Google Scholar
  22. Pattengale, N.D., Gottlieb, E.J., Moret, B.M.E.: Efficiently Computing the Robinson-Foulds Metric. J. of Computational Biology 14(6), 724–735 (2007)
    Article CAS Google Scholar
  23. Rambaut, A., Drummond, A.: Tracer MCMC Trace Analysis Tool version 1.3 (2004)
    Google Scholar
  24. Robinson, D.F., Foulds, L.R.: Comparison of Weighted Labelled Trees. Lecture Notes in Mathematics 748, 119–126 (1979)
    Article Google Scholar
  25. Robinson, D.F., Foulds, L.R.: Comparison of Phylogenetic Trees. Math. Biosc. 53(1), 131–147 (1981)
    Article Google Scholar
  26. Ronquist, F., Huelsenbeck, J.P.: MrBayes 3: Bayesian Phylogenetic Inference under Mixed Models. Bioinformatics 19(12), 1572–1574 (2003)
    Article CAS PubMed Google Scholar
  27. Soltis, D.E., Gitzendanner, M.A., Soltis, P.S.: A 567-taxon Data Set for Angiosperms: The Challenges Posed by Bayesian Analyses of Large Data Sets. Int’l J. Plant Sci. 168(2), 137–157 (2007)
    Article CAS Google Scholar
  28. Soltis, D.E., Soltis, P.S.: Applying the Bootstrap in Phylogeny Reconstruction. Statist. Sci. 18(2), 256–267 (2003)
    Article Google Scholar
  29. Stamatakis, A.: RAxML-VI-HPC: Maximum Likelihood-based Phylogenetic Analyses with Thousands of Taxa and Mixed Models. Bioinformatics 22(21), 2688–2690 (2006)
    Article CAS PubMed Google Scholar
  30. Stamatakis, A., Hoover, P., Rougemont, J.: A Rapid Bootstrap Algorithm for the RAxML Web Servers. Sys. Biol. (2008) (in press)
    Google Scholar
  31. Stamatakis, A., Meier, H., Ludwig, T.: New Fast and Accurate Heuristics for Inference of Large Phylogenetic Trees. In: Proc. of IPDPS 2004, HICOMB Workshop, Proceedings on CD, Santa Fe, New Mexico (2004)
    Google Scholar
  32. Whelan, S.: New Approaches to Phylogenetic Tree Search and Their Application to Large Numbers of Protein Alignments. Sys. Biol. 56(5), 727–740 (2007)
    Article Google Scholar
  33. Zwickl, D.: Genetic Algorithm Approaches for the Phylogenetic Analysis of Large Biological Sequence Datasets under the Maximum Likelihood Criterion. PhD thesis, University of Texas at Austin (April 2006)
    Google Scholar

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of New Mexico, Albuquerque, NM, USA
    Nicholas D. Pattengale
  2. Laboratory for Computational Biology and Bioinformatics, EPFL (École Polytechnique Fédérale de Lausanne), Switzerland
    Masoud Alipour & Bernard M. E. Moret
  3. AG Systematik und Evolutionsbiologie, Institut für Biologie und Umweltwissenschaften, University of Oldenburg, Germany
    Olaf R. P. Bininda-Emonds
  4. Swiss Institute of Bioinformatics, Lausanne, Switzerland
    Bernard M. E. Moret
  5. The Exelixis Lab, Department of Computer Science, Technische Universität München, Germany
    Alexandros Stamatakis

Authors

  1. Nicholas D. Pattengale
    You can also search for this author inPubMed Google Scholar
  2. Masoud Alipour
    You can also search for this author inPubMed Google Scholar
  3. Olaf R. P. Bininda-Emonds
    You can also search for this author inPubMed Google Scholar
  4. Bernard M. E. Moret
    You can also search for this author inPubMed Google Scholar
  5. Alexandros Stamatakis
    You can also search for this author inPubMed Google Scholar

Editor information

Editors and Affiliations

  1. Computer Science Department, James H. Clark Center, 318 Campus Drive, RM S266, CA 94305-5428,, Stanford, USA
    Serafim Batzoglou

Rights and permissions

© 2009 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Pattengale, N.D., Alipour, M., Bininda-Emonds, O.R.P., Moret, B.M.E., Stamatakis, A. (2009). How Many Bootstrap Replicates Are Necessary?. In: Batzoglou, S. (eds) Research in Computational Molecular Biology. RECOMB 2009. Lecture Notes in Computer Science(), vol 5541. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02008-7\_13

Download citation

Publish with us