Meta-analysis and the science of research synthesis (original) (raw)

References

  1. Jennions, M. D ., Lortie, C. J. & Koricheva, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 23, 364–380 (Princeton Univ. Press, 2013)
    Article Google Scholar
  2. Roberts, P. D ., Stewart, G. B. & Pullin, A. S. Are review articles a reliable source of evidence to support conservation and environmental management? A comparison with medicine. Biol. Conserv. 132, 409–423 (2006)
    Article Google Scholar
  3. Bastian, H ., Glasziou, P . & Chalmers, I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 7, e1000326 (2010)
    Article PubMed PubMed Central Google Scholar
  4. Borman, G. D. & Grigg, J. A. in The Handbook of Research Synthesis and Meta-analysis 2nd edn (eds Cooper, H. M . et al.) 497–519 (Russell Sage Foundation, 2009)
  5. Ioannidis, J. P. A. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 94, 485–514 (2016)
    Article PubMed PubMed Central Google Scholar
  6. Koricheva, J . & Gurevitch, J. Uses and misuses of meta-analysis in plant ecology. J. Ecol. 102, 828–844 (2014)
    Article Google Scholar
  7. Littell, J. H . & Shlonsky, A. Making sense of meta-analysis: a critique of “effectiveness of long-term psychodynamic psychotherapy”. Clin. Soc. Work J. 39, 340–346 (2011)
    Article Google Scholar
  8. Morrissey, M. B. Meta-analysis of magnitudes, differences and variation in evolutionary parameters. J. Evol. Biol. 29, 1882–1904 (2016)
    Article CAS PubMed Google Scholar
  9. Whittaker, R. J. Meta-analyses and mega-mistakes: calling time on meta-analysis of the species richness-productivity relationship. Ecology 91, 2522–2533 (2010)
    Article PubMed Google Scholar
  10. Begley, C. G . & Ellis, L. M. Drug development: Raise standards for preclinical cancer research. Nature 483, 531–533 (2012); clarification 485, 41 (2012)
    Article CAS ADS PubMed Google Scholar
  11. Hillebrand, H . & Cardinale, B. J. A critique for meta-analyses and the productivity-diversity relationship. Ecology 91, 2545–2549 (2010)
    Article PubMed Google Scholar
  12. Moher, D . et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med. 6, e1000097 (2009).This paper provides a consensus regarding the reporting requirements for medical meta-analysis and has been highly influential in ensuring good reporting practice and standardizing language in evidence-based medicine, with further guidance for protocols, individual patient data meta-analyses and animal studies.
    Article PubMed PubMed Central Google Scholar
  13. Moher, D . et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst. Rev. 4, 1 (2015)
    Article PubMed PubMed Central Google Scholar
  14. Nakagawa, S . & Santos, E. S. A. Methodological issues and advances in biological meta-analysis. Evol. Ecol. 26, 1253–1274 (2012)
    Article Google Scholar
  15. Nakagawa, S ., Noble, D. W. A ., Senior, A. M. & Lagisz, M. Meta-evaluation of meta-analysis: ten appraisal questions for biologists. BMC Biol. 15, 18 (2017)
    Article PubMed PubMed Central Google Scholar
  16. Hedges, L. & Olkin, I. Statistical Methods for Meta-analysis (Academic Press, 1985)
  17. Viechtbauer, W. Conducting meta-analyses in R with the metafor package. J. Stat. Softw. 36, 1–48 (2010)
    Article Google Scholar
  18. Anzures-Cabrera, J . & Higgins, J. P. T. Graphical displays for meta-analysis: an overview with suggestions for practice. Res. Synth. Methods 1, 66–80 (2010)
    Article PubMed Google Scholar
  19. Egger, M ., Davey Smith, G ., Schneider, M. & Minder, C. Bias in meta-analysis detected by a simple, graphical test. Br. Med. J. 315, 629–634 (1997)
    Article CAS Google Scholar
  20. Duval, S . & Tweedie, R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56, 455–463 (2000)
    Article CAS MATH PubMed Google Scholar
  21. Leimu, R . & Koricheva, J. Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology. Proc. R. Soc. Lond. B 271, 1961–1966 (2004)
    Article Google Scholar
  22. Higgins, J. P. T . & Green, S. (eds) Cochrane Handbook for Systematic Reviews of Interventions: Version 5.1.0 (Wiley, 2011).This large collaborative work provides definitive guidance for the production of systematic reviews in medicine and is of broad interest for methods development outside the medical field.
  23. Lau, J ., Rothstein, H. R . & Stewart, G. B. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 25, 407–419 (Princeton Univ. Press, 2013)
    Article Google Scholar
  24. Lortie, C. J ., Stewart, G ., Rothstein, H. & Lau, J. How to critically read ecological meta-analyses. Res. Synth. Methods 6, 124–133 (2015)
    Article PubMed Google Scholar
  25. Murad, M. H . & Montori, V. M. Synthesizing evidence: shifting the focus from individual studies to the body of evidence. J. Am. Med. Assoc. 309, 2217–2218 (2013)
    Article Google Scholar
  26. Rasmussen, S. A ., Chu, S. Y ., Kim, S. Y ., Schmid, C. H . & Lau, J. Maternal obesity and risk of neural tube defects: a meta-analysis. Am. J. Obstet. Gynecol. 198, 611–619 (2008)
    Article PubMed Google Scholar
  27. Littell, J. H ., Campbell, M ., Green, S . & Toews, B. Multisystemic therapy for social, emotional, and behavioral problems in youth aged 10–17. Cochrane Database Syst. Rev. https://doi.org/10.1002/14651858.CD004797.pub4 (2005)
  28. Schmidt, F. L. What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. Am. Psychol. 47, 1173–1181 (1992)
    Article Google Scholar
  29. Button, K. S . et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013); erratum 14, 451 (2013)
    Article CAS PubMed Google Scholar
  30. Parker, T. H . et al. Transparency in ecology and evolution: real problems, real solutions. Trends Ecol. Evol. 31, 711–719 (2016)
    Article PubMed Google Scholar
  31. Stewart, G. Meta-analysis in applied ecology. Biol. Lett. 6, 78–81 (2010)
    Article PubMed Google Scholar
  32. Sutherland, W. J ., Pullin, A. S ., Dolman, P. M . & Knight, T. M. The need for evidence-based conservation. Trends Ecol. Evol. 19, 305–308 (2004)
    Article PubMed Google Scholar
  33. Lowry, E . et al. Biological invasions: a field synopsis, systematic review, and database of the literature. Ecol. Evol. 3, 182–196 (2013)
    Article PubMed Central Google Scholar
  34. Parmesan, C . & Yohe, G. A globally coherent fingerprint of climate change impacts across natural systems. Nature 421, 37–42 (2003)
    Article CAS ADS PubMed Google Scholar
  35. Jennions, M. D ., Lortie, C. J . & Koricheva, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 24, 381–403 (Princeton Univ. Press, 2013)
    Article Google Scholar
  36. Balvanera, P . et al. Quantifying the evidence for biodiversity effects on ecosystem functioning and services. Ecol. Lett. 9, 1146–1156 (2006)
    Article PubMed Google Scholar
  37. Cardinale, B. J . et al. Effects of biodiversity on the functioning of trophic groups and ecosystems. Nature 443, 989–992 (2006)
    Article CAS ADS PubMed Google Scholar
  38. Rey Benayas, J. M ., Newton, A. C ., Diaz, A. & Bullock, J. M. Enhancement of biodiversity and ecosystem services by ecological restoration: a meta-analysis. Science 325, 1121–1124 (2009)
    Article ADS PubMed CAS Google Scholar
  39. Leimu, R ., Mutikainen, P. I. A ., Koricheva, J. & Fischer, M. How general are positive relationships between plant population size, fitness and genetic variation? J. Ecol. 94, 942–952 (2006)
    Article Google Scholar
  40. Hillebrand, H. On the generality of the latitudinal diversity gradient. Am. Nat. 163, 192–211 (2004)
    Article PubMed Google Scholar
  41. Gurevitch, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 19, 313–320 (Princeton Univ. Press, 2013)
    Article Google Scholar
  42. Rustad, L . et al. A meta-analysis of the response of soil respiration, net nitrogen mineralization, and aboveground plant growth to experimental ecosystem warming. Oecologia 126, 543–562 (2001)
    Article CAS ADS PubMed Google Scholar
  43. Adams, D. C. Phylogenetic meta-analysis. Evolution 62, 567–572 (2008)
    Article PubMed Google Scholar
  44. Hadfield, J. D . & Nakagawa, S. General quantitative genetic methods for comparative biology: phylogenies, taxonomies and multi-trait models for continuous and categorical characters. J. Evol. Biol. 23, 494–508 (2010)
    Article CAS PubMed Google Scholar
  45. Lajeunesse, M. J. Meta-analysis and the comparative phylogenetic method. Am. Nat. 174, 369–381 (2009)
    Article PubMed Google Scholar
  46. Rosenberg, M. S ., Adams, D. C . & Gurevitch, J. MetaWin: Statistical Software for Meta-Analysis with Resampling Tests Version 1 (Sinauer Associates, 1997)
  47. Wallace, B. C . et al. OpenMEE: intuitive, open-source software for meta-analysis in ecology and evolutionary biology. Methods Ecol. Evol. 8, 941–947 (2016)
    Article Google Scholar
  48. Gurevitch, J ., Morrison, J. A . & Hedges, L. V. The interaction between competition and predation: a meta-analysis of field experiments. Am. Nat. 155, 435–453 (2000)
    Article PubMed Google Scholar
  49. Adams, D. C ., Gurevitch, J . & Rosenberg, M. S. Resampling tests for meta-analysis of ecological data. Ecology 78, 1277–1283 (1997)
    Article Google Scholar
  50. Gurevitch, J . & Hedges, L. V. Statistical issues in ecological meta-analyses. Ecology 80, 1142–1149 (1999)
    Article Google Scholar
  51. Schmid, C. H . & Mengersen, K. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 11, 145–173 (Princeton Univ. Press, 2013)
    Article Google Scholar
  52. Eysenck, H. J. Exercise in mega-silliness. Am. Psychol. 33, 517 (1978)
    Article Google Scholar
  53. Simberloff, D. Rejoinder to: Don’t calculate effect sizes; study ecological effects. Ecol. Lett. 9, 921–922 (2006)
    Article Google Scholar
  54. Cadotte, M. W ., Mehrkens, L. R . & Menge, D. N. L. Gauging the impact of meta-analysis on ecology. Evol. Ecol. 26, 1153–1167 (2012)
    Article Google Scholar
  55. Koricheva, J ., Jennions, M. D. & Lau, J. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J . et al.) Ch. 15, 237–254 (Princeton Univ. Press, 2013)
    Article Google Scholar
  56. Lau, J ., Ioannidis, J. P. A ., Terrin, N ., Schmid, C. H . & Olkin, I. The case of the misleading funnel plot. Br. Med. J. 333, 597–600 (2006)
    Article Google Scholar
  57. Vetter, D ., Rucker, G. & Storch, I. Meta-analysis: a need for well-defined usage in ecology and conservation biology. Ecosphere 4, 1–24 (2013)
    Article Google Scholar
  58. Mengersen, K ., Jennions, M. D. & Schmid, C. H. in The Handbook of Meta-analysis in Ecology and Evolution (eds Koricheva, J. et al.) Ch. 16, 255–283 (Princeton Univ. Press, 2013)
    Article Google Scholar
  59. Patsopoulos, N. A ., Analatos, A. A. & Ioannidis, J. P. A. Relative citation impact of various study designs in the health sciences. J. Am. Med. Assoc. 293, 2362–2366 (2005)
    Article CAS Google Scholar
  60. Kueffer, C . et al. Fame, glory and neglect in meta-analyses. Trends Ecol. Evol. 26, 493–494 (2011)
    Article PubMed Google Scholar
  61. Cohnstaedt, L. W. & Poland, J. Review Articles: The black-market of scientific currency. Ann. Entomol. Soc. Am. 110, 90 (2017)
    Article Google Scholar
  62. Longo, D. L. & Drazen, J. M. Data sharing. N. Engl. J. Med. 374, 276–277 (2016)
    Article PubMed Google Scholar
  63. Gauch, H. G. Scientific Method in Practice (Cambridge Univ. Press, 2003)
  64. Science Staff. Dealing with data: introduction. Challenges and opportunities. Science 331, 692–693 (2011)
  65. Nosek, B. A . et al. Promoting an open research culture. Science 348, 1422–1425 (2015)
    Article CAS ADS PubMed PubMed Central Google Scholar
  66. Stewart, L. A . et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: the PRISMA-IPD statement. J. Am. Med. Assoc. 313, 1657–1665 (2015)
    Article Google Scholar
  67. Saldanha, I. J . et al. Evaluating Data Abstraction Assistant, a novel software application for data abstraction during systematic reviews: protocol for a randomized controlled trial. Syst. Rev. 5, 196 (2016)
    Article PubMed PubMed Central Google Scholar
  68. Tipton, E. & Pustejovsky, J. E. Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression. J. Educ. Behav. Stat. 40, 604–634 (2015)
    Article Google Scholar
  69. Mengersen, K ., MacNeil, M. A . & Caley, M. J. The potential for meta-analysis to support decision analysis in ecology. Res. Synth. Methods 6, 111–121 (2015)
    Article PubMed Google Scholar
  70. Ashby, D. Bayesian statistics in medicine: a 25 year review. Stat. Med. 25, 3589–3631 (2006)
    Article MathSciNet PubMed Google Scholar
  71. Senior, A. M . et al. Heterogeneity in ecological and evolutionary meta-analyses: its magnitude and implications. Ecology 97, 3293–3299 (2016)
    Article PubMed Google Scholar
  72. McAuley, L ., Pham, B ., Tugwell, P . & Moher, D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 356, 1228–1231 (2000)
    Article CAS PubMed Google Scholar
  73. Koricheva, J ., Gurevitch, J . & Mengersen, K. (eds) The Handbook of Meta-Analysis in Ecology and Evolution (Princeton Univ. Press, 2013)This book provides the first comprehensive guide to undertaking meta-analyses in ecology and evolution and is also relevant to other fields where heterogeneity is expected, incorporating explicit consideration of the different approaches used in different domains.
  74. Lumley, T. Network meta-analysis for indirect treatment comparisons. Stat. Med. 21, 2313–2324 (2002)
    Article PubMed Google Scholar
  75. Zarin, W . et al. Characteristics and knowledge synthesis approach for 456 network meta-analyses: a scoping review. BMC Med. 15, 3 (2017)
    Article PubMed PubMed Central Google Scholar
  76. Elliott, J. H . et al. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap. PLoS Med. 11, e1001603 (2014)
    Article PubMed PubMed Central Google Scholar
  77. Vandvik, P. O ., Brignardello-Petersen, R . & Guyatt, G. H. Living cumulative network meta-analysis to reduce waste in research: a paradigmatic shift for systematic reviews? BMC Med. 14, 59 (2016)
    Article PubMed PubMed Central Google Scholar
  78. Jarvinen, A. A meta-analytic study of the effects of female age on laying date and clutch size in the Great Tit Parus major and the Pied Flycatcher Ficedula hypoleuca. Ibis 133, 62–67 (1991)
    Article Google Scholar
  79. Arnqvist, G. & Wooster, D. Meta-analysis: synthesizing research findings in ecology and evolution. Trends Ecol. Evol. 10, 236–240 (1995)
    Article CAS PubMed Google Scholar
  80. Hedges, L. V ., Gurevitch, J . & Curtis, P. S. The meta-analysis of response ratios in experimental ecology. Ecology 80, 1150–1156 (1999)
    Article Google Scholar
  81. Gurevitch, J ., Curtis, P. S. & Jones, M. H. Meta-analysis in ecology. Adv. Ecol. Res 32, 199–247 (2001)
    Article CAS Google Scholar
  82. Lajeunesse, M. J. phyloMeta: a program for phylogenetic comparative analyses with meta-analysis. Bioinformatics 27, 2603–2604 (2011)
    CAS PubMed Google Scholar
  83. Pearson, K. Report on certain enteric fever inoculation statistics. Br. Med. J. 2, 1243–1246 (1904)
    Article Google Scholar
  84. Fisher, R. A. Statistical Methods for Research Workers (Oliver and Boyd, 1925)
  85. Yates, F. & Cochran, W. G. The analysis of groups of experiments. J. Agric. Sci. 28, 556–580 (1938)
    Article Google Scholar
  86. Cochran, W. G. The combination of estimates from different experiments. Biometrics 10, 101–129 (1954)
    Article Google Scholar
  87. Smith, M. L . & Glass, G. V. Meta-analysis of psychotherapy outcome studies. Am. Psychol. 32, 752–760 (1977)
    Article CAS PubMed Google Scholar
  88. Glass, G. V. Meta-analysis at middle age: a personal history. Res. Synth. Methods 6, 221–231 (2015)
    Article PubMed Google Scholar
  89. Cooper, H. M ., Hedges, L. V . & Valentine, J. C. (eds) The Handbook of Research Synthesis and Meta-analysis 2nd edn (Russell Sage Foundation, 2009).This book is an important compilation that builds on the ground-breaking first edition to set the standard for best practice in meta-analysis, primarily in the social sciences but with applications to medicine and other fields.
  90. Rosenthal, R. Meta-analytic Procedures for Social Research (Sage, 1991)
  91. Hunter, J. E ., Schmidt, F. L. & Jackson, G. B. Meta-analysis: Cumulating Research Findings Across Studies (Sage, 1982)
  92. Gurevitch, J ., Morrow, L. L ., Wallace, A . & Walsh, J. S. A meta-analysis of competition in field experiments. Am. Nat. 140, 539–572 (1992).This influential early ecological meta-analysis reports multiple experimental outcomes on a longstanding and controversial topic that introduced a wide range of ecologists to research synthesis methods.
    Article Google Scholar
  93. O’Rourke, K. An historical perspective on meta-analysis: dealing quantitatively with varying study results. J. R. Soc. Med. 100, 579–582 (2007)
    Article PubMed PubMed Central Google Scholar
  94. Shadish, W. R . & Lecy, J. D. The meta-analytic big bang. Res. Synth. Methods 6, 246–264 (2015)
    Article PubMed Google Scholar
  95. Glass, G. V. Primary, secondary, and meta-analysis of research. Educ. Res. 5, 3–8 (1976)
    Article Google Scholar
  96. DerSimonian, R . & Laird, N. Meta-analysis in clinical trials. Control. Clin. Trials 7, 177–188 (1986)
    Article CAS PubMed Google Scholar
  97. Lipsey, M. W . & Wilson, D. B. The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. Am. Psychol. 48, 1181–1209 (1993)
    Article CAS PubMed Google Scholar
  98. Chalmers, I. & Altman, D. G. Systematic Reviews (BMJ Publishing Group, 1995)
  99. Moher, D . et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet 354, 1896–1900 (1999)
    Article CAS PubMed Google Scholar
  100. Higgins, J. P. & Thompson, S. G. Quantifying heterogeneity in a meta-analysis. Stat. Med. 21, 1539–1558 (2002)
    Article PubMed Google Scholar

Download references