Power failure: why small sample size undermines the reliability of neuroscience (original) (raw)

• Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005). This study demonstrates that many (and possibly most) of the conclusions drawn from biomedical research are probably false. The reasons for this include using flexible study designs and flexible statistical analyses and running small studies with low statistical power.
  Article PubMed PubMed Central Google Scholar
• Fanelli, D. Negative results are disappearing from most disciplines and countries. Scientometrics 90, 891–904 (2012).
  Article Google Scholar
• Greenwald, A. G. Consequences of prejudice against the null hypothesis. Psychol. Bull. 82, 1–20 (1975).
  Article Google Scholar
• Nosek, B. A., Spies, J. R. & Motyl, M. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631 (2012).
  Article PubMed Google Scholar
• Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011). This article empirically illustrates that flexible study designs and data analysis dramatically increase the possibility of obtaining a nominally significant result. However, conclusions drawn from these results are almost certainly false.
  Article PubMed Google Scholar
• Sullivan, P. F. Spurious genetic associations. Biol. Psychiatry 61, 1121–1126 (2007).
  Article CAS PubMed Google Scholar
• Begley, C. G. & Ellis, L. M. Drug development: raise standards for preclinical cancer research. Nature 483, 531–533 (2012).
  Article CAS PubMed Google Scholar
• Prinz, F., Schlange, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nature Rev. Drug Discov. 10, 712 (2011).
  Article CAS Google Scholar
• Fang, F. C. & Casadevall, A. Retracted science and the retraction index. Infect. Immun. 79, 3855–3859 (2011).
  Article CAS PubMed PubMed Central Google Scholar
• Munafo, M. R., Stothart, G. & Flint, J. Bias in genetic association studies and impact factor. Mol. Psychiatry 14, 119–120 (2009).
  Article CAS PubMed Google Scholar
• Sterne, J. A. & Davey Smith, G. Sifting the evidence — what's wrong with significance tests? BMJ 322, 226–231 (2001).
  Article CAS PubMed PubMed Central Google Scholar
• Ioannidis, J. P. A., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011).
  Article PubMed Google Scholar
• Ioannidis, J. P. A. Why most discovered true associations are inflated. Epidemiology 19, 640–648 (2008).
  Article PubMed Google Scholar
• Tversky, A. & Kahneman, D. Belief in the law of small numbers. Psychol. Bull. 75, 105–110 (1971).
  Article Google Scholar
• Masicampo, E. J. & Lalande, D. R. A peculiar prevalence of p values just below .05. Q. J. Exp. Psychol. 65, 2271–2279 (2012).
  Article CAS Google Scholar
• Carp, J. The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage 63, 289–300 (2012). This article reviews methods reporting and methodological choices across 241 recent fMRI studies and shows that there were nearly as many unique analytical pipelines as there were studies. In addition, many studies were underpowered to detect plausible effects.
  Article PubMed Google Scholar
• Dwan, K. et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 3, e3081 (2008).
  Article CAS PubMed PubMed Central Google Scholar
• Sterne, J. A. et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ 343, d4002 (2011).
  Article PubMed Google Scholar
• Joy-Gaba, J. A. & Nosek, B. A. The surprisingly limited malleability of implicit racial evaluations. Soc. Psychol. 41, 137–146 (2010).
  Article Google Scholar
• Schmidt, K. & Nosek, B. A. Implicit (and explicit) racial attitudes barely changed during Barack Obama's presidential campaign and early presidency. J. Exp. Soc. Psychol. 46, 308–314 (2010).
  Article Google Scholar
• Evangelou, E., Siontis, K. C., Pfeiffer, T. & Ioannidis, J. P. Perceived information gain from randomized trials correlates with publication in high-impact factor journals. J. Clin. Epidemiol. 65, 1274–1281 (2012).
  Article PubMed Google Scholar
• Pereira, T. V. & Ioannidis, J. P. Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. J. Clin. Epidemiol. 64, 1060–1069 (2011).
  Article PubMed Google Scholar
• Faul, F., Erdfelder, E., Lang, A. G. & Buchner, A. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav. Res. Methods 39, 175–191 (2007).
  Article PubMed Google Scholar
• Babbage, D. R. et al. Meta-analysis of facial affect recognition difficulties after traumatic brain injury. Neuropsychology 25, 277–285 (2011).
  Article PubMed Google Scholar
• Bai, H. Meta-analysis of 5, 10-methylenetetrahydrofolate reductase gene poymorphism as a risk factor for ischemic cerebrovascular disease in a Chinese Han population. Neural Regen. Res. 6, 277–285 (2011).
  Google Scholar
• Bjorkhem-Bergman, L., Asplund, A. B. & Lindh, J. D. Metformin for weight reduction in non-diabetic patients on antipsychotic drugs: a systematic review and meta-analysis. J. Psychopharmacol. 25, 299–305 (2011).
  Article PubMed Google Scholar
• Bucossi, S. et al. Copper in Alzheimer's disease: a meta-analysis of serum, plasma, and cerebrospinal fluid studies. J. Alzheimers Dis. 24, 175–185 (2011).
  Article CAS PubMed Google Scholar
• Chamberlain, S. R. et al. Translational approaches to frontostriatal dysfunction in attention-deficit/hyperactivity disorder using a computerized neuropsychological battery. Biol. Psychiatry 69, 1192–1203 (2011).
  Article PubMed Google Scholar
• Chang, W. P., Arfken, C. L., Sangal, M. P. & Boutros, N. N. Probing the relative contribution of the first and second responses to sensory gating indices: a meta-analysis. Psychophysiology 48, 980–992 (2011).
  Article PubMed Google Scholar
• Chang, X. L. et al. Functional parkin promoter polymorphism in Parkinson's disease: new data and meta-analysis. J. Neurol. Sci. 302, 68–71 (2011).
  Article CAS PubMed Google Scholar
• Chen, C. et al. Allergy and risk of glioma: a meta-analysis. Eur. J. Neurol. 18, 387–395 (2011).
  Article CAS PubMed Google Scholar
• Chung, A. K. & Chua, S. E. Effects on prolongation of Bazett's corrected QT interval of seven second-generation antipsychotics in the treatment of schizophrenia: a meta-analysis. J. Psychopharmacol. 25, 646–666 (2011).
  Article PubMed Google Scholar
• Domellof, E., Johansson, A. M. & Ronnqvist, L. Handedness in preterm born children: a systematic review and a meta-analysis. Neuropsychologia 49, 2299–2310 (2011).
  Article PubMed Google Scholar
• Etminan, N., Vergouwen, M. D., Ilodigwe, D. & Macdonald, R. L. Effect of pharmaceutical treatment on vasospasm, delayed cerebral ischemia, and clinical outcome in patients with aneurysmal subarachnoid hemorrhage: a systematic review and meta-analysis. J. Cereb. Blood Flow Metab. 31, 1443–1451 (2011).
  Article CAS PubMed PubMed Central Google Scholar
• Feng, X. L. et al. Association of FK506 binding protein 5 (FKBP5) gene rs4713916 polymorphism with mood disorders: a meta-analysis. Acta Neuropsychiatr. 23, 12–19 (2011).
  Article PubMed Google Scholar
• Green, M. J., Matheson, S. L., Shepherd, A., Weickert, C. S. & Carr, V. J. Brain-derived neurotrophic factor levels in schizophrenia: a systematic review with meta-analysis. Mol. Psychiatry 16, 960–972 (2011).
  Article CAS PubMed Google Scholar
• Han, X. M., Wang, C. H., Sima, X. & Liu, S. Y. Interleukin-6–74G/C polymorphism and the risk of Alzheimer's disease in Caucasians: a meta-analysis. Neurosci. Lett. 504, 4–8 (2011).
  Article CAS PubMed Google Scholar
• Hannestad, J., DellaGioia, N. & Bloch, M. The effect of antidepressant medication treatment on serum levels of inflammatory cytokines: a meta-analysis. Neuropsychopharmacology 36, 2452–2459 (2011).
  Article CAS PubMed PubMed Central Google Scholar
• Hua, Y., Zhao, H., Kong, Y. & Ye, M. Association between the MTHFR gene and Alzheimer's disease: a meta-analysis. Int. J. Neurosci. 121, 462–471 (2011).
  Article CAS PubMed Google Scholar
• Lindson, N. & Aveyard, P. An updated meta-analysis of nicotine preloading for smoking cessation: investigating mediators of the effect. Psychopharmacology 214, 579–592 (2011).
  Article CAS PubMed Google Scholar
• Liu, H. et al. Association of 5-HTT gene polymorphisms with migraine: a systematic review and meta-analysis. J. Neurol. Sci. 305, 57–66 (2011).
  Article CAS PubMed Google Scholar
• Liu, J. et al. PITX3 gene polymorphism is associated with Parkinson's disease in Chinese population. Brain Res. 1392, 116–120 (2011).
  Article CAS PubMed Google Scholar
• MacKillop, J. et al. Delayed reward discounting and addictive behavior: a meta-analysis. Psychopharmacology 216, 305–321 (2011).
  Article CAS PubMed PubMed Central Google Scholar
• Maneeton, N., Maneeton, B., Srisurapanont, M. & Martin, S. D. Bupropion for adults with attention-deficit hyperactivity disorder: meta-analysis of randomized, placebo-controlled trials. Psychiatry Clin. Neurosci. 65, 611–617 (2011).
  Article PubMed Google Scholar
• Ohi, K. et al. The SIGMAR1 gene is associated with a risk of schizophrenia and activation of the prefrontal cortex. Prog. Neuropsychopharmacol. Biol. Psychiatry 35, 1309–1315 (2011).
  Article CAS PubMed Google Scholar
• Olabi, B. et al. Are there progressive brain changes in schizophrenia? A meta-analysis of structural magnetic resonance imaging studies. Biol. Psychiatry 70, 88–96 (2011).
  Article PubMed Google Scholar
• Oldershaw, A. et al. The socio-emotional processing stream in Anorexia Nervosa. Neurosci. Biobehav. Rev. 35, 970–988 (2011).
  Article CAS PubMed Google Scholar
• Oliver, B. J., Kohli, E. & Kasper, L. H. Interferon therapy in relapsing-remitting multiple sclerosis: a systematic review and meta-analysis of the comparative trials. J. Neurol. Sci. 302, 96–105 (2011).
  Article CAS PubMed Google Scholar
• Peerbooms, O. L. et al. Meta-analysis of MTHFR gene variants in schizophrenia, bipolar disorder and unipolar depressive disorder: evidence for a common genetic vulnerability? Brain Behav. Immun. 25, 1530–1543 (2011).
  Article CAS PubMed Google Scholar
• Pizzagalli, D. A. Frontocingulate dysfunction in depression: toward biomarkers of treatment response. Neuropsychopharmacology 36, 183–206 (2011).
  Article PubMed Google Scholar
• Rist, P. M., Diener, H. C., Kurth, T. & Schurks, M. Migraine, migraine aura, and cervical artery dissection: a systematic review and meta-analysis. Cephalalgia 31, 886–896 (2011).
  Article PubMed PubMed Central Google Scholar
• Sexton, C. E., Kalu, U. G., Filippini, N., Mackay, C. E. & Ebmeier, K. P. A meta-analysis of diffusion tensor imaging in mild cognitive impairment and Alzheimer's disease. Neurobiol. Aging 32, 2322.e5–2322.e18 (2011).
  Article Google Scholar
• Shum, D., Levin, H. & Chan, R. C. Prospective memory in patients with closed head injury: a review. Neuropsychologia 49, 2156–2165 (2011).
  Article PubMed Google Scholar
• Sim, H. et al. Acupuncture for carpal tunnel syndrome: a systematic review of randomized controlled trials. J. Pain 12, 307–314 (2011).
  Article PubMed Google Scholar
• Song, F. et al. Meta-analysis of plasma amyloid-β levels in Alzheimer's disease. J. Alzheimers Dis. 26, 365–375 (2011).
  Article CAS PubMed PubMed Central Google Scholar
• Sun, Q. L. et al. Correlation of E-selectin gene polymorphisms with risk of ischemic stroke A meta-analysis. Neural Regen. Res. 6, 1731–1735 (2011).
  CAS Google Scholar
• Tian, Y., Kang, L. G., Wang, H. Y. & Liu, Z. Y. Meta-analysis of transcranial magnetic stimulation to treat post-stroke dysfunction. Neural Regen. Res. 6, 1736–1741 (2011).
  Google Scholar
• Trzesniak, C. et al. Adhesio interthalamica alterations in schizophrenia spectrum disorders: a systematic review and meta-analysis. Prog. Neuropsychopharmacol. Biol. Psychiatry 35, 877–886 (2011).
  Article PubMed Google Scholar
• Veehof, M. M., Oskam, M. J., Schreurs, K. M. & Bohlmeijer, E. T. Acceptance-based interventions for the treatment of chronic pain: a systematic review and meta-analysis. Pain 152, 533–542 (2011).
  Article PubMed Google Scholar
• Vergouwen, M. D., Etminan, N., Ilodigwe, D. & Macdonald, R. L. Lower incidence of cerebral infarction correlates with improved functional outcome after aneurysmal subarachnoid hemorrhage. J. Cereb. Blood Flow Metab. 31, 1545–1553 (2011).
  Article PubMed PubMed Central Google Scholar
• Vieta, E. et al. Effectiveness of psychotropic medications in the maintenance phase of bipolar disorder: a meta-analysis of randomized controlled trials. Int. J. Neuropsychopharmacol. 14, 1029–1049 (2011).
  Article CAS PubMed Google Scholar
• Wisdom, N. M., Callahan, J. L. & Hawkins, K. A. The effects of apolipoprotein E on non-impaired cognitive functioning: a meta-analysis. Neurobiol. Aging 32, 63–74 (2011).
  Article CAS PubMed Google Scholar
• Witteman, J., van Ijzendoorn, M. H., van de Velde, D., van Heuven, V. J. & Schiller, N. O. The nature of hemispheric specialization for linguistic and emotional prosodic perception: a meta-analysis of the lesion literature. Neuropsychologia 49, 3722–3738 (2011).
  Article PubMed Google Scholar
• Woon, F. & Hedges, D. W. Gender does not moderate hippocampal volume deficits in adults with posttraumatic stress disorder: a meta-analysis. Hippocampus 21, 243–252 (2011).
  Article PubMed Google Scholar
• Xuan, C. et al. No association between APOE ε 4 allele and multiple sclerosis susceptibility: a meta-analysis from 5472 cases and 4727 controls. J. Neurol. Sci. 308, 110–116 (2011).
  Article CAS PubMed Google Scholar
• Yang, W. M., Kong, F. Y., Liu, M. & Hao, Z. L. Systematic review of risk factors for progressive ischemic stroke. Neural Regen. Res. 6, 346–352 (2011).
  Google Scholar
• Yang, Z., Li, W. J., Huang, T., Chen, J. M. & Zhang, X. Meta-analysis of Ginkgo biloba extract for the treatment of Alzheimer's disease. Neural Regen. Res. 6, 1125–1129 (2011).
  CAS Google Scholar
• Yuan, H. et al. Meta-analysis of tau genetic polymorphism and sporadic progressive supranuclear palsy susceptibility. Neural Regen. Res. 6, 353–359 (2011).
  Google Scholar
• Zafar, S. N., Iqbal, A., Farez, M. F., Kamatkar, S. & de Moya, M. A. Intensive insulin therapy in brain injury: a meta-analysis. J. Neurotrauma 28, 1307–1317 (2011).
  Article PubMed Google Scholar
• Zhang, Y. G. et al. The −1082G/A polymorphism in IL-10 gene is associated with risk of Alzheimer's disease: a meta-analysis. J. Neurol. Sci. 303, 133–138 (2011).
  Article CAS PubMed Google Scholar
• Zhu, Y., He, Z. Y. & Liu, H. N. Meta-analysis of the relationship between homocysteine, vitamin B(12), folate, and multiple sclerosis. J. Clin. Neurosci. 18, 933–938 (2011).
  Article CAS PubMed Google Scholar
• Ioannidis, J. P. & Trikalinos, T. A. An exploratory test for an excess of significant findings. Clin. Trials 4, 245–253 (2007). This study describes a test that evaluates whether there is an excess of significant findings in the published literature. The number of expected studies with statistically significant results is estimated and compared against the number of observed significant studies.
  Article PubMed Google Scholar
• Ioannidis, J. P. Excess significance bias in the literature on brain volume abnormalities. Arch. Gen. Psychiatry 68, 773–780 (2011).
  Article PubMed Google Scholar
• Pfeiffer, T., Bertram, L. & Ioannidis, J. P. Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias. PLoS ONE 6, e18362 (2011).
  Article CAS PubMed PubMed Central Google Scholar
• Tsilidis, K. K., Papatheodorou, S. I., Evangelou, E. & Ioannidis, J. P. Evaluation of excess statistical significance in meta-analyses of 98 biomarker associations with cancer risk. J. Natl Cancer Inst. 104, 1867–1878 (2012).
  Article CAS PubMed Google Scholar
• Ioannidis, J. Clarifications on the application and interpretation of the test for excess significance and its extensions. J. Math. Psychol. (in the press).
• David, S. P. et al. Potential reporting bias in small fMRI studies of the brain. PLoS Biol. (in the press).
• Sena, E. S., van der Worp, H. B., Bath, P. M., Howells, D. W. & Macleod, M. R. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biol. 8, e1000344 (2010).
  Article CAS PubMed PubMed Central Google Scholar
• Ioannidis, J. P. Extrapolating from animals to humans. Sci. Transl. Med. 4, 151ps15 (2012).
  Article PubMed Google Scholar
• Jonasson, Z. Meta-analysis of sex differences in rodent models of learning and memory: a review of behavioral and biological data. Neurosci. Biobehav. Rev. 28, 811–825 (2005).
  Article PubMed Google Scholar
• Macleod, M. R. et al. Evidence for the efficacy of NXY-059 in experimental focal cerebral ischaemia is confounded by study quality. Stroke 39, 2824–2829 (2008).
  Article PubMed Google Scholar
• Sena, E., van der Worp, H. B., Howells, D. & Macleod, M. How can we improve the pre-clinical development of drugs for stroke? Trends Neurosci. 30, 433–439 (2007).
  Article CAS PubMed Google Scholar
• Russell, W. M. S. & Burch, R. L. The Principles of Humane Experimental Technique (Methuen, 1958).
  Google Scholar
• Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol. 8, e1000412 (2010).
  Article CAS PubMed PubMed Central Google Scholar
• Bassler, D., Montori, V. M., Briel, M., Glasziou, P. & Guyatt, G. Early stopping of randomized clinical trials for overt efficacy is problematic. J. Clin. Epidemiol. 61, 241–246 (2008).
  Article PubMed Google Scholar
• Montori, V. M. et al. Randomized trials stopped early for benefit: a systematic review. JAMA 294, 2203–2209 (2005).
  Article CAS PubMed Google Scholar
• Berger, J. O. & Wolpert, R. L. The Likelihood Principle: A Review, Generalizations, and Statistical Implications (ed. Gupta, S. S.) (Institute of Mathematical Sciences, 1998).
  Google Scholar
• Vesterinen, H. M. et al. Systematic survey of the design, statistical analysis, and reporting of studies published in the 2008 volume of the Journal of Cerebral Blood Flow and Metabolism. J. Cereb. Blood Flow Metab. 31, 1064–1072 (2011).
  Article PubMed Google Scholar
• Smith, R. A., Levine, T. R., Lachlan, K. A. & Fediuk, T. A. The high cost of complexity in experimental design and data analysis: type I and type II error rates in multiway ANOVA. Hum. Comm. Res. 28, 515–530 (2002).
  Article Google Scholar
• Perel, P. et al. Comparison of treatment effects between animal experiments and clinical trials: systematic review. BMJ 334, 197 (2007).
  Article CAS PubMed Google Scholar
• Nosek, B. A. & Bar-Anan, Y. Scientific utopia: I. Opening scientific communication. Psychol. Inquiry 23, 217–243 (2012).
  Article Google Scholar
• Open-Science-Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect. Psychol. Sci. 7, 657–660 (2012). This article describes the Reproducibility Project — an open, large-scale, collaborative effort to systematically examine the rate and predictors of reproducibility in psychological science. This will allow the empirical rate of replication to be estimated.
• Simera, I. et al. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 8, 24 (2010).
  Article PubMed PubMed Central Google Scholar
• Ioannidis, J. P. The importance of potential studies that have not existed and registration of observational data sets. JAMA 308, 575–576 (2012).
  Article CAS PubMed Google Scholar
• Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H. & Ioannidis, J. P. Public availability of published research data in high-impact journals. PLoS ONE 6, e24357 (2011).
  Article CAS PubMed PubMed Central Google Scholar
• Ioannidis, J. P. et al. Repeatability of published microarray gene expression analyses. Nature Genet. 41, 149–155 (2009).
  Article CAS PubMed Google Scholar
• Ioannidis, J. P. & Khoury, M. J. Improving validation practices in “omics” research. Science 334, 1230–1232 (2011).
  Article CAS PubMed PubMed Central Google Scholar
• Chambers, C. D. Registered Reports: A new publishing initiative at Cortex. Cortex 49, 609–610 (2013).
  Article PubMed Google Scholar
• Ioannidis, J. P., Tarone, R. & McLaughlin, J. K. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22, 450–456 (2011).
  Article PubMed Google Scholar
• Siontis, K. C., Patsopoulos, N. A. & Ioannidis, J. P. Replication of past candidate loci for common diseases and phenotypes in 100 genome-wide association studies. Eur. J. Hum. Genet. 18, 832–837 (2010).
  Article CAS PubMed PubMed Central Google Scholar
• Ioannidis, J. P. & Trikalinos, T. A. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J. Clin. Epidemiol. 58, 543–549 (2005).
  Article PubMed Google Scholar
• Ioannidis, J. Why science is not necessarily self-correcting. Perspect. Psychol. Sci. 7, 645–654 (2012).
  Article PubMed Google Scholar
• Zollner, S. & Pritchard, J. K. Overcoming the winner's curse: estimating penetrance parameters from case-control data. Am. J. Hum. Genet. 80, 605–615 (2007).
  Article CAS PubMed PubMed Central Google Scholar