Accuracy of Effect Size Estimates From Published Psychological Experiments Involving Multiple Trials (original) (raw)
Related papers
Multiple trials may yield exaggerated effect size estimates
The Journal of general …, 2010
Published psychological research attempting to support the existence of small and medium effect sizes may not have enough participants to do so accurately and, thus, repeated trials or the use of multiple items may be used in an attempt to obtain significance. Through a series of Monte-Carlo simulations, this paper describes the results of multiple trials or items on effect size estimates when the averages and aggregates of a dependent measure are analyzed. The simulations revealed a large increase in observed effect size estimates when the numbers of trials or items in an experiment were increased. Overestimation effects are mitigated by correlations between trials or items but remain substantial in some cases. Some concepts such as a P300
Australian Journal of Psychology, 2012
Estimation based on effect sizes, confidence intervals, and meta-analysis usually provides a more informative analysis of empirical results than does statistical significance testing, which has long been the conventional choice in psychology. The sixth edition of the American Psychological Association Publication Manual now recommends that psychologists should, wherever possible, use estimation and base their interpretation of research results on point and interval estimates. We outline the Manual's recommendations and suggest how they can be put into practice: adopt an estimation framework, starting with the formulation of research aims as 'How much?' or 'To what extent?' questions. Calculate from your data effect size estimates and confidence intervals to answer those questions, then interpret. Wherever appropriate, use meta-analysis to integrate evidence over studies. The Manual's recommendations can assist psychologists improve they way they do their statistics and help build a more quantitative and cumulative discipline.
2019
Effect sizes are underappreciated and often misinterpreted – the most common mistakes being to describe them in ways that are uninformative (e.g., using arbitrary standards) or misleading (e.g., squaring effect size r’s). We propose that effect sizes can be usefully evaluated in comparison with well-understood benchmarks or in terms of concrete consequences. In that light, we conclude that, when reliably estimated (a critical consideration), an effect of r =.05 is “very small” for the explanation of single events but potentially consequential in the not-very long run, r = .10 is still “small” at the level of single events but potentially more ultimately consequential; r = .20 is “medium” and of some use even in the short run and therefore even more important; and an effect size of r = .30 is “large” and potentially powerful in the short and long run. A “very large” effect size (r = .40 or greater) in the context of psychological research is likely to be a gross overestimate rarely f...
Effect size estimates: Current use, calculations, and interpretation
2012
The Publication Manual of the American Psychological Association (American Psychological Association, 2001, 2010) calls for the reporting of effect sizes and their confidence intervals. Estimates of effect size are useful for determining the practical or theoretical importance of an effect, the relative contributions of factors, and the power of an analysis. We surveyed articles published in 2009 and 2010 in the Journal of Experimental Psychology: General, noting the statistical analyses reported and the associated reporting of effect size estimates. Effect sizes were reported for fewer than half of the analyses; no article reported a confidence interval for an effect size. The most often reported analysis was analysis of variance, and almost half of these reports were not accompanied by effect sizes. Partial 2 was the most commonly reported effect size estimate for analysis of variance. For t tests, 2/3 of the articles did not report an associated effect size estimate; Cohen's d was the most often reported. We provide a straightforward guide to understanding, selecting, calculating, and interpreting effect sizes for many types of data and to methods for calculating effect size confidence intervals and power analysis.
Evaluating Effect Size in Psychological Research: Sense and Nonsense
Advances in Methods and Practices in Psychological Science
Effect sizes are underappreciated and often misinterpreted—the most common mistakes being to describe them in ways that are uninformative (e.g., using arbitrary standards) or misleading (e.g., squaring effect-size rs). We propose that effect sizes can be usefully evaluated by comparing them with well-understood benchmarks or by considering them in terms of concrete consequences. In that light, we conclude that when reliably estimated (a critical consideration), an effect-size r of .05 indicates an effect that is very small for the explanation of single events but potentially consequential in the not-very-long run, an effect-size r of .10 indicates an effect that is still small at the level of single events but potentially more ultimately consequential, an effect-size r of .20 indicates a medium effect that is of some explanatory and practical use even in the short run and therefore even more important, and an effect-size r of .30 indicates a large effect that is potentially powerful...
Communication Monographs, 2009
Meta-analysis involves cumulating effects across studies in order to qualitatively summarize existing literatures. A recent finding suggests that the effect sizes reported in meta-analyses may be negatively correlated with study sample sizes. This prediction was tested with a sample of 51 published meta-analyses summarizing the results of 3,602 individual studies. The correlation between effect size and sample size was negative in almost 80 percent of the meta-analyses examined, and the negative correlation was not limited to a particular type of research or substantive area. This result most likely stems from a bias against publishing findings that are not statistically significant. The primary implication is that meta-analyses may systematically overestimate population effect sizes. It is recommended that researchers routinely examine the nÁr scatter plot and correlation, or some other indication of publication bias and report this information in meta-analyses.
The Effect Size Statistic: Overview of Various Choices
2000
Over the years, methodologists have been recommending that researchers use magnitude of effect estimates in result interpretation to highlight the distinction between statistical and practical significance (cf. R. Kirk, 1996). A magnitude of effect statistic (i.e., effect size) tells to what degree the dependent variable can be controlled, predicted, or explained by the independent variable (P. Snyder and S. Lawson, 1993). There are a number of ways one can compute an effect size statistic as part of data analysis. There is no concept of "one size fits all" (B. Thompson, 1999), so it is up to the smart researcher to choose the index best suited for a particular research endeavor. It has now become necessary that such a statistic always be included to enable other researchers to carry out meta-analyses and to inform judgment regarding the practical significance of results. This paper provides a tutorial summary of some of the effect size choices so that researchers will be able to follow the recommendations of the American Psychological Association (APA) publication manual, those of the APA Task Force on Statistical Inference, and the publication requirements of some journals. (Contains 3 tables and 11 references.) (Author/SLD) Reproductions supplied by EDRS are the best that can be made from the original document.
Examining the Reproducibility of Meta-Analyses in Psychology: A Preliminary Report
Meta-analyses are an important tool to evaluate the literature. It is essential that meta-analyses can easily be reproduced to allow researchers to evaluate the impact of subjective choices on meta-analytic effect sizes, but also to update meta-analyses as new data comes in, or as novel statistical techniques (for example to correct for publication bias) are developed. Research in medicine has revealed meta-analyses often cannot be reproduced. In this project, we examined the reproducibility of meta-analyses in psychology by reproducing twenty published meta-analyses. Reproducing published meta-analyses was surprisingly difficult. 96% of meta-analyses published in 2013-2014 did not adhere to reporting guidelines. A third of these meta-analyses did not contain a table specifying all individual effect sizes. Five of the 20 randomly selected meta-analyses we attempted to reproduce could not be reproduced at all due to lack of access to raw data, no details about the effect sizes extrac...