Discussion and comments : strong versus weak significance tests and the role of meta-analytic procedures (original) (raw)
Related papers
The use of meta-analytic statistical significance testing
Research Synthesis Methods, 2014
Meta-analysis multiplicity, the concept of conducting multiple tests of statistical significance within one review, is an underdeveloped literature. We address this issue by considering how Type I errors can impact meta-analytic results, suggest how statistical power may be affected through the use of multiplicity corrections, and propose how meta-analysts should analyze multiple tests of statistical significance. The context for this study is a meta-review of meta-analyses published in two leading review journals in education and psychology. Our review of 130 meta-analyses revealed a strong reliance on statistical significance testing without consideration of Type I errors or the use of multiplicity corrections. In order to provide valid conclusions, meta-analysts must consider these issues prior to conducting the review. F Meta-regression: parameter estimates H 0 : β = 0; tests whether the slope of the regression line is greater than sampling error z or t 100% 50% 10% 1% Figure 1. Type I error rate for different proportions of true null hypotheses. Lines represent assumptions of the percentage of true null hypotheses.
Critical Examination of the Ongoing Use of Significance Tests in Psychology
Null hypothesis significance testing is entrenched within the psychological enterprise. The hybrid theory underlying significance testing contributes to misinterpretation. The amalgamation of the Ronald Fisher’s null hypothesis test and the Neyman-Pearson predetermined percentage error within a frequentist probabilistic framework results in the propagation of fallacies concerning probability, the p value and reportage of experimental conclusions. Such misconceptions arise from logically flawed theoretical constructs. As fallacies are closely associated with individual beliefs, misinterpretations infiltrate psychological research and are presented within hypothetical constructs and conclusions. The American Psychological Association has advised that significance testing incorporate confidence intervals and effect sizes to minimise potential confusion and adhere to simple testing processes to combat ambiguity.
Journal of Informetrics, 2013
Meta-analysis refers to the statistical methods used in research synthesis for combining and integrating results from individual studies. In this regard meta-analytical studies share with narrative reviews the goal of synthesizing the scientific literature on a particular topic, while as in the case of standard articles they present new results. This study aims to identify the potential similarities and differences between meta-analytical studies, reviews and standard articles as regards their impact and structural features in the field of ...
Researchers Should Make Thoughtful Assessments Instead of Null-Hypothesis Significance Tests
Organization Science, 2011
Abstract Null-hypothesis significance tests (NHSTs) have received much criticism, especially during the last two decades. Yet many behavioral and social scientists are unaware that NHSTs have drawn increasing criticism, so this essay summarizes key criticisms. The essay also recommends alternative ways of assessing research findings. Although these recommendations are not complex, they do involve ways of thinking that many behavioral and social scientists find novel.
Organization Science, 2011
Null-hypothesis significance tests (NHSTs) have received much criticism, especially during the last two decades. Yet many behavioral and social scientists are unaware that NHSTs have drawn increasing criticism, so this essay summarizes key criticisms. The essay also recommends alternative ways of assessing research findings. Although these recommendations are not complex, they do involve ways of thinking that many behavioral and social scientists find novel. Instead of making NHSTs, researchers should adapt their research assessments to specific contexts and specific research goals, and then explain their rationales for selecting assessment indicators. Researchers should show the substantive importance of findings by reporting effect sizes and should acknowledge uncertainty by stating confidence intervals. By comparing data with naïve hypotheses rather than with null hypotheses, researchers can challenge themselves to develop better theories. Parsimonious models are easier to unders...
In recent decades psychological researchers have overemphasized exploratory research with little regard for well-powered confirmatory research. Cumming advocates that the resulting research problems be addressed with “new statistics” that emphasize estimation and meta-analyses and avoid hypothesis tests. He advocates that researchers avoid “dichotomous thinking” and associated conclusions about the validity of hypotheses. Unfortunately, this continues to overemphasize exploratory research and to underemphasize confirmation. Hypothesis tests and estimation both have a role in effective statistical methodology. Hypothesis tests are optimal for (a) research when human life is directly involved and answers are urgently needed, (b) controversial areas of research such as parapsychology, and (c) any case when researchers want to provide the most convincing evidence that they understand and can control an effect. Meta-analysis is post hoc analysis that involves correlational analysis of observational data. Like other types of post hoc analyses, meta-analyses have not been effective at resolving scientific controversies. A clear distinction between exploratory research and confirmatory research is needed, as has been established for clinical trials. A group of well-designed, adequately powered confirmatory studies using hypothesis tests provides the strongest scientific evidence that an effect is valid. The “new statistics” appear to place priority on generating academic publications rather than drawing strong inferences about the validity of effects.
Australian Journal of Psychology, 2012
Estimation based on effect sizes, confidence intervals, and meta-analysis usually provides a more informative analysis of empirical results than does statistical significance testing, which has long been the conventional choice in psychology. The sixth edition of the American Psychological Association Publication Manual now recommends that psychologists should, wherever possible, use estimation and base their interpretation of research results on point and interval estimates. We outline the Manual's recommendations and suggest how they can be put into practice: adopt an estimation framework, starting with the formulation of research aims as 'How much?' or 'To what extent?' questions. Calculate from your data effect size estimates and confidence intervals to answer those questions, then interpret. Wherever appropriate, use meta-analysis to integrate evidence over studies. The Manual's recommendations can assist psychologists improve they way they do their statistics and help build a more quantitative and cumulative discipline.
Examining the Reproducibility of Meta-Analyses in Psychology: A Preliminary Report
Meta-analyses are an important tool to evaluate the literature. It is essential that meta-analyses can easily be reproduced to allow researchers to evaluate the impact of subjective choices on meta-analytic effect sizes, but also to update meta-analyses as new data comes in, or as novel statistical techniques (for example to correct for publication bias) are developed. Research in medicine has revealed meta-analyses often cannot be reproduced. In this project, we examined the reproducibility of meta-analyses in psychology by reproducing twenty published meta-analyses. Reproducing published meta-analyses was surprisingly difficult. 96% of meta-analyses published in 2013-2014 did not adhere to reporting guidelines. A third of these meta-analyses did not contain a table specifying all individual effect sizes. Five of the 20 randomly selected meta-analyses we attempted to reproduce could not be reproduced at all due to lack of access to raw data, no details about the effect sizes extrac...
Meta-analysis in Psychological Research
International Journal of …, 2010
Meta-analysis is a research methodology that aims to quantitatively integrate the results of a set of empirical studies about a given topic. With this purpose, effect-size indices are obtained from the individual studies and the characteristics of the studies are coded in order to examine their relationships with the effect sizes. Statistical analysis in meta-analysis requires the weighting of each effect estimate as a function of its precision, by assuming a fixed-or a randomeffects model. This paper outlines the steps required for carrying out the statistical analyses in a meta-analysis, the different statistical models that can be assumed, and the consequences of the assumptions in interpreting their results. The statistical analyses are illustrated with a real example.
Meta-Analyses in Psychology Often Overestimate Evidence for and Size of Effects
Adjusting for publication bias is essential when drawing meta-analytic inferences. However,most methods that adjust for publication bias are sensitive to the particular researchconditions, such as the degree of heterogeneity in effect sizes across studies. Sladekovaet al. (2022) tried to circumvent this complication by selecting the methods that are mostappropriate for a given set of conditions, and concluded that publication bias on averagecauses only minimal over-estimation of effect sizes in psychology. However, this approachsuffers from a “catch-22” problem — to know the underlying research conditions, one needsto have adjusted for publication bias correctly, but to correctly adjust for publication bias,one needs to know the underlying research conditions. To alleviate this problem weconduct an alternative analysis, Robust Bayesian meta-analysis (RoBMA), which is notbased on model-selection but on model-averaging. In RoBMA, models that predict theobserved results better are give...