Estimating the mean effect size in meta-analysis: Bias, precision, and mean squared error of different weighting methods (original) (raw)
Related papers
A Unified Approach to the Estimation of Effect Size in Meta-Analysis
1989
Parametric measures to estimate J. Cohen's effect size (1966) from a single experiment or for a single study in meta-analysis are investigated. The main objective was to examine the principal statistical properties of this effect size-delta-under variance homogeneity, variance heterogeneity with known variance ratios, and for the Behrens-Fisher problem. Derived estimators were compared according to the criteria of ..heir magnitudes, unbiasedness, and mean-square errors. General properties of the derived estimators were examined by means of Monte Carlo results. Results tend to confirm the recommendation of theoretical analysis that two identified estimators, "h" and "d(sub T)," should be used in conducting a meta-analytic study. These estimators better ensure unbiasedness and minimum mean-square error than do other: derived. Eight tables and four graphs illustrate study data. (SLD) *
The accuracy of effect-size estimates under normals and contaminated normals in meta-analysis
Heliyon
This article evaluates the accuracy of effect-size estimates for some estimation procedures in meta-analysis. The dilemma of which effect-size estimate is suitable is still a problem in meta-analysis. Monte Carlo simulations were used to generate random variables from a normal distribution or contaminated normal distribution for primary studies. The primary studies were hypothesised to have equal variance under different population effect sizes. The primary studies were also hypothesised to have unequal variance. Meta-analysis was done on the simulated hypothesized-primary-studies. The effect sizes for the simulated design of the primary studies were estimated using Cohen's , Hedges' , Glass' △, Cliff's delta and the Probability of Superiority. Their corresponding standard error and confidence interval were computed and a comparison of an efficient estimator was done using statistical bias, percentage error and confidence interval width. The statistical bias, percentage error and confidence interval width pointed to Probability of Superiority as an accurate effect size estimate under contaminated normal distribution, and Hedges' as the most accurate effect size estimates compared to Cohen's and Glass' △ when equal variance assumptions are violated. This study suggests that the accuracy of effect size estimates depends on the details of the primary studies included in the metaanalysis.
A comparison of the effect size estimators in meta-analysis
2009
The objective of a meta-analysis is usually to estimate the overall treatment effect and make inferences about the difference between the efffects of the two treatments. This article presents several forms of effect size estimators and compares these effect size estimators and the variance of overall treatment effect estimator within each group. as outcome measures, standarized difference is considered. Four modes of effect size estimators are discussed. Effect size estimators by Glass, Hedges, The Maximum Likelihood, and Shrunken Estimators of Effect Size are employed in this study. Finally, with the help of a software the results of these four effect size estimators are discussed. Estimators are illustrated using a comparison of the effectiveness of amlodipine and placebo on work capacity.
Consequences of effect size heterogeneity for meta-analysis: a Monte Carlo study
Statistical Methods and Applications, 2010
In this article we use Monte Carlo analysis to assess the small sample behaviour of the OLS, the weighted least squares (WLS) and the mixed effects metaestimators under several types of effect size heterogeneity, using the bias, the mean squared error and the size and power of the statistical tests as performance indicators. Specifically, we analyse the consequences of heterogeneity in effect size precision (heteroskedasticity) and of two types of random effect size variation, one where the variation holds for the entire sample, and one where only a subset of the sample of studies is affected. Our results show that the mixed effects estimator is to be preferred to the other two estimators in the first two situations, but that WLS outperforms OLS and mixed effects in the third situation. Our findings therefore show that, under circumstances that are quite common in practice, using the mixed effects estimator may be suboptimal and that the use of WLS is preferable.
Improving the Meta-Analytic Assessment of Effect Size Variance With an Informed Bayesian Prior
Journal of Management, 2014
Meta-analytic estimation of effect size variance is critical for determining the degree to which a relationship or finding generalizes across contexts. In most meta-analyses, population effect size variability is estimated by subtracting expected sampling error variance from observed variance, using only information from a limited set of available studies. We propose an improved Bayesian variance estimation technique that incorporates findings from previous meta-analytic research through an informed prior distribution of likely levels of effect size variance. The logic of exchangeability as a conceptual foundation for using an informed prior is explicated. Based on Monte Carlo simulations, we find the traditional method of meta-analytic variance estimation the most biased and least accurate technique across all sizes of meta-analyses considered. The Bayesian methodology incorporating an informed prior proved to be most accurate and overall least biased of all estimation methods. Conceptual advantages and limitations that must be taken into account when incorporating an informed prior to estimate variability of effect sizes in a meta-analysis are also discussed.
The Impact of Effect Size Heterogeneity on Meta-Analysis: A Monte Carlo Experiment
SSRN Electronic Journal, 2000
In this paper we use Monte Carlo simulation to investigate the impact of effect size heterogeneity on the results of a meta-analysis. Specifically, we address the small sample behaviour of the OLS, the fixed effects regression and the mixed effects meta-estimators under three alternative scenarios of effect size heterogeneity. We distinguish heterogeneity in effect size variance, heterogeneity due to a varying true underlying effect across primary studies, and heterogeneity due to a non-systematic impact of omitted variable bias in primary studies. Our results show that the mixed effects estimator is to be preferred to the other two estimators in the first two situations. However, in the presence of random effect size variation due to a non-systematic impact of omitted variable bias, using the mixed effects estimator may be suboptimal. We also address the impact of sample size and show that meta-analysis sample size is far more effective in reducing meta-estimator variance and increasing the power of hypothesis testing than primary study sample size. JEL-codes: C12; C15; C40
Psychological Bulletin, 1992
Combined significance tests (combined p values) and tests of the weighted mean effect size are both used to combine information across studies in meta-analysis. This article compares a combined significance test (the Stouffer test) with a test based on the weighted mean effect size as tests of the same null hypothesis. The tests are compared analytically in the case in which the withingroup variances are known and compared through large-sample theory in the more usual case in which the variances are unknown. Generalizations suggested are then explored through a simulation study. This work demonstrates that the test based on the average effect size is usually more powerful than the Stouffer test unless there is a substantial negative correlation between withinstudy sample size and effect size. Thus the test based on the average effect size is generally preferable, and there is little reason to also calculate the Stouffer test.
Methods of estimating the pooled effect size under meta-analysis: A comparative appraisal
Clinical Epidemiology and Global Health, 2020
Present study has compared methods of synthesizing the pooled effect estimate under meta-analysis, namely Fixed Effect Method (FEM), Random Effects Method (REM) and a recently proposed Weighted Least Square (WLS) method. Methods: Three methods of estimating pooled effect estimates under meta-analysis were compared on the basis of coverage probability and width of confidence interval. These methods were compared for seven outcomes with varying heterogeneity and sample size using real data of systematic review comparing neo-adjuvant chemotherapy with adjuvant chemotherapy involving 'hazard ratio' and 'risk ratio' as effect size. Results: WLS method was found to be superior to FEM having higher coverage probability in case of heterogeneity. Further, WLS with similar coverage probability was found to be superior to REM with more precise confidence interval. Conclusion: Unrestricted WLS method needs to be preferred unconditionally over fixed effect method and random effects method.
rESCMA: A brief summary on effect size conversion for meta-analysis
OSF Preprints, 2020
Effect sizes are highly relevant in quantitative research. It facilitates the comparison and quantitative synthesis of scientific studies. The main objective of this report is to present: a) a brief summary of the formulas used for conversion between the three main effect sizes used in the meta-analysis: the correlation coefficient, the standardized mean difference and the odds ratio; and b) the Rapid Effect Size Converter for Meta-Analysis (rESCMA), a open-source and browser-based app for efficient conversion and bulk-conversion of effect sizes and their variances based on the formulas proposed in this report. In addition, a table summarizing the formulas is presented for easy accessibility and use.