How to assess publication bias: funnel plot, trim-and-fill method and selection models (original) (raw)
How to assess publication bias: funnel plot, trim-and-fill method and selection models
Free
- Dimitris Mavridis1,2,
- Georgia Salanti1
- 1Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece
- 2Department of Primary Education, University of Ioannina, Ioannina, Greece
- Correspondence to Dr Dimitris Mavridis, Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece; dimi.mavridis{at}googlemail.com
Statistics from Altmetric.com
Request Permissions
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
This new section of the Journal is aimed at providing the essential information readers should know about the topics that are addressed in the “Statistics in practice” paper published in the same issue of the journal. This stand-alone section has to be seen as an articulated summary of the main notions clinicians have to know about some basic concepts in statistics, which may be useful for their evidence based practice. After going through these notes, readers are encouraged to read the “Statistics in practice” articles. Of course, we welcome any feedback from you (via email or Twitter) about this!
The EBMH Editors
A funnel plot is a scatter plot of the treatment effect estimates from individual trials against a measure of study's precision (usually the standard error (SE)).1
- Asymmetry in the funnel plot should not be automatically equated with publications bias.
- There are various alternative explanations for funnel plot asymmetry, such as heterogeneity, selective outcome reporting and chance.
- A sizeable number of studies (by convention, usually at least 10 though more may be needed in the presence of substantial heterogeneity) are needed to obtain a visual assessment of the funnel plot.
- Funnel plots should be better seen as a means for exploring small-study effects; small studies showing systematically larger effects than large studies.
- Adding contours of statistical significance to the funnel plot may help distinguish publication bias from other causes of funnel plot asymmetry.
Trim-and-fill method
The trim-and-fill is a funnel plot-derived, two-step method aimed at both identifying publication bias and adjusting results for it.1 Phase 1 (Trimming): to exclude small studies in order to have a symmetrical plot and then estimate an adjusted summary effect considering only the larger studies. Phase 2 (Filling): to replicate the funnel plot replacing the excluded studies with their ‘missing’ counterparts around the adjusted summary estimate.
- The trim-and-fill method provides a summary effect adjusted for publication bias.
- It allows estimating the number of unpublished studies.
- The trim-and-fill method assumes publication bias as the only reason for funnel plot asymmetry which is an unrealistic assumption. It should be used as a sensitivity analysis as its inventors suggested.
Selection models
Selection models focus on the selection process, that is, the mechanism by which trials are selected for publication.1 Using selection models, researchers can estimate the likely impact the missing studies would have, had they been included in the meta-analysis. One of the key assumptions in the selection models is that the included sample of studies is not at random. The studies have been included because they have some characteristics that increase their propensity for publication, therefore the overall estimate is conditional to the observed studies that have been published and identified. Taking this into account, it is possible to calculate the marginal effect size, which is the effect size unconditional to the publication status.
- Assumptions are needed about the factors that influence the probability of publication for a study. The probability of publication is typically assumed to be a function of a study's p value or sample size.
- A selection process does not necessarily entail bias. If the effect sizes are comparable in small and large studies, then even a strong selection process will not alter the results of meta-analysis.
- In meta-analysis, selection models condition the observed effect size in each study to its propensity for publication and they estimate the unconditional summary that pertains to all studies that have been carried out, either published or unpublished.
- Copas selection model is the most sophisticated selection model because it assumes that probability of publication depends on both the study's effect size and SE. It allows us to estimate the correlation between the probability of publication of a study and the effect size. If different from zero, this means that the selection process has produced a publication bias.
- Unlike approaches based on funnel plot asymmetry, selection models test and adjust for publication bias, without being confounded by heterogeneity.
- The mechanism of the selection process is unknown and a sensitivity analysis is advocated in which the intervention effect is estimated under different assumptions about the severity of selection bias.