Evaluation of Reproducible Research Practices in Oncology Systematic Reviews With Meta-analyses Referenced by National Comprehensive Cancer Network Guidelines - PubMed (original) (raw)
Evaluation of Reproducible Research Practices in Oncology Systematic Reviews With Meta-analyses Referenced by National Comprehensive Cancer Network Guidelines
Cole Wayant et al. JAMA Oncol. 2019.
Abstract
Importance: Reproducible research practices are essential to biomedical research because these practices promote trustworthy evidence. In systematic reviews and meta-analyses, reproducible research practices ensure that summary effects used to guide patient care are stable and trustworthy.
Objective: To evaluate the reproducibility in theory of meta-analyses in oncology systematic reviews cited by the 49 National Comprehensive Cancer Network (NCCN) guidelines for the treatment of cancer by site and evaluate whether Cochrane reviews or systematic reviews that report adherence to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines use more reproducible research practices.
Design, setting, and participants: A cross-sectional investigation of all systematic reviews with at least 1 meta-analysis and at least 1 included randomized clinical trial (RCT) that are cited by NCCN guidelines for treatment of cancer by site. We scanned the reference list of all NCCN guidelines (n = 49) for potential systematic reviews and meta-analyses. All retrieved studies were screened, and data were extracted, independently and in duplicate. The analysis was carried out between May 6, 2018, and January 28, 2019.
Main outcomes and measures: The frequency of reproducible research practices, defined as (1) effect estimate and measure of precision (eg, hazard ratio with 95% confidence interval); (2) clear list of studies included for each analysis; and (3) for subgroup and sensitivity analyses, it must be clear which studies were included in each group or level.
Results: We identified 1124 potential systematic reviews, and 154 meta-analyses comprising 3696 meta-analytic effect size estimates were included. Only 2375 of the 3696 meta-analytic estimates (64.3%), including subgroup and sensitivity analyses, were reproducible in theory. Forest plots appear to improve the reproducibility of meta-analyses. All meta-analytic estimates were reproducible in theory in 100 systematic reviews (64.9%), and in 15 systematic reviews (9.7%), no meta-analytic estimates could potentially be reproduced. Data were said to be imputed in 29 meta-analyses, but none specified which data. Only 1 meta-analysis included a link to an online data set.
Conclusions and relevance: More reproducible research practices are needed in oncology meta-analyses, as suggested by those that are cited by the NCCN. Reporting meta-analyses in forest plots and requirements for full data sharing are recommended.
Conflict of interest statement
Conflict of Interest Disclosures: Dr Vassar reported grants from Oklahoma Center for the Advancement of Science and Technology outside the submitted work. No other disclosures were reported. Dr Page is supported by an Australian National Health and Medical Research Council Early Career Fellowship (1088535). No other conflicts are reported.
Figures
Figure 1.. PRISMA Diagram
IPD indicates individual patient data; NCCN, National Comprehensive Cancer Network; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses; RCT, randomized clinical trial; SR, systematic review.
Figure 2.. Forest Plot of Studies Stratified by Mention of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Guidelines
NCCN indicates National Comprehensive Cancer Network; uRR, unadjusted risk ratio.
Figure 3.. Forest Plot of Studies Stratified by Time Frame
PRISMA indicates Preferred Reporting Items for Systematic Reviews and Meta-Analyses; uRR, unadjusted risk ratio.
Comment in
- doi: 10.1001/jamaoncol.2019.2664
Similar articles
- Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas.
Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, Moraleda C, Rogers L, Daniels K, Green P. Crider K, et al. Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article. - The future of Cochrane Neonatal.
Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834 - Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions.
Page MJ, McKenzie JE, Kirkham J, Dwan K, Kramer S, Green S, Forbes A. Page MJ, et al. Cochrane Database Syst Rev. 2014 Oct 1;2014(10):MR000035. doi: 10.1002/14651858.MR000035.pub2. Cochrane Database Syst Rev. 2014. PMID: 25271098 Free PMC article. - Reproducible research practices are underused in systematic reviews of biomedical interventions.
Page MJ, Altman DG, Shamseer L, McKenzie JE, Ahmadzai N, Wolfe D, Yazdi F, Catalá-López F, Tricco AC, Moher D. Page MJ, et al. J Clin Epidemiol. 2018 Feb;94:8-18. doi: 10.1016/j.jclinepi.2017.10.017. Epub 2017 Nov 4. J Clin Epidemiol. 2018. PMID: 29113936 Review. - The Benefits and Challenges of Using Multiple Sources of Information about Clinical Trials [Internet].
Dickersin K, Mayo-Wilson E, Li T. Dickersin K, et al. Washington (DC): Patient-Centered Outcomes Research Institute (PCORI); 2018 Mar. Washington (DC): Patient-Centered Outcomes Research Institute (PCORI); 2018 Mar. PMID: 37043591 Free Books & Documents. Review.
Cited by
- Advances in the field of developing biomarkers for re-irradiation: a how-to guide to small, powerful data sets and artificial intelligence.
Huma C, Hawon L, Sarisha J, Erdal T, Kevin C, Valentina KA. Huma C, et al. Expert Rev Precis Med Drug Dev. 2024;9(1):3-16. doi: 10.1080/23808993.2024.2325936. Epub 2024 Mar 11. Expert Rev Precis Med Drug Dev. 2024. PMID: 38550554 - [The PRISMA 2020 statement: an updated guideline for reporting systematic reviewsDeclaración PRISMA 2020: una guía actualizada para la publicación de revisiones sistemáticas].
Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, Chou R, Glanville J, Grimshaw JM, Hróbjartsson A, Lalu MM, Li T, Loder EW, Mayo-Wilson E, McDonald S, McGuinness LA, Stewart LA, Thomas J, Tricco AC, Welch VA, Whiting P, Moher D. Page MJ, et al. Rev Panam Salud Publica. 2022 Dec 30;46:e112. doi: 10.26633/RPSP.2022.112. eCollection 2022. Rev Panam Salud Publica. 2022. PMID: 36601438 Free PMC article. Portuguese. - Sex, Racial, and Ethnic Representation in COVID-19 Clinical Trials: A Systematic Review and Meta-analysis.
Xiao H, Vaidya R, Liu F, Chang X, Xia X, Unger JM. Xiao H, et al. JAMA Intern Med. 2023 Jan 1;183(1):50-60. doi: 10.1001/jamainternmed.2022.5600. JAMA Intern Med. 2023. PMID: 36469312 Free PMC article. - Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: cross sectional meta-research study.
Nguyen PY, Kanukula R, McKenzie JE, Alqaidoom Z, Brennan SE, Haddaway NR, Hamilton DG, Karunananthan S, McDonald S, Moher D, Nakagawa S, Nunan D, Tugwell P, Welch VA, Page MJ. Nguyen PY, et al. BMJ. 2022 Nov 22;379:e072428. doi: 10.1136/bmj-2022-072428. BMJ. 2022. PMID: 36414269 Free PMC article. - How often do cancer researchers make their data and code available and what factors are associated with sharing?
Hamilton DG, Page MJ, Finch S, Everitt S, Fidler F. Hamilton DG, et al. BMC Med. 2022 Nov 9;20(1):438. doi: 10.1186/s12916-022-02644-2. BMC Med. 2022. PMID: 36352426 Free PMC article.
References
LinkOut - more resources
Full Text Sources