The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies - PubMed (original) (raw)
The Evidence Project risk of bias tool: assessing study rigor for both randomized and non-randomized intervention studies
Caitlin E Kennedy et al. Syst Rev. 2019.
Abstract
Background: Different tools exist for assessing risk of bias of intervention studies for systematic reviews. We present a tool for assessing risk of bias across both randomized and non-randomized study designs. The tool was developed by the Evidence Project, which conducts systematic reviews and meta-analyses of behavioral interventions for HIV in low- and middle-income countries.
Methods: We present the eight items of the tool and describe considerations for each and for the tool as a whole. We then evaluate reliability of the tool by presenting inter-rater reliability for 125 selected studies from seven published reviews, calculating a kappa for each individual item and a weighted kappa for the total count of items.
Results: The tool includes eight items, each of which is rated as being present (yes) or not present (no) and, for some items, not applicable or not reported. The items include (1) cohort, (2) control or comparison group, (3) pre-post intervention data, (4) random assignment of participants to the intervention, (5) random selection of participants for assessment, (6) follow-up rate of 80% or more, (7) comparison groups equivalent on sociodemographics, and (8) comparison groups equivalent at baseline on outcome measures. Together, items (1)-(3) summarize the study design, while the remaining items consider other common elements of study rigor. Inter-rater reliability was moderate to substantial for all items, ranging from 0.41 to 0.80 (median κ = 0.66). Agreement between raters on the total count of items endorsed was also substantial (κw = 0.66).
Conclusions: Strengths of the tool include its applicability to a range of study designs, from randomized trials to various types of observational and quasi-experimental studies. It is relatively easy to use and interpret and can be applied to a range of review topics without adaptation, facilitating comparability across reviews. Limitations include the lack of potentially relevant items measured in other tools and potential threats to validity of some items. To date, the tool has been applied in over 30 reviews. We believe it is a practical option for assessing risk of bias in systematic reviews of interventions that include a range of study designs.
Keywords: Critical appraisal; Quality assessment; Rigor assessment; Rigor score; Risk of bias; Study quality; Study rigor.
Conflict of interest statement
Authors’ information
CEK is an Associate Professor and Director of the Social and Behavioral Interventions Program, Department of International Health, Johns Hopkins Bloomberg School of Public Health. She currently serves as Co-Investigator for the Evidence Project.
VAF is an Assistant Professor in the Department of Psychiatry and Behavioral Sciences at the Medical University of South Carolina. She currently serves as Co-Investigator for the Evidence Project.
KSA is a Statistician in the Department of Psychiatry and Behavioral Sciences at the Medical University of South Carolina. He currently serves as statistician for the Evidence Project.
JAD is an Associate Professor in the Social and Behavioral Interventions Program, Department of International Health, Johns Hopkins Bloomberg School of Public Health. She helped develop the risk of bias tool as the original study coordinator for the Evidence Project.
PTY is a Research Associate in the Social and Behavioral Interventions Program, Department of International Health, Johns Hopkins Bloomberg School of Public Health. She currently serves as study coordinator for the Evidence Project.
KRO is a Clinical Associate Professor in the Department of Psychiatry and Behavioral Sciences at the Medical University of South Carolina. He jointly founded the Evidence Project and currently serves as Co-Investigator.
MDS is a Professor in the Department of Psychiatry and Behavioral Sciences at the Medical University of South Carolina. He jointly founded the Evidence Project and currently serves as Principal Investigator.
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Similar articles
- Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas.
Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, Moraleda C, Rogers L, Daniels K, Green P. Crider K, et al. Cochrane Database Syst Rev. 2022 Feb 1;2(2022):CD014217. doi: 10.1002/14651858.CD014217. Cochrane Database Syst Rev. 2022. PMID: 36321557 Free PMC article. - Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials: a meta-epidemiological study.
Toews I, Anglemyer A, Nyirenda JL, Alsaid D, Balduzzi S, Grummich K, Schwingshackl L, Bero L. Toews I, et al. Cochrane Database Syst Rev. 2024 Jan 4;1(1):MR000034. doi: 10.1002/14651858.MR000034.pub3. Cochrane Database Syst Rev. 2024. PMID: 38174786 - Validity and Inter-Rater Reliability Testing of Quality Assessment Instruments [Internet].
Hartling L, Hamm M, Milne A, Vandermeer B, Santaguida PL, Ansari M, Tsertsvadze A, Hempel S, Shekelle P, Dryden DM. Hartling L, et al. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012 Mar. Report No.: 12-EHC039-EF. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012 Mar. Report No.: 12-EHC039-EF. PMID: 22536612 Free Books & Documents. Review. - Healthcare outcomes assessed with observational study designs compared with those assessed in randomized trials.
Anglemyer A, Horvath HT, Bero L. Anglemyer A, et al. Cochrane Database Syst Rev. 2014 Apr 29;2014(4):MR000034. doi: 10.1002/14651858.MR000034.pub2. Cochrane Database Syst Rev. 2014. PMID: 24782322 Free PMC article. Updated. Review. - Behavioral interventions to reduce risk for sexual transmission of HIV among men who have sex with men.
Johnson WD, Diaz RM, Flanders WD, Goodman M, Hill AN, Holtgrave D, Malow R, McClellan WM. Johnson WD, et al. Cochrane Database Syst Rev. 2008 Jul 16;(3):CD001230. doi: 10.1002/14651858.CD001230.pub2. Cochrane Database Syst Rev. 2008. PMID: 18646068 Review.
Cited by
- Meta-analysis of Pregnancy Events in Biomedical HIV Prevention Trials in Sub-Saharan Africa: Implications for Gender Transformative Trials.
Lorenzetti L, Dinh N, Whitcomb C, Martinez A, Chatani M, Lievense B, Nhamo D, Slack C, Eley N, MacQueen K. Lorenzetti L, et al. AIDS Behav. 2024 Aug 17. doi: 10.1007/s10461-024-04459-z. Online ahead of print. AIDS Behav. 2024. PMID: 39153025 Review. - Family Planning Counseling for Women Living with HIV in Low- and Middle-Income Countries: A Systematic Review of the Impact on Contraceptive Uptake, Intention to Use Contraception and Pregnancy Incidence, 2011 to 2022.
O'Reilly KR, Yeh PT, Kennedy CE, Fonner VA, Sweat MD. O'Reilly KR, et al. AIDS Behav. 2024 Aug;28(8):2477-2491. doi: 10.1007/s10461-024-04319-w. Epub 2024 Apr 25. AIDS Behav. 2024. PMID: 38662281 Free PMC article. Review. - Effects of exercise on life satisfaction of people diagnosed with cancer: a systematic review and meta-analysis.
Fernández-Sánchez J, Trujillo-Colmena D, Rodríguez-Castaño A, Lavín-Pérez AM, Del Coso J, Casado A, Collado-Mateo D. Fernández-Sánchez J, et al. Support Care Cancer. 2024 Apr 19;32(5):297. doi: 10.1007/s00520-024-08486-3. Support Care Cancer. 2024. PMID: 38637349 Free PMC article. - Treatments and interventions addressing chronic somatic pain in torture survivors: A systematic review.
Oren T, Ercanli N, Maayan O, Tham S, Wright D, Kaur G. Oren T, et al. PLOS Glob Public Health. 2024 Mar 28;4(3):e0003070. doi: 10.1371/journal.pgph.0003070. eCollection 2024. PLOS Glob Public Health. 2024. PMID: 38547161 Free PMC article. - The methodological quality of systematic reviews regarding the Core Outcome Set (COS) development.
Cao H, Chen Y, Yang Z, Lan J, Sum-Wing Kwong J, Zhang R, Zhao H, Hu L, Wang J, Sun S, Tan S, Cao J, He R, Zheng W, Zhang J. Cao H, et al. BMC Med Res Methodol. 2024 Mar 11;24(1):65. doi: 10.1186/s12874-024-02182-w. BMC Med Res Methodol. 2024. PMID: 38468223 Free PMC article.
References
- Deeks JJ, Dinnes J, D'Amico R, et al. Evaluating non-randomised intervention studies. Health Technol Assessment (Winchester) 2003;7(27):iii–iix. - PubMed
Publication types
MeSH terms
LinkOut - more resources
Full Text Sources