Toward systematic review automation: a practical guide to using machine learning tools in research synthesis - PubMed (original) (raw)
Editorial
Toward systematic review automation: a practical guide to using machine learning tools in research synthesis
Iain J Marshall et al. Syst Rev. 2019.
Abstract
Technologies and methods to speed up the production of systematic reviews by reducing the manual labour involved have recently emerged. Automation has been proposed or used to expedite most steps of the systematic review process, including search, screening, and data extraction. However, how these technologies work in practice and when (and when not) to use them is often not clear to practitioners. In this practical guide, we provide an overview of current machine learning methods that have been proposed to expedite evidence synthesis. We also offer guidance on which of these are ready for use, their strengths and weaknesses, and how a systematic review team might go about using them in practice.
Keywords: Evidence synthesis; Machine learning; Natural language processing.
Conflict of interest statement
The authors declare that they have no competing interests.
Figures
Fig. 1
Classifying text using machine learning, in this example logistic regression with a ‘bag of words’ representation of the texts. The system is ‘trained’, learning a coefficient (or weight) for each unique word in a manually labelled set of documents (typically in the 1000s). In use, the learned coefficients are used to predict a probability for an unknown document
Fig. 2
Bag of words modelling for classifying RCTs. Top left: Example of bag of words for three articles. Each column represents a unique word in the corpus (a real example would likely contain columns for 10,000s of words). Top right: Document labels, where 1 = relevant and 0 = irrelevant. Bottom: Coefficients (or weights) are estimated for each word (in this example using logistic regression). In this example, high +ve weights will increase the predicted probability that an unseen article is an RCT where it contains the words ‘random’ or ‘randomized’. The presence of the word ‘systematic’ (with a large negative weight) would reduce the predicted probability that an unseen document is an RCT
Fig. 3
Schematic of a typical data extraction process. The above illustration concerns the example task of extracting the study sample size. In general, these tasks involve labelling individual words. The word (or ‘token’) at position t is represented by a vector. This representation may encode which word is at this position and likely also communicates additional features, e.g. whether the word is capitalized or if the word is (inferred to be) a noun. Models for these kinds of tasks attempt to assign labels all T words in a document and for some tasks will attempt to maximize the joint likelihood of these labels to capitalize on correlations between adjacent labels
Fig. 4
Typical workflow for semi-automated abstract screening. The asterisk indicates that with uncertainty sampling, the articles which are predicted with least certainty are presented first. This aims to improve the model accuracy more efficiently
Similar articles
- Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR).
Beller E, Clark J, Tsafnat G, Adams C, Diehl H, Lund H, Ouzzani M, Thayer K, Thomas J, Turner T, Xia J, Robinson K, Glasziou P; founding members of the ICASR group. Beller E, et al. Syst Rev. 2018 May 19;7(1):77. doi: 10.1186/s13643-018-0740-7. Syst Rev. 2018. PMID: 29778096 Free PMC article. - A question of trust: can we build an evidence base to gain trust in systematic review automation technologies?
O'Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B. O'Connor AM, et al. Syst Rev. 2019 Jun 18;8(1):143. doi: 10.1186/s13643-019-1062-0. Syst Rev. 2019. PMID: 31215463 Free PMC article. - Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed.
Tóth B, Berek L, Gulácsi L, Péntek M, Zrubka Z. Tóth B, et al. Syst Rev. 2024 Jul 8;13(1):174. doi: 10.1186/s13643-024-02592-3. Syst Rev. 2024. PMID: 38978132 Free PMC article. - Tools to support the automation of systematic reviews: a scoping review.
Khalil H, Ameen D, Zarnegar A. Khalil H, et al. J Clin Epidemiol. 2022 Apr;144:22-42. doi: 10.1016/j.jclinepi.2021.12.005. Epub 2021 Dec 8. J Clin Epidemiol. 2022. PMID: 34896236 Review. - A narrative review of recent tools and innovations toward automating living systematic reviews and evidence syntheses.
Schmidt L, Sinyor M, Webb RT, Marshall C, Knipe D, Eyles EC, John A, Gunnell D, Higgins JPT. Schmidt L, et al. Z Evid Fortbild Qual Gesundhwes. 2023 Sep;181:65-75. doi: 10.1016/j.zefq.2023.06.007. Epub 2023 Aug 16. Z Evid Fortbild Qual Gesundhwes. 2023. PMID: 37596160 Review.
Cited by
- An exploration of available methods and tools to improve the efficiency of systematic review production: a scoping review.
Affengruber L, van der Maten MM, Spiero I, Nussbaumer-Streit B, Mahmić-Kaknjo M, Ellen ME, Goossen K, Kantorova L, Hooft L, Riva N, Poulentzas G, Lalagkas PN, Silva AG, Sassano M, Sfetcu R, Marqués ME, Friessova T, Baladia E, Pezzullo AM, Martinez P, Gartlehner G, Spijker R. Affengruber L, et al. BMC Med Res Methodol. 2024 Sep 18;24(1):210. doi: 10.1186/s12874-024-02320-4. BMC Med Res Methodol. 2024. PMID: 39294580 Review. - (Semi)automated approaches to data extraction for systematic reviews and meta-analyses in social sciences: A living review.
Legate A, Nimon K, Noblin A. Legate A, et al. F1000Res. 2024 Jun 20;13:664. doi: 10.12688/f1000research.151493.1. eCollection 2024. F1000Res. 2024. PMID: 39220382 Free PMC article. Review. - A question-answering framework for automated abstract screening using large language models.
Akinseloyin O, Jiang X, Palade V. Akinseloyin O, et al. J Am Med Inform Assoc. 2024 Sep 1;31(9):1939-1952. doi: 10.1093/jamia/ocae166. J Am Med Inform Assoc. 2024. PMID: 39042516 Free PMC article. - How to optimize the systematic review process using AI tools.
Fabiano N, Gupta A, Bhambra N, Luu B, Wong S, Maaz M, Fiedorowicz JG, Smith AL, Solmi M. Fabiano N, et al. JCPP Adv. 2024 Apr 23;4(2):e12234. doi: 10.1002/jcv2.12234. eCollection 2024 Jun. JCPP Adv. 2024. PMID: 38827982 Free PMC article. Review. - Evaluating the OpenAI's GPT-3.5 Turbo's performance in extracting information from scientific articles on diabetic retinopathy.
Gue CCY, Rahim NDA, Rojas-Carabali W, Agrawal R, Rk P, Abisheganaden J, Yip WF. Gue CCY, et al. Syst Rev. 2024 May 16;13(1):135. doi: 10.1186/s13643-024-02523-2. Syst Rev. 2024. PMID: 38755704 Free PMC article.
References
- Johnston E. How quickly do systematic reviews go out of date? A survival analysis. J Emerg Med. 2008;34:231. doi: 10.1016/j.jemermed.2007.11.022. - DOI
Publication types
MeSH terms
LinkOut - more resources
Full Text Sources