Empirical evidence of design-related bias in studies of diagnostic tests - PubMed (original) (raw)
Empirical evidence of design-related bias in studies of diagnostic tests
J G Lijmer et al. JAMA. 1999.
Erratum in
- JAMA 2000 Apr 19;283(15):1963
Abstract
Context: The literature contains a large number of potential biases in the evaluation of diagnostic tests. Strict application of appropriate methodological criteria would invalidate the clinical application of most study results.
Objective: To empirically determine the quantitative effect of study design shortcomings on estimates of diagnostic accuracy.
Design and setting: Observational study of the methodological features of 184 original studies evaluating 218 diagnostic tests. Meta-analyses on diagnostic tests were identified through a systematic search of the literature using MEDLINE, EMBASE, and DARE databases and the Cochrane Library (1996-1997). Associations between study characteristics and estimates of diagnostic accuracy were evaluated with a regression model.
Main outcome measures: Relative diagnostic odds ratio (RDOR), which compared the diagnostic odds ratios of studies of a given test that lacked a particular methodological feature with those without the corresponding shortcomings in design.
Results: Fifteen (6.8%) of 218 evaluations met all 8 criteria; 64 (30%) met 6 or more. Studies evaluating tests in a diseased population and a separate control group overestimated the diagnostic performance compared with studies that used a clinical population (RDOR, 3.0; 95% confidence interval [CI], 2.0-4.5). Studies in which different reference tests were used for positive and negative results of the test under study overestimated the diagnostic performance compared with studies using a single reference test for all patients (RDOR, 2.2; 95% CI, 1.5-3.3). Diagnostic performance was also overestimated when the reference test was interpreted with knowledge of the test result (RDOR, 1.3; 95% CI, 1.0-1.9), when no criteria for the test were described (RDOR, 1.7; 95% CI, 1.1-2.5), and when no description of the population under study was provided (RDOR, 1.4; 95% CI, 1.1-1.7).
Conclusion: These data provide empirical evidence that diagnostic studies with methodological shortcomings may overestimate the accuracy of a diagnostic test, particularly those including nonrepresentative patients or applying different reference standards.
Comment in
- Measuring the quality of trials: the quality of quality scales.
Berlin JA, Rennie D. Berlin JA, et al. JAMA. 1999 Sep 15;282(11):1083-5. doi: 10.1001/jama.282.11.1083. JAMA. 1999. PMID: 10493209 No abstract available. - Using meta-analysis to answer clinical questions.
Laheij RJ. Laheij RJ. JAMA. 2001 Dec 5;286(21):2669; author reply 2669-70. JAMA. 2001. PMID: 11730436 No abstract available. - Improving reports of studies of diagnostic tests: the STARD initiative.
Rennie D. Rennie D. JAMA. 2003 Jan 1;289(1):89-90. doi: 10.1001/jama.289.1.89. JAMA. 2003. PMID: 12503983 No abstract available.
Similar articles
- Evidence of bias and variation in diagnostic accuracy studies.
Rutjes AW, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PM. Rutjes AW, et al. CMAJ. 2006 Feb 14;174(4):469-76. doi: 10.1503/cmaj.050090. CMAJ. 2006. PMID: 16477057 Free PMC article. - [Methodological bias and variation of systematic reviews on diagnostic test accuracy].
Li ZX, Yang ZR, Xiang X, Gao P, Shu Z, Huang YS, Cao Y, Sun F, Zhan SY. Li ZX, et al. Zhonghua Liu Xing Bing Xue Za Zhi. 2016 Feb;37(2):286-90. doi: 10.3760/cma.j.issn.0254-6450.2016.02.027. Zhonghua Liu Xing Bing Xue Za Zhi. 2016. PMID: 26917532 Chinese. - Evidence of reference standard related bias in studies of plain radiograph reading performance: a meta-regression.
Brealey SD, Scally AJ, Hahn S, Godfrey C. Brealey SD, et al. Br J Radiol. 2007 Jun;80(954):406-13. doi: 10.1259/bjr/41006673. Epub 2006 Dec 6. Br J Radiol. 2007. PMID: 17151064 Review. - Diagnostic test accuracy of nutritional tools used to identify undernutrition in patients with colorectal cancer: a systematic review.
Håkonsen SJ, Pedersen PU, Bath-Hextall F, Kirkpatrick P. Håkonsen SJ, et al. JBI Database System Rev Implement Rep. 2015 May 15;13(4):141-87. doi: 10.11124/jbisrir-2015-1673. JBI Database System Rev Implement Rep. 2015. PMID: 26447079 Review. - Empirical assessment of bias in machine learning diagnostic test accuracy studies.
Crowley RJ, Tan YJ, Ioannidis JPA. Crowley RJ, et al. J Am Med Inform Assoc. 2020 Jul 1;27(7):1092-1101. doi: 10.1093/jamia/ocaa075. J Am Med Inform Assoc. 2020. PMID: 32548642 Free PMC article.
Cited by
- Biomarkers of a five-domain translational substrate for schizophrenia and schizoaffective psychosis.
Fryar-Williams S, Strobel JE. Fryar-Williams S, et al. Biomark Res. 2015 Feb 6;3:3. doi: 10.1186/s40364-015-0028-1. eCollection 2015. Biomark Res. 2015. PMID: 25729574 Free PMC article. - The Diagnostic Accuracy of Mutant KRAS Detection from Pancreatic Secretions for the Diagnosis of Pancreatic Cancer: A Meta-Analysis.
Patel N, Petrinic T, Silva M, Soonawalla Z, Reddy S, Gordon-Weeks A. Patel N, et al. Cancers (Basel). 2020 Aug 20;12(9):2353. doi: 10.3390/cancers12092353. Cancers (Basel). 2020. PMID: 32825312 Free PMC article. - Large-Scale SARS-CoV-2 Antigen Testing With Real-World Specimens.
Parikh A, Cooper L, Frogel D, Le Benger K, Cooper CK, Parvu V. Parikh A, et al. Front Public Health. 2022 Apr 5;10:836328. doi: 10.3389/fpubh.2022.836328. eCollection 2022. Front Public Health. 2022. PMID: 35450121 Free PMC article. - A meta-analysis comparing the prognostic accuracy of six diagnostic tests for predicting perioperative cardiac risk in patients undergoing major vascular surgery.
Kertai MD, Boersma E, Bax JJ, Heijenbrok-Kal MH, Hunink MG, L'talien GJ, Roelandt JR, van Urk H, Poldermans D. Kertai MD, et al. Heart. 2003 Nov;89(11):1327-34. doi: 10.1136/heart.89.11.1327. Heart. 2003. PMID: 14594892 Free PMC article.
MeSH terms
LinkOut - more resources
Full Text Sources
Other Literature Sources
Medical
Miscellaneous