Utilizing a digital swarm intelligence platform to improve consensus among radiologists and exploring its applications (original) (raw)
Related papers
Journal of Intelligence
Medical errors have a huge impact on clinical practice in terms of economic and human costs. As a result, technology-based solutions, such as those grounded in artificial intelligence (AI) or collective intelligence (CI), have attracted increasing interest as a means of reducing error rates and their impacts. Previous studies have shown that a combination of individual opinions based on rules, weighting mechanisms, or other CI solutions could improve diagnostic accuracy with respect to individual doctors. We conducted a study to investigate the potential of this approach in cardiology and, more precisely, in electrocardiogram (ECG) reading. To achieve this aim, we designed and conducted an experiment involving medical students, recent graduates, and residents, who were asked to annotate a collection of 10 ECGs of various complexity and difficulty. For each ECG, we considered groups of increasing size (from three to 30 members) and applied three different CI protocols. In all cases, ...
Radiologists in the loop: the roles of radiologists in the development of AI applications
European Radiology
Objectives To examine the various roles of radiologists in different steps of developing artificial intelligence (AI) applications. Materials and methods Through the case study of eight companies active in developing AI applications for radiology, in different regions (Europe, Asia, and North America), we conducted 17 semi-structured interviews and collected data from documents. Based on systematic thematic analysis, we identified various roles of radiologists. We describe how each role happens across the companies and what factors impact how and when these roles emerge. Results We identified 9 roles that radiologists play in different steps of developing AI applications: (1) problem finder (in 4 companies); (2) problem shaper (in 3 companies); (3) problem dominator (in 1 company); (4) data researcher (in 2 companies); (5) data labeler (in 3 companies); (6) data quality controller (in 2 companies); (7) algorithm shaper (in 3 companies); (8) algorithm tester (in 6 companies); and (9)...
Boosting medical diagnostics by pooling independent judgments
Collective intelligence refers to the ability of groups to outperform individual decision makers when solving complex cognitive problems. Despite its potential to revolutionize decision making in a wide range of domains, including medical, economic, and political decision making, at present, little is known about the conditions underlying collective intelligence in real-world contexts. We here focus on two key areas of medical diagnostics, breast and skin cancer detection. Using a simulation study that draws on large real-world datasets, involving more than 140 doctors making more than 20,000 diagnoses, we investigate when combining the independent judgments of multiple doctors outperforms the best doctor in a group. We find that similarity in diagnostic accuracy is a key condition for collective intelligence: Aggregating the independent judgments of doctors outperforms the best doctor in a group whenever the diagnostic accuracy of doctors is relatively similar, but not when doctors' diagnostic accuracy differs too much. This intriguingly simple result is highly robust and holds across different group sizes, performance levels of the best doctor, and collective intelligence rules. The enabling role of similarity, in turn, is explained by its systematic effects on the number of correct and incorrect decisions of the best doctor that are overruled by the collective. By identifying a key factor underlying collective intelligence in two important real-world contexts, our findings pave the way for innovative and more effective approaches to complex real-world decision making, and to the scientific analyses of those approaches. collective intelligence | groups | medical diagnostics | dermatology | mammography C ollective intelligence, that is, the ability of groups to outperform individual decision makers when solving complex cognitive problems, is a powerful approach for boosting decision accuracy (1-7). However, despite its potential to boost accuracy in a wide range of contexts, including lie detection, political forecasting, investment decisions, and medical decision making , little is known about the conditions that underlie the emergence of collective intelligence in real-world domains. Which features of decision makers and decision contexts favor the emergence of collective intelligence? Which decision-making rules permit this potential to be harnessed? We here provide answers to these important questions in the domain of medical diagnostics.
Experimental evidence of effective human–AI collaboration in medical decision-making
Scientific Reports
Artificial Intelligence (ai) systems are precious support for decision-making, with many applications also in the medical domain. The interaction between mds and ai enjoys a renewed interest following the increased possibilities of deep learning devices. However, we still have limited evidence-based knowledge of the context, design, and psychological mechanisms that craft an optimal human–ai collaboration. In this multicentric study, 21 endoscopists reviewed 504 videos of lesions prospectively acquired from real colonoscopies. They were asked to provide an optical diagnosis with and without the assistance of an ai support system. Endoscopists were influenced by ai ($$\textsc {or}=3.05$$ O R = 3.05 ), but not erratically: they followed the ai advice more when it was correct ($$\textsc {or}=3.48$$ O R = 3.48 ) than incorrect ($$\textsc {or}=1.85$$ O R = 1.85 ). Endoscopists achieved this outcome through a weighted integration of their and the ai opinions, considering the case-by-case ...
Current State of Community-Driven Radiological AI Deployment in Medical Imaging
arXiv (Cornell University), 2022
Background: Artificial Intelligence (AI) has become commonplace to solve routine everyday tasks. Because of the exponential growth in medical imaging data volume and complexity, the workload on radiologists is steadily increasing. We project that the gap between the number of imaging exams and the number of expert radiologist readers required to cover this increase will continue to expand, consequently, introducing a demand for tools that improve the efficiency with which radiologists can comfortably interpret these exams. Methods: AI has been shown to improve efficiency in medical image generation, processing, and interpretation, and a variety of such AI models have been developed across research labs worldwide. However, very few of these, if any, find their way into routine clinical use, a discrepancy that reflects the divide between AI research and successful AI translation. To address the barrier to clinical deployment, we have formed the Medical Open Network for Artificial Intelligence (MONAI) Consortium, an open source community which is building standards for AI deployment in hospitals, and developing tools and infrastructure to facilitate their implementation. This paper represents several years of weekly discussions and hands-on problem solving experience by groups of industry experts and clinicians. Findings: We identify barriers between AI model development in research labs and subsequent clinical deployment and propose solutions. Our report provides guidance on processes which take an imaging AI model from development to clinical implementation in a healthcare institution. We discuss various AI integration points in a clinical radiology workflow. We also present a taxonomy of radiology AI use cases. Interpretation: Through this paper, we intend to educate the stakeholders in healthcare and AI (AI researchers, radiologists, imaging informaticists, and regulators) about cross-disciplinary challenges and possible solutions. Funding: MONAI is an open-source project. Funding for authors was made available through the respective affiliated institutional salaries.
Using a symbiotic man/machine approach to evaluating visual clinical research data
Journal of Medical Systems, 1988
Some candidate medical expert system applications have a significant visual component. Knowledge engineers usually dismiss such task domains as potential expert systems applications. Our success in developing ESCA, a system for evaluating serial coronary angiograms, shows that such task domains should not be dismissed so quickly. We used a symbiotic approach between man and machine, where technologists provide the visual skills with an expert system imitating the conceptual skills of the expert, to produce a partially automated system that is more consistent and cost effective than one that is fully manual. The agreement between the system's conclusions and that of a panel of experts is good. The expert system actually has a slightly higher agreement rate with the expert panel than the agreement rate between two expert panel teams evaluating the same film pair.
Applied Swarm-based medicine: collecting decision trees for patterns of algorithms analysis
BMC Medical Research Methodology, 2017
Background: The objective consensus methodology has recently been applied in consensus finding in several studies on medical decision-making among clinical experts or guidelines. The main advantages of this method are an automated analysis and comparison of treatment algorithms of the participating centers which can be performed anonymously. Methods: Based on the experience from completed consensus analyses, the main steps for the successful implementation of the objective consensus methodology were identified and discussed among the main investigators. Results: The following steps for the successful collection and conversion of decision trees were identified and defined in detail: problem definition, population selection, draft input collection, tree conversion, criteria adaptation, problem re-evaluation, results distribution and refinement, tree finalisation, and analysis. Conclusion: This manuscript provides information on the main steps for successful collection of decision trees and summarizes important aspects at each point of the analysis.