One of the most lauded promises of medical artificial intelligence (AI) is its ability to assist human clinicians in more accurately interpreting images from X-rays and CT scans, thereby producing more precise diagnostic reports and enhancing the performance of radiologists.
But Is This the Reality?
Research collaborations between Harvard Medical School, Massachusetts Institute of Technology, and Stanford University suggest that the effectiveness of AI tools in image interpretation may vary among clinicians.
In other words, whether it’s beneficial or not still largely depends on human judgement. The research, recently published in Nature Medicine, highlights that individual differences among clinicians can affect the interaction between humans and machines in ways not fully understood by AI experts yet.
Considering Individual Factors of Doctors
The study indicates that AI use can sometimes interfere with a radiologist’s performance and impact their interpretation accuracy.
While previous studies showed AI assistants could improve diagnostic performance, they treated doctors as a homogeneous group, not accounting for individual differences. However, in clinical practice, each doctor’s judgment is 100% for their patients.
This new study focuses on individual factors of clinicians—such as specialty, years of practice, and prior experience with AI tools—to analyze how these factors play a role in human-machine collaboration.
Researchers analyzed how AI affected the performance of 140 radiologists across 15 X-ray diagnostic tasks, involving 324 cases of patients with 15 different conditions, to determine AI’s impact on doctors’ ability to identify and correctly diagnose issues using advanced computational methods.
The results showed inconsistent effects of AI assistance among radiologists, with some experiencing improved performance and others experiencing deterioration.
Bhavnit K. Lakhtakia, Assistant Professor of Biomedical Informatics at the Royal College of Medicine’s Bravanik Institute, confirmed this finding, suggesting that doctors should not be viewed as a uniform group, focusing only on the ‘average’ impact of AI on their performance.
However, this does not mean that doctors and clinics should cease adopting AI. Instead, it highlights the need for a better understanding of human-AI interaction and designing methods that enhance rather than impair human performance.
Unpredictable AI “Assistants”
Given that radiology is considered one of the clinical fields where AI can provide significant assistance, the findings are particularly significant.
Interestingly, factors such as years of experience, specialization in chest radiology, or prior use of AI devices did not reliably predict the impact of AI tools on work performance.
Another counterintuitive finding was that clinicians with baseline lower performance did not consistently benefit from AI assistance. Overall, radiologists with lower baseline performance remained lower, regardless of AI assistance, and the same was true for those with higher baseline performance.
More accurate AI improved radiologists’ performance, while average AI reduced diagnostic accuracy, emphasizing the importance of testing and verifying AI tools before clinical deployment to ensure that subpar AI does not interfere with clinicians’ judgment and delay patient care.
Impacting the Future of Clinical Medicine
Since clinicians have varying levels of expertise, experience, and decision-making styles, ensuring AI reflects this diversity is crucial for targeted treatment implementation. Individual factors and variations should be key to ensuring AI progress, not an interference that ultimately affects diagnostics.
Interestingly, the findings do not explain why AI impacts clinicians’ performance differently, but understanding this becomes increasingly important as AI’s influence on clinical medicine grows, a challenge that AI experts are still addressing.
The research team suggests that future interactions between radiologists and AI should be tested in simulated real-world scenarios, reflecting actual patient populations. Besides improving the accuracy of AI tools, training radiologists to detect inaccurate AI, review, and question AI diagnoses is also essential.
In other words, before AI can help you, you need to improve yourself.