An AI algorithm that examined teleradiology studies for signs of intracranial hemorrhage had mixed performance in a new study in Radiology: Artificial Intelligence. AI helped detect ICH cases that might have been missed, but false positives slowed radiologists down.
AI is being touted as a tool that can detect unseen pathology and speed up the workflow of radiologists facing an environment of limited resources and growing image volume.
- This dynamic is particularly evident at teleradiology practices, which frequently see high volumes during off-hour shifts; indeed, a recent study found that telerad cases had higher rates of patient death and more malpractice claims than cases read by traditional radiology practices.
So teleradiologists could use a bit more help. In the new study, researchers from the VA’s National Teleradiology Program assessed Avicenna.ai’s CINA v1.0 algorithm for detecting ICH on STAT non-contrast head CT studies.
- AI was used to analyze 58.3k CT exams processed by the teleradiology service from January 2023 to February 2024, with a 2.7% prevalence of ICH.
Results were as follows…
- AI flagged 5.7k studies as positive for acute ICH and 52.7k as negative
- Final radiology reports confirmed that 1.2k exams were true positives for a sensitivity of 76% and a positive predictive value of 21%
- There were 384 false negatives (missed ICH cases), for a specificity of 92% and a negative predictive value of 99.3%
- The algorithm’s performance at the VA was a bit lower than in previously published literature
- Cases that the algorithm falsely flagged as positive took over a minute longer to interpret than prior to AI deployment
- Overall, case interpretation times were slightly lower after AI than before
One issue to note is that the CINA algorithm is not intended for small hemorrhages with volumes < 3 mL; the researchers did not exclude these cases from their analysis, which could have reduced its performance.
- Also, at 2.7% the VA’s teleradiology program ICH prevalence was lower than the 10% prevalence Avicenna has used to rate its performance.
The Takeaway
The new findings aren’t exactly a slam dunk for AI in the teleradiology setting, but in terms of real-world results they are exactly what’s needed to assess the true value of the technology compared to outcomes in more tightly controlled environments.
