AI Models Go Head-to-Head in Project AIR Study

One of the biggest challenges in assessing the performance of different AI algorithms is the varying conditions under which AI research studies are conducted. A new study from the Netherlands published this week in Radiology aims to correct that by testing a variety of AI algorithms head-to-head under similar conditions. 

There are over 200 AI algorithms on the European market (and even more in the US), many of which address the same clinical condition. 

  • Therefore, hospitals looking to acquire AI can find it difficult to assess the diagnostic performance of different models. 

The Project AIR initiative was launched to fill the gap in accurate assessment of AI algorithms by creating a Consumer Reports-style testing environment that’s consistent and transparent.

  • Project AIR researchers have assembled a validated database of medical images for different clinical applications, against which multiple AI algorithms can be tested; to ensure generalizability, images have come from different institutions and were acquired on equipment from different vendors. 

In the first test of the Project AIR concept, a team led by Kicky van Leeuwen of Radboud University Medical Centre in the Netherlands invited AI developers to participate, with nine products from eight vendors validated from June 2022 to January 2023: two models for bone age prediction and seven algorithms for lung nodule assessment (one vendor participated in both tests). Results included:

  • For bone age analysis, both of the tested algorithms (Visiana and Vuno) showed “excellent correlation” with the reference standard, with an r correlation coefficient of 0.987-0.989 (1 = perfect agreement)
  • For lung nodule analysis, there was a wider spread in AUC between the algorithms and human readers, with humans posting a mean AUC of 0.81
  • Researchers found superior performance for Annalise.ai (0.90), Lunit (0.93), Milvue (0.86), and Oxipit (0.88)

What’s next on Project AIR’s testing agenda? Van Leeuwen told The Imaging Wire that the next study will involve fracture detection. Meanwhile, interested parties can follow along on leaderboards for both bone age and lung nodule use cases. 

The Takeaway

Head-to-head studies like the one conducted by Project AIR may make many AI developers squirm (several that were invited declined to participate), but they are a necessary step toward building clinician confidence in the performance of AI algorithms that needs to take place to support the widespread adoption of AI. 

Top 12 Radiology Trends for 2024

What will be the top radiology trends for 2024? We talked to key opinion leaders across the medical imaging spectrum to get their opinions on the technologies, clinical applications, and regulatory developments that will shape the specialty for the next 12 months.

AI – Generative AI to Reduce Radiology’s Workload: “New generative AI methods will summarize complex medical records, draft radiology reports from images, and explain radiology reports to patients using language they understand. These innovative systems will reduce our workload and will provide more time for us to connect with our colleagues and our patients.” — Curtis Langlotz, MD, PhD, Stanford University and president, RSNA 2024

AI – Generative AI Will Get Multimodal: “In 2024, we can expect continued innovations in generative AI with a greater emphasis on integrating GenAI into existing and new radiology and patient-facing applications with growing interests in retrieval-augmented generation, fine-tuning, smaller models, multi-model routing, and AI assistants. Medicine being multimodal, the term ‘multimodal’ will become more ubiquitous.” — Woojin Kim, MD, CMIO at Rad AI

AI – Will AI Really Reduce Radiology Burnout? “Burnout will continue to be a huge issue in radiology with no solution in sight. AI vendors will offer algorithms as solutions to burnout with catchy slogans such as ‘buy our lung nodule detector and become the radiologist your parents wanted you to be.’ Their enthusiasm will cause even more burnout.” — Saurabh Jha, MBBS, AKA RogueRad, Hospital of the University of Pennsylvania

Breast Imaging – Prepare Now for Density Reporting: “The FDA ‘dense breast’ reporting standard to patients becomes effective on September 10, 2024, and breast imaging centers should be prepared for new patient questions and conversations. A plan for a consistent approach to recommending supplemental screening and facilitating ordering of additional imaging from referring providers should be put into action.” — JoAnn Pushkin, executive director, DenseBreast-info.org

Breast Imaging – Density Reporting to Spur Earlier Detection: “In March 2023, FDA issued a national requirement for reporting breast density to patients and referring providers after mammography. Facilities performing mammograms must meet the September 2024 deadline incorporating breast density type and associated breast cancer risk in their reporting. This change can lead to earlier breast cancer detection as these patients will be informed of supplemental screening as it relates to their breast density and [will] choose to pursue it.” — Stamatia Destounis, MD, Elizabeth Wende Breast Care and chair, ACR Breast Imaging Commission

CT – Lung Cancer Screening to Build Momentum: “Uptake of LDCT screening for lung cancer will increase in the US and worldwide. AI-enabled cardiac evaluation, even on non-gated scans, will allow for prediction of illnesses such as AFib and heart failure.  Quantifying measurement error across platforms will become an important aspect of nodule management.” — David Yankelevitz, MD, Icahn School of Medicine at Mount Sinai Health System

CT – Photon-Counting CT to Expand: “In 2024, we will continue to see many papers published on photon-counting CT, strengthening the body of scientific evidence as to its many strengths. Results from clinical trials involving multiple manufacturers’ systems will also increase in number, perhaps leading to more commercial systems entering the market.” — Cynthia McCollough, PhD, director, CT Clinical Innovation Center, Mayo Clinic

Enterprise Imaging – Time is Ripe for Cloud and AI: “Healthcare has an opportunity for change in 2024, and imaging is ripe for disruption, with burnout, staffing challenges, and new technology needs. Many organizations are expanding their enterprise imaging strategy and are asking how and where they can take the plunge into cloud and AI. Vendors have got the message; now it’s time to push the gas and deliver.” — Monique Rasband, VP of strategy & research, imaging/oncology at KLAS

Imaging IT – Data Brokerage to Go Mainstream: “A new market will hit the mainstream in 2024 – radiology data brokerage. As data-hungry LLMs scale up and the use of companion diagnostics in lifesciences proliferates, health systems will look to cash in on curated radiology data. This will also be an even bigger driver for migration to cloud-based imaging IT.” — Steve Holloway, managing director, Signify Research     

MRI – Prostate MRI to Reduce Biopsies: “Prostate MRI in conjunction with PSMA PET will explode in 2024 and reduce the number of unnecessary biopsies for patients.” — Stephen Pomeranz, MD, CEO of ProScan Imaging and chair, Naples Florida Community Hospital Network 

Theranostics – New Radiotracers to Drive Diagnosis & Treatment: “Through 2024, nuclear medicine theranostics will increasingly be integrated into standard global practice. With many new radiopharmaceuticals in development, theranostics promise early diagnosis and precision treatment for a broadening range of cancers, expanding options for patients resistant to traditional therapies. Treatments will be enhanced by personalized dosimetry, artificial intelligence, and combination therapies.” — Helen Nadel, MD, Stanford University and president, SNMMI 2023-2024

Radiology Operations – Reimbursement Challenges Continue: “In 2024, we will continue to experience recruitment challenges coupled with decreases in reimbursement. Now, more than ever, every radiologist needs to be diligent in advocating for the specialty, focus on business plan diversification, and ensure all services rendered are optimally documented and billed.” — Rebecca Farrington, chief revenue officer, Healthcare Administrative Partners 

The Takeaway
To paraphrase Robert F. Kennedy, radiology is indeed living in interesting times – times of “danger and uncertainty,” but also times of unprecedented creativity and innovation. In 2024, radiology will get a much better glimpse of where these trends are taking us.

Get every issue of The Imaging Wire, delivered right to your inbox.

You might also like..

Select All

You're signed up!

It's great to have you as a reader. Check your inbox for a welcome email.

-- The Imaging Wire team

You're all set!