Real-World AI Experiences

Clinical studies showing that AI helps radiologists interpret medical images are great, but how well does AI work in the real world – and what do radiologists think about it? These questions are addressed in a new study in Applied Ergonomics that takes a deep dive into the real-world implementation of a commercially available AI algorithm at a German hospital. 

A slew of clinical studies supporting AI were published in 2023, from the MASAI study on AI for breast screening to smaller studies on applications like opportunistic screening or predicting who should get lung cancer screening

  • But even an AI algorithm with the best clinical evidence behind it could fall flat if it’s difficult to use and doesn’t integrate well with existing radiology workflow.

To gain insight into this issue, the new study tracked University Hospital Bonn’s implementation of Quantib’s Prostate software for interpreting and documenting prostate MRI scans (Quantib was acquired by RadNet in January 2022). 

  • Researchers described the solution as providing partial automation of prostate MRI workflow, such as helping segment the prostate, generating heat maps of areas of interest, and automatically producing patient reports based on lesions it identifies. 

Prostate was installed at the hospital in the spring of 2022, with nine radiology residents and three attending physicians interviewed before and after implementation, finding…

  • All but one radiologist had a positive attitude toward AI before implementation and one was undecided 
  • After implementation, seven said their attitudes were unchanged, one was disappointed, and one saw their opinion shift positively
  • Use of the AI was inconsistent, with radiologists adopting different workflows and some using it all the time with others only using it occasionally
  • Major concerns cited included workflow delays due to AI use, additional steps required such as sending images to a server, and unstable performance

The findings prompted the researchers to conclude that AI is likely to be implemented and used in the real world differently than in clinical trials. Radiologists should be included in AI algorithm development to provide insights into workflow where the tools will be used.

The Takeaway

The new study is unique in that – rather than focusing on AI algorithm performance – it concentrated on the experiences of radiologists using the software and how they changed following implementation. Such studies can be illuminating as AI developers seek broader clinical use of their tools. 

AI Models Go Head-to-Head in Project AIR Study

One of the biggest challenges in assessing the performance of different AI algorithms is the varying conditions under which AI research studies are conducted. A new study from the Netherlands published this week in Radiology aims to correct that by testing a variety of AI algorithms head-to-head under similar conditions. 

There are over 200 AI algorithms on the European market (and even more in the US), many of which address the same clinical condition. 

  • Therefore, hospitals looking to acquire AI can find it difficult to assess the diagnostic performance of different models. 

The Project AIR initiative was launched to fill the gap in accurate assessment of AI algorithms by creating a Consumer Reports-style testing environment that’s consistent and transparent.

  • Project AIR researchers have assembled a validated database of medical images for different clinical applications, against which multiple AI algorithms can be tested; to ensure generalizability, images have come from different institutions and were acquired on equipment from different vendors. 

In the first test of the Project AIR concept, a team led by Kicky van Leeuwen of Radboud University Medical Centre in the Netherlands invited AI developers to participate, with nine products from eight vendors validated from June 2022 to January 2023: two models for bone age prediction and seven algorithms for lung nodule assessment (one vendor participated in both tests). Results included:

  • For bone age analysis, both of the tested algorithms (Visiana and Vuno) showed “excellent correlation” with the reference standard, with an r correlation coefficient of 0.987-0.989 (1 = perfect agreement)
  • For lung nodule analysis, there was a wider spread in AUC between the algorithms and human readers, with humans posting a mean AUC of 0.81
  • Researchers found superior performance for Annalise.ai (0.90), Lunit (0.93), Milvue (0.86), and Oxipit (0.88)

What’s next on Project AIR’s testing agenda? Van Leeuwen told The Imaging Wire that the next study will involve fracture detection. Meanwhile, interested parties can follow along on leaderboards for both bone age and lung nodule use cases. 

The Takeaway

Head-to-head studies like the one conducted by Project AIR may make many AI developers squirm (several that were invited declined to participate), but they are a necessary step toward building clinician confidence in the performance of AI algorithms that needs to take place to support the widespread adoption of AI. 

Top 12 Radiology Trends for 2024

What will be the top radiology trends for 2024? We talked to key opinion leaders across the medical imaging spectrum to get their opinions on the technologies, clinical applications, and regulatory developments that will shape the specialty for the next 12 months.

AI – Generative AI to Reduce Radiology’s Workload: “New generative AI methods will summarize complex medical records, draft radiology reports from images, and explain radiology reports to patients using language they understand. These innovative systems will reduce our workload and will provide more time for us to connect with our colleagues and our patients.” — Curtis Langlotz, MD, PhD, Stanford University and president, RSNA 2024

AI – Generative AI Will Get Multimodal: “In 2024, we can expect continued innovations in generative AI with a greater emphasis on integrating GenAI into existing and new radiology and patient-facing applications with growing interests in retrieval-augmented generation, fine-tuning, smaller models, multi-model routing, and AI assistants. Medicine being multimodal, the term ‘multimodal’ will become more ubiquitous.” — Woojin Kim, MD, CMIO at Rad AI

AI – Will AI Really Reduce Radiology Burnout? “Burnout will continue to be a huge issue in radiology with no solution in sight. AI vendors will offer algorithms as solutions to burnout with catchy slogans such as ‘buy our lung nodule detector and become the radiologist your parents wanted you to be.’ Their enthusiasm will cause even more burnout.” — Saurabh Jha, MBBS, AKA RogueRad, Hospital of the University of Pennsylvania

Breast Imaging – Prepare Now for Density Reporting: “The FDA ‘dense breast’ reporting standard to patients becomes effective on September 10, 2024, and breast imaging centers should be prepared for new patient questions and conversations. A plan for a consistent approach to recommending supplemental screening and facilitating ordering of additional imaging from referring providers should be put into action.” — JoAnn Pushkin, executive director, DenseBreast-info.org

Breast Imaging – Density Reporting to Spur Earlier Detection: “In March 2023, FDA issued a national requirement for reporting breast density to patients and referring providers after mammography. Facilities performing mammograms must meet the September 2024 deadline incorporating breast density type and associated breast cancer risk in their reporting. This change can lead to earlier breast cancer detection as these patients will be informed of supplemental screening as it relates to their breast density and [will] choose to pursue it.” — Stamatia Destounis, MD, Elizabeth Wende Breast Care and chair, ACR Breast Imaging Commission

CT – Lung Cancer Screening to Build Momentum: “Uptake of LDCT screening for lung cancer will increase in the US and worldwide. AI-enabled cardiac evaluation, even on non-gated scans, will allow for prediction of illnesses such as AFib and heart failure.  Quantifying measurement error across platforms will become an important aspect of nodule management.” — David Yankelevitz, MD, Icahn School of Medicine at Mount Sinai Health System

CT – Photon-Counting CT to Expand: “In 2024, we will continue to see many papers published on photon-counting CT, strengthening the body of scientific evidence as to its many strengths. Results from clinical trials involving multiple manufacturers’ systems will also increase in number, perhaps leading to more commercial systems entering the market.” — Cynthia McCollough, PhD, director, CT Clinical Innovation Center, Mayo Clinic

Enterprise Imaging – Time is Ripe for Cloud and AI: “Healthcare has an opportunity for change in 2024, and imaging is ripe for disruption, with burnout, staffing challenges, and new technology needs. Many organizations are expanding their enterprise imaging strategy and are asking how and where they can take the plunge into cloud and AI. Vendors have got the message; now it’s time to push the gas and deliver.” — Monique Rasband, VP of strategy & research, imaging/oncology at KLAS

Imaging IT – Data Brokerage to Go Mainstream: “A new market will hit the mainstream in 2024 – radiology data brokerage. As data-hungry LLMs scale up and the use of companion diagnostics in lifesciences proliferates, health systems will look to cash in on curated radiology data. This will also be an even bigger driver for migration to cloud-based imaging IT.” — Steve Holloway, managing director, Signify Research     

MRI – Prostate MRI to Reduce Biopsies: “Prostate MRI in conjunction with PSMA PET will explode in 2024 and reduce the number of unnecessary biopsies for patients.” — Stephen Pomeranz, MD, CEO of ProScan Imaging and chair, Naples Florida Community Hospital Network 

Theranostics – New Radiotracers to Drive Diagnosis & Treatment: “Through 2024, nuclear medicine theranostics will increasingly be integrated into standard global practice. With many new radiopharmaceuticals in development, theranostics promise early diagnosis and precision treatment for a broadening range of cancers, expanding options for patients resistant to traditional therapies. Treatments will be enhanced by personalized dosimetry, artificial intelligence, and combination therapies.” — Helen Nadel, MD, Stanford University and president, SNMMI 2023-2024

Radiology Operations – Reimbursement Challenges Continue: “In 2024, we will continue to experience recruitment challenges coupled with decreases in reimbursement. Now, more than ever, every radiologist needs to be diligent in advocating for the specialty, focus on business plan diversification, and ensure all services rendered are optimally documented and billed.” — Rebecca Farrington, chief revenue officer, Healthcare Administrative Partners 

The Takeaway
To paraphrase Robert F. Kennedy, radiology is indeed living in interesting times – times of “danger and uncertainty,” but also times of unprecedented creativity and innovation. In 2024, radiology will get a much better glimpse of where these trends are taking us.

RSNA 2023 Video Highlights

That’s a wrap! 

RSNA 2023 just concluded, and by most accounts it was a successful conference. Preliminary figures indicate that attendance was up 11% over 2022. While short of the glory days of RSNA, the numbers indicate that the meeting’s recovery from the COVID-19 pandemic will be slow but steady.

As expected, AI was a dominant theme at McCormick Place, and that’s reflected in our video coverage of the technical exhibit floor. AI busted out of the AI Showcase to permeate both exhibit halls, a sign of the technology’s growing influence on radiology.

We profiled many of the most intriguing companies that were exhibiting at RSNA 2023 – some of them dominant players in the field while others are new entries looking to secure a foothold. 

We hope you enjoy watching our coverage as much as we enjoyed producing it! Check out the links below or visit the Shows page on our website.

Reimbursement Drives AI Adoption

It’s no secret that insurance reimbursement drives adoption of new medical technology. But a new analysis in NEJM AI shows exactly how reimbursement is affecting the diffusion into clinical practice of perhaps the newest medical technology – artificial intelligence. 

Researchers analyzed a database of over 11B CPT claims from January 2018 to June 2023 to find out how often reimbursement claims are being submitted for the use of the over 500 AI devices that had been approved by the FDA at the time the paper was finalized. 

  • The authors chose to focus on CPT claims rather than claims under the NTAP program for new technologies because CPT codes are used by both public and private payors in inpatient and outpatient settings, while NTAP only applies to Medicare inpatient payments. 

They found 16 medical AI procedures billable under CPT codes; of these, 15 codes were created since 2021 and the median age of a CPT code was about 374 days, indicating the novelty of medical AI.

  • Also, only four of the 16 had more than 1k claims submitted, leading the authors to state “overall utilization of medical AI products is still limited and focused on a few leading procedures,” such as coronary artery disease and diabetic retinopathy.

The top 10 AI products and number of CPT claims submitted are as follows:

  1. HeartFlow Analysis for coronary artery disease (67,306)
  2. LumineticsCore for diabetic retinopathy (15,097)
  3. Cleerly for coronary atherosclerosis (4,459)
  4. Perspectum LiverMultiScan for liver MRI (2,428)
  5. Perspectum CoverScan for multiorgan MRI (591)
  6. Koios DS for breast ultrasound (552)
  7. Anumana for ECG cardiac dysfunction (435)
  8. CADScor for cardiac acoustic waveform recording (331)
  9. Perspectum MRCP for quantitative MR cholangiopancreatography (237)
  10. CompuFlo for epidural infusion (67)

While radiology may rule in terms of the sheer number of FDA-approved AI products (79% in a recent analysis), the list shows that cardiology is king when it comes to paying the bills. 

The Takeaway

Amid the breathless hype around medical AI, the NEJM AI study comes as a bit of a wake-up call, showing how the cold reality of healthcare economics can limit technology diffusion – a finding also indicated in other studies of economic barriers to AI

On the positive side, it shows that a rosy future lies ahead for those AI algorithms – like HeartFlow Analysis – that can make the leap.

Accessing Quality Data for AI Training

One of the biggest roadblocks in medical AI development is the lack of high-quality, diverse data for these technologies to train on.

What Is the Issue with Data Access?

Artificial Intelligence (AI) has emerged as a game-changer in the realm of medical imaging, with immense potential to revolutionize clinical practices. AI-powered medical imaging can efficiently identify intricate patterns within data and provide quantitative assessments of disease biomarkers. This technology not only enhances the accuracy of diagnosis but can also significantly speed up the diagnostic process, ultimately improving patient outcomes.

While the landscape is promising, medical innovators grapple with challenges in accessing high-quality, diverse, and timely data, which is vital for training AI and driving progress.

A 2019 study from the Massachusetts Institute of Technology found that over half of medical AI studies predominantly relied on databases from high-income countries, particularly the United States and China. If models trained on homogenous data are used clinically in diverse populations, then it could pose a risk to patients and worsen health inequalities experienced by underrepresented groups. In the United States, If the Food and Drug Administration deems these risks to be too high, then they could even reject a product’s application for approval. 

In trying to get hold of the best training data, AI developers, particularly startups and individual researchers, face a web of complexities, including legal, ethical, and technical considerations. Issues like data privacy, security, interoperability, and data quality compound these challenges, all of which are crucial in the effective and responsible utilization of healthcare data.

One company working to overcome these hurdles in hope of accelerated and high-quality innovations is Gradient Health.

Gradient Health’s Approach

Gradient Health offers AI developers instant access to one of the world’s largest libraries of anonymized medical images, sourced from hundreds of global hospitals, clinics, and research centers. This data is meticulously de-identified for compliance and can be tailored by vendors to suit their project’s needs and exported in machine learning-ready DICOM + JSON formats.

By partnering with Gradient Health, innovators can use these extensive, diverse datasets to train and validate their AI algorithms, mitigating bias in medical AI and advancing the development of precise, high-quality medical solutions.

Gaining access to top-tier data at the outset of the development process promises long-term benefits. Here’s how:

  • Expand Market Presence: Access the latest cross-vendor datasets to develop medical innovations, expanding your market share.
  • Global Expansion: Enter new regions swiftly with locally sourced data from your target markets, accelerating your global reach.
  • Competitive Edge: Obtain on-demand training data for imaging modalities and disease areas, facilitating product portfolio expansion.
  • Speed to Market: Quickly acquire data for product training and validation, reducing sourcing time and expediting regulatory clearances for faster patient delivery.

“After looking for a data provider for many weeks, I was not able to get even a sample delivery within one month. I was immensely glad to work with Gradient and go from first contact to final delivery within one week!” said Julien Schmidt, chief operations officer and co-founder at Mango Medical.

The Outlook

In recent years, medical AI has experienced significant growth. Innovations in medical imaging in particular have played a pivotal role in enabling healthcare professionals to identify diseases earlier and more accurately in patients with a range of conditions. 

Gradient Health offers a data-compliant, intuitive platform for AI developers, facilitating access to the essential data required to train these critical technologies. This approach holds the potential to save time, resources, and, most importantly, lives. 

More information about Gradient Health is available on the company’s website. They will also be exhibiting at RSNA 2023 in booth #5149 in the South Hall.

Unpacking the Biden Administration’s New AI Order

It seems like watershed moments in AI are happening on a weekly basis now. This time, the big news is the Biden Administration’s sweeping executive order that directs federal regulation of AI across multiple industries – including healthcare. 

The order comes as AI is becoming a clinical reality for many applications. 

  • The number of AI algorithms cleared by the FDA has been surging, and clinicians – particularly radiologists – are getting access to new tools on an almost daily basis.

But AI’s rapid growth – and in particular the rise of generative AI technologies like ChatGPT – have raised questions about its future impact on patient care and whether the FDA’s existing regulatory structure is suitable for such a new technology. 

The executive order appears to be an effort to get ahead of these trends. When it comes to healthcare, its major elements are summarized in a succinct analysis of the plan by Health Law Advisor. In short, the order: 

  • Calls on HHS to work with the VA and Department of Defense to create an HHS task force on AI within 90 days
  • Requires the task force to develop a strategic plan within a year that could include regulatory action regarding the deployment and use of AI for applications such as healthcare delivery, research, and drug and device safety
  • Orders HHS to develop a strategy within 180 days to determine if AI-enabled technologies in healthcare “maintain appropriate levels of quality” – basically, a review of the FDA’s authorization process
  • Requires HHS to set up an AI safety program within a year, in conjunction with patient safety organizations
  • Tells HHS to develop a strategy for regulating AI in drug development

Most analysts are viewing the executive order as the Biden Administration’s attempt to manage both risk and opportunity. 

  • The risk is that AI developers lose control of the technology, with consequences such as patients potentially harmed by inaccurate AI. The opportunity is for the US to become a leader in AI development by developing a long-term AI strategy. 

The Takeaway

The question is whether an industry that’s as fast-moving as AI – with headlines changing by the week – will lend itself to the sort of centralized long-term planning envisioned in the Biden Administration’s executive order. Time will tell.

Predicting the Future of Radiology AI

Making predictions is a messy business (just ask Geoffrey Hinton). So we’re always appreciative whenever key opinion leaders stick their necks out to offer thoughts on where radiology is headed and the major trends that will shape the specialty’s future. 

Two of radiology’s top thought leaders on AI and imaging informatics – Curtis Langlotz, MD, PhD, and Paul Chang, MD – gaze into the crystal ball in two articles published this week in Radiology as part of the journal’s centennial celebration. 

Langlotz offers 10 predictions on radiology AI’s future, briefly summarized below:

  • Radiology will continue its leadership position when it comes to AI adoption in medicine, as evidenced by its dominance of FDA marketing authorizations
  • Virtual assistants will help radiologists draft reports – and reduce burnout
  • Radiology workstations will become cloud-based cockpits that seamlessly unify image display, reporting, and AI
  • Large language models like ChatGPT will help patients better understand their radiology reports
  • The FDA will reform its regulation of AI to be more flexible and speed AI authorizations (see our article in The Wire below)
  • Large databases like the Medical Imaging and Data Resource Center (MIDRC) will spur data sharing and, in turn, more rapid AI development

Langlotz’s predictions are echoed by Chang’s accompanying article in Radiology in which he predicts the future of imaging informatics in the coming age. Like Langlotz, Chang sees the new array of AI-enabled tools as beneficial agents that will help radiologists manage growing workloads through dashboards, enhanced radiology reports, and workflow automation. 

The Takeaway

This week’s articles are required reading for anyone following the meteoric growth of AI in radiology. Far from Hinton’s dystopian view of a world without radiologists, Langlotz and Chang predict a future in which AI and IT technologies assist radiologists to do their jobs better and with less stress. We know which vision we prefer.

What’s Fueling AI’s Growth

It’s no secret that the rapid growth of AI in radiology is being fueled by venture capital firms eager to see a payoff for early investments in startup AI developers. But are there signs that VCs’ appetite for radiology AI is starting to wane?

Maybe. And maybe not. While one new analysis shows that AI investments slowed in 2023 compared to the year before, another predicts that over the long term, VC investing will spur a boom in AI development that is likely to transform radiology. 

First up is an update by Signify Research to its ongoing analysis of VC funding. The new numbers show that through Q3 2023, the number of medical imaging AI deals has fallen compared to Q3 2022 (24 vs. 40). 

  • Total funding has also fallen for the second straight year, to $501M year-to-date in 2023. That compares to $771M through the third quarter of 2022, and $1.1B through the corresponding quarter of 2021. 

On the other hand, the average deal size has grown to an all-time high of $20.9M, compared to 2022 ($15.4M) and 2021 ($18M). 

  • And one company – Rapid AI – joined the exclusive club of just 14 AI vendors that have raised over $100M with a $75M Series C round in July 2023. 

In a look forward at AI’s future, a new analysis in JACR by researchers from the ACR Data Science Institute (DSI) directly ties VC funding to healthcare AI software development, predicting that every $1B in funding translates into 11 new product approvals, with a six-year lag between funding and approval. 

  • And the authors forecast long-term growth: In 2022 there were 69 FDA-approved products, but by 2035, funding is expected to reach $31B for the year, resulting in the release of a staggering 350 new AI products that year.

Further, the ACR DSI authors see a virtuous cycle developing, as increasing AI adoption spurs more investment that creates more products available to help radiologists with their workloads. 

The Takeaway

The numbers from Signify and ACR DSI don’t match up exactly, but together they paint a picture of a market segment that continues to enjoy massive VC investment. While the precise numbers may fluctuate year to year, investor interest in medical imaging AI will fuel innovation that promises to transform how radiology is practiced in years to come.

Autonomous AI for Medical Imaging is Here. Should We Embrace It?

What is autonomous artificial intelligence, and is radiology ready for this new technology? In this paper, we explore one of the most exciting autonomous AI applications, ChestLink from Oxipit. 

What is Autonomous AI? 

Up to now, most interpretive AI solutions have focused on assisting radiologists with analyzing medical images. In this scenario, AI provides suggestions to radiologists and alerts them to suspicious areas, but the final diagnosis is the physician’s responsibility.

Autonomous AI flips the script by having AI run independently of the radiologist, such as by analyzing a large batch of chest X-ray exams for tuberculosis to screen out those certain to be normal. This can significantly reduce the primary care workload, where healthcare providers who offer preventive health checkups may see up to 80% of chest X-rays with no abnormalities. 

Autonomous AI frees the radiologist to focus on cases with suspicious pathology – with the potential of delivering a more accurate diagnosis to patients in real need.

One of the first of this new breed of autonomous AI is ChestLink from Oxipit. The solution received the CE Mark in March 2022, and more than a year later it is still the only AI application capable of autonomous performance. 

How ChestLink Works

ChestLink produces final chest X-ray reports on healthy patients with no involvement from human radiologists. The application only reports autonomously on chest X-ray studies where it is highly confident that the image does not include abnormalities. These studies are automatically removed from the reporting workflow. 

ChestLink enables radiologists to report on studies most likely to have abnormalities. In current clinical deployments, ChestLink automates 10-30% of all chest X-ray workflow. The exact percentage depends on the type of medical institution, with primary care facilities having the most potential for automation.

ChestLink Clinical Validation

ChestLink was trained on a dataset with over 500k images. In clinical validation studies, ChestLink consistently performed at 99%+ sensitivity.

A recent study published in Radiology highlighted the sensitivity of the application.

“The most surprising finding was just how sensitive this AI tool was for all kinds of chest disease. In fact, we could not find a single chest X-ray in our database where the algorithm made a major mistake. Furthermore, the AI tool had a sensitivity overall better than the clinical board-certified radiologists,” said study co-author Louis Lind Plesner, MD, from the Department of Radiology at the Herlev and Gentofte Hospital in Copenhagen, Denmark.

In this study ChestLink autonomously reported on 28% of all normal studies.

In another study at the Oulu University Hospital in Finland, researchers concluded that AI could reliably remove 36.4% of normal chest X-rays from the reporting workflow with a minimal number of false negatives, leading to effectively no compromise on patient safety. 

Safe Path to AI Autonomy

Oxipit ChestLink is currently used in healthcare facilities in the Netherlands, Finland, Lithuania, and other European countries, and is in the trial phase for deployment in one of the leading hospitals in England.

ChestLink follows a three-stage framework for clinical deployment.

  • Retrospective analysis. ChestLink analyzes a couple of years worth (100k+) of historic chest x-ray studies at the medical institution. In this analysis the product is validated on real-world data. It also realistically estimates what fraction of reporting scope can be automated.
  • Semi-autonomous operations. The application moves into prospective settings, analyzing images in near-real time. ChestLink produces preliminary reports for healthy patients, which may then be approved by a certified clinician.
  • Autonomous operations. The application autonomously reports on high-confidence healthy patient studies. The application performance is monitored in real-time with analytical tools.

Are We There Yet?

ChestLink aims to address the shortage of clinical radiologists worldwide, which has led to a substantial decline in care quality.

In the UK, the NHS currently faces a massive 33% shortfall in its radiology workforce. Nearly 71% of clinical directors of UK radiology departments feel that they do not have a sufficient number of radiologists to deliver safe and effective patient care.

ChestLink offers a safe pathway into autonomous operations by automating a significant and somewhat mundane portion of radiologist workflow without any negative effects for patient care. 

So should we embrace autonomous AI? The real question should be, can we afford not to? 

Get every issue of The Imaging Wire, delivered right to your inbox.

You might also like..

Select All

You're signed up!

It's great to have you as a reader. Check your inbox for a welcome email.

-- The Imaging Wire team

You're all set!