|
Grading AI Report Quality | Google’s Medical AI Strategy August 7, 2023
|
|
|
|
Together with
|
|
|
“The key will be how do we integrate all these AI outputs to reach the end clinician on time. I can identify a nodule, I can triage a hemorrhage, but if that still sits on the worklist of a radiologist, it’s not helped the patient.”
|
Chiranjiv Singh, vice president and general manager at Arterys, a Tempus Labs company, in the latest edition of The Imaging Wire Show.
|
|
What’s happening at the intersection of AI and precision medicine? In this Imaging Wire Show, we talked to Chiranjiv Singh, vice president and general manager at Arterys, a Tempus Labs company, about their vision for building a 360° view of the patient by combining clinical, molecular, radiology, and pathology data and applying AI on top of it..
|
|
|
One of the most exciting new use cases for medical AI is in generating radiology reports. But how can you tell whether the quality of a report generated by an AI algorithm is comparable to that of a radiologist?
In a new study in Patterns, researchers propose a technical framework for automatically grading the output of AI-generated radiology reports, with the ultimate goal of producing AI-generated reports that are indistinguishable from those of radiologists.
Most radiology AI applications so far have focused on developing algorithms to identify individual pathologies on imaging exams.
- While this is useful, helping radiologists streamline the production of their main output – the radiology report – could have a far greater impact on their productivity and efficiency.
But existing tools for measuring the quality of AI-generated narrative reports are limited and don’t match up well with radiologists’ evaluations.
- To improve that situation, the researchers applied several existing automated metrics for analyzing report quality and compared them to the scores of radiologists, seeking to better understand AI’s weaknesses.
Not surprisingly, the automated metrics fell short in several ways, including false prediction of findings, omitting findings, and incorrectly locating and predicting the severity of findings.
- These shortcomings point out the need for better scoring systems for gauging AI performance.
The researchers therefore proposed a new metric for grading AI-generated report quality, called RadGraph F1, and a new methodology, RadCliQ, to predict how well an AI report would measure up to radiologist scrutiny.
- RadGraph F1 and RadCliQ could be used in future research on AI-generated radiology reports, and to that end the researchers have made the code for both metrics available as open source.
Ultimately, the researchers see the construction of generalist medical AI models that could perform multiple complex tasks, such as conversing with radiologists and physicians about medical images.
- Another use case could be applications that are able to explain imaging findings to patients in everyday language.
The Takeaway
It’s a complex and detailed paper, but the new study is important because it outlines the metrics that can be used to teach machines how to generate better radiology reports. Given the imperative to improve radiologist productivity in the face of rising imaging volume and workforce shortages, this could be one more step on the quest for the Holy Grail of AI in radiology.
|
|
|
Understanding the Platform Approach to AI
Here’s a quick introduction to Blackford Analysis’ dedicated AI platform and its service for the selection, deployment, orchestration, and use of imaging applications and AI.
|
|
How Valley View Integrated PocketHealth with PowerShare
Valley View Hospital in Colorado wanted to share images and records with patients while continuing to use its existing PowerShare implementation for providers. Find out how PocketHealth helped them do it in this article.
|
|
- Google’s Medical AI Strategy: A pair of Google executives who are leading the search behemoth’s medical AI business outlined their approach to medical AI in a blog post, pointing to new papers describing how to apply large language model (LLM) AI algorithms to medical imaging. One of them describes ELIXR, an LLM running on Google’s PaLM 2 model for classifying chest X-rays, while another outlines a generalist biomedical AI based on PaLM-E called Med-Palm M that will work with a variety of medical data.
- AI Finds Missed PE Lesions: An AI algorithm that analyzed chest CT scans was able to detect cases of incidental acute pulmonary embolism (PE) that might have been missed by radiologists. In a study of 3.1k patients in European Radiology, the algorithm from Aidoc detected 25 cases of acute PE that would have gone unreported, or 37% of all positive PE cases. The findings show that AI could act as a backstop to help radiologists avoid missing acute cases while not overwhelming them with false-positive findings.
- Docs Get Surprise Billing Victory: Physicians got a victory in the battle over implementation of the federal No Surprises Act. A Texas judge ruled in favor of physician groups suing to stop CMS from raising the fee for managing billing disputes under the act’s independent dispute resolution (IDR) process from $50 to $350. The judge nullified the increase, prompting CMS to halt the IDR process and put a hold on new billing disputes within the Medicare and Medicaid systems.
- How Diverse is Radiology? Not very. A new study in JAMA shows that diagnostic radiology has a much higher percentage of male trainee physicians than female (73.1% vs. 26.5%), compared to a median of 53.6% vs. 43.8% for all other nonsurgical specialties. Only 8% of radiology trainees come from groups that are underrepresented in medicine, versus 10.4% for all nonsurgical specialties. Researchers also found that diversity was lower in specialties with higher faculty pay – which was the case in radiology.
- More Women Doctors in US: By point of comparison, another recent survey found that women made up 37% of the total US physician workforce in 2022, up from 30% in 2010. The total physician workforce grew 23% over the last decade, to a total of 1,044,734 doctors. State medical boards issued a record high of 129k new licenses in 2022 (up 27% from 2020), a trend driven predominantly by expanded telehealth use throughout the pandemic.
- Good Times Ahead for CT Sales: A new analysis from Signify Research values the global CT market at $6B in 2022, with growth last year “more subdued” compared to 2021. In the medium term, workforce shortages and weaker economic conditions could slow the CT replacement cycle in some markets. But over the long term, good times are ahead for radiology’s workhorse modality, as Signify foresees modernization in emerging markets and new technologies like AI reconstruction driving global revenue at a CAGR of 6.7% between 2022 and 2027.
- Single-use Contrast Vials Slash Waste: Using single-use vials of intravenous contrast media optimized for individual procedures rather than larger multiple-use vials slashed contrast waste by over 50% in a new paper published in JACR. Researchers reviewed contrast use in over 40k administrations in CT scans of 26k patients, finding that optimally sized single-use vials generated less waste per use (11mL) compared to both 100mL vials (26mL) and multi-use vials (13mL). Rather than stock a single vial size of contrast, CT departments may want to have various sizes available, especially given the 2022 contrast shortage.
- Misdiagnoses Linked to 370k Deaths Annually: A recent study in BMJ Quality and Safety drew attention after finding that the full impact of misdiagnoses in the U.S. is likely being seriously underestimated. Researchers estimated that 371k people die every year following a misdiagnosis, and 424k are permanently disabled – meaning nearly 800k people suffer “serious harm” annually.
- Butterfly Grows AI Garden: Handheld ultrasound developer Butterfly Network plans to launch Butterfly Garden, a new marketplace for third-party AI developers to host applications for the company’s installed base. Butterfly simultaneously launched an SDK to help third parties develop custom applications for the marketplace. The SDK can be used to develop apps ranging from image acquisition assistance to education and simulation.
- The Cost of Data Breaches: The healthcare sector reclaimed its unenviable title as the “most expensive industry for data breaches” for the 13th consecutive year, with the average cost of a healthcare data breach reaching nearly $11M in 2023 (a whopping 53% jump since 2020). High-profile incidents at HCA and CommonSpirit helped drag up the total, and last week’s attack on facilities run by Prospect Medical Holdings won’t help healthcare’s standing.
- CMS Finalizes 3.1% Hospital Bump: CMS finalized a 3.1% payment increase for hospitals for 2024, a slight increase from the 2.8% bump proposed earlier this year. Despite the upwards revision, hospital groups have been outspoken against the final rule and say the increase falls short. Still, hospitals are doing better than physicians, for whom CMS is proposing a 3.36% reduction in payments under the Medicare Physician Fee Schedule.
- Physicians Lack Trust in Leadership: A new survey found that only 36% of physicians at not-for-profit health systems agree that their leaders are honest and transparent (vs. half of physicians at investor-owned systems). Only half of physicians trust leadership decision-making when it comes to operations and patient care, and instead place the highest levels of trust in their peer physicians. When asked what would improve trust, “transparency” was unsurprisingly the most common response.
- Cleerly’s 10-Year ASCVD Predictions: A new JACC study found that Cleerly’s coronary CTA AI solution can effectively gauge CAD patients’ 10-year CV event risks. Researchers used the Cleerly AI to classify the plaque burdens from 536 patients with suspected CAD (stages: 0-3), and then analyzed 10-year outcomes. Patients with stage-3 plaque burdens had an over 3X higher risk of MACE than stage 0 or 1 patients (adjusted HR: 3.57), while Cleerly AI improved 10-year MACE prediction accuracy compared to a model using clinical risk factors and CAC scores (AUC: 0.82 vs. 0.73).
|
|
Improve Your Cardiovascular Image Data Management
Every 34 seconds, a patient dies of cardiovascular disease. Find out how you can better manage cardiovascular image data and improve patient care with syngo Dynamics from Siemens Healthineers.
|
|
New Developments in Enterprise Imaging
What’s the latest news from Merge by Merative? Get an update on the company’s activities in enterprise imaging, VNA and viewers, workflow orchestration, and radiology departmental solutions from General Manager Ashish Sant in this interview from SIIM 2023.
|
|
Strategies for Running an Efficient Radiology Department
Do you see room for efficiency improvement in your radiology department? Find out how you can leverage advanced technology like Enlitic’s Curie|ENDEX solution to make sure your radiologists are reporting efficiently.
|
|
- Knowing an individual’s mammographic breast density is key to understanding their risk for cancer. A new article from Visage Imaging explains how an AI-based solution can improve breast density assessment.
- Imaging AI is evolving fast, but radiology leaders’ expectations for their AI technologies might be evolving even faster. In this Imaging Wire Show with Dr. Charlene Liew of SingHealth and Dr. Nina Kottler of Radiology Partners, we explore radiology leaders’ current and future expectations for AI, and the central role platforms play in their AI roadmaps.
- With ongoing radiologist shortages and higher rates of burnout, there’s a great need for fast, effective, efficient medical imaging technologies – and those factors are driving medical imaging’s biggest trends detailed in this Arterys report.
- AI automates what radiologists can’t stand, surfaces what radiologists can’t see, and identifies what radiologists can’t miss. But only if it’s implemented in the way radiologists work. See how Nuance helps radiologists achieve these results through a single, streamlined, end-to-end AI experience.
- See how cloud-native imaging avoids traditional software’s resource utilization constraints and eliminates unexpected disruptions in this Change Healthcare animation.
- How is Clearpath improving the healthcare experience for patients? Learn more about the company and its solutions for ditching the disc when sharing images and records with patients.
- Annalise.ai doubled down on its comprehensive AI strategy with the launch of its Annalise Enterprise CTB solution, which identifies a whopping 130 different non-contrast brain CT findings. Annalise Enterprise CTB analyzes brain CTs as they are acquired, prioritizes urgent cases, and provides radiologists with details on each finding (types, locations, likelihood).
- Bayer’s cloud-based Calantic Digital Solutions AI platform features a suite of disease-specific AI apps that integrate into radiologist workflows, helping radiology teams scale AI deployment and improve efficiency and quality of care.
- Interested in learning more on cardiac CT protocols? Hear from Luis Landeras, MD, and technologist Michael Mason from the University of Chicago as they share common challenges, clinical cases, and customized CT protocols they utilize with cardiac patients on their GE HealthCare Revolution Apex CT scanner.
- Subtle Medical has been named to CB Insights’ 2023 list of top 100 AI companies worldwide. The laureates were picked from a pool of nearly 9k companies, and were chosen based on a variety of criteria developed by CB Insights.
- Crouse Hospital is a nationally recognized cardiac care center in the Syracuse, NY, area, but the hospital’s cardiovascular service relied on separate data islands. That is, until Crouse Hospital adopted the HealthView CVIS from Intelerad’s Lumedx business.
|
|
Share The Imaging Wire
|
Spread the news & help us grow ⚡
|
Refer colleagues with your unique link and earn rewards.
|
|
|
Or copy and share your custom referral link: *|SHAREURL|*
|
You currently have *|REFERRALS|* referrals.
|
|
|
|
|