|
Audit Excitement | LCS Saves Lives
April 10, 2022
|
|
|
|
|
Together with
|
|
|
|
“I can’t decide if I’m scared or excited.”
|
|
A tweet from Emory’s Hari Trivedi MD after Oxipit ChestLink became the first regulatory-approved autonomous imaging AI product
|
|
|
|
A new Lancet Digital Health study could have become one of the many “AI rivals radiologists” papers that we see each week, but it instead served as an important lesson that traditional performance tests might not prove that AI models are actually safe for clinical use.
The Model – The team developed their proximal femoral fracture detection DL model using 45.7k frontal X-rays performed at Australia’s Royal Adelaide Hospital (w/ 4,861 fractures).
The Validation – They then tested it against a 4,577-exam internal set (w/ 640 fractures), 400 of which were also interpreted by five radiologists (w/ 200 fractures), and against an 81-image external validation set from Stanford.
The Results – All three tests produced results that a typical study might have viewed as evidence of high-performance:
- The model outperformed the five radiologists (0.994 vs. 0.969 AUCs)
- It beat the best performing radiologist’s sensitivity (95.5% vs. 94.5%) and specificity (99.5% vs 97.5%)
- It generalized well with the external Stanford data (0.980 AUC)
The Audit – Despite the strong results, a follow-up audit revealed that the model might make some predictions for the wrong reasons, suggesting that it is unsafe for clinical deployment:
- One false negative X-ray included an extremely displaced fracture that human radiologists would catch
- X-rays featuring abnormal bones or joints had a 50% false negative rate, far higher than the reader set’s overall false negative rate (2.5%)
- Salience maps showed that AI decisions were almost never based on the outer region of the femoral neck, even with images where that region was clinically relevant (but it still often made the right diagnosis)
- The model scored a high AUC with the Stanford data, but showed a substantial model operating point shift
The Case for Auditing – Although the study might have not started with this goal, it ended up becoming an argument for more sophisticated preclinical auditing. It even led to a separate paper outlining their algorithmic auditing process, which among other things suggested that AI users and developers should co-own audits.
The Takeaway
Auditing generally isn’t the most exciting topic in any field, but this study shows that it’s exceptionally important for imaging AI. It also suggests that audits might be necessary for achieving the most exciting parts of AI, like improving outcomes and efficiency, earning clinician trust, and increasing adoption.A new Lancet Digital Health study could have become one of the many “AI rivals radiologists” papers that we see each week, but it instead served as an important lesson that traditional performance tests might not prove that AI models are actually safe for clinical use.
|
|
|
Purpose-Built for the Cloud
Imaging’s cloud evolution didn’t happen all at once. This Change Healthcare animation details the history of digital imaging architectures, and how cloud-native imaging improves stability and scalability, ease of management, patient data security, and operating costs.
|
|
Reinventing How Medical Imaging is Done
Explore how United Imaging is reinventing the medical imaging business, including downtime rebates, lifetime upgrades, and making sure their customers truly are successful.
|
|
- LCS Effectiveness: A new BMJ study highlighted the US lung cancer screening program’s significant impact since the USPSTF recommended annual exams in 2013. Data from 763k patients diagnosed with NSCLC lung cancer between 2010 and 2018 (ages 55-80) showed that the percent of patients diagnosed with stage one cancer increased by 3.9% each year from 2014 to 2018, while the median survival rate increased by 11.9% annually. That’s a big improvement compared to the 2010-2013 period when there were no significant increases in stage one diagnoses and survival rates.
- Viz.ai’s Unicorn Status: Viz.ai completed a $100M Series D round, bringing its total capital raised to over $251M and its valuation to $1.2B, while making it one of just a handful of imaging unicorns. Those are some big numbers for an imaging AI startup, justified by Viz.ai’s strong adoption (over 1k hospitals) and the fact that Viz LVO’s care coordination capabilities and reimbursements appear to be having a significant clinical impact. Viz.ai will use its new funding to try to keep that momentum going, with a focus on expanding its platform to new diseases and growing its global customer base.
- Aduhelm’s Big Setback: Biogen’s Aduhelm Alzheimer’s treatment experienced its biggest setback yet, after Medicare decided to officially limit the controversial treatment to clinical trial patients. Aduhelm was originally expected to drive a wave of imaging exams (it requires brain PET for diagnosis & ongoing MRIs for monitoring), but the new drug has been surrounded by criticism about its effectiveness, costliness, and questionable FDA approval process. In fact, the new Medicare restriction follows decisions from a number of major health systems’ to not provide or prescribe Aduhelm (MGH, Cleveland Clinic, Mount Sinai, the VA, and more).
- Arterys’ Cardio AI Expansion: Arterys expanded its Cardio AI cardiac MRI analysis application, launching three brand-new modules and announcing the FDA clearance of three existing modules (Arterys’ 8th FDA). The net new modules include the 4D Flow-powered Strain + AI (for myocardial velocity and strain, research only), Atrial Volumes (quantifies left and right atria volumes), and Wall Thickness (automatically measures wall thickness). The now FDA-cleared Cardio AI modules offer a dedicated workflow for T1 and ECV, and T2 quantification.
- Autonomous AI Goes Viral: Oxipit’s ChestLink solution becoming the first regulatory-approved autonomous imaging AI product was the hottest topic in radiology last week, driving coverage by mainstream news outlets and industry blogs while prompting some heated online conversations (here are a few of them). That’s not surprising given the significance of this milestone, and the wide range of opinions about this approval weren’t that surprising either (e.g. impressed, concerned for patients, concerned for careers, concerned about malpractice implications, doubtful it would work).
- Burnout & Outcomes: A recent Health Affairs study suggests that physician burnout might have a different impact on patient outcomes than some might think. The study linked surveys from 1,064 family physicians to Medicare claims, finding no statistically significant relationship between self-reported burnout and a range of negative patient outcomes (admissions, ED visits, readmissions, or costs). In fact, physicians who reported burnout were consistently associated with fewer negative events, suggesting that these doctors “may nevertheless be able to create better outcomes for their patients.”
- Aidoc’s Brain Aneurysm FDA: Aidoc announced the FDA 510(k) clearance of its new brain aneurysm solution, which detects suspected brain aneurysms and coordinates care across relevant teams (radiologists, neuroendovascular surgeons, neurologists). The Brain Aneurysm solution becomes Aidoc’s 9th FDA-cleared AI product (not including platform partner solutions) and further expands the company’s broad neuroscience suite (also includes ICH, LVO, and C-spine fractures).
- A Case for 89Zr-PSMA-DFO: A JNM study out of Germany showed that the 89Zr-PSMA-DFO PET tracer might catch prostate cancer lesions that are missed with standard PSMA tracers. The researchers performed 89Zr-PSMA-DFO PET/CT exams on 14 prostate cancer patients with biochemical recurrence who recently had negative results with existing PSMA tracers (68Ga-PSMA-11 or 18F-JK-PSMA-7). The 89Zr-PSMA-DFO follow-ups revealed 15 PSMA-positive lesions in 8 of the patients (3 localized recurrence, 3 metastases in lymph nodes, 2 lesions at distant sites), leading to new therapies with seven of the patients (5 lesion-targeted radiotherapies, 2 androgen deprivation therapies).
- Rock Health’s Q1 Slowdown: The digital health investment slowdown has officially arrived, as Rock Health’s Q1 2022 report revealed the sector’s first year-over-year Q1 decrease since 2019 ($6B vs. $6.7B in Q1 2021). Rock Health also noted that it’s been a “no good, very bad time” for publicly traded digital health companies (-38% since July 2021) causing many startups to delay their IPO plans. Although a relatively small portion of Q1’s funding went to imaging companies, two of the six most-funded clinical indications involved imaging (oncology #2 & cardiovascular #3, both ~$600M).
- DIY Obstetrics AI: A new NEJM study suggests that the rising interest we’ve seen in developing world-targeted obstetrics ultrasound AI is warranted. The researchers had 4,695 pregnant volunteers in North Carolina and Zambia perform their own ultrasound exams with a Butterfly handheld and receive standard biometry exams from trained sonographers, using the data to train and test an AI model. Using handheld ultrasounds and AI, the women were able to estimate gestational age with a lower mean absolute error rate than the sonographer exams (3.9 vs. 4.7 days), with minimal variations between the NC and Zambia-based women (−0.6 & −1.0 days).
- Probo Acquires Canute Medical: Probo Medical put its new private equity funding to work, acquiring Canada-based imaging equipment installation, deinstallation, and remarketing company Canute Medical. The acquisition comes just a few months after Probo similarly acquired install/deinstall provider REMETRONIX, and follows a string of acquisitions that significantly expanded Probo’s sales and service capabilities.
|
|
Us2.ai Automates the Fight Against Heart Disease
See how Us2.ai cuts echocardiography’s manual work, subjectivity, and turnaround times to automate the fight against heart disease.
|
|
- In this Novarad video, interventional oncologist Gary M. Onik, MD shares how Novarad’s AR surgical navigation system, OpenSight, helps his team accurately assess and treat tumors.
- Enterprise Imaging has come a long way, and it has a long way to go. This Intelerad white paper details the five pillars organizations should prioritize in order to realize the full potential of EI’s next evolution.
- The flow of new AI applications makes it hard for radiology groups to determine which tools would help them and how IT teams can handle increased AI adoption. In this Blackford Analysis white paper, radiology and IT leaders from NYU and Canopy Partners share how a platform approach alongside a curated marketplace can help solve these challenges.
- This Riverain Technologies case study details how Einstein Medical Center adopted ClearRead CT enterprise-wide (all 13 CT scanners) and how the solution allowed Einstein radiologists to identify small nodules faster and more reliably.
- Faced with the task of monitoring the thousands of exams its algorithms analyze each day, Qure.ai leveraged CARPL.ai’s validation workflow to create a real-time performance dashboard. See how they did it here.
- Learn how new GE Healthcare MRI technologies are making PI-RADS v2.1 compliant prostate imaging at 1.5T a game changer, with improved image quality, shorter scan times, and a better overall experience for the patient.
- Check out this editorial by Nuance EVP Peter Durlach on how AI-augmented diagnostic imaging is driving new approaches to collaborative, patient-specific precision care.
- Women’s imaging has come a long way, but operational efficiency remains a challenge for many facilities. To help address this challenge, this Fujifilm post details the five questions women’s imaging facilities should ask when evaluating workflow management solutions.
|
|
|
|
|
