The False Hope of Explainable AI

Many folks view explainability as a crucial next step for AI, but a new Lancet paper from a team of AI heavyweights argues that explainability might do more harm than good in the short-term, and AI stakeholders would be better off increasing their focus on validation.

The Old Theory – For as long as we’ve been covering AI, really smart and well-intentioned people have warned about the “black-box” nature of AI decision making and forecasted that explainable AI will lead to more trust, less bias, and greater adoption.

The New Theory – These black-box concerns and explainable AI forecasts might be logical, but they aren’t currently realistic, especially for patient-level decision support. Here’s why:

  • Explainability methods describe how AI systems work, not how decisions are made
  • AI explanations can be unreliable and/or superficial
  • Most medical AI decisions are too complex to explain in an understandable way
  • Humans over-trust computers, so explanations can hurt their ability to catch AI mistakes
  • AI explainability methods (e.g heat maps) require human interpretation, risking confirmation bias
  • Explainable AI adds more potential error sources (AI tool + AI explanation + human interpretation)
  • Although we still can’t fully explain how acetaminophen works, we don’t question whether it works, because we’ve tested it extensively

The Explainability Alternative – Until suitable explainability methods emerge, the authors call for “rigorous internal and external validation of AI models” to make sure AI tools are consistently making the right recommendations. They also advised clinicians to remain cautious when referencing AI explanations and warned that policymakers should resist making explainability a requirement. 

Explability’s Short-Term Role – Explainability definitely still has a role in AI safety, as it’s “incredibly useful” for model troubleshooting and systems audits, which can improve model performance and identify failure modes or biases.

The Takeaway – It appears we might not be close enough to explainable AI to make it a part of short-term AI strategies, policies, or procedures. That might be hard to accept for the many people who view the need for AI explainability as undebatable, and it makes AI validation and testing more important than ever.

ImageBiopsy Lab & UCB’s AI Alliance

Global pharmaceutical company UCB recently licensed its osteoporosis AI technology to MSK AI startup ImageBiopsy Lab, representing an interesting milestone for several emerging AI business models.

The UCB & ImageBiopsy Lab Alliance – ImageBiopsy Lab will use UCB’s BoneBot AI technology to develop and commercialize a tool that screens CT scans for “silent” spinal fractures to identify patients who should be receiving osteoporosis treatments. The new tool will launch by 2023 as part of ImageBiopsy Lab’s ZOO MSK platform.

UCB’s AI Angle – UCB produces an osteoporosis drug that would be prescribed far more often if detection rates improve (over 2/3 of vertebral fractures are currently undiagnosed). That’s why UCB developed and launched BoneBot AI in 2019 and it’s why the pharma giant is now working with ImageBiopsy Lab to bring it into clinical use.

The PharmaAI Trend – We’re seeing a growing trend of drug and device companies working with AI developers to help increase treatment demand. The list is getting pretty long, including quite a few PharmaAI alliances targeting lung cancer treatment (Aidence & AstraZeneca, Qure.ai & AstraZeneca, Huma & Bayer, Optellum & J&J) and a diverse set of AI alliances with medical device companies (Imbio & Olympus for emphysema, Aidoc & Inari for PE, Viz.ai & Medtronic for stroke).

The Population Health AI Trend – ImageBiopsy Lab’s BoneBot AI licensing is also a sign of AI’s growing momentum in population health, following increased interest from academia and major commercial efforts from Cleerly (cardiac screening) and Zebra Medical Vision (cardiac and osteoporosis screening… so far). This alliance also introduces a new type of population health AI beneficiary (pharma companies), in addition to risk holders and patients.

The Takeaway – ImageBiopsy Lab and UCB’s new alliance didn’t get a lot of media attention last week, but it tells an interesting story about how AI business models are evolving beyond triage, and how those changes are bringing some of healthcare’s biggest names into the imaging AI arena.

Growing with Desert Radiology

Desert Radiology executive Matt Grimes starred in a recent Aunt Minnie webinar, detailing the Las Vegas radiology group’s operational and growth strategy, and sharing some very relevant takeaways for imaging providers and vendors. 

About Desert Radiology – Desert Radiology (DR) operates 11 imaging centers, services 14 acute care hospitals across Southern Nevada, and staffs over 80 radiologists and 500 clinical/support teammates. DR was founded nearly 55 years ago, but it has nearly doubled its imaging centers and radiologist workforce in the last five years.

Challenges – Desert Radiology faces more than its share of challenges, some of which are unique to the Las Vegas area (large managed care population, no local radiology med school programs), and some that are common across the country (radiologist hiring/recruitment, competition, declining reimbursements, rising volumes).

Growth Starts from Within – In order to grow without burning out its team, DR restructured its shift schedules to better fit staff members’ needs and diversified its radiologist career paths to match their personal goals (e.g. multiple partner tracks, an associate track, and a telerad track). 

Engagement Pivot – DR previously relied on radio and billboard ads to reach new patients but pivoted towards a community engagement strategy, with a focus on outreach, charity work, and deepening its relationship with local providers and partners.

Population Health Partnerships – Because of Las Vegas’ high concentration of managed care patients, Desert Radiology places considerable focus on reducing unnecessary imaging and achieving early/accurate diagnoses. This patient environment has also driven DR to deepen its local healthcare relationships, leading to new population health-appropriate agreement structures and referrer programs.

Selecting The Right OEMs – When evaluating new scanners, DR first examines image quality by having its radiologists evaluate images while blinded to the scanner brands (avoiding bias). It then evaluates the proposed scanners’ ease of use, workflow fit, and overall value, before making a final decision.

DR’s Case for United Imaging – Grimes also detailed how Desert Radiology has benefitted from working with United Imaging (the webinar’s sponsor), specifically highlighting the value of UIH’s “Software for Life” (scanners automatically updated with future software) and “All-In” (scanners include all possible features/packages) policies.

The Takeaway – We get plenty of insights from the commercial and academic side of radiology each week, but operational insights are still rare, making this webinar particularly useful for the many imaging groups with similar goals and challenges as Desert Radiology.

Who Owns AI Evaluation and Monitoring?

Imaging AI evaluation and monitoring just became even hotter topics, following a particularly revealing Twitter thread and a pair of interesting new papers.

Rads Don’t Work for AI – A Mayo Clinic Florida neuroradiologist took his case to Twitter after an FDA-approved AI tool missed 6 of 7 hemorrhages in a single shift and he was asked to make extra clicks to help improve the algorithm. No AI tool is perfect, but many folks commenting on this thread didn’t take kindly to the idea of being asked to do pro-bono work to improve an algorithm that they already paid for. 

AI Takes Work – A few radiologists with strong AI backgrounds clarified that this “extra work” is intended to inform the FDA about postmarket performance, while monitoring healthcare tools and providing feedback is indeed physicians’ job. They also argued that radiology practices should ensure that they have the bandwidth to monitor AI before deciding to adopt it.

The ACR DSI Gets It – Understanding that “AI algorithms may not work as expected when used beyond the institutions in which they were trained, and model performance may degrade over time” the ACR Data Science Institute (DSI) released a helpful paper detailing how radiologists can evaluate AI before and during clinical use. In an unplanned nod to the above Twitter thread, the DSA paper also noted that AI evaluation/monitoring is “ultimately up to the end users” although many “practices will not be able to do this on their own.” The good news is the ACR DSI is developing tools to help them.

DLIR Needs Evaluation Too – Because measuring whether DL-reconstructed scans “look good” or allow reduced dosage exams won’t avoid errors (e.g. false tumors or removed tumors), a Washington University in St. Louis-led team is developing a framework for evaluating DLIR tools before they are introduced into clinical practice. The new framework comes from some big-name intuitions (WUSTL, NIH, FDA, Cleveland Clinic, UBC), all of whom also appear to agree that AI evaluation is up to the users.

The Takeaway – At least among AI insiders it’s clear that AI users are responsible for algorithm evaluation and monitoring, even if bandwidth is limited and many evaluation/monitoring tools are still being developed. Meanwhile, many AI users (who are crucial for AI to become mainstream) want their FDA-approved algorithms to perform correctly and aren’t eager to do extra work to help improve them. That’s a pretty solid conflict, but it’s also a silver lining for AI vendors who get good at streamlining evaluations and develop low-labor ways to monitor performance.

Intelerad Acquires Ambra

Intelerad just got a whole lot bigger, acquiring Ambra Health to create one of the industry’s most comprehensive image management companies.

Acquisition Details – The acquisition values the combined companies at $1.7b, expands their reach to nearly 2k global customers (including all of the US’ top 10 hospitals), and brings their headcount to roughly 1k team members. Ambra CEO, Morris Panner, will become Intelerad’s president and will lead the company alongside CEO, Mike Lipps.

Intelerad + Ambra Portfolio – The acquisition combines Intelerad’s PACS portfolio with Ambra’s cloud VNA, image exchange, custom integration services, and research and pathology capabilities. 

Competitive Impact – At least in terms of portfolio breadth, this acquisition moves Intelerad into enterprise imaging’s top tier (radiology, cardiology, archive, sharing), helping it expand beyond its radiology practice legacy and deeper into hospitals. However, the star of this acquisition may prove to be combining Ambra’s cloud VNA with Intelerad’s cloud PACS, which as we’ve seen from Visage and Change’s recent cloud takeovers, can be a very effective combination.

Intelerad Growth – Intelerad has taken full advantage of its PE-backing, making a series of acquisitions since mid-2020 that allowed expansions into new specialties (cardiac & OB/GYN), regions (UK), and technologies (cloud). Ambra is clearly its biggest investment and most significant expansion yet.

The Diagnostic Gap

The Lancet Commission on Diagnostics just put a spotlight on the developing world’s alarmingly low access to diagnostics and how this situation can be addressed. 

The LMIC Gap – An unbelievable 47% of the global population has little to no access to diagnostics, with the vast majority of this diagnostic gap concentrated in low and middle-income countries (LMICs). This problem is greatest through LMICs’ primary care facilities (19% of people can access PCs w/ diagnostics), but also exists in hospitals (60%-70% of people can access hospitals w/ diagnostics).

The Impact – About 50% of people living with any of six key conditions in LMICs are undiagnosed (hypertension, type 2 diabetes, HIV, tuberculosis, syphilis & hepatitis B virus infection in pregnant women), making diagnostic access the world’s single greatest barrier to care. If undiagnosed rates for these six conditions were reduced to 10% in LMICs, it would avoid 1.1m premature deaths annually.

The Imaging Gap – There is a significant lack of imaging access in LMICs. Imaging access is lowest in primary care where only 5% of basic facilities and 12% of advanced facilities have ultrasound (never mind more advanced imaging). Meanwhile only 36% to 87% of hospitals in LMICs have working X-ray systems and just 2% to 29% of hospitals have a CT scanner (depending on the country).

The Problem – The authors largely blamed the developing world’s diagnostic gap on a lack of visibility and prioritization, although there’s a long list of other factors (corruption, costs, infrastructure, workforce).

The Solutions – The Lancet Commission believes that recent technology and informatics innovations could accelerate government efforts to improve diagnostic access. Until then, they recommend that LMICs develop national diagnostics strategies, ensure that standard diagnostic tests are available at various healthcare tiers (e.g. ultrasound at all primary care facilities), and prioritize improving diagnostic access through primary care facilities.

One Takeaway – Half the world doesn’t have access to diagnostics. This is mainly an economic problem, but imaging could play an outsized role in the solution considering that many of the latest imaging innovations are well suited for low-resource areas (e.g. handheld POCUS, AI diagnostics/guidance, portable MRI, teleradiology, etc.).

Aidoc & Riverain’s Platform Partnership

Aidoc and Riverain Technologies announced a new partnership that will make Riverain’s ClearRead CT and ClearRead Xray solutions available on the Aidoc platform, while advancing the companies’ respective platform strategies. 

The Chest AI Package – In addition to offering Riverain’s AI tools individually, Aidoc will provide them as part of an ‘integrated chest AI package’ that also includes Aidoc’s modules for PE, incidental PE, and rib fractures. 

Riverain’s Platform Push – Riverain has amassed a solid network of AI marketplace and OEM partners over the years, and it now appears to be expanding its channel to complementary AI vendors. Riverain’s new Aidoc alliance comes just a few weeks after a similar partnership with Volpara that combines ClearRead CT with the Volpara Lung platform.

Aidoc’s Platform Portfolio – After years of building out its homegrown AI portfolio (7 products) and customer base (600 health centers), Aidoc is evolving into an AI platform company. Over the last year, Aidoc has assembled a solid AI portfolio that combines its own triage products with solutions that it doesn’t offer (Imbio, Icometrix, Subtle Medical, Riverain), allowing its clients to expand their AI stack without overhauling their infrastructure with each new tool.

The Takeaway – We’re at an interesting time in the AI space with a small handful of diversified AI players (e.g. Aidoc, Qure.ai), a group of focused category leaders (e.g. Riverain w/ thoracic, ScreenPoint w/ mammography), and an AI customer base that would prefer not to support multiple AI infrastructures. Although marketplaces also solve this problem, it’s easy to see how complementary vendor partnerships like these could play a growing role in how AI is delivered going forward.

Siemens’ First Photon-Counting CT

The FDA announced the 510(k) clearance of Siemens Healthineers’ NAEOTOM Alpha photon-counting CT scanner, calling it “the first new major technological improvement for Computed Tomography (CT) imaging in nearly a decade.” Those are some big words from a federal agency not known for hyperbole, and it doesn’t appear they are exaggerating.

About Photon-Counting CTs – Photon-counting CTs (PCCTs) produce far higher quality images than traditional CTs, with lower radiation and contrast dosage. Unlike standard CTs that simultaneously measure the total energy from many X-rays (at the expense of image info, clarity, and contrast), photon-counting CT detectors directly convert each individual X-ray photon into digital electrical signals that are then “counted.”

The NAEOTOM Alpha – The NAEOTOM Alpha might be the first PCCT scanner, but the star of this announcement is its detector. The new photon-counting detector leverages a CdTe active detection layer to achieve PCCT’s targeted image/dosage/contrast advantages, and it could be the foundation of Siemens’ PCCT portfolio for years to come. 

The PCCT Race – The other major OEMs seem to be doing everything they can to earn a spot among the PCCT leaders. GE Healthcare and Canon both acquired PCCT detector makers within the last year and are planning their own PCCT launches, while Philips appears to have ramped up its PCCT R&D.

The Takeaway – PCCT has been viewed as the “future” of CT technology for quite a while, and that future just became a lot closer with last week’s announcement. We’re going to see similar PCCT launches from the other major OEMs, but Siemens Healthineers will enjoy its role as the only player with an FDA-approved PCCT scanner until then.

Get every issue of The Imaging Wire, delivered right to your inbox.

You're signed up!

It's great to have you as a reader. Check your inbox for a welcome email.

-- The Imaging Wire team