A new study published in European Heart Journal – Digital Health suggests that AI can detect aortic stenosis (AS) in chest X-rays, which would be a major breakthrough if confirmed, but will be met with plenty of skepticism until then.
The Models – The Japan-based research team trained/validated/tested three DL models using 10,433 CXRs from 5,638 patients (all from the same institution), using echocardiography assessments to label each image as AS-positive or AS-negative.
The Results – The best performing model detected AS-positive patients with an 0.83 AUC, while achieving 83% sensitivity, 69% specificity, 71% accuracy, and a 97% negative predictive value (but… a 23% PPV). Given the widespread use and availability of CXRs, these results were good enough for the authors to suggest that their DL model could be a valuable way to detect aortic stenosis.
The Response – The folks on radiology/AI Twitter found these results “hard to believe,” given that human rads can’t detect aortic stenosis in CXRs with much better accuracy than a coin flip, and considering that these models were only trained/validated/tested with internal data. The conversation also revealed a growing level of AI study fatigue that will likely become worse if journals don’t start enforcing higher research standards (e.g. external validation, mentioning confounding factors, addressing the 23% PPV, maybe adding an editorial).
The Takeaway – Twitter’s MDs and PhDs love to critique study methodology, but this thread was a particularly helpful reminder of what potential AI users are looking for in AI studies — especially studies that claim AI can detect a condition that’s barely detectable by human experts.
