health · digital · family: the regulator demands evidence that cannot exist
approvedwithout proof
Over 1,250 AI medical devices cleared by the FDA — nearly half lack public clinical evidence
Problem statement
More than 1,400 AI-enabled medical devices have been authorized for marketing in the United States (FDA AI-enabled device list, March 2026 update). A cross-sectional study of the 903 devices on the FDA's list through August 2024 (Windecker et al., JAMA Network Open, 2025) found that only 505 (55.9%) reported clinical performance studies at the time of authorization, and 218 (24.1%) explicitly stated that no clinical performance studies were conducted. Most (877, or 97.1%) were cleared through the 510(k) pathway, which establishes substantial equivalence to an already-authorized device without a compulsory need for clinical testing: of the reported studies, 38.2% were retrospective, only 8.1% were prospective and 2.4% randomized, and for the remaining 51.3% the design was not specified (Windecker et al.), echoing an earlier analysis of 130 FDA-approved AI devices that found evaluations were overwhelmingly retrospective (Wu et al., Nature Medicine, 2021). The evidence bar is low even for the De Novo pathway that first-of-a-kind devices use: among the 63 novel moderate-risk therapeutic devices authorized via De Novo between 2011 and 2019 that had public decision summaries (65 were authorized in all) — therapeutic devices generally, not AI devices specifically — 31% of pivotal studies (17 studies) failed to meet at least one primary effectiveness endpoint, yet the devices were authorized (Johnston et al., JAMA Internal Medicine, 2020). Clinicians integrating AI tools into diagnostic and treatment workflows often cannot independently assess from the public record whether the device performs as claimed in their specific patient population, clinical setting, and workflow context.
Why this matters
AI-enabled devices are used across radiology (stroke detection, pulmonary embolism triage, fracture detection), cardiology (ECG interpretation), pathology (cancer detection), and ophthalmology (diabetic retinopathy screening) — specialties where clinicians make time-critical decisions based on AI outputs. Evaluating a model at a single site can mask weaknesses in generalizability: in a case study by Wu et al., pneumothorax-detection models trained on one hospital's data lost an average of 0.072 AUC when evaluated at two other sites. If an AI device has a higher false-positive rate in a specific demographic or clinical setting than its clearance data suggests, patients may undergo unnecessary interventions or experience delayed treatment for life-threatening conditions.
What’s been tried and why it hasn’t worked
The 510(k) pathway requires demonstration of substantial equivalence to a predicate device, not independent clinical validation — and for AI devices, the predicate may use an entirely different underlying technology, making the comparison structurally weak. In the same De Novo analysis, 19% of the novel therapeutic devices (12 of 63) were never evaluated in pivotal studies at all (Johnston et al., 2020) — a study its authors restricted to therapeutic devices and describe as not generalizable to diagnostic devices. The FDA's AI/ML action plan and Predetermined Change Control Plan (PCCP) guidance address how algorithms can be updated post-market but do not address the baseline clinical validation gap. Some academic institutions have begun independent validation studies, but these require access to clinical datasets that raise privacy and cost barriers. Much of what manufacturers submit stays out of public view: the FDA says the public summaries linked from its list "do not include most of the information that may be submitted in an application," and Windecker et al. note that additional information may have been given to the FDA without being made public. Reporting guidelines for the early clinical evaluation of AI decision-support systems have appeared (DECIDE-AI among them), but Windecker et al. judge that they "provide only a limited framework for the external validation of AI-enabled tools."
What would unlock progress
A regulatory mechanism that requires minimum clinical evidence thresholds for AI devices affecting clinical decision-making — without requiring the full PMA pathway that would stifle innovation — would close the gap. A standardized, publicly accessible performance reporting framework (building on TRIPOD+AI) that enables clinicians to compare AI devices by clinical setting, patient demographics, and workflow integration would transform purchasing and adoption decisions. Federated validation approaches, where performance is tested across institutional datasets without centralizing patient data, could resolve the privacy-versus-transparency tension.
Entry points for student teams
A student team could build a systematic database of FDA-cleared AI medical devices and their publicly available clinical evidence, extending existing cross-sectional analyses and creating a tool clinicians could use to evaluate devices before adoption. A team with machine learning and clinical skills could conduct an independent validation study of one or more FDA-cleared AI diagnostic devices using institutional or publicly available imaging datasets, documenting performance across demographic subgroups. A design-oriented team could prototype a "nutrition label" format for AI device performance that presents accuracy, sensitivity, specificity, and demographic breakdown in a standardized, clinician-friendly format.
Genes
Structural cousins: stuck the same way, in other fields
Sources
Windecker D, Baj G, Shiri I, et al., "Generalizability of FDA-Approved AI-Enabled Medical Devices for Clinical Use," *JAMA Network Open*, 2025;8(4):e258052. DOI: 10.1001/jamanetworkopen.2025.8052. Accessed 2026-08-20; Johnston JL, Dhruva SS, Ross JS, Rathi VK, "Clinical Evidence Supporting US Food and Drug Administration Clearance of Novel Therapeutic Devices via the De Novo Pathway Between 2011 and 2019," *JAMA Internal Medicine*, 2020;180(12):1701–1703. DOI: 10.1001/jamainternmed.2020.3214. Accessed 2026-08-20; Wu E, Wu K, Daneshjou R, et al., "How medical AI devices are evaluated: limitations and recommendations from an analysis of FDA approvals," *Nature Medicine*, 2021. Accessed 2026-08-20; MedTech Dive, "AI in medtech is booming. Track new devices here." (tracker of the FDA AI-enabled medical device list; more than 1,400 devices authorized as of the FDA's March 2026 update). Accessed 2026-08-20; Applied Radiology, "Gaps in Clinical Data for FDA-Approved AI-Enabled Medical Devices" (news summary of Windecker et al.). Accessed 2026-08-20 go to source 1 ↗ go to source 2 ↗ go to source 3 ↗ go to source 4 ↗ go to source 5 ↗
Working notes
Notes made while writing and checking this brief, shown as written. They use working shorthand.
Primary sources include the FDA Map / Applied Radiology cross-sectional analysis (2024), Nature Digital Medicine analysis of De Novo pathway performance (2024), and the FDA's draft guidance on "Artificial Intelligence-Enabled Device Software Functions" (January 2025). The "regulatory-mismatch" failure tag captures the core structural issue: the 510(k) pathway's legal standard is substantial equivalence, not clinical effectiveness, and changing this requires Congressional action. The "worsening" temporal tag reflects that AI device authorizations are accelerating (the 331 AI devices authorized in 2025 were the most in the agency's history, and the list passed 1,400 with the March 2026 update, per MedTech Dive) while the evidence gap compounds. The "design-proposal" tractability tag reflects that the regulatory and reporting framework changes needed are well-characterized but not yet implemented. Related briefs may include algorithmic bias, clinical decision support, or medical device regulation problems.
Reconciliation 2026-08-20: re-sourced after the citation-drift sweep flagged that the sole citation was a popular Applied Radiology news article while the body carried four precise study statistics. All four numbers were traced to their actual peer-reviewed sources and verified against them: (1) the 55.9%-with-clinical-performance-data figure (505 of 903 devices, plus 24.1% explicitly reporting no studies, 38.2% retrospective / 8.1% prospective / 2.4% randomized designs) is from Windecker D, Baj G, Shiri I, et al., JAMA Network Open 2025;8(4):e258052 (doi:10.1001/jamanetworkopen.2025.8052), a cross-sectional study of the FDA list through 2024-08-31 — the Applied Radiology piece was a news summary of this study, now kept only as supplementary; (2) the "one-third of De Novo devices failed primary effectiveness endpoints" and "one-fifth never evaluated in pivotal studies" figures are from Johnston JL, Dhruva SS, Ross JS, Rathi VK, JAMA Internal Medicine 2020;180(12):1701–1703 (doi:10.1001/jamainternmed.2020.3214) — verified as 31% of pivotal studies failing at least one primary effectiveness endpoint and 19% (12/63) of devices with no pivotal study; the body previously implied these were AI devices, but the Johnston cohort is novel moderate-risk therapeutic devices generally, and the body now says so explicitly; (3) the device count is updated from "over 1,250" to "more than 1,400" per the FDA's March 2026 list update as reported by MedTech Dive's tracker (the FDA's July 2025 update listed 1,247, per the FDA Law Blog — the title's "over 1,250" remains accurate); (4) the retrospective-evaluation claim is additionally anchored to Wu et al., Nature Medicine 2021 (130-device analysis). The first paragraph of these Source Notes predates this reconciliation: its "Nature Digital Medicine analysis of De Novo pathway performance (2024)" does not match any source found — the De Novo numbers come from the 2020 JAMA Internal Medicine research letter above. Genome tags untouched.
Reconciliation 2026-09-30: Whole write-up read against its sources, all fetched 2026-09-30, by a first reader and, the same day, a second reader who went over every repair independently. Read in full: Windecker et al. 2025 (open copy at https://pmc.ncbi.nlm.nih.gov/articles/PMC12044510/; the JAMA page refuses automated requests); Johnston et al. 2020 (https://pmc.ncbi.nlm.nih.gov/articles/PMC7551221/, for the same reason); Wu et al. 2021 (the Nature page is paywalled, but the authors' project page it links, https://ericwu09.github.io/medical-ai-evaluation, leads to their GitHub repository, which carries the manuscript at https://github.com/ericwu09/medical-ai-evaluation/blob/main/docs/paper.pdf; the second reader read it in full); the MedTech Dive tracker and the Applied Radiology article at the links on the Source line; the FDA's own list page, which the text names (https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices), the FDA pages for the two guidance documents named here, and the FDA's January 2021 AI/ML-Based SaMD Action Plan PDF (https://www.fda.gov/media/145022/download); and the FDA Law Blog post of July 29, 2025 that the 2026-08-20 note cites for the 1,247 figure (https://www.thefdalawblog.com/2025/07/fdas-latest-lists-for-digital-health-technologies/), which confirms it. Changes: (1) The text said most devices were cleared through 510(k) "based on bench testing or retrospective dataset performance, without prospective clinical validation demonstrating real-world diagnostic accuracy." Windecker et al. do not say that. They report that 877 of the 903 devices (97.1%) were cleared through 510(k), a pathway that establishes substantial equivalence "without the compulsory need for clinical testing"; the sentence now says that. (2) The same sentence gave only the prospective (8.1%) and randomized (2.4%) shares, which reads as if the rest were retrospective. The paper gives 38.2% retrospective and 51.3% with no design stated; both figures were added. (3) "Among the 63 novel moderate-risk therapeutic devices authorized": Johnston et al. count 65 such devices, of which 63 had public decision summaries and were analyzed; the sentence now says so, and gives the number of studies behind the 31% (17). (4) "Clinicians ... cannot independently assess" is now "often cannot independently assess from the public record"; the paper's wording is that the public information was "frequently insufficient." (5) "No equivalent published accounting yet exists for AI devices specifically" is not in any source; it was replaced with the limitation Johnston et al. state themselves (the study covers therapeutic devices and is not generalizable to diagnostic devices). (6) "Manufacturers consider performance data proprietary and competitive, actively resisting transparency mandates" is not in any source. It was replaced with what the sources do say: the FDA list page states that its public summaries "do not include most of the information that may be submitted in an application," and Windecker et al. note that information given to the FDA was not all made public. (7) "No standardized reporting framework exists ... analogous to STARD ..., though TRIPOD+AI is emerging": neither STARD nor TRIPOD+AI appears in any source, and Windecker et al. say reporting guidelines do exist (they cite DECIDE-AI among others) but "provide only a limited framework for the external validation of AI-enabled tools"; the sentence now says that. TRIPOD+AI is still named in the unlock section as this write-up's own suggestion, not as something the sources propose. (8) In the first paragraph of these notes, "the count has grown from ~100 in 2020 to over 1,250" matched no source: on the FDA list as it stands today, 346 devices carry decision dates up to the end of 2020, and 114 were decided in 2020 itself. It was replaced with MedTech Dive's two figures (331 authorized in 2025, the most in any year; more than 1,400 by the March 2026 update). (9) "When independent academic validation studies have been conducted, they frequently find performance below manufacturer claims in real-world settings" is not in any source. It was replaced with what Wu et al. report: single-site evaluation "can mask weaknesses in generalizability," and in their pneumothorax case study models trained at one hospital lost an average of 0.072 AUC at two other sites. Undone by the second reader: the first reader had replaced "for AI devices, the predicate may use an entirely different underlying technology, making the comparison structurally weak" with a point about recalled devices serving as predicates, on the ground that the claim was in no source read; Wu et al., once read, make exactly this point (image-enhancement devices built on convolutional neural networks "all cite predicates that do not use ML algorithms," so "ML-specific performance defects may be overlooked due to an inadequate regulatory standard set by non-ML predicate devices"), and the original sentence was restored. Confirmed in the sources and left as written: 903 devices through August 31, 2024; 505 (55.9%) with clinical performance studies and 218 (24.1%) stating none; 8.1% prospective and 2.4% randomized; the 130 devices in Wu et al. and their finding that evaluations were overwhelmingly retrospective (126 of 130 "only perform retrospective studies"); 31% of pivotal studies missing a primary effectiveness endpoint and 12 of 63 devices (19%) with no pivotal study; more than 1,400 devices at the March 2026 update; the 1,247 devices on the FDA's July 2025 list, per the FDA Law Blog; the citations on the Source line; the January 2025 date and title of the FDA draft guidance; the PCCP guidance and the action plan's "Predetermined Change Control Plan" being about modifications to devices, with none of the action plan's five actions setting a clinical-evidence requirement; and stroke, fracture, ECG, pathology and ophthalmic devices appearing on the FDA list. Not in any source read and left as written, as background the write-up does not attribute to a source: that academic validation efforts face privacy and cost barriers, and that changing the 510(k) standard "requires Congressional action." Two notes on the rest of the page. The long title's "over 1,250" is lower than the body's "more than 1,400," and the FDA's page now says over 1,600 as of September 2026; the title is still true and was not changed, and its "nearly half" is the 44.1% of the 903 studied devices with no reported clinical performance study, not a share of the whole list. In the first paragraph of these notes, "FDA Map / Applied Radiology cross-sectional analysis (2024)" should be read as the Applied Radiology news article dated May 2, 2025, which summarizes Windecker et al.; no source called "FDA Map" was found. Genome tags untouched.