perfect on competition day, blind in the clinic

Problem statement

Deep learning models for diabetic retinopathy (DR) screening achieve >95% sensitivity and >90% specificity on curated competition datasets (Kaggle 2015, APTOS 2019), yet real-world deployments consistently underperform. Google Health's deployment in Thai clinics found that 21% of images were rejected as ungradable (vs. <5% in competitions), nurses struggled with the camera equipment, internet connectivity was unreliable, and patients left before receiving results. The gap is not algorithmic — it is a system-level mismatch between competition conditions and clinical reality.

Why this matters

Diabetic retinopathy affects ~100 million people globally and is the leading cause of preventable blindness in working-age adults. Screening by trained ophthalmologists is effective but infeasible at scale in LMICs where ophthalmologist-to-patient ratios can exceed 1:500,000. AI screening promised to close this gap, but the competition-to-deployment failure has slowed adoption by years and eroded clinical trust in AI diagnostics more broadly. The pattern extends beyond DR to other imaging-based screening applications (cervical cancer, skin cancer, tuberculosis).

What’s been tried and why it hasn’t worked

Competition models are trained on high-quality fundus photographs taken by skilled technicians with standardized cameras in controlled lighting. Real-world images come from diverse camera models (desktop fundoscopes, smartphone attachments, handheld devices), are taken by minimally trained staff, and include artifacts from poor dilation, media opacities, and patient movement. Domain adaptation and image quality filtering help but create a tradeoff: strict quality filters reject too many images (defeating the purpose of screening), while permissive filters let through images that generate false diagnoses. Transfer learning on local datasets requires ground truth labels that are expensive to obtain in exactly the settings where AI screening is most needed. The Beede et al. study showed that even when the algorithm performed well, workflow failures (internet outages, nurse unfamiliarity, patient flow disruptions) degraded end-to-end performance.

What would unlock progress

Three complementary approaches: (1) camera-agnostic model architectures that explicitly handle image quality variation as an input feature rather than a rejection criterion; (2) offline-capable deployment systems that don't depend on cloud connectivity for inference; (3) co-design of the screening workflow with actual clinical staff in target settings before, not after, algorithm development. The deeper lesson is that AI medical device development must integrate human factors engineering from the start rather than optimizing accuracy on clean data and hoping deployment works.

Entry points for student teams

A team can reproduce the deployment gap at a desk instead of buying cameras: take a public DR-graded fundus set and apply a simulated acquisition-degradation pipeline — defocus and motion blur, uneven or reduced illumination, JPEG compression, and cropped field of view — then measure how a model trained on clean images loses sensitivity at each degradation level and how many images its quality filter rejects along the way. The DDR dataset (https://github.com/nkicsl/DDR-dataset, MIT-licensed, direct download) and the 45-image CC-BY High-Resolution Fundus set (https://www5.cs.fau.de/research/data/fundus-images/) need no registration; the EyeQ re-annotation supplies Good/Usable/Reject quality labels for 28,792 EyePACS images (https://github.com/HzFu/EyeQ) so the simulated degradation can be calibrated against real clinical quality grades, though those images must be pulled from the Kaggle EyePACS competition (https://www.kaggle.com/c/diabetic-retinopathy-detection, free account plus rules acceptance) and Messidor-2 requires a signed registration form (https://www.adcis.net/en/third-party/messidor2/). Alternatively, a team could design a screening workflow prototype that addresses the specific failure modes identified in the Beede study — image-quality feedback at the moment of capture, offline operation, nurse-friendly interface — and test it in a mock clinic with role-played patient flow rather than in a live one. Skills: machine learning, human-computer interaction, clinical workflow design, mobile development.