The computer will see you now: AI technology produces mixed results

Artificial intelligence (AI) produced a mixed bag of results when screening diabetic retinopathy in a "real-world" clinical setting, researchers say, setting expectations for autonomous grading systems.

Published online in JAMA Network Open, the Australian study described cautious promise for AI-based grading of diabetic retinopathy and subsequent specialist referral after the technology correctly identified a pair of severe disease cases but also misclassified 15 false positives. The study comes at a time when U.S. federal regulators are beginning to clear the way for integrating AI systems into health care, with diabetic retinopathy screening at the forefront.

Diabetic retinopathy, a progressive retinal disease, is the most common cause of vision loss among people with diabetes and a leading cause of blindness among working-age adults. The result of uncontrolled swelling in the retina from leaking blood vessels, diabetic retinopathy can often go unnoticed by patients until vision loss occurs. That's why early detection and intervention through regular, comprehensive eye exams can reduce the risk of blindness by 95%.

Likewise, it's also why some look to AI with potential as another screening option. But enthusiasm surrounding AI isn't the same as demonstrated performance, which is what researchers set out to determine in this most recent study.

According to the study, researchers trained nurses at a primary care practice to use tele-retinal AI screening software in conjunction with a color fundus camera over the course of six months. That system would take 1-3 macula-centered images per eye in about 10-15 minutes before taking another 3 minutes for the AI to generate a grade.

Overall, the AI system screened and graded 193 patients and those results were compared to an ophthalmologist's review. While the ophthalmologist noted 183 patients with no signs of retinopathy, 8 with non-proliferative diabetic retinopathy (NPDR) and 2 with clinically significant diabetic retinopathy, the AI system identified 176 patients with no signs of retinopathy and 17 patients with clinically significant diabetic retinopathy. Although the AI classified all 8 mild diabetic retinopathy cases correctly, and the 2 clinically significant diabetic retinopathy patients, the system classified 15 false positives.

Researchers attribute these false positives to poor image quality and drusen similar in appearance to exudates or sheen reflections around the optic disc, the papillomacular area and the macula, for a positive predictive value of just 12%.

"Our evaluation demonstrates both the promise and challenges of using AI systems to identify (diabetic retinopathy) in clinical practice," the authors note. "Evaluations of AI systems should be conducted in real-world clinical practice before they are deployed widely."

AI devices becoming a norm?

In April, the U.S. Food and Drug Administration (FDA) approved an AI device, called IDx-DR, that analyzes retinal images and provides "a screening decision without the need for a clinician to also interpret images or results." If those images are of sufficient quality for grading, IDx-DR reportedly can detect mild diabetic retinopathy or more and refer those patients to an eye care professional.

That same month, FDA Commissioner Scott Gottlieb noted at the Health Datapalooza in Washington, D.C., that the FDA is working toward updated regulatory framework to keep up with technology. He even went as far as saying, " we expect to see an increasing number of AI-based submissions in the coming years, starting with medical imaging devices, and we're working with experts in the field," quoted The Hill.

Paul Barney, O.D., AOA New Technology Committee chair, has followed the progression of AI screening devices for diabetic retinopathy and sees a "good fit" for such technology triaging diabetic patients into eye care. Comments such as Gottlieb's only firms Dr. Barney's opinion that AI technology in eye care will progressively develop over the next few years.

"Artificial intelligence-based technology is coming, and it will find a place in eye care," Dr. Barney notes. "Doctors of optometry should try to embrace this technology as it develops and, if possible, incorporate it into their practice."

Devices intended specifically for a primary medical office setting, such as the AI technology reviewed in the JAMA study, do present some unique challenges, including the high false-positive turnout noted by the study. In that case, doctors of optometry might avail themselves to such practices by reviewing images prior to specialist referral to curb unnecessary referrals and health care costs, Dr. Barney suggests.

It's important to note, too, that diabetic patients have other eye care issues than simply retinopathy, and though AI technology is progressing to the point of analyzing diabetic fundus examinations, doctors of optometry still are crucial to furnishing that care.

"Those patients frequently have more fluctuation in their refractive error, have more ocular surface issues, and if they wear contact lenses, need closer monitoring of their corneal health," Dr. Barney says. "Again, the proactive optometrist could make their expertise and services for those conditions known to the primary medical provider, and in turn, become part of the medical team managing the diabetic patient's eye care needs."

Read more about optometry's essential, expanding role in diabetes care in AOA Focus and access the AOA's evidence-based clinical practice guideline, Eye Care of the Patient with Diabetes Mellitus—Second Edition.