![SonomaEye-full-color-vector[2].png](https://static.wixstatic.com/media/bed00b_da20b9cc704044b0b048ac8ef9190470~mv2_d_3024_2195_s_2.png/v1/fill/w_177,h_123,al_c,q_85,usm_0.66_1.00_0.01,enc_avif,quality_auto/SonomaEye-full-color-vector%5B2%5D.png)
Thank you for your submission!
Can AI replace the Uveitis Specialist? The Role of LLM’s in Uveitis Diagnosis and Treatment.
Taylor Crook
Presenter:
Taylor Crook1, Pooya Khosravi1,2, Jordan Tang1, Paul Zhou MD1, Samir Nazarali MD1, Olivia L Lee MD1
Authors:
Affiliation:
1 Department of Ophthalmology, School of Medicine, University of California, Irvine, CA 92697, USA
2 Department of Computer Science, Donald Bren School of Information and Computer Sciences, University of California, Irvine, CA 92697, USA
Purpose: Incorporating artificial intelligence into patient care has increased at unprecedented levels. The nationwide shortage of uveitis specialist's limits access to subspecialty care for uveitis patients, particularly in non-urban areas. This study looked at whether we could train an open-source large language model (LLM), Llama 3.2, to generate accurate diagnosis and clinically appropriate plan of care for patients presenting to a tertiary care uveitis clinic.
Method: A retrieval-augmented generation (RAG) system was trained in diagnosis and management of uveitis using published ophthalmology reference materials (AAO BCSC Section 9 and The Wills Eye Manual). When given the patient's clinical data, including history, exam findings and available test results, the LLM then generated a diagnosis and plan of care. Deepseek-R1, another LLM, acted as a judge and generated assessments with the ground truth based on appropriateness, harmfulness to patients, clinical context, and accuracy on a scale of 1-5.
Results: After training, Llama 3.2 was applied to 39 eyes (20 patients) of uveitis derived from de-identified medical records while excluding diagnosis, assessment and plan; it achieved a mean score of 3.33 ± 0.93, 3.28 ± 0.94, 2.97 ± 1.2, and 3.03 ± 0.93 for accuracy, appropriateness, harmfulness, and clinical context. Some of the irrelevant LLM derived plans included treatments for systemic problems without attention to ophthalmological problems. This shows that the model was able to learn about uveitis treatment and diagnosis but is still far from the skill and expertise of a human uveitis specialist.
Conclusions: These results provide us with initial findings that it is possible and feasible to teach an LLM to read clinical notes, labs, and the history of a patient and provide a diagnosis with treatment plans. While the LLM achieved promising results, further refinement is needed to improve its overall performance. Future studies should focus on further fine-tuning the LLM's performance, exploring strategies to improve its generalizability across diverse patient populations, improving prompt engineering, and investigating its potential as a valuable tool for clinicians in uveitis care.
Attachments:
