Stanford researchers have identified potential for large language models to improve medical diagnoses and clinical reasoning, according to a new study detailed by Adam Hadhazy for Stanford HAI.
The research team presented a series of cases based on actual patients to ChatGPT-4 and to 50 physicians from Stanford University, Beth Israel Deaconess Medical Center, and the University of Virginia. Half of the physicians used conventional diagnostic resources, while the other half had access to ChatGPT as a diagnostic aid.
According to the study, ChatGPT scored approximately 92 on its own—equivalent to an "A" grade. Physicians in both the non-AI and AI-assisted groups earned median scores of 74 and 76 respectively, indicating they did not express as comprehensive a series of diagnoses-related reasoning steps.
Study co-lead author Ethan Goh, a postdoctoral scholar in Stanford's School of Medicine and research fellow at Stanford's Clinical Excellence Research Center, told Stanford HAI, "Our study shows that ChatGPT has potential as a powerful tool in medical diagnostics, so we were surprised to see its availability to physicians did not significantly improve clinical reasoning."
Jonathan H. Chen, Stanford assistant professor at the School of Medicine and the paper's senior author, explained to Stanford HAI, "What is very possible is that once a human feels like they've got a diagnosis, they don't 'waste time or space' on explaining more of the steps for why. There's also a real phenomenon that often human experts cannot themselves explain exactly why they made correct decisions."
The study, published in JAMA Network Open, found that physicians with ChatGPT access completed their individual case assessments more than a minute faster on average than those without the AI tool.
As reported by Stanford HAI, the findings have led to the formation of ARiSE (AI Research and Science Evaluation), a bi-coastal AI evaluation network involving Stanford University, Beth Israel Deaconess Medical Center, the University of Virginia, and the University of Minnesota.