The Oct. 8 NEJM article includes a typical case presentation along with a carefully reasoned differential diagnosis from expert diagnostician Gurpreet Dhaliwal of San Francisco Veterans Affairs Medical Center and the University of California, San Francisco, whom Manrai describes as \u201ca real, modern Dr. House.\u201d After that, Dr. CaBot\u2019s differential diagnosis appears. <\/p>\n
Manrai and Buckley were encouraged to see that although Dr. CaBot reasoned through the case differently than Dhaliwal, it reached a comparable final diagnosis.<\/p>\n
From Dr. Cabot to Dr. CaBot<\/p>\n
During graduate school, Manrai became fascinated by how CPCs demystify the process that physicians use to arrive at a diagnosis. They reminded him of the mystery novels he enjoyed growing up. <\/p>\n
More recently, his lab and others have studied the accuracy of AI models for providing patient diagnoses. Manrai wondered whether it was possible to design a system that could go further. <\/p>\n
The core of Dr. CaBot is OpenAI\u2019s o3 large language reasoning model. In building the system, Buckley, who is a Dunleavy Fellow in HMS\u2019 AI in Medicine track, needed to augment o3 with new abilities. <\/p>\n
One is Dr. CaBot\u2019s ability to efficiently search millions of clinical abstracts from high-impact journals, which helps it properly cite its work and avoid factual hallucinations. Dr. CaBot can also search its \u201cbrain\u201d of several thousand CPCs and use these examples to replicate the style of an expert diagnostician in NEJM. The team is working closely with clinician collaborators at Beth Israel Deaconess Medical Center and other Harvard-affiliated hospitals to continue refining the system. <\/p>\n
Dr. CaBot delivers two main products. <\/p>\n
The first is a roughly five-minute, narrated, slide-based video presentation of a case, in which the system explains how it reasoned through the possibilities to come to a diagnosis. The presentations are \u201csurprisingly lifelike,\u201d Buckley said, complete with filler words like \u201cum,\u201d \u201cuh,\u201d and \u201cyou know\u201d as well as colloquial phrases. <\/p>\n
During the team\u2019s demonstrations, \u201cthe realness of the narrated presentation seems to connect with physicians,\u201d Manrai said. <\/p>\n
The other is a detailed written version of Dr. CaBot\u2019s reasoning and diagnosis.<\/p>\n
Taking Dr. CaBot on the road<\/p>\n
The researchers are eager for physicians to engage with Dr. CaBot and provide expert feedback. To this end, they are planning more demonstrations at local hospitals, and they published a paper describing the system<\/a> on a preprint server. They see the NEJM CPC as another opportunity for input.<\/p>\n \u201cDr. CaBot\u2019s AI-generated discussion has not been analyzed for correctness; any factual errors present have been retained so that the reader can observe the strengths and limitations of the system,\u201d the editor\u2019s note on the CPC reads, concluding, \u201cwhether AI has a legitimate use in clinical decision making is up to the reader to determine.\u201d<\/p>\n A narrated video presentation of a challenging medical case produced by Dr. CaBot. Video: Manrai lab <\/p>\n