In a  study by Dr. Adam Rodman, a Boston-based internal medicine expert, 50 licensed physicians were tested to determine if ChatGPT could enhance their diagnostic capabilities and were less likely to recommend a course of leeches [we made the last bit up].

The surprising results showed that "Doctors who were given ChatGPT-4 along with conventional resources did only slightly better than doctors who did not have access to the bot."

ChatGPT alone outperformed the doctors scoring an impressive average of 90 per cent when diagnosing a medical condition from a case report and explaining its reasoning. In contrast, doctors using the chatbot scored an average of 76 per cent, while those without access to it had an average score of 74 per cent.

The study highlighted not only the chatbot's superior performance but also doctors' sometimes unwavering belief in their diagnoses, even when a chatbot suggests a potentially better one.

The study illustrated a significant gap in doctors' ability to exploit artificial intelligence tools in their work. Many failed to use AI systems' capability to solve complex diagnostic problems and provide detailed explanations for their diagnoses.

Dr. Rodman emphasised that AI systems should act as "doctor extenders," offering valuable second opinions on diagnoses.

"The results were similar across subgroups of different training levels and experience with the chatbot. These results suggest that access alone to LLMs will not improve overall physician diagnostic reasoning in practice," the study said.

The findings are particularly relevant as many health systems offer Health Insurance Portability and Accountability Act-compliant chatbots that physicians can use in clinical settings, often with little to no training on using these tools.