Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum
John W. Ayers, PhD, MA; Adam Poliak, PhD; Mark Dredze, PhD; et al
Who?
Fun study out of UCSD comparing ChatGPT and physician responses on www.Reddit.com/r/askdocs. The researchers found the AI responses were rated as more empathetic. Unfortunately, with each social media posting, the article’s findings become grander and grander. I have been seeing a lot of “ChatGPT more empathetic than doctors” posts lately. Fun study, but it requires a few clarifiers and discussion of its large limitations.
Limitations
For one, I never thought I’d see /r/askdocs used as a comparison group! These are unverified physicians providing medical guidance on a social media forum. This is a much different scenario from in office or MyChart messaging.
Another limitation is the variable for length of responses and is an interesting confounder. For better or worse, ChatGPT generates long responses. I’d be curious to see if when comparing responses of approximate equal word count, if the ChatGPT responses are still seen as more empathetic.
Further Thoughts
Does suggest one way to clear out your inbox. Prompt: “You are a physician, write an empathetic response to an electronic message from a patient that cannot be answered via messaging and warrants an inpatient examination. Also provide instructions on how to schedule an appointment.”
ChatGPT Response
“Dear [Patient’s Name],
Thank you for reaching out to me with your concerns. I understand how distressing it can be to experience unexplained symptoms, and I want you to know that I am here to help you. Your wellbeing is of utmost importance to me, and I appreciate the trust you’ve placed in my care.
Based on the information you’ve shared, it’s difficult to provide an accurate assessment or recommendations through electronic messaging. In order to thoroughly evaluate your situation and provide the best possible care, I would like to see you for an in-person examination. This will allow me to take a closer look at your symptoms, perform any necessary tests, and discuss your concerns in detail.
To schedule an appointment, please call our office at 999-999-9999. Our staff will be happy to assist you in finding a convenient time and date for your visit. If you have any difficulties or need assistance, don’t hesitate to reach out, and we will do our best to accommodate your needs.
Once again, thank you for sharing your concerns with me. I look forward to seeing you soon and working together to address your health issues. Your health and peace of mind are my top priorities, and I am committed to providing you with the care and support you need.
Warm regards,
[Your Name, MD] [Your Specialization] [Your Contact Information]
References:
https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2804309
