Comparison of quality, empathy and readability of physician responses versus chatbot responses to common cerebrovascular neurosurgical questions on a social media platform.

Résumé

BACKGROUND: Social media platforms are utilized by patients prior to scheduling formal consultations and also serve as a means of pursuing second opinions. Cerebrovascular pathologies require regular surveillance and specialized care. In recent years, chatbots have been trained to provide information on neurosurgical conditions. However, their ability to answer questions in vascular neurosurgery have not been evaluated in comparison to physician responses. Our study is a pilot study evaluating the accuracy, completeness, empathy, and readability of responses provided by ChatGPT 3.5 (Open AI, San Francisco) to standard specialist physician responses on social media.METHODS: We identified the top 50 cerebrovascular questions and their verified physician responses from Reddit. These questions were inputted into ChatGPT. Responses were anonymized and ranked on a Likert scale for accuracy regarding neurosurgical guidelines, completeness and empathy by four independent reviewers. Readability was assessed using standardized indexes (Flesch Reading Ease, Flesch Kincaid Grade, Gunning Fox Index, Simple Measure of "Gobbledygook" (SMOG) Index, Automated Readability Index and Coleman Liau Index).RESULTS: Responses provided by ChatGPT had significantly higher ratings of completeness (median (IQR) 3 (2-3) vs. 2 (1-3) and empathy 4 (3-5) vs. 2 (1-3) compared to physician responses, respectively (p