Comparing ChatGPT and physicians' answers to endometriosis questions on Reddit: A blind expert evaluation.

Résumé

OBJECTIVES: To compare the perceived quality, safety, and relevance of ChatGPT responses to those provided by verified physicians on Reddit, a large online discussion platform, in response to questions related to endometriosis.METHODS: We selected 30 endometriosis-related questions posted on Reddit's r/AskDocs forum, each answered by a verified physician. Using the same question prompts, ChatGPT (GPT-3.5) generated matched-length responses. Responses were anonymized, randomized (A/B format), and assessed blindly by three university-affiliated physicians using a 11-item Likert-scale questionnaire covering medical accuracy, safety, clarity, empathy, and alignment with best practices. Evaluators also indicated which response they considered most pertinent and whether they suspected AI authorship.RESULTS: ChatGPT responses were rated significantly higher than physicians' responses on most items, including medical coherence (mean 3.89 ± 0.89 vs. 3.08 ± 0.92), clarity (3.93 ± 0.95 vs. 3.04 ± 0.99), and empathy (3.91 ± 0.93 vs. 2.76 ± 1.09), all with p-values