(58) Assessing Readability and Stigma of ChatGPT’s Opioid Use Disorder FAQs

Background & Introduction: Artificial intelligence (AI) chatbots such as ChatGPT are increasingly used for patient education, including topics related to opioid use disorder (OUD). Given the heightened sensitivity of OUD, educational materials must balance depth of information, readability, and non-stigmatizing language. However, AI-generated content may inadvertently increase text complexity and include stigmatizing terms, potentially undermining its utility. This study aimed to assess whether ChatGPT-generated responses to common OUD FAQs (1) achieve appropriate readability and understandability, (2) adhere to recommended non-stigmatizing language guidelines, and (3) provide sufficient yet accessible detail for patient education.

Methods: Using multiple search engines, we screened the top 40 webpages for OUD FAQs based on: (1) reputable affiliation (e.g., CDC), (2) at least five OUD-related FAQs, and (3) original, non-duplicated content. After excluding pages with repetitive questions, 50 unique OUD FAQs were identified. ChatGPT (model GPT-4o, 2024-11-13) generated answers to each FAQ. We then used online software (e.g., online-utility.org) to compare character count, word count, sentence count, lexical density, and readability metrics (Gunning Fog, SMOG, Flesch Reading Ease) between ChatGPT’s responses and the published FAQs. Statistical comparisons employed a two-tailed Student’s t test (significance set at p< 0.05). Stigmatizing language was detected using Voyant Tools, applying the National Institute on Drug Abuse (NIDA) “Words Matter” guidelines. Any response with ≥1 stigmatizing term was labeled as stigmatizing, enabling systematic analysis of term frequency, context, and distribution.

Results: ChatGPT’s answers were significantly longer than existing FAQ responses (mean 1,435 vs. 379 characters). Readability indices indicated ChatGPT-generated text was more complex (Gunning Fog: 18.7 vs.16.3; SMOG: 16.1 vs.14.7; p< 0.001) and had a lower Flesch Reading Ease score, where higher scores mean easier to read (19.7 vs. 34.9; p< 0.001). Stigmatizing language appeared in 49 of 521 ChatGPT sentences (9.4%) compared with 14 of 233 existing FAQ sentences (6.0%).The frequency of stigmatizing language did not significantly differ between ChatGPT (0.98; 95% CI, -1.29-3.25) and published FAQs (0.28; 95% CI, -0.42-0.98).

Conclusion & Discussion: While ChatGPT can rapidly generate OUD FAQs with comprehensive detail, its outputs are more difficult to read and sometimes include terms that could potentially stigmatize people with substance use disorders. Although it offers detailed information, refinements are needed to improve readability and reduce stigma in AI-generated patient education materials. Limitations include potential AI involvement in original FAQs, a relatively small set of 50 questions that may limit generalizability, and the use of a single GPT model version. Future research with larger, more diverse question sets and updated AI models is warranted to refine AI-generated patient education materials for sensitive conditions like OUD.

References: National Institute on Drug Abuse. (n.d.). Words matter: Preferred language for talking about addiction. Retrieved January 22, 2025, from https://nida.nih.gov/research-topics/addiction-science/words-matter-preferred-language-talking-about-addiction

Kelly, J. F., Wakeman, S. E., & Saitz, R. (2015). Stop talking ‘dirty’: Clinicians, language, and quality of care for the leading cause of preventable death in the United States. American Journal of Medicine, 128(1), 8–9. https://doi.org/10.1016/j.amjmed.2014.07.043

Soliman, P., Amaefuna, I., Gully, B. J., & Haass-Koffler, C. L. (2024). Evaluating the readability of online patient-facing resources for alcohol use disorder. Alcohol, 115, 1–4. https://doi.org/10.1016/j.alcohol.2023.08.012

Bellanda, V. C. F., Santos, M. L. D., Ferraz, D. A., Jorge, R., & Melo, G. B. (2024). Applications of ChatGPT in the diagnosis, management, education, and research of retinal diseases: A scoping review. International Journal of Retina and Vitreous, 10(1), 79. https://doi.org/10.1186/s40942-024-00595-9

Samaan, J. S., Yeo, Y. H., Rajeev, N., Hawley, L., Abel, S., Ng, W. H., et al. (2023). Assessing the accuracy of responses by the language model ChatGPT to questions regarding bariatric surgery. Obesity Surgery, 33(6), 1790–1796. https://doi.org/10.1007/s11695-023-06413-2