(15) Extracting Post-Opioid Overdose Care from ED Encounters Using Large Language Models

Background & Introduction: Naloxone distribution is a cornerstone of overdose prevention and is associated with reductions in opioid-related mortality. Patients discharged from the emergency department (ED) after an overdose face particularly high risk of recurrent overdose and mortality, with a substantial proportion of deaths occurring within 48 hours of discharge. Emergency Department encounters after opioid overdose (OOD) provide a critical opportunity to deliver harm-reduction interventions such as take-home naloxone (THN) and to initiate treatment and provide linkage to ongoing care. However, variation in ED naloxone provision workflows often results in documentation limited to free-text clinical notes, leading to incomplete capture of naloxone provision when relying only on structured data. This is of critical importance as naloxone provision continues to be an evolving measure of quality in the evaluation of emergency care for patients with OUD and OOD. Surveillance efforts often rely on administrative data or pharmacy records, which may undercount THN provision when naloxone is dispensed directly in the ED. Large language models (LLMs) offer a promising approach to automate extraction of these elements from narrative documentation for quality measurement and improvement. We sought to develop and evaluate a LLM approach to identify post-overdose care processes from unstructured ED documentation.

Methods: We conducted a multisite retrospective chart review of ED encounter level notes (clinicians, nursing, social work) for nonfatal opioid overdose across three Connecticut hospitals from January 1, 2022 to December 31, 2023, identified using ICD-10 overdose codes. Using a multi-stage iterative prompt-refinement framework, we developed GPT-4 prompts to extract three outcomes from free text: (1) documentation of substance use/intoxication, (2) treatment discussions/offers (e.g., buprenorphine or methadone, community treatment resources, etc) and (3) THN provision or naloxone prescription at discharge.

From the randommized notes were 108 were allocated for iterative prompt development and evaluation. The development set was further divided into three equal subsets of 36 notes each, which were used sequentially across prompt-refinement cycles. Two development sets (36 encounters each) were used for iterative testing and refinement and a held-out evaluation set (n=36) was used for final assessment. Gold-standard labels were assigned via blinded manual chart review; the evaluation set was double-reviewed, and inter-rater reliability was assessed using Cohen’s kappa. Model performance was evaluated using sensitivity, specificity, positive predictive value (PPV), and F1 score.

Results: Across the three EDs, there were N=[1774] ED encounters that met ICD-10 criteria for opioid overdose/poisoning during the study period. From the eligible pool, we randomly sampled 108 encounters into two development sets (n=36 each) and a held-out evaluation set (n=36).

Across iterative development cycles, prompt refinement improved performance and reduced error patterns. In the held-out evaluation set (n=36), overall performance met or exceeded sensitivity ≥0.96 and specificity of 1.00 across outcomes, with PPV of 1.00 for each. F1 scores were 0.981 for substance use documentation and 1.00 for both treatment discussion and naloxone/THN identification.

Inter-rater reliability for gold-standard labels in the evaluation set was assessed with Cohen’s kappa between the two manual reviewers (κ = 0.87 for substance use; κ = 0.94 for treatment discussion; κ = 0.91 for naloxone/THN).

Conclusion & Discussion: We developed and evaluated a multi-stage prompt-refinement framework using a LLM to extract three clinically meaningful components of ED post-overdose care from unstructured documentation: substance use/intoxication documentation, treatment discussions or offers, and naloxone/THN at discharge. This work demonstrated an iteratively refined, prompt-based GPT-4 method can accurately extract post-OOD processes including THN from free-text ED documentation, supporting scalable surveillance and quality improvement where structured data undercounts key harm-reduction activities. Our results are consistent with a growing body of evidence demonstrating that LLMs and modern NLP approaches can extract substance use–related information from clinical text with high fidelity, including near–state-of-the-art performance for multiple substance categories and nuanced documentation patterns. Further external validation across additional health systems and documentation practices is warranted for future studies.

References: Strang J, McDonald R, Campbell G, et al. Take-home naloxone for the emergency interim management of opioid overdose: the public health application of an emergency medicine. Drugs. 2019;79(13):1395-1418. doi:10.1007/s40265-019-01154-5

Gage CB, Powell JR, Ulintz A, et al. Layperson-administered naloxone trends reported in emergency medical service activations, 2020-2022. JAMA Netw Open. 2024;7(10):e2439427. doi:10.1001/jamanetworkopen.2024.39427

Hwai H, Ho YJ, Wang CH, Huang CH. Large language model application in emergency medicine and critical care. J Formos Med Assoc. 2025;124(8):696-699. doi:10.1016/j.jfma.2024.08.032

Gireesh H, Shukla L, Shivaprakash P, Mukherjee A, Chand P, Murthy P. Language Models for Standardising Clinical Notes and Information Extraction in Addiction Psychiatry-An Empirical Study. Drug Alcohol Rev. 2026;45(1):e70059. doi:10.1111/dar.70059