Applying Large Language Models in Healthcare: Lessons from the Field

When it comes to deploying large language models (LLMs) in healthcare, precision is not just a goal — it’s a necessity. Few understand this better than David Talby and his team at John Snow Labs, a leading provider of medical-specific LLMs. Their work has set a gold standard for integrating advanced natural language processing (NLP) into clinical settings.

The stakes in healthcare are high, requiring accuracy far beyond general-purpose AI models. Misinterpreting a patient record or overlooking a drug interaction can have life-threatening consequences. This article delves into two real-world use cases from John Snow Labs — understanding clinical documents and reasoning over patient timelines — to uncover critical lessons for data professionals working with large language models in healthcare.

Measuring LLM Success

Evaluating large language models in healthcare often starts with:

• Benchmark performance on standardized NLP datasets.
• Peer-reviewed research to validate theoretical accuracy.
• Case studies and real-world deployments are often the toughest but most revealing tests.

While benchmarks and papers offer valuable insights, the true test comes when LLMs encounter the messiness of clinical data — handwritten notes, shorthand, and jargon-laden reports — all under the weight of privacy regulations like HIPAA. Success in these settings demands not only technical excellence but also robust systems for compliance, scalability, and handling incomplete or inconsistent data.

Use Case 1: Understanding Clinical Documents

Extracting Adverse Events from Unstructured Text

Adverse drug events, particularly those linked to opioids, are often underreported. John Snow Labs collaborated with the FDA to tackle this problem by analyzing free-text progress notes to detect opioid-related adverse events.

The technical complexity here is immense:

• Medical notes often describe what didn’t happen, making negation detection crucial.
• Identifying causality — distinguishing whether a condition resulted from a medication or pre-existed — is equally challenging.
• The process involves three distinct NLP tasks: event classification, named entity recognition (NER), and relationship extraction.

GPT Models Fall Short in Information Extraction

Despite the buzz around GPT models like GPT-4, their performance on these specialized extraction tasks lags behind task-specific, fine-tuned models. Peer-reviewed studies have consistently shown that general-purpose LLMs struggle to accurately pull structured information from clinical text.

Social Determinants of Health (SDOH)

SDOH — factors like housing stability or employment — significantly impact health outcomes. In one study, GPT-4 made substantially more mistakes than fine-tuned models when extracting SDOH from clinical notes. The problem? GPT-4 wasn’t trained to handle subtle variations in social context-specific medical text.

Clinical Entity Recognition (CER)

Pathology reports contain critical data on cancer staging, tumor size, and biomarkers. Here too, GPT-4 underperformed — making twice as many mistakes as task-specific models, even with prompt tuning.

Mapping Terms to Medical Codes

Accuracy in healthcare also requires mapping extracted terms to standard vocabularies like ICD-10 or SNOMED. GPT models are not designed for this level of structured data alignment, further reducing their utility in this context.

De-identification

Protecting patient privacy is non-negotiable. John Snow Labs compared GPT models to specialized de-identification systems and found a stark gap in accuracy. Worse, GPT-based solutions were prohibitively expensive due to token-based pricing, making them inefficient for large-scale data anonymization.

Use Case 2: Reasoning About Whole Patient Timelines

Why Timelines Matter

Healthcare decisions rarely hinge on a single visit. Chronic conditions, medication side effects, or treatment responses unfold over months or years. Analyzing a patient’s longitudinal record is often key to spotting patterns.

Montelukast (Singulair) Study

Partnering with Oracle and the FDA Sentinel program, John Snow Labs investigated mental health side effects in children taking Montelukast. The study required mining unstructured notes for neuropsychiatric events — data not captured in standard billing codes. This highlighted the necessity of combining:

• Structured data (medication records, diagnoses)
• Unstructured data (progress notes, psychiatric evaluations)

Building a Unified Patient Timeline

Integrating data across modalities, timeframes, and coding systems into a single patient timeline is essential. This approach allows for natural language queries like, “Has this patient shown signs of depression since starting Montelukast?” — transforming how clinicians interact with data.

Key Learnings from Patient Timeline Systems

1. Multimodal Data Integration is Critical

Relying solely on structured EHR data risks missing up to 80% of patient context. Combining notes, lab results, imaging data, and prescription histories give a fuller picture — vital for accurate risk prediction and decision support.

2. MVPs in Healthcare Aren’t Simple

Healthcare professionals rarely ask one-dimensional questions. Real-world inquiries often require cross-referencing symptoms, medications, and test results across time — demanding systems that handle complex queries spanning different data types.

3. General-Purpose LLMs Aren’t Built for This

While GPT models excel at summarizing and drafting text, they falter in clinical reasoning:

• Accuracy: GPT-4 frequently generates incorrect SQL queries when tasked with answering patient-level questions from structured health data.
• Consistency: Variability in responses undermines clinician trust.
• Speed: Handling large patient histories stretches GPT models, even with long context windows, and demands pre-optimized databases.

Addressing LLM Sensitivity in Healthcare

Tailoring Models to Medicine

General-purpose LLMs often fail to grasp the nuances of medical language — like abbreviations or specialty-specific terms. Developing healthcare-specific models requires:

• Pre-training on clinical data.
• Synthetic example generation to bolster edge cases.
• Careful pruning to eliminate low-quality data.
• Exposure to specialty-specific writing styles (e.g., oncology vs. psychiatry).

The diversity of clinician note-taking styles poses a unique challenge. A robust healthcare LLM must adapt across sub-specialties — something general GPT models struggle with.

Key Takeaways

• Healthcare-specific LLMs outperform GPT models in information extraction and clinical reasoning.
• Combining structured and unstructured data is essential for accurate insights.
• General-purpose models lack the precision and consistency required for real-world clinical use.
• Patient-level reasoning demands specialized solutions, with multimodal data fusion and custom query engines.

Conclusion on Large Language Models in Healthcare

Applying large language models in healthcare requires more than adopting the latest GPT iteration. Success lies in developing specialized models trained on medical data, tested against real-world complexity, and integrated into patient-centered workflows.

The future of healthcare AI will be driven not by one-size-fits-all models, but by collaboration between data scientists and clinicians, tailoring tools to the unique demands of patient care.