Elevating Healthcare Documentation: The Significance of SOAP, BIRP, and the Promise of Generative AI

In today’s hectic healthcare environment, precise and comprehensive documentation is necessary to provide seamless service delivery and foster effective communication. Studies have shown that physicians and clinicians spend a significant portion of their workday, dedicated to generating clinical notes post-patient encounters. It can range between one to three hours a day on average. According to the American College of Physicians report and a national study conducted in the US, it is evident that documentation burden may increase clinician burnout, medical errors, threats to patient safety, and inferior documentation quality.

In this post, we explore how we can use generative AI (Artificial Intelligence) to produce clinical notes from patient visits and enhance the overall quality of documentation. To demonstrate how simple it is to design such a solution, we have shared a sample of Python[1] code.

Before we begin, let us understand what is generative AI, or genAI as we call it in short. Gen AI refers to artificial intelligence(AI) systems that are capable of generating new content, such as text, images, audio, or other media, based on the training data they have been exposed to. These large models have the ability to combine and recombine elements from their training data in new and innovative ways, enabling them to produce creative and desired outputs. In the context of natural language processing(NLP), models like GPT-3 (Generative Pre-trained Transformer 3)[1] can generate human-like text, including articles, stories, scripts, and even computer code, based on prompts or human inputs. On the other hand, models like DALL-E[1] and Stable Diffusion[2] can create realistic and imaginative images by combining and synthesizing elements from their training data based on input text or prompts. These models are powerful, as they hold tremendous potential to transform clinical documentation practices, leading to improved efficiency, accuracy, standardization, and quality of care in healthcare settings.

In a healthcare setting, SOAP[1] (Subjective, Objective, Assessment, Plan) notes are typically written after each patient encounter or therapy session. It allows professionals to methodically capture the patient’s subjective complaints, objective findings from examinations and tests, their assessment or diagnosis, and the plan of care.

Take for example the following SOAP note, which was generated with ChatGPT[1]–

Subjective (S):
Patient complains of persistent abdominal pain in the lower right quadrant, rated 7/10 on the pain scale. Reports nausea and lack of appetite for the past two days.

Objective (O):
Vital signs: Temperature 99.5°F, Blood Pressure 120/80 mmHg, Pulse 80 bpm, Respiratory Rate 18 breaths/min. Abdominal examination reveals tenderness and guarding in the right lower quadrant.

Assessment (A):
Diagnosis: Suspected appendicitis based on clinical presentation and physical examination findings.

Plan (P):

1. Order abdominal ultrasound to confirm diagnosis.
2. Initiate IV fluids and keep NPO (nothing by mouth).
3. Administer IV antibiotics (ceftriaxone and metronidazole) for presumed appendicitis.
4. Consult general surgery for possible appendectomy.
5. Provide analgesics (acetaminophen) for pain relief.
6. Schedule follow-up in 24 hours for reassessment.

The subjective section records the patient’s chief concerns, symptoms, and medical history in their own words. While the objective section documents measurable data, such as vital signs, physical examination findings, and test results. The assessment section synthesizes the information gathered into potential diagnoses or differential diagnoses. Finally, the plan outlines the treatment approach, including medications, follow-up instructions, referrals, and any additional tests or procedures.

Although SOAP notes are widely recognized, BIRP[1](Behavior, Intervention, Response, Plan) note format has also gained traction in recent years especially with the mental health documentation process, emphasizing a more patient-centered approach to documentation. BIRP stands for “Bringing (B) Illness/Injury (I) into Relation (R) with the Patient (P)” and encourages healthcare professionals to document the patient’s unique circumstances, including their personal, social, and cultural contexts, and how these factors may impact their health condition and treatment plan. Following is a BIRP note example which was also generated using ChatGPT[1].

Behavior(B):
During the session, the patient exhibited signs of anxiety, including restlessness, fidgeting, and avoiding eye contact. The patient reported feeling overwhelmed by work-related stressors and expressed difficulty sleeping due to racing thoughts.

Intervention(I):
Utilized deep breathing exercises and mindfulness techniques to help the patient relax and reduce anxiety levels. Discussed cognitive-behavioral strategies to challenge negative thought patterns and promote positive coping mechanisms.

Response(R):
The patient responded positively to the relaxation exercises, reporting decreased muscle tension and a sense of calmness. Engaged actively in discussing stress management strategies and expressed motivation to practice techniques outside of therapy sessions.

Plan(P):
Agreed to continue practicing relaxation techniques daily and to monitor stress levels using a mood journal. Scheduled a follow-up appointment in two weeks to assess progress and further refine coping skills. Provided resources for additional support, including stress management workshops and relaxation apps.

Despite the clear benefits of the notes, you can see that generating such a document from patient and clinicians sessions can be time-consuming, and often stressful after patient care.

A cross-sectional national level study analyzed office-based physician responses to the 2019 National Electronic Health Records Survey in the US, and based on the response it highlights that — “Assuming a 5-day work week and 47-week work year, we estimated US physicians spent 125 million hours documenting outside office hours in 2019”[1]

Approach

By using natural language processing (NLP) and automatic speech recognition (ASR) technologies, an AI could transcribe the entire conversation as it happens. It would be able to understand the context and structure of what’s being said. Then, by using advanced prompting techniques or prompt engineering, the AI could take that transcript and turn it into a draft note, organizing the subjective information from the patient, the objective findings from the doctor’s exam, the assessment of the potential diagnosis, and the plan for treatment.

For BIRP notes specifically, the AI could also incorporate details about the patient’s personal context — their living situation, cultural background, and other factors that provide a more holistic picture, and do all these by protecting the patient’s personally identifiable information[1] (PII) or Protected Health Information[1] (PHI). Instead of the doctor or nurse having to type up notes after the fact, they have a solid first draft ready to review and finalize. This could save a tremendous amount of time and effort spent on documentation. The clinicians could focus more on the actual patient interaction.

Solution

The intention of the solution is not to endorse any specific product but to demonstrate how you can easily develop such a solution using the capabilities of machine learning and generative AI. Ultimately, the effectiveness of a production-grade implementation will be influenced by various factors such as accuracy, latency, cost, and others.

The next section outlines our methodical approach to solve the problem –

Step 1 is to get the transcript of the meetings between a patient and a clinician that were recorded as audio and/or video. Here, we’re going to assume that the patient interaction was recorded for future reference. However, there are other methods you can use to produce transcription in real-time as well. We used OpenAI’s Whisper-small model from Hugging Face for transcription generation. You have the flexibility to pick and choose your own transcription model using fully-managed AI services like Amazon Transcribe, Amazon Transcribe Medical, Amazon HealthScribe, Deepgram’s speech-to-text, or even choose open-source models such as vosk.

Step 2 We utilized Anthropic’s Claude 3 Sonnet model[1] via Amazon Bedrock as the large language model(LLM). We are using advanced prompting techniques to use the transcript from the Whisper model to generate the clinical notes via the LLM. You can access an LLM directly from third-party (3P) model providers like Anthropic or OpenAI, via cloud providers like AWS, Azure, GCP, or via Hugging Face.

How do you get started?

To begin, clone the GitHub repo or download it to your local computer. Create and activate a virtual environment, and then pip install -r requirements.txt to install the required packages.

Execute the run.py script found at the root of the project’s directory, and input the file path for the audio or video file. You can specify if SOAP notes are needed (yes/no), if BIRP notes are needed (yes/no), and optionally provide the path to an instruction file for additional guidance to the LLM.

In the run.py script, the generate_transcript_from_file(…) function from transcript_generator.py is invoked to produce the transcript utilizing the OpenAI/Whisper model. During this process, the function also examines whether SOAP notes, BIRP notes, or both have been chosen, and accordingly generates the appropriate prompt using prompt_generator.py.

Let us now look at the soap notes generator prompt to understand how you can use a LLM to generate clinical notes. But, before we do that let us quickly understand what is a prompt?

A prompt for a LLM is a text input provided to the model to guide its generation of text or completion of a task. The prompt serves as a starting point or instruction for the model, helping it understand the context and generate relevant output. It helps steer the model’s output in a desired direction based on the input provided by the user or system. Anthropic provides comprehensive documentation and guidance on how to prompt in their documentation.

The important points are that the model’s output is dependent upon the clarity of the instructions it receives. Claude constructs responses one character at a time based on the prompt and its previous outputs. The prompt length is limited by the model’s context window size. You can think of the context window as the “working memory” of the model. A larger context window enables the model to comprehend and address more intricate and lengthy prompts. Conversely, a smaller context window may restrict the model’s capacity to manage longer prompts effectively or sustain coherence throughout extended conversations. You can refer to the documentation to find out more about context windows and how it varies for each model. For additional learning, you can also refer to Anthropic’s prompt library, which offers a collection of prompts for various use cases, covering both work and play, and also explore the experimental metaprompt tool, which can help generate prompts based on user-provided guidelines.

Now coming back to the prompt we used, let us dissect each section thoroughly and go over it in detail. Understanding each section and its impact on the output of a LLM is important.

1. Specifying the SOAP format:

• Example: “Please write a SOAP note for a patient encounter based on the following information and the provided transcript below….”
• Importance: Defines the SOAP format and its components (Subjective, Objective, Assessment, Plan) for documenting patient encounters.
• Impact: Guides the LLM to structure the SOAP note according to professional standards, ensuring completeness and organization in the output. Poorly structured SOAP notes may fail to effectively communicate important information about the patient’s condition, history, and treatment plan and can lead to misunderstandings and errors in patient management. Incomplete documentation can hinder continuity of care, compromise patient safety, and increase the risk of medical errors. It may fail to meet documentation standards mandated by healthcare regulatory bodies, leading to potential legal liabilities or disciplinary actions.

2. Highlighting the comprehensive note requirement:

• Example: “Please follow the SOAP format and provide a comprehensive yet concise note..”
• Importance: This emphasizes the need for a thorough yet concise SOAP note that covers all relevant aspects of the patient encounter.
• Impact: It encourages the LLM to be succinct yet to the point in each part. It fosters effective communication, efficient workflow, legal compliance, and patient-centered care. It balances the need for comprehensive documentation with the imperative of brevity, ensuring that critical information is communicated clearly and efficiently for optimal patient outcomes.

3. Additional Information to include:

• Example: “Additional information you may want to include: Patient’s name, age, and gender; Relevant medical history, including allergies and current medications; Timing and duration of symptoms…”
• Importance: This instruction lists additional details to enhance the completeness and accuracy of the SOAP note. You can use it to add any organization specific information, or add instructions to modify the output format.
• Impact: It guides the LLM to incorporate relevant patient information, medical history, exam findings, and treatment rationale into the generated note, ensuring relevance and clinical utility. You may also decide to anonymise certain information based on your requirements.

4. Maintaining confidentiality:

• Example: “Remember to maintain confidentiality by avoiding the use of identifiable patient information.”
• Importance: This serves as a reminder to maintain patient confidentiality and privacy. It is a critical factor and could be a determining factor to choose your LLM. There are certain LLMs which are highly efficient in this task like the Anthropic’s Claude model family. It is built with responsible AI in mind.
• Impact: It encourages the LLM to generate a de-identified SOAP note that respects patient privacy and complies with healthcare regulations and ethical standards.

5. Including the transcript for context:

• Example: “<Transcript>\n\n{transcript}\n\n</Transcript>”
• Importance: This is the basis for generating the SOAP note, ensuring alignment between the note and the patient encounter details.
• Impact: It allows the LLM to analyze the provided information in the transcript and incorporate relevant clinical findings into the SOAP note, enhancing accuracy and relevance purely based on the transcript. It will help to mitigate hallucination.

6. Example SOAP note for reference:

• Example: “<example> … </example>”
• Importance: The <example></example> offers a sample SOAP note as a reference for the LLM to follow when generating the note from the provided transcript. You can provide additional instruction or formatting that you might be looking for in this section. You can even instruct the LLM to generate the output in JSON format, or in any particular format which can be consumed by other business processes downstream.
• Impact: It provides a model for structuring the SOAP note and formulating assessment and treatment plans, promoting consistency and professionalism in the output.

Now let us look at a real-life example of a case study: ‘First session with a client with symptoms of social anxiety (CBT model) — link) shared by Judith Johnson on YouTube[1]0

This is an example used for educational purposes only. It aims to illustrate some of the triggers, thoughts, feelings, and responses related to problematic social anxiety. The video represents the first 12 minutes of an initial therapy session. The video features an actor (Malak El-Gonemy) playing the character of Hannah. The dialogue is real and unscripted, reflecting a therapeutic exchange that takes place naturally during a session. Judith Johnson is conducting the session as a qualified Clinical Psychologist. Her efforts and agreement to share the videos publicly have allowed us to hear what a real session may sound like.

The following section takes a closer look at the output when we use the session video and run our scripts to produce the SOAP and BIRP notes. Let’s take a close look at each section, beginning with the SOAP note that the LLM produced.

Subjective (S):

Hannah is a 25-year-old female presenting with complaints of anxiety difficulties that started around 5–6 years ago. She recently moved out of her parents’ home for the first time, which has exacerbated her anxiety symptoms. Hannah reports feeling increasingly isolated, staying in her house most of the time, and avoiding social situations and work opportunities due to anxiety. She is a freelance photographer and finds it challenging to interact with people during photoshoots, leading her to turn down job offers. Hannah experiences physical symptoms like sweating, shaking hands, and feeling flushed when in anxiety-provoking situations. She worries about being judged negatively by others and feeling like “a freak.”

Objective (O):

Vital signs are within normal limits. Hannah appears anxious during the interview, fidgeting in her chair and avoiding eye contact at times. No abnormal physical findings are noted.

Assessment (A):

The primary diagnosis is Generalized Anxiety Disorder (GAD), characterized by excessive anxiety and worry about various situations, along with physical symptoms of anxiety. The recent life stressor of moving out of her parents’ home has likely contributed to the worsening of her symptoms. Social Anxiety Disorder is also a possible diagnosis given her significant fear and avoidance of social and performance situations.

Plan (P):

1. Initiate cognitive-behavioral therapy (CBT) to help Hannah identify and challenge her negative thought patterns and develop coping strategies for anxiety-provoking situations.
2. Consider prescribing a selective serotonin reuptake inhibitor (SSRI) such as sertraline or escitalopram for pharmacological management of anxiety symptoms.
3. Encourage Hannah to engage in regular exercise and practice relaxation techniques like deep breathing or progressive muscle relaxation.
4. Recommend joining a support group or seeking online resources to connect with others experiencing similar challenges.
5. Schedule a follow-up appointment in 4 weeks to monitor progress and adjust the treatment plan as needed.

The process of obtaining the transcript and generating the SOAP and BIRP notes for the model took approximately 84 seconds(approximately). Upon reviewing the SOAP note, it becomes evident that it serves as a solid foundation for clinicians. Rather than starting from square one, clinicians can readily utilize this note, allowing them to efficiently craft more precise and comprehensive documentation to enhance patient care.

Let’s now examine the BIRP note that was created below –

Behavior:

During the session, the client appeared visibly anxious, describing symptoms such as feeling hot, sweaty, and having difficulty breathing. They reported avoiding social situations and job opportunities due to fears of being negatively judged or perceived as incompetent by others. The client also mentioned engaging in safety behaviors, such as tightly gripping their camera and wearing layers of clothing to conceal signs of anxiety.

Intervention:

Utilized active listening and open-ended questioning to explore the client’s thoughts, emotions, and behavioral patterns related to their anxiety. Employed cognitive-behavioral techniques, such as identifying and challenging irrational beliefs and negative self-talk. Introduced relaxation strategies, including deep breathing exercises and progressive muscle relaxation, to help manage physiological symptoms of anxiety.

Response:

The client engaged actively in the therapeutic process, providing detailed accounts of their experiences and openly discussing their thoughts and behaviors. They demonstrated a willingness to learn and practice coping strategies, such as relaxation techniques, to better manage their anxiety symptoms. However, the client expressed concerns about the potential long-term impact of their anxiety on their personal and professional life.

Plan:

Scheduled a follow-up session to continue exploring the client’s anxiety triggers and reinforcing the use of cognitive-behavioral techniques and relaxation strategies. Recommended keeping a thought record to identify and challenge distorted thinking patterns. Discussed the potential benefits of exposure therapy to gradually face anxiety-provoking situations in a controlled and supportive environment. Provided resources on anxiety management, including self-help materials and support groups.

As a starting point, the BIRP note seems to be thorough and well-structured. What’s particularly intriguing is its ability to maintain confidentiality by refraining from disclosing any personal or healthcare information in the response.

Furthermore, generative AI models can be trained on vast amounts of data, enabling them to suggest relevant diagnoses, treatment plans, and follow-up instructions based on the patient’s symptoms, medical history, and examination findings. This can assist clinicians in making more informed decisions and ensuring comprehensive care plans. While the integration of generative AI into healthcare documentation holds immense promise, concerns surrounding data privacy, model bias, and the need for human oversight and validation must be carefully considered.

In conclusion, using generative AI to generate medical documentation offers a viable way to expedite the documentation process. Healthcare professionals can reduce administrative workloads and focus more on patient care by utilizing this cutting-edge technology. However, to preserve patient confidentiality and treatment quality, it is critical to approach this innovation cautiously and make sure that ethical considerations, legal compliance, and data privacy are given top priority. The responsible and significant deployment of generative AI in healthcare documentation will require cooperation between technology developers, healthcare practitioners, and regulatory agencies as we continue to research and innovate.

Thanks for reading.

For those eager to delve into our research paper, feel free to explore it through this arxiv link.

Written by Wrick Talukdar and Anajanava Biswas

While Wrick and Anjanava are AWS employees, opinions and statements on this blog are their own.

You can connect with Wrick and Anjan to stay updated on the latest trends, insights, and developments in the dynamic landscape of artificial intelligence and machine learning.

Originally posted on OpenDataScience.com

Read more data science articles on OpenDataScience.com, including tutorials and guides from beginner to advanced levels! Subscribe to our weekly newsletter here and receive the latest news every Thursday. You can also get data science training on-demand wherever you are with our Ai+ Training platform. Interested in attending an ODSC event? Learn more about our upcoming events here.