Building LLM-based Speech Recognition for Clinical Conversations

This post is part of a larger series on Abridge’s AI models. You can read the first post in the series here.

Conversations are complicated. We engage in them all day, every day, but when you write or map them out, they hardly resemble the structured information you might find in a spreadsheet, or even the simplified dialogue you might read in a book or see in a movie. They are full of false starts, half-said words, and physical gestures that augment our messages.

Clinical conversations are far more complex than day-to-day speech, full of specialized language and packed with context crucial to patient health. Often, these conversations occur with multiple speakers, in multiple languages (sometimes with translators), and in environments confounded by ambient sounds. They also contain the information needed to generate accurate clinical documentation to move care forward.

Consequently, converting hours of spoken dialogue into reliable structured records remains a challenging and high-stakes problem. Ambient AI systems, such as Abridge, depend on high-quality speech recognition to transform these conversations into transcripts that power downstream tasks, including clinical note generation. Automatic Speech Recognition (ASR) therefore serves as a foundational component of the clinical AI pipeline, and developing highly accurate and domain-specialized ASR models remains a critical problem for healthcare applications.

Recognition errors introduced during transcription can propagate through the entire pipeline, compromising the accuracy and completeness of generated documentation. Therefore, clinical ASR systems must accurately recognize specialized medical terminology, newly introduced medication names, multilingual conversations, and acoustically challenging environments and must perform reliably across the wide range of speakers and settings encountered in clinical care. In this post we describe our work on developing an LLM-based clinical ASR model tailored to clinical conversations. We show that the resulting model provides a strong foundation for clinical speech recognition, delivering meaningfully improved transcription quality and robustness in challenging healthcare scenarios.

Transforming that kind of messy, high-stakes spoken language into text is the job of ASR. Conventional ASR systems, including Connectionist Temporal Classification (CTC), Recurrent Neural Network Transducer (RNN-T), and attention-based encoder-decoder models, have achieved impressive performance by learning acoustic and linguistic patterns jointly from large-scale paired speech-text data. However, because these models are optimized specifically for speech recognition, their ability to leverage broad world knowledge, perform complex reasoning, or adapt to new domains through natural language instructions remains limited.

Over the past few years, Large Language Models (LLMs) have achieved remarkable progress across a broad spectrum of natural language processing tasks. Motivated by these advances, recent studies [1–4] have extended LLMs to ASR by incorporating acoustic information as prompts for transcription generation.

In this paradigm, an audio encoder first extracts acoustic representations and projects them into the LLM’s text embedding space. These projected audio embeddings are then integrated with text embeddings and provided to the LLM as input prompts for generating transcriptions. By combining robust acoustic representations with the strong contextual understanding and generative capabilities of LLMs, LLM-based ASR systems have demonstrated superior performance over conventional ASR models.

The model architecture of a Speech LLM consists of three main components that bridge acoustic speech signals and a pre-trained LLM:

We train the model in two stages: supervised fine-tuning (SFT) followed by reinforcement learning (RL).

This first stage is about breadth. The SFT stage teaches the model to transcribe speech accurately across a broad distribution of audio. We construct a curated training mixture that includes:

All training data has been de-identified and does not contain personally identifiable information. We apply LoRA (Low-Rank Adaptation) to both the LLM layers and the projection module, while keeping the audio encoder frozen. This choice leverages the strength of the pre-trained audio representations and reduces the risk of forgetting general acoustic features.

This second stage is about precision. Following SFT, we further optimize the model using Group Sequence Policy Optimization (GSPO) [5], a variant of Group Relative Policy Optimization (GRPO) that performs importance sampling at the sequence level rather than the token level, resulting in lower-variance policy gradient estimates.

During RL training, LoRA is applied only to the LLM, while the audio encoder and projection module remain frozen. For each audio utterance, the model generates multiple candidate transcriptions, which are evaluated by a reward function and used to update the policy.

The reward function combines several complementary objectives:

To preserve the transcription quality learned during supervised fine-tuning, RL optimization includes a KL-divergence penalty with respect to the SFT checkpoint, preventing the policy from drifting excessively from the supervised baseline.

The GSPO training framework can be optionally extended to support contextual biasing, which prepares the model by enabling it to leverage a user-provided list of domain-specific terms at inference time to improve transcription accuracy. During training, approximately 75% of utterances are augmented with a bias list inserted into the user prompt before the audio embeddings. Each bias list contains between 5 and 30 terms sampled from a curated vocabulary of medical concepts. The list consists of a mixture of terms extracted from the reference transcript and distractor terms that do not appear in the audio. Each reference term is included independently with a fixed probability, preventing the model from relying exclusively on the bias list while encouraging it to continue recognizing unhinted medical terms directly from the audio. To support contextual biasing, the reward function is augmented with two additional objectives:

At inference time, users may optionally provide a list of relevant terms (e.g., medications, procedures, or diagnoses) as part of the prompt. Because the model is explicitly trained to distinguish between relevant hints and distractors, it learns to utilize the provided context to improve recognition accuracy while avoiding the hallucination of terms that were not spoken.

We evaluate our model on an internal benchmark of real-world clinical conversational speech spanning a range of recording conditions and acoustic environments. The benchmark is designed to reflect the variability encountered in real-world clinical conversations and is annotated with high-quality reference transcripts and medical terminology.

We report two metrics:

Evaluating both WER and MTMR provides a more comprehensive assessment of clinical ASR quality. While WER measures overall transcription accuracy, MTMR directly quantifies the model’s ability to recognize medically salient terminology, which is often more critical for clinical documentation than small differences in general transcription accuracy.

The figure below compares our model against a range of open-source baselines and proprietary API-based speech recognition systems.

Our model achieves the lowest WER as well as the lowest MTMR score across all evaluated systems. The results highlight a key insight: general-purpose speech APIs, despite their scale, frequently miss rare medical terminology. By combining a curated clinical data mix with a reward signal that explicitly optimizes transcription quality, our model closes much of this gap. We also evaluated MedASR [6], a model specifically pre-trained for medical dictation and transcription. MedASR exhibits a 50% higher WER than the worst-performing baseline. This is likely because MedASR was trained primarily on radiology, internal medicine, and family medicine dictation rather than cross-specialty clinical conversations. This further highlights that strong performance requires not just “medical” data, but specifically clinical conversation data.

We also evaluate contextual biasing on a benchmark of 50 newly released drug names, representing out-of-vocabulary medications. Success is defined as correctly transcribing the target drug name anywhere in the transcript. Without biasing, the model correctly transcribes only 60% of the drug names, reflecting the difficulty of recognizing previously unseen compounds. With 10-term biasing—where the target drug name is included alongside 9 randomly sampled rare medical terms—recall increases to 94%. With 30-term biasing, recall remains high at 86%, with the slight reduction likely attributable to the target drug name being diluted among a larger set of distractor terms. These results demonstrate that contextual biasing is highly effective for recognizing rare, out-of-vocabulary drug names. Moreover, injecting rare medical terms as bias prompts does not meaningfully degrade transcription quality on either clinical or general speech.

Using these training methods, we’ve managed to achieve models that more accurately capture clinical conversations, meaning better downstream documentation for clinicians and their patients.

This work also represents an important step toward applying modern LLM post-training methods to clinical speech recognition. Future work includes exploring tighter integration with downstream clinical documentation tasks.

This post is part of a larger series on Abridge’s AI models. You can read the first post in the series here.