Offline Evaluations to Improve Our Systems

To protect patient and clinician privacy, in all our blog posts we use representative synthetic examples.

All new features and product improvements at Abridge undergo “offline” pre-deployment evaluation. These evaluations are conducted on systems that are not yet deployed with our partners (not yet “online”), enabling experimentation without risking adverse impacts.

Offline evaluations enable us to compare new system versions to existing ones across a range of quality dimensions. While improvements to existing products often directly target a subset of quality dimensions—such as reducing misattribution rates—it is important to evaluate more broadly to be able to detect and address unexpected regressions along non-focal quality dimensions.

A core signal used in offline evaluations is expert feedback, often coming from clinicians or professional medical coders. We regularly ask experts to assess the quality of proposed system versions along different dimensions (for more info about these blinded head-to-head comparisons, please see “Pioneering the Science of AI Evaluation”). However, this approach is constrained by the cost and speed of human assessments; this impedes rapid system iteration cycles. One way we address this challenge is to conduct research on sample-efficient methods for combining expert annotations with automated annotations from LLM judges (for more details, please see “No Free Lunch: The Risks and Rewards of Incorporating LLM Judges”). The second way is to leverage expertise not for annotation but for developing automated offline evaluations.

This blog post describes one of our automated evaluation strategies, in which we create internal rubric-based benchmarks (for more details about our other offline evaluation strategies, please see “AI Evaluation at Abridge”). Our rubric-based benchmarks consist of encounter-specific, clinician-written rules that a high quality note must satisfy. These benchmarks codify clinicians’ judgment of what makes a high quality note for a given encounter into a set of rules that LLMs can automatically verify for any generated note.  This strategy provides high-quality feedback in a fraction of the time, facilitating rapid product iteration against a high-quality evaluation signal.

Our rubric-based benchmarking approach is similar to those used elsewhere to evaluate our clinical decision support systems, and to approaches in the literature for evaluating clinical chatbot conversations.^{1, 2, 3, 4} To ensure our benchmarks are reflective of the deployment setting, we construct them using de-identified data from Abridge encounters. These examples thus reflect all of the ambiguity and complexity of real-world clinical settings where Abridge is used, and provide the strongest validation of our systems.

The characteristics of a high-quality note are often specialty-dependent. For example, if in an encounter the patient mentions that they have been using Dove soap, this must be documented for a Dermatology encounter, but it should not be documented for an Ophthalmology encounter. Therefore we construct separate benchmarks for different specialties.

To build each benchmark, we enlist clinicians who have expertise in the respective specialty. For each encounter, the clinicians create a set of rules that describe the criteria a high-quality note documenting the encounter must meet. These rules form encounter-specific rubrics that can be re-used to grade notes for the same encounter but produced by different systems. On average, encounters in our benchmarks contain ten rules, but longer and more complex encounters can include up to fifty rules. By using LLMs to automate the grading, this approach allows us to evaluate new note generation system versions without requiring further expert annotation.

For example, we may have an encounter in which the patient presents with abdominal pain:

For this transcript, clinicians may write a rubric with the following rules:

Different candidate systems will produce different notes:

The rubric can be re-used across these candidate systems to detect errors in the notes.

We’ve created rubric-based benchmarks for 20+ specialties and we continue to add new specialties. Collectively, these benchmarks include thousands of encounters. These benchmarks are designed to be challenging for our systems–this ensures that our benchmarks are far from saturated, and allows us to detect improvements to our systems. To ensure that the benchmarks remain challenging, we continually add new encounters and rubrics, focusing on encounters for which earlier system versions have produced unsatisfactory notes.

The specificity of the rubrics in our benchmarks provides interpretable, actionable evaluations of our notes. We can examine the specific rules our notes fail, identify consistent patterns, and apply targeted mitigations.

To enable rapid iteration, we use an LLM-based rule evaluator to assess generated notes against each rubric rather than relying on clinician review. The evaluator takes in the generated note and rubric for each encounter, and produces a pass/fail score for each rule. To ensure this evaluator accurately reflects expert judgments, we validated it on clinician verdicts for 300 rules across 40 notes. The rule-checking model had high agreement with clinicians (0.78 TPR, 0.07 FPR). While the rule-checking model is far from perfect, it provides a directional quality signal that is useful for system development.

Naively iterating against the rubrics in a benchmark risks over-fitting: we might build systems that work very well on this specific set of encounters, but perform poorly when used in production on new encounters. To prevent over-fitting, we split each dataset into a train set for iteratively improving system quality, and a test set for evaluating quality on new encounters. These benchmarks have helped us measurably improve our systems. In one instance, we used one of our benchmarks to iterate on an agent that pulls information from prior notes in order to improve the note for the current encounter. The rubric-based evaluation revealed specific errors in which the agent pulled unnecessary prior content, placed information in the wrong section, and omitted important prior content. Using this offline quality signal, we rapidly iterated on the agent, and in particular iterated much faster than a clinician review between each iteration would have allowed. After these iterations led to a final candidate system, we enlisted clinicians to directly validate the improvements. The clinicians strongly preferred the outputs from the updated agent: in a blinded comparison between the original and updated outputs, clinicians preferred the updated outputs in 91% of comparisons.

Automated offline evaluations enable us to rapidly iterate on our systems to improve quality.  Our internal rubric-based benchmarks are just one piece of our offline evaluation strategy, which also includes clinician review, LLM judges, and adversarial testing. Each of these pieces provides a different view of system quality. Together, they help us improve our systems and prevent errors before they ever reach our partners.