Continual Monitoring of Note Quality At Scale:
Validating and Using LLM Judges

To protect patient and clinician privacy, in all our blog posts we use representative synthetic examples.

Our online evaluations run continually on our deployed note generation systems to ensure that we maintain a high standard of note quality. Every day, we evaluate thousands of notes across many different quality dimensions. We conduct these evaluations across every specialty we support at our many partner health systems.

These online evaluations involve two complementary types of quality signals: user feedback and LLM judges (LLJs). User feedback carries signal of what clinicians care about in real encounters. But the signal is often only implicit (e.g., inferrable from note edits), or skewed due to selection bias in terms of which clinicians provide direct feedback and for which encounters. LLJs, on the other hand, can be applied to any set of encounters, but must be validated to ensure that their judgments align with those of clinicians.

In this post, we focus on the second of these quality signals: LLM judges. We discuss our process for developing and rigorously validating LLJs to ensure they align with expert judgments. The companion to this post (“Continual Monitoring of Note Quality At Scale, Part 2: Identifying Signals in User Feedback”) describes how we use user feedback both to ground our evaluations in what clinician users value, and as an independent validation of our other quality signals.

We monitor note quality across more than 50 dimensions, which are defined and operationalized in collaboration with our in-house clinicians (as described in “What Makes a Great Clinical Note?”). Each day, we monitor these dimensions across thousands of production encounters. This scale of encounters is necessary to perform disaggregated evaluations that assess quality stratified by speciality and health system. While clinician review is our most trusted quality measure, it is infeasible to get high-quality expert scores at this volume and speed required for continual monitoring.

To evaluate at scale, we use LLJs. An LLJ is an evaluator that uses an LLM to score an input according to a set of instructions. LLJs are commonly used to evaluate GenAI outputs, from ambient scribe outputs^{1, 2} to non-clinical applications such as chatbot responses.³ In our evaluations, we use one LLJ for each note quality dimension. For example, one LLJ assesses whether words said by a patient are misattributed to the clinician, one assesses whether words said by the clinician are misattributed to the patient, and yet another assesses whether the note is overly verbose. LLJs apply encounter-agnostic criteria to each note. This enables us to use the LLJs to score new notes in continual monitoring (in contrast to the offline rubric-based benchmarks we describe in “Offline evaluations to improve our systems”).

While LLJs are a powerful tool for scaling up evaluations, they are only useful if they show high agreement with expert judgment. A large body of research has shown that this is not always the case.^{4, 5, 6}

It is therefore important to validate that LLJs align with clinician judgment prior to deploying them for online monitoring. This begins with collecting a dataset of hundreds of notes that experts score on each dimension of quality for which we wish to develop and validate an LLJ. We then select a validation metric—a measure of LLJ-expert agreement tailored to the particular use case. Lastly, we optimize each LLJ using an in-house framework, and verify that the LLJ is sufficiently well-validated for the intended use case according to the validation metric.

Once our in-house clinicians have defined a quality dimension (as described in “What Makes a Great Clinical Note?”), we incorporate the definition into annotation guidelines that we use to collect hundreds of expert annotations from board-certified clinicians and professional medical coders. This dataset is used to develop and validate our LLJs.

When building the dataset, we must first determine the set of notes we will ask experts to annotate. A simple random sample of notes is often insufficient for a good annotation dataset. For example, we often want to detect and fix relatively rare errors. If an error occurs in 0.5% of notes and we randomly sample 400 notes for annotation, we will on average sample only two notes that actually contain the error. This class imbalance means that we will have very little data to understand how our LLJ behaves on notes that contain the error. We use weighted sampling methods to produce datasets that include examples of different expected quality, from different specialties, and generated by different versions of our systems. It is important that our LLJ development and validation datasets span multiple versions of the system we plan to evaluate; we have observed multiple cases in which LLJs have strong agreement with expert annotations for some system versions but not others. Once assembled, these datasets enable us to validate LLJ performance across different types of examples.

Even among domain experts, there may be differences in how they interpret each dimension, so we work with in-house clinicians to write annotation guides that aim to minimize ambiguity while not being overly reductive. Multiple clinicians then score each note according to the annotation guide. To ensure that we have reliable, high-quality annotations, we examine inter-annotator agreement metrics, and also manually review a subsample of annotations.

Once we have collected expert annotations, we use them to determine whether an LLJ can reliably proxy for expert judgement in certain evaluation use cases. Operationally, this requires metrics quantifying LLJ-expert alignment.

Many metrics have been used to evaluate alignment between LLJ and human labels, such as the agreement rate, Cohen’s Kappa, Kendall’s Tau, and the intraclass correlation coefficient (ICC). However, these metrics can be misleading in certain situations. For example, looking at the agreement rate for binary quality dimensions can be misleading if the label distribution is very skewed. If experts score 90% of examples as “pass”, an LLJ could achieve a seemingly high (90%) agreement rate simply by systematically returning “pass” for every example. This LLJ would not be useful in practice: it would score all notes from all system versions as being error-free irrespective of the system’s true error rate. While metrics such as Cohen’s Kappa improve upon the agreement rate metric by adjusting for the likelihood of chance agreement, they have their own limitations.

Metrics commonly used to assess agreement on ordinal (e.g., Likert-scale) and numeric scoring tasks, such as Kendall’s Tau and ICC, can also run into similar limitations. As we now illustrate, we can have a very high value of the metric, but still have the LLJ disagree with the experts on the overarching question of whether one proposed system is better than another.

To see how these metrics can be misleading, consider the following hypothetical example of a dataset where both an LLJ and a clinician scored outputs from two systems, “A” and “B”:

Based on standard agreement metrics, the LLJ appears to be in nearly perfect ordinal agreement with expert scores—much higher agreement than what is typically reported in studies of LLJ-expert inter-annotator agreement.

But if we were to rely on the LLJ to assess which system produces higher quality notes on average, we would arrive at the wrong conclusion. On average, clinicians prefer the original system A to version B by a large margin. Yet despite the high inter-annotator agreement metrics, the LLJ slightly prefers version B to version A.

No evaluation metric is perfect for all situations, so the specific metric we use depends on how we plan to use the LLJ. We therefore do not categorize a given judge as “validated” or “unvalidated” in an absolute sense, but rather for a specific set of use cases. For example, if our goal is to estimate the prevalence of a certain type of error across our notes, we care most about the average bias of the LLJ across our notes. If instead, our goal is to estimate whether one version of a system is better than another, what we care about is the difference of the LLJ bias between the two systems. An LLJ that has an identical upward bias across all system versions will lead us to systematically overestimate the true error prevalence, but the bias, no matter how large, will cancel out when comparing across systems. In practice, we often find that LLJ bias varies at least somewhat across the systems being compared. In such cases, we might seek to bound the bias. For instance, if we are confident that the LLJ is on average within 5% of the true average expert-assessed score across systems, then the LLJ is well-validated enough to differentiate between systems that score 0.8 and 0.92, but not between systems that score 0.8 and 0.82.

LLJ bias is one source of error; sampling variation is another. Because all metrics—including measures of LLJ-expert alignment and LLJ-based measures of system performance—are computed on limited datasets, there is sampling uncertainty around the metric values themselves. Our measurements of LLJ validation incorporate recent research on the statistical validation of LLJs to account for both miscalibration (bias) and sampling variation (variance) that arise throughout the judge validation and system evaluation process.^{7, 8} This requires additional care to adjust for non-uniformity in the sampling design used to select the annotation data.

Creating a good LLJ requires selecting the right LLM, choosing the values for settings such as temperature and effort, and then performing careful prompt optimization. Optimizing all the pieces of an LLJ is time-consuming to do by hand, and prior work suggests that LLMs themselves may be well-suited to this optimization task.⁹

As with any training or tuning procedure, APO is at risk of over-fitting to the dataset used to optimize the LLJ: finding a prompt that achieves high validation with clinician judgment on the training dataset, but which is poorly validated when applied to new notes. To prevent over-fitting, our APO framework first splits the dataset into a “train” set which is used to develop the LLJ, and a “test” set which is used to measure how well-validated the LLJ is. The train set is further split into a “critique” set to determine the failure modes of candidate LLJs, and a “validation” set to choose the best candidates.

The prompt optimization itself is done through iterative beam search. An “Initializer” module creates candidate LLJs with a specific prompt, LLM, and LLM settings. These candidates are passed to an iterative “Critique-and-Edit” beam search stage. In each iteration, each candidate LLJ is run on the critique set, a “Criticizer” module is used to characterize the types of errors that the LLJ makes, and an “Editor” module creates new candidate LLJs. At the end of each iteration, the validation set is used to choose the best candidates to pass to the next iteration.

The final validation of the APO-selected LLJ is measured on a held-out test set. After we’ve optimized the LLJ, we periodically ask in-house clinicians to review LLJ outputs to ensure that the LLJ continues to produce reasonable scores.

This APO process does not always succeed: Sometimes the validation metric remains low even after many iterations, or our in-house clinicians disagree with the LLJ’s scores on a large proportion of examples. When this happens, we explore potential root causes, such as ambiguity in how we defined the dimension, overlooked inconsistency in expert annotation quality, or an APO issue that signals the need for additional human expertise in optimizing the LLJ.

We’ve experimented with thousands of expert annotations and dozens of quality dimensions to refine our modules, metaprompts, and architecture. Over the course of these experiments, we’ve found that APO helps improve validation in the vast majority of cases we have applied it to, while also reducing manual development time by up to 90%.