What Makes a Great Clinical Note?
Note generation was the first product we built at Abridge, the first product with which we earned the trust of our clinicians, and the first product on which we honed our evaluation framework. This post introduces a series deep-diving into note generation evaluations. In this first post, we show how we answer a central question introduced in our first post—“what do we evaluate”—for note generation. Answering that question requires first answering a more fundamental question: what makes a great clinical note?
This question does not have a single straightforward answer. A clinical note is a piece of unstructured free text and there is no ground-truth “best clinical note” to compare to. A note also has multiple audiences using it for different purposes, further complicating matters. A doctor who has been seeing the same patient for many years might value a concise note with a brief history, while a doctor seeing a new patient for the first time might value a detailed note with a thorough history. A great cardiology note might look different than a great pediatric note, and a hospital billing specialist will be looking for completely different things in a note than a clinician.
In natural language processing, tasks like note generation are known as “abstractive summarization tasks.” An abstractive summarization task is one where the model generates a summary in its own words, as opposed to generating a summary by excerpting the available documents. Common metrics for abstractive summarization look at word overlap, like ROUGE and BLEU, or at semantic overlap, like BERTScore. But using these metrics to determine if a clinical note is an effective summary of the transcript of an encounter can be misleading in our setting. Two clinical notes might use very different words to describe the same information, leading to very different ROUGE and BLEU scores when compared to the transcript, but be of the same quality for clinical documentation purposes. Including irrelevant details from the patient-provider conversation will increase semantic overlap, driving up BERTScore, but lead to a note that clinicians have to comb through for the information they actually need. At the same time, these metrics are often insensitive to negation and other clinically significant single-word differences between texts. And no abstractive summarization metric will flag an error such as a clinician’s words being attributed to the patient. All of this means that these metrics have large blind spots and are poorly suited to measuring clinical documentation quality.
Other metrics specific to clinical summarization like the Physician Documentation Quality Instrument (PDQI-9), Provider Documentation Summarization Quality Instrument (PDSQI-9), and SCRIBE provide structured rubrics for measuring note quality. These instruments ask raters to provide Likert scores for several axes of quality, like comprehensibility and succinctness. They are straightforward to administer and applicable to a variety of summarization tasks, but give too coarse-grained a signal on note quality for our purposes. For example, feedback from our clinicians at partner systems and consultation with our in-house clinicians has shown that clinicians generally prefer notes that use active voice over notes that use passive voice, but this preference is not reflected in existing rubrics.
We instead build our evaluation of note quality around a large number of granular, interpretable, and actionable axes. These axes are chosen by collaborating with our in-house clinicians, analyzing the edits clinicians make to the notes we generate, and collecting on-the-ground feedback from our partner success teams. For example, a partner success team might repeatedly hear from clinicians that our notes are misattributing statements made by one speaker to another speaker, and bring this to our attention internally. We would then identify the specific notes the clinicians flagged to better understand the misattributions present in them, and also search for other notes with possible misattributions to determine whether this is a dimension of quality we should be evaluating.
Once we decide to evaluate a specific dimension of quality, we iterate with experts to define it as precisely as possible. This process includes examining our production data to surface edge cases of notes that debatably have the issue, grouping them into thematic categories, and asking our clinician scientists for a verdict on whether each category should be considered to have the issue or not. Similarly, we collect expert annotations of whether or not notes have the particular issue, and ask our clinician scientists to give a final ruling on cases where annotators disagreed with each other, further sharpening the definition.
We often find through this process that a quality definition that seems clear enough at the outset turns out to be very ambiguous when applied to real data.
For the case of misattributions, we found through the process above that defining misattribution broadly as attributing the words of one speaker to a different speaker led to poor specificity in our measurements and low actionability. To define a misattribution more precisely, we separate out several different kinds of misattribution:
Each of these categories must be further refined. For example, if a clinician observes a symptom that the patient then confirms and the clinical note documents that the patient reports that symptom, should we flag this as a misattribution? There are also specialty-specific edge cases to adjudicate. In a pediatric visit, if a parent says that an infant has an ear infection and the note summarizes this as the patient reporting an ear infection, is this a case we should count as a misattribution error? These are the kinds of boundaries we must delineate to arrive at a definition of note quality that is granular enough to be actionable.
Another dimension of note quality that seems simple at first glance but requires care to define effectively is the use of medical terminology rather than lay language. We cannot simply replace every instance of lay language with a corresponding technical term. One acceptable use of lay language is when patients are reporting their own symptoms in non-technical terms. Another is in cases where the use of a technical term implies a formal diagnosis: the word “flu” should be used when this is what the patient self-reports, while “influenza” should be used only when the patient has received this diagnosis. Our definition of whether or not a note uses appropriate medical terminology must reflect these subtleties.
By repeatedly identifying issues based on partner feedback and iterating with in-house clinicians and annotators to define them precisely, we have constructed 50+ different dimensions of note quality, including:
We believe that generating great clinical notes requires evaluating all of these, and more. Our list of dimensions is not exhaustive: even if two notes are both satisfactory along all of the 50+ dimensions we have enumerated, clinicians may uniformly prefer one to the other because of some other dimension of note quality we have not yet defined and measured. There are also factors specific to a particular encounter, such as capturing a specific important detail of patient history, that cannot be represented by a general dimension of note quality.
These remaining gaps are why we continue to grow our list of measurable dimensions and to supplement them with other kinds of evaluations, such as those described in “Offline Evaluations to Improve Our Systems”.
Once we have defined these actionable measures of note quality, the next step is to act on them. In the next two posts, we’ll turn to the question of how we evaluate, how we measure whether a system is improving along one of these dimensions, and how we measure these dimensions for the notes we generate in production.