Continual Monitoring of Note Quality At Scale, Part 2:
Identifying Signals in User Feedback
To protect patient and clinician privacy, in all our blog posts we use representative synthetic examples.
Our online evaluations run continually on our deployed note generation systems to ensure that we maintain a high standard of note quality after our systems are deployed. Every day, we evaluate thousands of notes across many different quality dimensions. We conduct these evaluations across every specialty we support at our many partner health systems.
These online evaluations involve two complementary types of quality signals: LLM judges (LLJs), which we discussed in a companion post (“Continual Monitoring of Note Quality At Scale: Validating and Using LLM Judges”); and user feedback on specific notes. This post focuses on user feedback.
User feedback carries signal of what clinicians care about in real encounters. The challenge is that this signal is often only implicit (e.g., inferrable from note edits), or skewed due to selection bias in which clinicians provide direct feedback and for which encounters. User feedback also enables us to identify quality dimensions that are important to clinicians but are not yet included among the quality dimensions we already track in existing evaluations.
This post describes how we approach the scientific challenges that come with understanding user feedback at scale. We discuss both how we use user feedback as an independent quality signal of what clinicians care about in documenting a given encounter, and how we identify unmeasured dimensions of note quality.
When clinicians review and revise our notes, we observe direct feedback in the form of free-text comments, as well as implicit feedback in the form of edits to the generated note. Free-text comments often point to the most salient changes or new features a clinician wants. In contrast, edits provide implicit but fine-grained feedback about issues or personal preferences that may be harder to verbalize, or that a clinician may not have time to comment on. These data are valuable signals of what clinicians care most about in their clinical documentation.
Implicit and explicit user feedback falls into many different categories. Some feedback points to note issues, such as incorrect terminology or clinically important missing details. Other feedback reflects a clinician’s personal stylistic preference, such as using dashes instead of bullet points. Clinicians also use the feedback box to request new features or offer praise for a recent release.
We analyze user feedback through a combination of manual review and automated feedback analysis pipelines. This hybrid strategy enables us to process the large volume of feedback signals that we receive each day to surface both aggregate trends, and individual pieces of feedback for manual review.
When reviewing user feedback, we often notice shared themes between different items of feedback, even when the items are phrased differently. For example, two comments from different clinicians may refer to similar underlying misattribution errors:
The same type of feedback might also come through an in-note edit:
Aggregating feedback across similar themes helps our in-house clinicians discern the underlying errors in our systems, and figure out how to fix them. For these themes to be actionable, we design them to match the granularity of what a single system update could address. We developed an error taxonomy in collaboration with our in-house clinicians and partner success team. This taxonomy is designed for the level of specificity that we require at Abridge. For example, whereas many error taxonomies involve an “inaccuracies” category, our analyses warranted more granular categories. Inaccuracy errors related to clinical facts require different fixes from errors related to attribution; among errors related to attribution, errors related to incorrectly labeling the source of information require different fixes from errors related to misattributing a clinician statement to the patient. Our taxonomy currently includes over 50 categories, and we continually update the taxonomy as we identify new themes and new opportunities for improving our systems.
Taxonomy Of Feedback Labels
Each piece of feedback arrives as unstructured text data that needs to be categorized according to our existing taxonomy, or identified as a new feedback category that our taxonomy does not yet cover. To help categorize the tens of thousands of pieces of feedback, we develop automated analysis pipelines.
One immediate challenge is that each comment or edit may reflect multiple problems. For example, a comment might mention multiple issues, in addition to offering praise:
Edits in a note might address multiple problems, and also add information that was not grounded in the conversation
To address this challenge, we decompose each item of feedback into “atomic units.” Each atomic unit reflects a specific motivation for the edit, such as correcting a diagnosis error, adding a missing medication, or adjusting formatting. For example, we could decompose “love the new a&p format. much cleaner. but missed a1c results and misattributed meds update” into
The first stage of our automated pipeline takes as input a comment or edit, and decomposes it into its constituent atomic units.
love the new a&p format. much cleaner. but missed a1c results and misattributed meds update
Mr. Lee is a 58-year-old male with a history of type 2 diabetes and hypertension who presents for follow-up. Patient states Patient’s mother reports that he has stopped metformin approximately one month ago due to GI upset. He has not been checking his blood sugars at home since stopping the medication. He continues to take lisinopril daily without issue.
He also presents with a new musculoskeletal complaint. Patient reports right left knee pain for three weeks, worse with stairs and prolonged standing. Denies swelling. Mild effusion noted on exam. No history of trauma or prior injury to the knee. He has been taking ibuprofen as needed with partial relief.
Once atomic units are extracted, the next stage of our pipeline classifies each unit into the error taxonomy that we developed with our in-house clinicians.
Most units fall under an existing category within the taxonomy. For example, “changed source of medication change details from patient to patient’s mother” would be an instance of the “Misattribution: Incorrect Source Labeling” category.
However, feedback from users sometimes surfaces issues that do not correspond to an existing category. For example, when we initially created our error taxonomy we did not yet have a personalization feature. After personalization was launched, we received error reports as well as items of praise that did not fit into our existing feedback categories, such as
In cases such as these, the classifier flags the feedback for manual review, and these flagged examples feed into continual updates to our feedback taxonomy.
We’ve validated our feedback analysis pipeline against hundreds of expert annotations and found that the classifications agree with experts on the feedback category 78% of the time (compared to a 17% majority-class baseline).
Our automated pipelines run on many thousands of encounters each day to categorize user feedback, direct feedback for manual review, and identify previously untracked feedback categories. Many of the tracked categories reflect distinct error types, allowing us to continually monitor error prevalence, detect regressions in note quality, and verify that expected improvements are reflected in an increase of positive feedback or a decreased prevalence of negative feedback. These pipelines are one of the key signals we rely on in iteratively improving product quality, and monitoring to ensure that quality remains high.