Tech Blog
Contact Us

Introduction

A few months ago, I had the privilege of sitting down for a filmed roundtable with another Abridge clinician builder and two Abridge product leaders. The topic was Clinical Decision Support (CDS), a new product that surfaces evidence-based information during the patient encounter from trusted sources like the New England Journal of Medicine, JAMA, and other leading medical journals and association guidelines. You can ask it a question at any time during the visit and its response will be informed by the context of the clinical conversation.

This was a full-circle moment for me. It was only a few months prior that we had started work on this new AI feature, and now we were having a very public retrospective discussion (the roundtable has since been published on YouTube). Something was said by one of my colleagues—Chaitanya Asawa, head of AI engineering for the project—that stuck with me:

“When I don’t have those real-world earned insights, I’m always looking for a way to get them.”In this case, I was the “way.”

He went on to explain, "If I'm not my own user, I'm always looking for the gap. What is the gap? What are the insights that I don't have? And I think this is why the clinician-in-the-loop model at Abridge is so relevant."

At Abridge, all new products are developed with clinicians and engineers working hand-in-hand to build something that is both clinically useful as well as technologically sound. So when a doctor uses it in a clinical environment, it just works.

The Feedback Loop Only a Physician Can Close

The fundamental problem with building clinical AI without clinicians is that engineers may not know if the output is correct, whether it’s a clinical note, orders, or the response to a contextual clinical question.

As a clinician, you can probably intuit what healthcare software was built with you in mind, and which wasn’t. An engineering team can make changes to a product that surfaces a response to a clinical question, rerun it, and have no way of knowing whether what they did made it better or worse. Clinically, a response can be beautifully formatted and confidently wrong, and nothing in the code will tell you that if you don’t have the training and experience of a clinician.

When someone building the product knows what the output should look like, the feedback loop is immediate.

I can read a CDS response and know within seconds whether it's what I'd trust at the bedside, and, importantly, whether it’s better or worse than the last round of iteration on the product.

It helps that I've sat on both sides of that desk. I'm a hospitalist, trained in internal medicine and pediatrics, and I still practice. I also studied computer science as an undergrad and worked as an electronic health record developer before medical school. I speak enough of the engineers' language to be useful in their process, and I’ve practiced enough medicine to know when their process has produced something I wouldn't use on a patient.

What ‘Trusted’ Evidence Actually Means

When developing CDS, my expertise extended beyond answering the question, “is this output accurate?” Before even getting to the response to a clinical question, we had to answer a more fundamental question: What does “trusted” evidence actually mean?

Abridge CDS is grounded in a wide range of gold-standard sources, like the New England Journal of Medicine and JAMA. In addition, we’re now incorporating specialty journals and guidelines from organizations like the American Heart Association, American Diabetes Association, American Family Physician, Journal of Clinical Oncology, and Neurology, with more being added regularly.

Within each of these content sources, not all evidence has equal weight for each clinical situation.

Is the right source a specialty society guideline? The primary literature? A review article? An opinion piece? Those carry very different weight at the point of care, and knowing how to rank them, and when, is not a question an algorithm can settle on its own.

This clinical input shaped CDS at every level: which content partnerships we pursued, how the search prioritizes across those sources, and how results are displayed to a working clinician.

The scarce ingredient here is clinical expertise and taste, knowing what the output should look like before it exists.

Where My Own Judgment Was Lacking

Early versions of CDS reflected my personal preferences. I like my clinical information verbose and comprehensive, so that's how CDS responded: thorough and exhaustive. Then we put it in front of the other clinicians at Abridge for internal testing, and they disliked it. Too long, too much, not what a working clinician wants mid-encounter. We made it more concise.

I include this story because it's the most honest illustration of how this actually works. The clinician-in-the-loop model isn't about installing one doctor's instincts into a product. It's a process, with many clinicians testing, disagreeing, and correcting each other, so that no single perspective, including mine, becomes a blind spot.

The same loop extends far beyond our walls. When we launched CDS, it incorporated the patient conversation into its responses, but did not have access to the full electronic health record. Users across the country told us that this would make CDS far more useful. We made it a priority. If your institution uses Abridge, your feedback is part of how this product gets built.

How to Get Started

AI tools have made it so that anyone can build. What they can't supply is what clinicians already have: domain expertise and taste, informed by the lived knowledge of what a meaningful product looks like inside a real healthcare system. Doctors are at an incredible advantage in technology development right now, and I'm not sure enough of us know it.

You don't have to choose one world, either. I still see patients in the hospital, where the impact of my work is direct and human and right in front of me. At Abridge, what I build scales to millions of patient encounters. That’s not to say it isn’t challenging—this is one of the most challenging periods of my career, but it’s also among the most rewarding.

The first step may not require making a drastic change in your life. When your organization evaluates or pilots new software and AI products, get involved. Be the clinician in the room who knows what the output should look like. Many third-party companies that work directly with AI platforms are also looking to leverage clinical expertise. Your experiences and the challenges you encounter caring for patients are valuable assets you can use to inform the next wave of clinical AI innovation.

Share

In this series:

PT.01What It Takes to Be a Clinician-Builder at AbridgeDr. Michal Nedoszytko
PT.02 · CurrentWhat a Clinician-Builder at Abridge Actually Builds: Inside Clinical Decision SupportDr. Chase Yarbrough

Recent posts

Clinical conversations are full of specialized language, multiple speakers, and noisy environments — and transcription errors propagate through the whole pipeline. How we built an LLM-based ASR model, tuned with reinforcement learning and contextual biasing, to recognize medical terminology more reliably…

No single model is the right tool for every clinical task, which is why we use frontier models and specialized in-house models together. Why we build in-house clinical models, why it’s hard, and how we approach the work — so clinicians never have to think about which model is behind their tools…

Beyond offline evaluation and online monitoring, “in-line” evaluations run inside the note generation pipeline itself. How our Editors detect and correct issues before a clinician ever sees the draft — and how we validate the Editors themselves…

User feedback carries signal of what clinicians care about in real encounters — but it arrives as unstructured comments and note edits. How we decompose that feedback into atomic units and classify it against an error taxonomy of over 50 categories…

Our online evaluations run continually on our deployed note generation systems. Every day we evaluate thousands of notes across many quality dimensions — work that relies on LLM judges, and on validating those judges against expert clinician judgment…

All new features and product improvements at Abridge undergo “offline” pre-deployment evaluation. These evaluations are conducted on systems that are not yet deployed with our partners, enabling experimentation without risking adverse impacts…

Note generation was the first product we built at Abridge, the first product with which we earned the trust of our users, and the first product on which we honed our evaluation framework. Here we show how we answer the question of what makes a great clinical note…

Reliable evaluation is a baseline requirement for developing AI products that users can trust. At Abridge, where our products are trusted by tens of thousands of clinicians, we honor and maintain that trust through rigorous evaluation of those products…

A few months ago, I sat down for a filmed roundtable with another Abridge clinician builder and two product leaders. The topic was Clinical Decision Support, a new product that surfaces evidence-based information during the patient encounter…

A patient arrives in the ED with a bleeding scalp laceration. Abridge surfaces a perfect match for a clinical trial evaluating machine-learning-based methods to assess bleeding — on paper, correct; in practice, wrong…

The question isn’t whether clinicians will use AI for clinical guidance. It’s whether that AI has been rigorously evaluated for enterprise readiness…