Does Your AI Scribe Get Better Over Time? What Actually Improves

The claim every vendor makes

Almost every AI scribe vendor will tell you their product gets better over time. The phrasing varies: "continual learning," "adaptive customization," "your personal AI scribe gets to know your style," "platform improvements." The implication is that what you experience in month 12 will be meaningfully better than what you experience in month 1.

This claim is partly true. It's also partly marketing. Sorting which is which matters because it affects how much weight you give to "the platform will improve" when evaluating a vendor that isn't quite good enough today.

This article is the honest read on what actually improves, what doesn't, and what to expect over a 12 to 24 month adoption arc.

What actually improves

Three categories of improvement are real:

1. Your customizations and templates. This is the largest source of improvement and it's mostly under your control. Over the first 3 months, you customize templates, add vocabulary, set preferences. After 3 months, the AI's output reflects your style much more than it did on day one. This isn't the AI learning; it's you teaching the platform's customization layer. Real, durable, and significant.

2. Platform-wide model improvements from the vendor. The vendor periodically updates the underlying model. New version handles edge cases better, transcribes specific accents better, generates better plan sections. These improvements happen 2 to 4 times per year for active vendors. The cumulative effect over 12 to 18 months is real, sometimes equivalent to a generation jump.

3. Platform feature additions. New integrations, new templates, new workflow features. These aren't AI improvements per se but they make the platform more useful over time. A practice using the platform actively gets to benefit from these as they're released.

These three categories combined are why a long-tenured customer's experience is genuinely better than a new customer's. Not because their specific AI has been individualized to them in some deep way, but because the customizations and the platform have both evolved.

What doesn't improve as much as marketing implies

A few things vendors imply will improve that don't really, or improve only marginally:

The AI does not learn from your individual edits in real time. When you correct the AI's mishearing of a specific drug name, the platform may add it to your custom vocabulary (improvement under your control), but the underlying model doesn't "learn" from your correction the way some marketing implies. The next provider in the practice making the same correction also has to make it; the global model isn't personalized at runtime.

Accuracy on rare conditions or rare medications usually doesn't improve much over time at a given vendor. Long-tail conditions and rare drug names require training data the vendor has to source. A vendor that's weak on (say) cystic fibrosis vocabulary in month 1 is usually still weak in month 12 unless they've made a specific investment in that area.

Diarization in difficult environments doesn't improve much over time. A vendor that struggles with multi-speaker pediatric visits, or with phone interpreter visits, or with noisy environments, usually has those weaknesses persist. The fixes are platform-level and require deliberate engineering, not "the AI learning."

Integration depth doesn't improve unless the vendor invests in it. Your EHR integration in month 1 is roughly what it is in month 12, unless the vendor releases an explicit integration update.

The "we'll get better" claim during evaluation

A common vendor pitch during evaluation: "I know our [specific feature] isn't where you'd like it, but we're improving rapidly; by the time you renew, this will be solved."

The honest response: maybe. Some vendors do follow through; some don't. Evaluating this claim requires looking at:

The vendor's release cadence. Have they shipped meaningful updates in the past 6 months? Quarterly is healthy. Twice a year is okay. Once a year or less is not enough to support "we're improving rapidly."

The vendor's track record on specific commitments. If they promised something to a previous customer 6 months ago, did they ship it? Check with reference customers.

The vendor's product team size. A small company with 3 engineers can't move on multiple fronts at once. A larger company with a dedicated product team has more capacity. Neither is automatically better; smaller teams sometimes move faster on specific issues.

The specific roadmap commitment. "We're working on it" is vague. "It's in the next release in March" is specific. The latter is what to ask for.

If a vendor's "we'll improve" claim is your reason for signing up despite specific weaknesses, you're betting on something you can't fully verify. Try to negotiate contract terms that protect you if the improvement doesn't materialize (shorter initial term, performance milestones, exit options).

The expected improvement curve

For a typical practice adopting an AI scribe in 2026, the realistic improvement arc:

Month 0 to 1: The slump. Accuracy and time savings are below where they'll settle. Most of the improvement in this period is in your workflow, not the platform.

Month 1 to 3: Customizations build. Templates fit your style. Vocabulary fits your specialty. Time savings reach steady state. The AI itself hasn't changed much; your relationship to it has.

Month 3 to 6: First platform update probably arrives. Generally an improvement; sometimes neutral, occasionally a small regression on something. Most providers don't notice meaningful change.

Month 6 to 12: Two to three platform updates. Cumulative effect is noticeable. Specific weaknesses you had at month 3 may have been addressed. Or may not have been; depends on the vendor.

Month 12 to 24: More platform updates, sometimes a major model upgrade. The platform you're using at month 24 may feel like a different tool than the platform at month 0. This is the genuine "AI getting better over time" effect.

Month 24 onward: Steady state with periodic updates. Significant new capability appearances are rarer once the platform has matured. Vendors at this stage often differentiate on features, not core accuracy.

What you can do to accelerate improvement

A few practices accelerate the improvement curve for their own use:

Active customization. The providers who customize aggressively get better outcomes faster than providers who accept defaults. The 30-minute customization session in week 1 and the periodic refinements over the first months compound.

Active feedback to the vendor. Specific, actionable feedback to vendor customer success helps them prioritize your concerns. Vague complaints ("it's not great with our specialty") don't move anything. Specific feedback ("medication X is consistently misheard as Y in our notes; here are three examples") can.

Beta participation. Some vendors offer beta access to new features. Practices that participate get earlier access and have more influence on the feature's final form. The downside is occasional instability.

Vendor relationship investment. A practice that the vendor sees as a strategic customer gets more attention, more responsive support, and more influence on the roadmap. This isn't unfair; it's how vendor relationships work. Be a good customer; the vendor responds in kind.

When to switch vendors instead of waiting

If the platform's specific weaknesses are real impediments to your practice and the vendor isn't shipping improvements that address them, switching is sometimes the right call.

Signals that switching beats waiting:

The weakness affects your high-volume visit types. A vendor that struggles with your most common visit type doesn't improve fast enough to fix the day-to-day pain. Switch.

The vendor's product team has been quiet for 6+ months. No releases, no roadmap updates, no engagement on your concerns. Often a sign of bigger problems at the vendor.

Other vendors offer what you need today. If a competitor handles your specific issue well now, switching is faster than waiting for your current vendor to catch up.

Pricing has gone up without commensurate improvement. If the renewal quote is higher and you haven't seen platform improvements that justify it, the value calculation has shifted.

See switching AI scribe vendors mid-year for the migration playbook.

What the vendor knows that you don't

Vendors know things about their own product trajectory that don't show up in marketing materials. A few things to ask vendors directly during contract negotiation:

What's in your roadmap for the next 6 months that you can share? A vendor with a real roadmap will share at least some of it. A vendor with no plan will hedge.

What's the biggest known weakness of your platform today? A vendor who can name a specific weakness honestly is one who knows their product. A vendor who insists their platform has no weaknesses is one who hasn't done the introspection.

What's the most common customer complaint? Patterns of complaint tell you about the platform's real friction points. Vendors who track this and can name it are honest; vendors who say "no specific complaints" are not being candid.

What was the most significant improvement you shipped in the last 12 months? This question separates vendors who can articulate their progress from vendors who are running in place.

These questions are awkward during a sales conversation but they produce information that customer references and demo experiences won't surface.

The honest summary

AI scribes do get better over time, but most of the improvement comes from:

• Your own customizations
• Platform feature additions
• Periodic model updates

The amount of improvement in 12 to 24 months can be significant, but it doesn't justify signing up with a vendor whose current product isn't acceptable. "We'll improve" is real for active vendors but isn't a substitute for "we're acceptable today."

Practices that pick well-positioned vendors and then engage actively (customize, give feedback, participate in beta features) get the most improvement. Practices that pick poorly-positioned vendors hoping they'll catch up usually wait longer than the patience available.

The honest evaluation is: what works for me today, and is the vendor moving in a direction that suggests they'll continue to be a fit in 24 months? Both questions matter; neither alone is enough.

---

For the related question of when to formally re-evaluate the vendor relationship, see six months in: what to re-evaluate. For the migration path if the vendor isn't keeping pace, switching AI scribe vendors mid-year covers the transition.