Beyond the Transcript: How Multimodal AI Scribes Are Learning to Capture the Exam

The blind spot in audio-only documentation

Today's AI scribes are, fundamentally, listeners. They turn the spoken encounter into a structured note, and they do it well. But a clinical visit is not only spoken. A physician points at an X-ray, gestures to the location of pain, examines a rash, demonstrates a range-of-motion limitation, and reviews a screen full of results. An audio-only scribe captures none of that directly. It captures it only insofar as the clinician narrates it out loud.

That narration gap is the next problem the field is trying to solve. Multimodal documentation, scribes that combine audio with vision and structured data, aims to capture the parts of the encounter that happen in silence or on a screen. It is one of the more interesting frontiers in clinical AI, and it comes with real promise and real privacy questions in equal measure.

What "multimodal" means in an exam room

A multimodal scribe brings together more than one type of input:

• Audio, the conversation, as today's scribes already handle.
• Vision, the ability to interpret images: a photograph of a skin lesion, a wound, a document, or a result on a monitor.
• Structured data, direct connections to device readings, vitals, and EHR results so the note reflects measured values, not just spoken ones.

The goal is a note that reflects the whole encounter rather than only its soundtrack. When a dermatologist photographs a lesion, the image and its measured dimensions become part of the record automatically. When a clinician reviews labs on screen, the relevant values flow into the note without dictation. When a physical therapist demonstrates an assessment, the structured findings are captured rather than narrated.

Where it genuinely helps

The value concentrates in visual and procedural specialties where the conversation is the smallest part of the visit.

Dermatology. Lesion documentation is inherently visual. A photo with measured dimensions and location is worth more than any verbal description, and tying it to the note streamlines both the record and longitudinal tracking.

Wound care. Serial photographs that document healing over time, paired with structured measurements, tell a story that text struggles to convey and that justifies the care plan.

Physical and occupational therapy. Range-of-motion and functional assessments are observed, not discussed. Capturing them as structured data reflects the actual work of the visit.

Procedural documentation. When the relevant detail is what was done with the hands rather than what was said, multimodal capture fills a gap that audio cannot.

In each case the pattern is the same: the most important clinical information was never going to be fully spoken, so a tool that only listens was always going to miss it.

The privacy stakes go up

Here is the part that deserves sober attention. Adding a camera to clinical documentation is not a small step. Audio is sensitive; images of patients are more so. Multimodal documentation expands the amount and the sensitivity of the protected health information the system handles, and the safeguards have to expand with it.

Several questions become essential, not optional:

• Consent. Patients consent to being recorded. Visual capture, especially of the body, requires its own clear, informed consent. A general "we use an AI scribe" notice is not enough when the AI is also looking.
• Minimization. Does the system capture only what is clinically necessary, or does it record continuously? The privacy-protective design captures a deliberate image when the clinician chooses, not an ambient video stream of the room.
• Storage and retention. Images are more identifying than transcripts. Where they are stored, how they are encrypted, and how long they are kept all matter more, not less.
• Access control. Who can view captured images, and is every access logged? The audit trail that applies to notes has to apply to images too.

These are the same privacy-by-design principles that govern any handling of protected health information, applied to a more sensitive data type. The technology being impressive does not lower the bar. If anything, it raises it.

Promise versus hype

It is worth being honest about maturity. Audio-based ambient documentation is proven and in daily use. Multimodal documentation is earlier. The vision components are powerful but not infallible, and interpreting a clinical image carries a higher risk of consequential error than transcribing a sentence. A scribe that misreads an image is a more serious problem than one that mishears a word.

That argues for the same discipline multimodal makes more important elsewhere: the clinician reviews and confirms. A multimodal scribe should present what it captured and interpreted, and the clinician should verify it before it counts. The image belongs in the record because the clinician decided it does, with the interpretation the clinician confirmed.

The honest near-term promise is modest: the AI captures the visual and measured parts of the visit so the record is complete, and the clinician confirms the interpretation. That alone beats narrating everything aloud, and it keeps clinical judgment where it belongs. Anyone selling it as autonomous image diagnosis is overselling it.

What to ask before adopting

If you are evaluating a documentation tool that adds visual or structured capture, four questions cut to the heart of it. Is image capture deliberate or continuous? Deliberate, clinician-triggered capture is far more defensible than always-on video. Is consent handled for visual capture specifically? It should be distinct from audio consent. How are images stored, encrypted, retained, and access-logged? At least as strictly as notes. Does the clinician confirm interpreted findings? There should be no path where an AI image interpretation reaches the record unreviewed.

The direction of travel

The single-sense scribe was always a starting point. A clinical visit is multimodal, so documentation that aspires to capture it fully will be too, and for visual specialties that cannot come soon enough. The catch is the one this whole piece keeps circling back to: a camera in the exam room only earns its place if consent, encryption, and clinician review come with it.

---

Transcribe Health is built on privacy-by-design infrastructure, with encryption, access logging, and mandatory clinician review at the center of every workflow. Learn how we handle patient data or try it free.

---

This article is for informational purposes only and does not constitute clinical, legal, or compliance advice. Any documentation tool that captures images or other patient data must be evaluated against your applicable privacy obligations, and AI-interpreted findings must be reviewed by the responsible provider.