Multimodal AI deep dive: Google AMIE vision latest diagnostic dialogue with skin photos

Multimodal AI deep dive: Google AMIE vision latest diagnostic dialogue with skin photos is one of the clearest signals yet that “chatbots” are evolvin...

Kevan Shah · · 10 min read

Multimodal AI deep dive: Google AMIE vision latest diagnostic dialogue with skin photos is one of the clearest signals yet that “chatbots” are evolving into stateful, evidence-seeking agents—systems that can notice what’s missing, request the right data at the right time, and update conclusions as new information arrives. In recent research updates, Google DeepMind and Google Research describe how AMIE (a research medical AI system) has been upgraded from text-only consultation to multimodal diagnostic dialogue, where it can ask for, interpret, and reason over medical images such as skin photos, as well as artifacts like ECGs and clinical documents during a live-style conversation.
Sources: https://research.google/blog/amie-gains-vision-a-research-ai-agent-for-multi-modal-diagnostic-dialogue/ and https://arxiv.org/html/2505.04653v1

For technology leaders and developers, AMIE’s upgrade isn’t just a healthcare story—it’s a blueprint for the next generation of AI assistants: systems that manage uncertainty, drive the conversation toward evidence, and integrate multiple data modalities. For organizations building AI products (or automating workflows with tools like n8n), the design patterns here—state-aware dialogue, mid-conversation data requests, and dynamic hypothesis revision—map directly to real-world enterprise automation problems.

Below is a technical, practical deep dive into what Google reported, what it doesn’t yet prove, and how to translate these ideas into safer, more reliable AI-driven workflows.

Multimodal AI deep dive: Google AMIE vision latest diagnostic dialogue with skin photos

What exactly changed: from “answering” to “asking for evidence”

According to Google’s research blog and the accompanying paper, the key update is not merely that AMIE can “see.” It’s that AMIE can:

That matters because many real workflows fail not due to lack of intelligence, but due to lack of the right inputs at the right time. In other words: the system’s ability to ask is as important as its ability to answer.

The “state-aware dialogue framework”: a structure for trustworthy conversations

Google reports AMIE uses a state-aware dialogue framework with explicit phases, including:

This is a major architectural signal. Instead of treating a conversation as a flat stream of messages, AMIE is described as tracking “where it is” in a clinical workflow, so it can decide when it needs additional evidence—like a skin photo—to close gaps.

In enterprise terms, this looks like a hybrid of:

  • A finite-state machine (phases),
  • A policy for what to request next (information-seeking behavior),
  • And a reasoning loop that updates as new multimodal data arrives.

If you’re building AI copilots for support, operations, compliance, or IT, this is the difference between:

  • “Here’s a best-effort answer,” and
  • “Here’s what I believe, here’s what I’m missing, and here’s what I need from you to proceed.”

Dermatology as the “killer demo”: why skin photos change the dialogue

In Google’s research blog, the example behavior is explicitly dermatology-oriented. If a patient reports a rash, AMIE can prompt for skin photos, including requesting additional views or better lighting when needed, and then incorporate those images into the ongoing diagnostic conversation.
Sources: https://research.google/blog/amie-gains-vision-a-research-ai-agent-for-multi-modal-diagnostic-dialogue/ and https://arxiv.org/html/2505.04653v1

Dermatology is a natural testbed for multimodal diagnostic dialogue because:

  1. The visual signal is often decisive (morphology, distribution, color, borders).
  2. Remote-first workflows already exist (teledermatology).
  3. Image quality varies widely—so the system must learn to request better evidence, not just accept what it gets.

The interesting design implication: AMIE isn’t just consuming an image; it’s managing an evidence acquisition process. That same approach generalizes to many business workflows:

  • “Please upload the invoice PDF.”
  • “I need a screenshot of the error.”
  • “Attach the contract clause you referenced.”
  • “Provide the ECG / lab values / form fields.”

Beyond photos: ECGs and clinical documents in the loop

The paper also states AMIE was tested with artifacts such as skin images, ECGs, and clinical documents in simulated telemedicine-style consultations.
Source: https://arxiv.org/html/2505.04653v1

Additional reporting likewise notes AMIE can incorporate ECGs and clinical documents into its reasoning.
Sources: https://www.news9live.com/technology/artificial-intelligence/google-amie-ai-can-see-medical-images-2846337 and https://arxiv.org/html/2505.04653v1

This is crucial because “multimodal” in the enterprise rarely means just images. It means combining:

  • Unstructured text (chat, notes),
  • Semi-structured docs (PDFs, forms),
  • Images (photos, scans),
  • Time series or signals (ECG-like traces),
  • And metadata (timestamps, device type, location, user role).

The hard problem is not simply model capability—it’s orchestration: ingesting inputs, validating quality, maintaining provenance, and ensuring the assistant’s reasoning stays anchored to the evidence.

Reported results: strong performance in controlled evaluations (and what that implies)

Google’s research post and third-party reporting describe benchmark performance that is compelling—in controlled settings.

One report states that multimodal AMIE achieved better or similar results to physicians on several clinical benchmarks, including top-3 diagnostic accuracy of 0.65 vs. 0.59 for human doctors in one comparison.
Source: https://www.news9live.com/technology/artificial-intelligence/google-amie-ai-can-see-medical-images-2846337

Another report discussing the same research claims AMIE reached 66% top-1 diagnostic accuracy in a teledermatology study on 963 cases involving 26 common skin conditions, compared with 63% for board-certified dermatologists, 44% for primary-care physicians, and 40% for nurse practitioners.
Source: https://www.digitalhealthnews.com/google-s-chatbot-can-now-diagnose-from-photos

From a technology strategy perspective, these numbers underscore a key point: multimodal+dialogue can outperform unimodal point predictions because the system can interactively reduce uncertainty. It’s not only “model accuracy,” it’s workflow intelligence—asking the right follow-ups and requesting the right evidence.

But it’s equally important to interpret these results correctly.

The biggest caveat: research-only, simulated/controlled settings—not routine care

Every reputable source in this cluster emphasizes the limitation: AMIE is still research-only and evaluated in simulated or controlled settings, not real-world clinical deployment.

Google’s own materials frame the work as a step toward more capable conversational medical AI, not an available diagnostic product for patients.
Source: https://research.google/blog/amie-gains-vision-a-research-ai-agent-for-multi-modal-diagnostic-dialogue/

And the sources do not establish:

For business leaders, this is the correct lens: high-potential architecture, early validation, not yet a deployable medical product.

What AMIE teaches AI builders: the “agentic multimodal” pattern

Even if you don’t build for healthcare, AMIE’s approach provides a valuable template for robust AI systems:

1) Evidence-seeking behavior beats “single-shot” answers

AMIE’s ability to request missing information mid-conversation is a direct antidote to brittle automation. In customer support, finance ops, HR, and IT, the AI should be trained (or prompted and orchestrated) to:

  • Detect missing fields (e.g., “invoice number is missing”),
  • Ask for the right attachment (e.g., “upload the purchase order PDF”),
  • Request higher-quality inputs (e.g., “please provide a clearer screenshot”).

2) State-aware workflows reduce hallucinations

The state-aware framework (history → diagnosis/management → follow-up) is essentially a structured workflow embedded in the conversation. This reduces the chance that the model jumps ahead without collecting prerequisites.

In practice, we can implement this by:

  • Tracking conversation state in a database,
  • Using policy rules for transitions,
  • Running validation checks before moving to the next phase.

3) Dynamic revision is a feature, not a bug

AMIE can update its diagnosis as new multimodal data arrives. That’s how real experts work: hypotheses evolve with evidence.

For enterprise AI, build systems that:

  • Persist intermediate hypotheses,
  • Record what changed and why,
  • Provide an audit trail of inputs → outputs.

How to operationalize these ideas with n8n workflows (real-world automation)

If your organization is adopting AI consulting and workflow automation, the practical question is: how do we build something AMIE-like—not a medical diagnostician, but a multimodal, evidence-seeking assistant—safely and reliably?

Here’s a concrete pattern we implement using n8n workflows:

Workflow pattern: “Evidence-Gated AI Agent”

  1. Trigger: user message arrives (web chat, Slack, Teams, ticketing system).
  2. State store: load a conversation state object (phase, missing info checklist, permissions).
  3. Classifier step: determine intent + required evidence (e.g., screenshot, log file, PDF).
  4. Request step: if evidence missing, ask user for it with clear instructions (lighting, angles, file type).
  5. Ingestion step: accept uploads; run preprocessing:
    • OCR for PDFs,
    • metadata extraction,
    • basic image quality checks (blur/brightness),
    • redaction if needed.
  6. Reasoning step: send structured prompt + extracted artifacts to the model.
  7. Revision step: compare new conclusions vs. previous; if changed, log “reason updated due to new evidence.”
  8. Routing step: either:
    • auto-resolve (low-risk, high-confidence), or
    • escalate to a human with a summarized evidence pack.

This is the enterprise translation of “AMIE requests skin photos mid-chat and revises the differential.”

Practical takeaways for tech professionals, developers, and innovators

For developers

  • Design for missingness: treat “missing evidence” as a first-class condition, not an error state.
  • Add input quality gates: for images (blur/lighting), documents (OCR confidence), signals (sampling rate, lead placement).
  • Log reasoning deltas: when new data arrives, store what changed. This is gold for debugging and governance.

For product and engineering leaders

  • Don’t chase multimodal for novelty—tie it to workflow ROI: multimodal makes sense when the image/document is a bottleneck for resolution time or accuracy.
  • Build escalation into the system: the safest AI agents know when to hand off.
  • Treat “simulated benchmark wins” as architecture validation, not deployment readiness: AMIE’s results are impressive, but the sources stress controlled evaluations.

For innovators in regulated industries

  • Provenance and auditability are non-negotiable: if an AI used an image or a document, you must be able to trace it.
  • Separate “assistant” from “decision”: design systems where AI supports professional judgment rather than replacing it, especially in high-stakes domains.

Why this matters to our AI consulting practice

AMIE’s evolution highlights what we’re seeing across industries: the winners won’t be teams that merely “add AI.” They’ll be teams that engineer agentic systems—AI that can:

  • Operate within a structured process,
  • Request the right artifacts,
  • Integrate multimodal inputs,
  • And update conclusions with an auditable chain of evidence.

That is exactly where strong AI consulting plus n8n workflow automation delivers value: turning promising model capabilities into production-grade systems with guardrails, observability, and measurable impact.

If you’re exploring multimodal AI—whether for document-heavy operations, visual QA, claims processing, technical support, or compliance—our team can help you design the orchestration layer that makes it reliable.

Call to action: build an evidence-seeking AI workflow, not just a chatbot

If you’re interested in applying the “multimodal diagnostic dialogue” pattern from Google AMIE to your business workflows—without the hype and with the right governance—contact us to discuss:

  • Multimodal AI architecture and agent design
  • n8n workflow automation for evidence collection, routing, and escalation
  • Production guardrails: audit logs, evaluation harnesses, and human-in-the-loop operations

Reach out to our AI consulting team to schedule a technical discovery session and we’ll map a practical, ROI-driven path from prototype to production.}