Failure Mode Taxonomy for Clinical GUI Agents

Healthcare administration has become one of the largest live deployment surfaces for AI agents, with the global AI in healthcare market growing rapidly year over year. GUI agents, systems that operate software the way a human would, by looking at a screen and clicking through it, are already processing prior authorizations, filing cla... That growth is not theoretical. GUI agents, systems that operate software the way a human would, by looking at a screen and clicking through it, are already processing prior authorizations, filing claims appeals, navigating payer portals, and parsing fax-based document intake in live operational settings, not sandboxes.

• Roughly half of healthcare organizations are currently using or actively evaluating AI agents in some part of their operations.
• A large share of health systems have at least one AI pilot running.
• Many of those systems have yet to complete a formal security review of the tools involved.
• The market's growth trajectory, from $39.34 billion in 2025 to a projected $56.01 billion in 2026, is outpacing the industry's ability to evaluate what it's deploying.

That last gap, between adoption and evaluation, is the real subject here. Before anyone can grade an agent's reliability, they need a precise vocabulary for how it fails. Not "it made a mistake," but which mistake, at which layer of the system, with which downstream consequence. That taxonomy is what this piece lays out.

How a clinical GUI agent works, and where the breakdown can begin

A GUI agent doesn't reason about a task in the abstract. It runs a loop: observe, ground, act, verify. It captures a screenshot or an accessibility tree or a DOM snapshot, maps whatever it's trying to do onto a specific clickable target on that screen, performs the click or the keystroke, and then looks again to decide if the world changed the way it expected. Every one of those four stages is a place where things can go sideways, but they don't go sideways evenly.

• Grounding, the step where intent gets translated into an exact UI target, is where most errors start. Clicking the correct form field versus clicking forty pixels to the left carries real clinical and financial weight inside an EHR or a payer portal, not just an inconvenience.
• Most agent memory systems store what they've observed as text. That strips out the visual information a human staff member absorbs instantly, including icons, layout, which region is currently selected, the state of an input field, and an error banner sitting quietly at the top of the screen.
• Healthcare GUI environments are harder than general enterprise software by a wide margin. Payer portals differ from insurer to insurer in field order, form logic, and what documentation each one demands before it accepts a submission.
• EHR interfaces are dense, field-heavy, and unforgiving of small misreads. Fax-based intake adds an entirely separate problem: optical noise that degrades whatever recognition pipeline the agent depends on before it even reaches the reasoning stage.

The first systematic healthcare GUI benchmark's findings on real-world reliability

HealthAdminBench is the first benchmark built specifically to test agents inside healthcare GUI environments rather than general office software. It covers four environments, an EHR, two separate payer portals, and a fax system, and it runs agents through 135 expert-defined tasks spanning prior authorization, appeals and denials management, and durable medical equipment (DME) order processing. Those tasks break down into 1,698 discrete evaluation points, which is what makes the benchmark's finding so hard to argue with.

The best end-to-end task success rate recorded, achieved by Claude Opus 4.6 CUA, was 36.3%.

• The best subtask success rate recorded, achieved by GPT-5.4 CUA, was 82.8%.

The distance between those two figures, 82.8% subtask accuracy against 36.3% full-task success, is the empirical core of the agents' unreliability at completing full tasks.

• An agent can execute most individual steps correctly and still fail the overall workflow in nearly two out of three complete attempts.

That gap deserves to be sat with for a moment. The failure isn't concentrated in any single click or field entry; something structural is happening between steps, at the seams of the workflow, where subtasks that each look clean on their own fail to add up to a completed task. It means something structural is happening between steps, at the seams of the workflow, where subtasks that each look clean on their own fail to add up to a completed task. The rest of this piece is an attempt to name what happens at those seams.

Failure mode 1: completing steps without completing tasks

Call it the subtask illusion. An agent executes every discrete action in a sequence correctly, by any reasonable measure, and still never arrives at a valid end state. Nothing in the individual steps was wrong. The workflow as a whole simply never resolved.

A prior authorization form gets filled out with accurate information, then submitted to the wrong payer portal.

• An appeals packet gets assembled correctly, document by document, and then abandoned when a session timeout interrupts the submission step before it completes.

Claim fields get populated without a single data-entry error, but no confirmation of receipt ever gets logged, so the claim goes untracked until a denial appears weeks later.

Multi-step administrative workflows are exposed to this failure mode more than almost any other category of software task, because prior authorization, appeals, and DME processing all depend on handoffs across payer portals, external systems, and document exchange. Every one of those handoffs is a seam, and at every seam, a technically flawless subtask can produce zero downstream value. Research on general-purpose GUI agents outside of healthcare finds the same underlying pattern: multi-step task failures trace back to breakdowns at the seams between discrete actions, where agents struggle to recover when a workflow does not proceed as expected.

Failure mode 2: misreading what is on the screen

Perception failure isn't one thing. It breaks into at least four distinct types, each with its own signature.

• Grounding or spatial error: the agent's semantic intent is correct, it knows what it's trying to click, but coordinate precision fails, landing in dead space just outside the target's actual bounding box.
• Visual hallucination: the agent interacts with a UI element that isn't there, targeting a ghost element left over from a previous screen state, or guessing at layout coordinates based on prior training data instead of what's actually on the current screen.
• Observation neglect: the agent scrolls, searches, or opens a menu hunting for an element that is already visible, in plain sight, on the current screen.
• Repetition-induced termination: without any mechanism to detect that something is wrong, the agent retries the same action against a non-responsive element over and over until a termination rule finally kicks in and the task fails.

Healthcare adds its own amplifiers on top of these four. Payer portals are not static environments, and an agent calibrated to one version of a portal's layout may misread a changed screen with misplaced confidence. EHR interfaces are dense enough that a misidentified dropdown or an overlapping modal window produces a wrong entry that looks entirely correct in the agent's own action log. Fax-based intake degrades whatever OCR or text-recognition step the agent relies on to know what it's looking at.

Text-based memory compounds all of it. Visual context that would alert a human to a problem can be lost when an observation is reduced to a text representation, leaving the agent without the cues it needs to detect that something is wrong. That's what makes this failure mode genuinely dangerous rather than merely annoying: a perceptual grounding error throws no error message. The agent logs the action as successful because, from inside its own state representation, it was successful. The mistake becomes visible only later, when a denial letter arrives or a claim comes back unprocessable, and by then the trail back to the original cause has gone cold.

Failure mode 3: losing the thread across system boundaries and session interruptions

Context loss happens when an agent crosses from one system into another, resumes work after an interruption, or re-enters a workflow that's been running for a while, and proceeds as though earlier steps are done when they aren't, or as though the context it learned earlier still holds when it has in fact changed since then.

• A single prior authorization can touch an EHR, a payer portal, a fax queue, and an internal tracking system, each with its own session, its own login, and its own version of where the workflow currently stands.
• Session timeouts are a structural hazard baked into this environment. Portals vary widely in how long a session lasts, and an agent that re-authenticates after a timeout may resume at the wrong stage of the workflow without ever detecting that a discontinuity happened.

Memory and context management is a recognized dimension of LLM agent failure. An agent can misread or misunderstand an instruction at an early step and silently carry that error forward across many downstream steps, eventually returning an output that's confident, well-formatted, and wrong.

• The workflow itself never announces the break. There's no dialog box that says "you have lost track of where you are." The agent just keeps acting on a version of reality that stopped being true several steps ago.

Failure mode 4: reasoning errors that propagate silently through a workflow

Agentic chains don't dampen a bad inference, they amplify it. A wrong conclusion at one step doesn't throw an error message; it produces a plausible-looking action that then becomes the input for every step that follows.

• The same four-dimensional taxonomy that names memory and context as a failure category also names reasoning and planning as a core failure dimension, alongside tool use and multi-agent orchestration. An agent can call a tool with slightly wrong parameters, get back a result, and proceed as though the call succeeded, while every downstream step compounds the original error further.
• Coding schema failures create a healthcare-specific version of this same risk. Agents operating inside payer portals are working against a code landscape that shifts on fixed calendar dates: CPT and ICD-10-CM codes are updated on a regular annual cycle.
• Coding errors are a recognized source of claim denials industry-wide, and an agent trained or fine-tuned on a pre-2026 code set will submit an outdated code without flagging anything, because from its own perspective, nothing went wrong.
• The CMS Interoperability and Prior Authorization Final Rule (CMS-0057-F), which took effect January 1, 2026, has in practice produced more denials at many practices, not fewer.

Overconfidence causes all of it, and its effect appears next in the reasoning error described. An agent that fails because of a reasoning error typically presents the exact same behavioral signature as an agent that succeeded: no exception thrown, no retry attempted, no uncertainty flagged anywhere in the log. Success and failure look identical from the outside until the consequence arrives in the mail.

Failure mode 5: security boundaries, prompt injection, and compliance violations

Computer-use agents introduce attack surfaces that simply have no analogue in earlier generations of AI security work. Security research on computer-use agents covers attack categories including visual attacks, abuse of plugin architectures, and supply-chain compromise carried out through tool descriptions.

Prompt injection is among the most critical risks identified in AI security frameworks, and inside an agentic system it's a fundamentally more dangerous problem than it is in a simple chat interface. A successful injection can hijack the agent's entire goal, manipulate the tool calls it makes, and propagate malicious behavior across an orchestrated multi-step system.

• OWASP's 2026 taxonomy for agentic applications identifies three distinct injection vectors: direct goal manipulation, where a prompt explicitly overrides the agent's stated objective; indirect instruction injection, where hidden instructions embedded in a document, a piece of retrieved content, or a tool's output quietly alter the agent's behavior; and recursive hijacking, where a goal modification propagates through the agent's reasoning chain or self-modifies as the workflow continues.
• Human-in-the-loop bypass is a recognized failure scenario in agentic security research, where a system designed to require human sign-off gets routed around it.
• Agentic systems are structurally harder to deploy safely at scale than earlier generations of automation, precisely because they demand deeper system integration, broader access rights, and stricter governance than a simple scripted workflow ever did.

Prior authorization as the convergence point for all five failure modes

Prior authorization is the highest-stakes target for GUI agent automation in healthcare, and it's also the workflow most often chosen by vendors building these tools, for the simple reason that the volume and the pain It's the highest-stakes target for GUI agent automation in healthcare, and it's also the workflow most often chosen by vendors building these tools, for the simple reason that the volume and the pain are both enormous.

• The vast majority of physicians report that prior authorization delays access to care their patients need.
• Roughly half of insured adults who required specialized care report experiencing a delay or an outright denial because of the prior authorization process standing between them and treatment.
• Physicians and their office staff collectively spend a substantial number of hours on prior authorization tasks, hours pulled directly away from patient care.
• Every failure mode described above, the subtask illusion, perception errors, lost context across system boundaries, silently propagating reasoning mistakes, and security boundary violations, converges on this single workflow, because prior authorization is exactly the kind of multi-system, multi-step, high-friction process where each of those five failure modes finds room to operate.

Naming these failure modes precisely is the prerequisite for building evaluation frameworks, monitoring systems, and deployment guardrails that actually match how these agents break in practice, rather than how anyone assumes they migh... It's the prerequisite for building evaluation frameworks, monitoring systems, and deployment guardrails that actually match how these agents break in practice, rather than how anyone assumes they might. HealthAdminBench's own numbers, an 82.8% subtask success rate sitting next to a 36.3% end-to-end task success rate, make clear that the industry has been measuring the wrong layer of the problem. Fixing that starts with the taxonomy, not with another benchmark run on the same blind spot.