Detecting Unexpected EHR State Mid-Workflow

An EHR workflow breaks the moment the screen in front of an agent stops matching the screen the agent planned for. That single condition, an unexpected state appearing mid-task, is the subject of this piece: how computer-use agents detect it, interpret it, and respond to it once a workflow is already underway.

Why EHR workflows break mid-execution

The EHR is a live document. Between the moment an agent begins a task and the moment it tries to act on it, a patient flag can change, a record can get locked by a concurrent user, an insurance field can update, a scheduling slot can disappear. None of these changes announce themselves to the agent in advance. Payer portals make the problem worse: layouts shift without notice, session tokens expire mid-task, and aggregator layers often hand back cached data that no longer matches what the payer's system would actually return if queried directly. The referral and scheduling surface shows what's at stake in plain terms. An agent that matches a referral against a record that has since been merged with a duplicate, or that reads a scheduling slot that has since been filled by someone else, doesn't just make a small error. It produces a misfired workflow with consequences for patient access and for revenue that someone downstream has to catch and fix by hand. The task for the rest of this piece is to explain how an agent can catch that kind of break before it happens, and what it should do once it does.

How agents construct their picture of current state

A computer-use agent doesn't know the state of an EHR record the way a database query knows it. The agent's picture of the world comes entirely from what it can see on screen at the instant it looks. That makes the freshness of each observation the foundation for every action the agent takes next: stale input produces a stale plan, no matter how good the reasoning layered on top of it is.

The mechanism itself is simple to describe. The agent takes a screenshot, runs that image through a vision model to interpret what's there, identifies the clickable and fillable elements by their visible labels and their position on the page, and picks the next action based on that reading. Nothing about a CSS selector or a hardcoded path enters into it. That's a meaningful design choice, and it has a direct consequence: an agent reading labels and layout survives small interface tweaks that would break a script tied to a fixed selector.

But that same reliance on live visual reading means every changed layout, every new modal, every locked field is a real event the agent has to notice and sort out, not a status code it can trap in a try-catch block. The agentic loop, perceive, plan, act, perceive again, means the agent re-checks the screen after every step. The value of that loop depends entirely on what happens when the new screenshot doesn't match what the prior plan assumed would be there. Multi-surface workflows raise the stakes further. A prior authorization agent might pull clinical data from the EHR, then move to a payer portal to submit it, carrying assumptions about the EHR record that may no longer hold by the time it gets there. The agent's picture of reality is only as current as its last look, and the gap between what it last saw and what's actually on screen now is where every failure mode described below begins.

The four categories of unexpected EHR state an agent encounters mid-workflow

Unexpected states aren't random noise. They sort into four categories, each with its own cause, its own tell, and its own way of breaking a workflow if the agent misses it.

Concurrent modification happens when another user or process has changed or locked the record the agent is working on. This is routine in multi-provider practices where front desk staff, billing staff, and clinical staff all touch the same patient chart over the course of a day. The tell is a lock indicator, an error banner, or a field that was editable in the previous screenshot and is now grayed out. If it's missed, the agent tries to write to a record that either silently rejects the write or overwrites someone else's change without anyone knowing until later.

Session interruption happens when a payer portal times out, an EHR session expires, or an aggregator connection drops, and the agent lands back on a login screen or an error page in the middle of a task. The tell is the absence of the page the agent expected to see, replaced by a login prompt, a timeout message, or a network error. If it's missed, the agent keeps trying to interact with elements that no longer exist, or submits data into a session that already closed.

Layout or UI change happens when a payer or EHR vendor updates its portal after the agent's workflow was last validated against it: fields move, labels get renamed, a new step gets inserted into a process that used to have one fewer step. The tell is that expected elements are missing or shifted, or that the labels on screen don't match what the agent's plan describes. If it's missed, the agent clicks the wrong control or skips a field it was supposed to fill, producing a submission that's malformed in a way no one notices until it's rejected.

Unanticipated modals and interrupts happen when a pop-up, an alert banner, a required acknowledgment, or a warning appears that the workflow was never built around, a new payer compliance notice, a duplicate-patient warning, a clinical flag a recent EHR update introduced. The tell is an overlay blocking the element the agent expected to interact with next. An agent that misses the modal tries to act through it, unable to see past it, or dismisses a warning that needed a person to actually read it.

How agents detect screen mismatches

Detecting an unexpected state is a judgment call, not an error code. At each step in the loop, the agent has to decide whether the screen in front of it matches what its plan assumed would be there, and it has to make that call fresh every time, because the answer can change between one screenshot and the next.

Visual grounding checks are the first layer. The agent compares what it observes against the elements its plan calls for, and if the target element is missing, blocked, or structurally different from what was expected, that mismatch is the signal. Research on the OSWorld-Human benchmark found that a meaningful share of agent errors trace back to poor visual grounding, and those grounding failures cause agents to burn far more steps completing a single task than they should need. Grounding quality isn't a side detail. It decides whether an unexpected state gets caught in time or gets acted on by mistake. Agents that read elements by label and context rather than by selector have a wider net to catch problems with: they can tell that a field labeled "Authorization Status" has moved or vanished entirely, not just that some underlying path in the code broke.

State comparison across the loop is the second layer. Because the agent re-perceives the screen after every action it takes, it can check the result against what it expected to happen: a write that failed, a field that didn't change, a screen that jumped somewhere unplanned all register as mismatches worth stopping for. This is where the reflection and judging steps in the loop earn their keep. OSWorld-Human found that reflection and judging account for a large share of a task's total time, and that cost is the price of knowing, step by step, whether the last action actually worked and whether the current screen is the one the plan expected. Skipping that cost to save time means skipping the only mechanism that catches a failure before it compounds.

Confidence thresholds are the third layer. When the agent's read on the current screen falls below some threshold of certainty, because the layout is unfamiliar, a modal is partly covering the content, or a label is ambiguous, a well-built agent treats that low confidence itself as a detection event that stops the flow for review. This logic appears directly in how prior authorization agents are built: an ambiguous clinical question or an uncertain portal state is designed to trigger a human review.

Absence detection is the fourth layer, and it's the one that's easiest to overlook. An agent that only checks for the presence of elements it expects will miss the case where a required element simply isn't there anymore. A locked-record indicator, a session-expiry screen, a modal overlay, all of these appear as absences, as something missing. An agent has to treat that absence as a real signal demanding a response, not as a null result to shrug off and proceed past.

What good mid-workflow response looks like

Catching the problem is only half the job. What the agent does next decides whether the detection was worth anything.

For session interruptions and transient portal timeouts, the right move is to re-authenticate, re-perceive the screen from a known starting point, and re-run the step that got interrupted. Picking up from a cached mid-stream assumption instead of a verified checkpoint risks acting on a screen state that's no longer true.

For states the agent can't confidently classify, an unfamiliar modal, a clinical flag that calls for judgment, a duplicate-patient warning, the right move is to pause, hold on to the work already completed, and hand the exception to a staff member with enough context to make the call. A prior authorization agent flags ambiguous clinical questions for human review before it proceeds any further, treating its own uncertainty as a normal part of the workflow.

The response that causes the most damage downstream is an agent that notices an unexpected state and keeps going anyway: submitting a prior authorization through a session that already expired, writing into a record another user has locked, clicking past a warning it never actually understood. Rule-based automation built on fixed scripts tends to fail in exactly this way. It acts on stale or wrong information without raising any flag, and the problem only becomes visible once a claim gets denied, a record gets corrupted, or a duplicate patient entry gets created somewhere downstream.

Interrupted workflows should resume from the last confirmed point of success, not start over from scratch. That requires the agent to keep a record of what it already completed that doesn't depend on reading the screen correctly after the fact. Syll's agent harness, for instance, turns execution into multimodal evidence, logs, keyframes, approval checkpoints, so the record of what happened can be inspected after the fact. It keeps memory, skills, routines, and governance as separate, persistent artifacts outside the agent itself. A workflow interrupted halfway through can be picked back up from a known-good point.

Auditability and HIPAA compliance

Every decision an agent makes mid-workflow while handling protected health information, to pause, to escalate, to retry, to proceed, has to trace back to a specific agent identity, a specific human authorization, and a log detailed enough to survive an OCR breach investigation. A generic automation log that just records "task completed" can't answer the question regulators actually ask: which PHI fields did this agent touch, under which instance, at what time, for what business reason. That's the minimum chain of custody required under HIPAA's audit controls, codified at 45 CFR §164.312(b).

The most common way this breaks down is the shared service account. When every agent instance logs in under the same credential, an unexpected state that sends the agent outside its intended scope leaves no distinct trail pointing back to which instance did it. The breach happens and the log can't show it happened. A compliant architecture gives each agent its own identity, ties every workflow to a named human who authorized it, and carries both of those through every entry in the log.

This connects directly back to the pause-and-escalate response described above. An unexpected state that causes an agent to access the wrong patient's record, a merged duplicate, a wrong match on name and date of birth, is a HIPAA breach and a patient safety failure at the same time. The detection and response mechanisms built to catch bad states mid-workflow are the mechanisms that prevent that kind of breach from happening. Pausing the workflow and logging the exception is the compliant choice when an agent hits a state it can't confidently read, not just the operationally sound one. Any vendor touching PHI as part of this kind of workflow has to operate under a signed Business Associate Agreement, and whatever the agent reads off a screen it didn't expect to see, including the exception itself, falls inside that agreement's scope.

How cross-system workflows amplify detection difficulty

The hardest version of this problem doesn't live inside one system. It lives at the seams between systems. A prior authorization workflow, for instance, spans the EHR, a payer portal, and an aggregator layer in between, and each of those three can report a different state at the same moment. The EHR might show a record as current, the aggregator might be serving cached data that's already out of date, and the payer portal might reflect a third state that doesn't match either one. An agent working across all three is holding three pictures of reality at once, and any of them can be wrong without the others showing it. That's the condition interoperability architecture has to be built to survive, because no amount of detection logic inside a single screen can fix a disagreement between systems that the agent has no way to see until it moves from one to the next.