Evaluation Metrics for Healthcare Administrative Agents

Healthcare administrative agents need evaluation metrics that measure complete work. Their value depends on whether they can finish a workflow the way a trained staff member would. A useful framework therefore starts with end-to-end task success, explains failures at each step, and measures the staff capacity recovered within the practice’s actual integration and compliance constraints.

Why generic AI benchmarks fail to capture what healthcare administrative agents actually do

Generic AI benchmarks test isolated question-answering or single-step accuracy, while healthcare administration needs a series of actions that span systems. A correct answer about a prior authorization requirement is not a completed authorization submission. The work also calls for finding the information needed for the submission, handling documents, and moving data between systems. Evaluating only one of those steps removes the handoffs where the workflow can fail.

HealthAdminBench makes that mismatch concrete through tasks involving prior authorization, appeals and denials management, and DME order processing. Its simulated GUI environments require agents to work across an EHR, payer portals, and a fax system without programmatic APIs. That benchmark constraint leaves open whether real-world counterparts have FHIR-based APIs. PatientAgentBench establishes that static tests miss failures that emerge during sustained, tool-using patient-facing conversations against realistic records. In both settings, a strong isolated result is insufficient evidence that an agent can carry work through to resolution.

What HealthAdminBench reveals about the gap between subtask accuracy and end-to-end task success

HealthAdminBench, published in April 2026 by Kinetic Systems / Stanford University, evaluates 135 expert-designed tasks across four GUI environments. Those environments comprise an EHR, two payer portals, and a fax system, with tasks decomposed into verifiable subtasks. This structure makes it possible to distinguish progress through a workflow from completion of the workflow. That distinction is the benchmark’s central operational lesson.

HealthAdminBench reports a best end-to-end task success rate of 36.3% among the evaluated agents, achieved by Claude Opus 4.6 CUA. The agent leading on subtask success does not lead on full-task completion. The rankings therefore answer different questions, and the more favorable number cannot substitute for the more consequential one. An agent can get each action right yet still need too much help to finish the amount of work a deployment requires.

HealthAdminBench finds that agents fail when handling documents and moving data between systems. These are not peripheral functions, because they connect the actions that make an administrative task complete. The benchmark also finds that fine-tuning an open-source model on a modest set of training tasks substantially improves held-out task success, surpassing the best closed-source agent. For evaluators, the implication is direct, because a vendor reporting subtask accuracy without end-to-end success is reporting the less meaningful number.

The layered metric structure that end-to-end workflow evaluation actually requires

End-to-end task success rate belongs at the top of the evaluation framework. It measures whether the agent completed the full workflow without human intervention, rather than how much of the workflow it attempted correctly. Resolution rate measures the same thing for patient issues: the share fully resolved without routing to a human. Neither, however, explains why a task failed.

Subtask error rate supplies that diagnosis by measuring errors at each workflow step. HealthAdminBench’s findings make document handling and cross-system transfer important points to examine, but the measurement must follow the steps of the workflow being deployed. A single score for overall accuracy can hide a weak handoff even when the other steps go well. Tracking errors by step makes it possible to isolate and fix the failure without rebuilding the entire workflow.

First Contact Resolution, or FCR, adds an efficiency measure by asking whether a patient’s issue was solved on the first interaction without follow-up or escalation. Resolution rate alone cannot distinguish that result from an issue resolved after further contact. Routing accuracy then measures whether requests reach the right workflow, queue, or staff member on the first try. A misrouted prior authorization request restarts the clock on time-sensitive work, so routing errors matter beyond the initial transfer. Practitioner frameworks cite a target above 90% for first-transfer accuracy, although routing performance does not establish that the destination workflow succeeds.

Staff capacity recovered measures the operational payoff of these results in administrative hours per month returned for exception handling and higher-value work. It answers a question that completion and efficiency rates cannot answer by themselves, which is how much staff time the deployment releases. That capacity should be reported with a per-workflow breakdown of the aggregate. A practice with a prior authorization bottleneck needs to see hours recovered on PA specifically.

Why prior authorization is the stress test for any administrative agent evaluation framework

Prior authorization concentrates the difficulties that make administrative automation demanding. It combines multi-portal navigation, document extraction, payer-specific rules, and time pressure within one workflow. HealthAdminBench includes it as one of three core task types because cross-system coordination exposes the gap between individual actions and completed work. A framework that cannot explain PA performance is missing some of the most consequential failure points in healthcare administration.

PA task success measures how well the agent can follow portal guidelines, check coverage criteria, and submit the right documentation. First-pass approval rate measures the outcome that completing those tasks is meant to help achieve. Allegheny Health Network achieved a first-pass approval rate of 96% for prior authorizations. A high first-pass rate means fewer denials to appeal and faster revenue realization, but it answers a different question from whether the agent completed its submission workflow.

The CMS Interoperability and Prior Authorization Final Rule, CMS-0057-F, adds a regulatory clock to this evaluation. From January 1, 2026, impacted payers must issue expedited and standard decisions within defined timeframes and publicly report PA metrics annually. The first report was due March 31, 2026, covering calendar year 2025. These requirements make turnaround performance more than an internal SLA concern, while the distinction between an agent’s task completion and a payer’s decision remains important. Evaluation must account for the workflow’s time-sensitive obligations without treating every downstream result as a direct measure of the agent.

From January 1, 2027, payers must operate FHIR-based Prior Authorization APIs, shifting the automation modality from portal navigation toward structured API interaction. A multi-year deployment therefore spans more than one interaction model. Evaluators should assess how the platform’s architecture will perform across both modalities. This makes PA a test of both present workflow execution and the integration conditions under which future results will be measured.

How EHR integration depth changes what the metrics actually measure

Integration depth determines what an agent can observe and act on at each workflow step. As a result, identical metric thresholds can describe different operating scopes. Computer-use agents read, click, and navigate EHR screens in the way a staff member would, rather than being limited to functions exposed through an API. That distinction matters in payer portal and legacy system contexts where no API exists. This also means that judging performance in a GUI benchmark requires knowing the environment where the agent will work.

A denominator that excludes the difficult work creates the central measurement risk. An agent with shallow integration may report high routing accuracy because it handles only the cases it can confidently route, while harder cases escalate to humans outside that denominator. Evaluation should therefore include workflow coverage, defined as the share of total task volume the agent handles unassisted.

This distinction also changes the interpretation of staff capacity recovered. Strong performance on a narrow slice of work does not establish that the practice’s main bottleneck has improved. Capacity must remain tied to the workflow and the volume the agent can handle within its integration environment. For a deployment spanning portal and API interaction, that environment is part of the evaluation standard.

Compliance and audit-readiness as evaluation dimensions, not procurement gates

Compliance evaluation must go beyond checking a credential during procurement to check how an agent behaves as it works. For a practice administrator, the operational question is whether the agent’s actions can be audited as a staff member’s actions would be. Every action involving PHI, including reading a record, submitting a form, and transferring a document, should appear in an audit-grade log. That log is an evaluation artifact as well as a legal safeguard. It provides the action-level basis for assessing whether workflow behavior reflects the controls claimed for the system.

CMS-0057-F also brings API obligations into the operating environment, with requirements covering FHIR-based Prior Authorization, Provider Access, Payer-to-Payer, and Patient Access APIs by January 1, 2027. Claims from a vendor about audit-readiness therefore need to be tested as part of checking how well regulated workflows run. SOC 2 Type II and HIPAA certification are required at a minimum under the procurement framework described here. The live evaluation still has to check whether the agent’s actions at each workflow step live up to those assurances. A credential and an auditable record of execution answer different questions.

Texas’s 2025 law prohibits utilization review agents from using an automated decision system to make an adverse determination wholly or partly, reserving that determination for a licensed physician or health care provider. Arizona requires independent medical-director review and bars the use of AI as the only basis for a medical necessity denial. Maryland’s law more broadly prohibits AI tools from denying, delaying, or modifying care. An agent operating in these states must demonstrate that its human-in-the-loop controls function as designed.

Confidence thresholds connect these controls to performance measurement. An agent must recognize when work exceeds its reliable operating range and route it to a human. A low escalation rate on genuinely ambiguous tasks is therefore a warning sign. The framework must distinguish avoidable staff intervention from necessary review, because maximizing autonomous completion cannot be the only objective.

How to build a workflow-specific KPI agreement before deployment begins

A pre-deployment KPI agreement turns the framework into a defined standard for judging results from day one. The practice’s actual bottleneck should guide the agreement, because ROI cannot be measured against a universal promise. A multi-location group with a prior authorization backlog needs different baselines and targets from a single-location practice with a scheduling gap. The agreement should name the workflows to be automated and record their current task volume, staff hours, error rate, and denial rate where denials are part of the work.

The next step is to agree on target thresholds for the layered metrics, the review cadence, and the escalation criteria that trigger human review. HealthAdminBench’s findings support making full-task completion explicit rather than allowing subtask accuracy to stand in for success. Because integration depth limits which tasks an agent can handle, workflow coverage must accompany performance on handled tasks. For PA, CMS-0057-F’s decision timeframes and the transition to FHIR-based APIs also belong in the evaluation context. Together, these terms define the work, the conditions under which it is performed, and the boundaries of autonomous operation.

The agreement must then cover each subtask as well as the outcome of each workflow. Prior authorization needs PA task success and first-pass approval rate, while scheduling needs resolution rate, routing accuracy, and cancellation recovery. Denial appeals need appeal success rate and turnaround time against regulatory SLAs. Staff capacity recovered adds how much staff time each workflow gives back to the practice. A framework agreed before deployment is the only way to distinguish an agent that performs like a trained staff member from one that merely looks like it does.