Evaluating and improving the agent experience of websites

The Last Mile loop

The three stages are:

• Evaluation — an evaluation suite and a structured process to measure agent performance, including an Index scoring the websites of representative players in the covered industry.
• Diagnosis — a diagnosis of where and why agents struggle.
• Optimization — guidance to implement best practices and emerging standards to improve the agent experience.

The framework is complemented by a playground (opens in a new tab) — a test website where you (or your agent) can experience different degrees of agent-friendliness. It demonstrates the use and impact of specific best practices for building agent-ready websites. Because we own that site, we can run the same tasks against the same journeys with agent-friendly practices switched on and off: across its three readiness profiles the Last Mile Index moves from 0% to 100%. Note that the bare link lands on the deliberately hostile default profile; add ?profile=ready to see the other end of the range.

The maturity model

Our maturity model describes the capabilities a website needs to support agents acting on behalf of users. It contains four connected maturity levels — discoverability, understanding, interaction, and transaction.

The first two determine whether agents can find and interpret a website. The second two determine whether they can act through it. Transaction sits beyond the "money line," where actions introduce payment, authorisation and additional trust requirements.

The maturity levels are universal and stable. Their specifics — the relevant best practices, failure modes, standards, and so on — can change. You can find a detailed, up-to-date specification of the maturity model on the interactive spec page, which is rendered directly from the same versioned data files the benchmark runs on, so the published spec can never drift from what is actually measured. You can also use the AI Radar's agent-readiness and Agentic Web pages (its names, not our category term) to monitor the related developments.

This section describes each maturity level, provides example tasks and summarizes the best practices.

Discoverability

Discoverability describes whether an agent can identify and reach the website, page or capability relevant to a user's request.

Example tasks that rely on discoverability

• Finding an airline that serves a requested route
• Locating its first-party booking experience
• Finding a specific capability such as baggage information or a booking tool

If the company or capability does not surface during the initial research done by the agent, it is unlikely to be considered in the rest of the journey. Discoverability therefore influences whether an agent reaches a first-party channel and whether the company's own information and services shape the result.

To increase discoverability, important content and capabilities should be easy to locate through clear navigation, stable URLs, accurate crawler guidance, complete sitemaps and machine-discoverable service information. Where access is controlled, publish clear authentication information and support the recognition of trusted agents.

What we measure here. We do not test open-web visibility: each run starts at the airline's own entry point, because the subject of the benchmark is the website, not the search engine or the model's training data. What we measure is whether an agent that has arrived can find the page or capability the task needs — through navigation and labelling it can actually read, or through a published machine-readable index of the site that names it. Open-market visibility is a real and related question, and the callout above says so; we simply do not claim to have measured it.

Understanding

Understanding describes whether an agent can accurately interpret a website's structure, content, offers, rules, inputs and errors.

Representative test tasks for understanding

• Identifying the baggage allowance attached to a fare
• Interpreting the conditions of returned flight options
• Understanding the fields or validation messages presented during a journey

Retrieving raw text is not enough for understanding. The agent must interpret the information correctly and associate it with the relevant context — for example a specific offer, action or customer request. When important details are unclear, inconsistent or difficult to extract, the agent may give an incorrect answer or abandon the journey.

Improving understanding for agents follows similar principles as for humans. Present important information as clean, accessible and consistently structured content. Use semantic HTML, appropriate structured data, clearly labelled inputs and explicit error messages. Avoid locking critical information inside images, inaccessible documents or unnecessarily complex client-side interfaces.

Interaction

Interaction describes whether an agent can actively operate a website to complete actions that do not move money.

Representative tasks

• Entering a route and dates to run a flight search
• Selecting a seat or service that does not require payment
• Completing check-in or another account action

Interaction is where agents begin to create direct operational value. An agent that can only read a website may answer questions, but it cannot complete the user's task. Reliable interaction can reduce customer effort, improve journey completion and allow companies to deliver existing services through agent-mediated channels. The evaluation considers successful completion and, where the site produced an error during the run, whether the agent could recover from it.

Use stable, accessible and clearly labelled controls that agents can operate programmatically. Preserve entered data when errors occur, provide actionable validation messages, expose clear success states and design recoverable paths for common failures. Important actions may also be exposed as structured WebMCP tools or APIs to improve reliability.

Where the site produced a genuine error along the way, we also record whether the agent recovered — but we do not inject faults into live third-party websites. Deliberate error-condition testing runs on the playground, where we control the site and can reproduce the condition on demand: today that covers invalid credentials, where a wrong booking reference and last name returns a machine-readable "Booking not found", and confirm-before-acting steps, where a tool call must be re-issued with an explicit confirmation.

Transaction Outlook

Transaction describes whether an authorised agent can complete a money-moving action on a customer's behalf. This is the ultimate goal for agent experiences. However, as of today, the related technologies are not yet widely established and hardened, and we are not explicitly testing this maturity level. To follow the developments in this space, you can monitor the Agentic Commerce topic in the AI Radar.

The maturity levels form a progression:

Earlier levels provide the foundation for those that follow. An agent must first discover a website before it can understand its content, and it must understand an interface before it can operate it or complete a transaction.

Each maturity level is diagnosed separately, but scored through the task. A task either completes or it does not; when it does not, the deterministic signals tell us which level broke it. That is how we can say a journey failed "at understanding" rather than merely "failed", and it is what makes the diagnosis actionable.

We deliberately do not publish four independent maturity-level scores per website. A level score with no task behind it is a checklist with a number on it — precisely what we are arguing against. What we publish per website is the completion rate, the grade, and the distribution of failing signals across its tasks.

Evaluation

This and the two sections that follow describe the loop of evaluation, diagnosis, and optimization which enables web developers to make their website agent-ready and keep up with new user expectations and emerging technologies. The evaluation methodology itself consists of two components — the evaluation suite and the protocol for running the evaluation.

Evaluation suite

The evaluation suite defines what is tested. It is organised around representative tasks from industry-specific customer journeys. Each task tests whether an agent can understand a user objective, navigate the relevant digital journey, complete the required action, and provide reliable evidence of the outcome. Each task is specified as:

• an id and a short name;
• the journey step it belongs to;
• its access type — which gate stands in front of it, and therefore how far the agent is expected to get;
• the prompt: the user request and the starting instruction the agent receives;
• the desired outcome in plain language;
• a deterministic success check: what an acceptable answer must contain, written before the run;
• a step budget: the maximum number of browser actions the agent may spend;
• discovery keywords, used to test whether the entry point was findable.

A note on the last row: maturity levels are a reading of the task, not a field that changes how it is scored. A task completes or it does not; the levels are how we describe what it exercised and, when it fails, where it broke.

For the airline industry we catalogue twenty tasks and run sixteen of them: eleven public and five booking-flow. Four are catalogued but parked from this first sweep — two are gated behind a booking reference or a sign-in, and two are post-purchase servicing tasks that sit outside the funnel we are measuring. Parking a task does not weaken it: the tasks stay in the catalogue with their success checks unchanged, flagged out of the run set, and re-admitting one is a one-line change once staged credentials exist.

The customer journey has ten steps. This first run set deliberately covers five of them — Inspiration & Planning, Compare & Select, Book Travel, Plan & Prepare, and Help — which together make up the pre-purchase funnel: inspiration, through comparison, to booking as far as it can be taken without a login. Day-of-travel and post-trip servicing are catalogued and will follow.

Running the evaluation

Each task from the evaluation suite is run against the relevant websites under documented conditions so that results are comparable across websites and runs. Every run is recorded so that a result can be re-examined rather than taken on trust. Each record carries:

• the website and the entry URL the run started from;
• the task, its access type, and how far that access type allows the agent to get;
• which model drove the agent, and its provider — a result is a statement about a website and the agent that met it, and we treat the model as a reported axis, not a hidden constant;
• the date and time of the run;
• the full ordered trace: every navigation, click, keystroke and read the agent performed, with the site's response;
• the answer the agent extracted, and whether the deterministic check accepted it;
• why the run ended — completed, gave up, ran out of its step budget, or failed on our infrastructure;
• the output of every deterministic probe;
• a screenshot of the final page state, and for cloud-driven runs a session replay link.

Runs are driven in a Chromium browser at a fixed desktop viewport, unauthenticated, and pinned to the target's own domain so an agent cannot wander onto another carrier's site and have the result attributed to the wrong brand. Consent banners that intercept clicks are dismissed so the run can proceed, but the fact that one was present is recorded as friction rather than erased.

What we do not yet vary or control: market, interface language, and prompt wording. Every task runs from one prompt version, in one locale, unauthenticated. That is a limitation of the current sweep, not a claim about the sites, and we would rather state it than imply a controlled matrix we have not run.

Access: what we can measure today

Tasks differ not only in what they ask the agent to do but in what gate stands in front of them. We classify every task by access type, and the access type decides how far the agent is expected to get:

Booking-reference, login, eligibility and hybrid tasks are catalogued and run to their wall, where they are recorded as Blocked and excluded from the Index. They are not failures and we do not present them as such: a booking-reference lookup that correctly asks for a booking reference is a site working as designed. Until we hold staged credentials, we report those journeys as "as far as access allows".

Two consequences worth stating plainly. First, the exemption is tied to the task's access type, not to the wall itself: a public task that runs into a sign-in prompt is a failure, because gating a public surface is the site's own choice and it stops the agent just as dead. Second, we stop at the purchase or commit step by design. The agent never completes a payment and never makes an irreversible change to a real booking — which is also why Transaction is scoped as future work rather than tested.

Who decides the outcome

The agent drives; it does not grade itself.

Every task carries a deterministic success check, written before the run: a literal string or a pattern that an acceptable answer must contain. When the agent believes it is done, it declares an answer. That declaration only nominates the text — the check then decides. An agent that declares success on text the check rejects is recorded as a failure, not a completion. The answer is also capped at a short length before checking, so an agent that pastes a whole page cannot have a stray price somewhere on it counted as the answer it was asked for.

Running alongside the agent is a set of deterministic probes that read the page directly: whether the site exposes a structured path for agents, whether a consent overlay intercepts interaction, whether date fields and controls can be operated programmatically, whether a failed action explained itself, whether the relevant entry point was findable. The probes contain no model judgement — they read page state. Their job is diagnosis, not adjudication: they explain why a task failed and which maturity level broke it. Deliberately, they cannot complete a task on the agent's behalf. Where a site exposes a structured tool that returns the right answer, we record that as a property of the site; we never let it stand in for the agent's own run, because a probe completing the task would be the benchmark marking its own homework.

Task outcomes and website grades

Task outcomes. Each task run against each website resolves to one of three outcomes:

Website grades. Fails, Struggles, Completes and Gated are the website-level grade, derived from those task outcomes. For each website we count its measurable tasks (every task run against it, minus the blocked ones) and its completed tasks:

The evidence floor of two is deliberate, and it is the whole difference between an honest "Struggles" and a flattering one: a single measured task cannot distinguish "reliably completes" from "got lucky once". It errs pessimistic, which is the correct bias for a benchmark whose credibility is the product.

Results

The full evaluation produces task-level and website-level results. At task level we record the outcome, the failing signal where there is one, the evidence captured, and the trace. At website level we aggregate into the completion rate, the grade, the pattern of failing signals, the journey steps where the site is strong or weak, and the recommended improvements. The website-level evaluation also goes into our Last Mile Index, a benchmark of representative websites in the considered industry.

The Last Mile Index

The Index is a completion rate over the runs we can actually measure: tasks completed ÷ measurable tasks, computed per website and reported for the industry as a whole.

We always publish the fraction next to the percentage. "7 of 11 measurable tasks completed · 3 blocked · 1 error" is the honest unit; the percentage is a convenience. A reader who wants to argue with our number can see exactly what went into the denominator, which is the point.

Diagnosis

The evaluation says what happened; the diagnosis says where it broke.

When a task fails, the record names the failing signal, chosen from the deterministic signals in a fixed order of severity: a hostile input control first, then a failure the site did not explain, then a consent wall, then discoverability. That order is not arbitrary — in our data a date field that rejects programmatic input and a dead end that says nothing kill a journey more decisively than a missing structured path. Where none of the specific signals fired, the failure is recorded against task completion: the agent simply did not get there and we do not invent a cause.

Signals co-occur, and we report all of them rather than only the first. On airline sites a date picker and a consent overlay are almost always both present, so the report names the primary cause and lists what else was present alongside it, instead of quietly hiding the second problem. Only failures carry a diagnosis: a completed run and a blocked run are clean boundaries and get none.

A diagnostic record may contain

• the task being attempted;
• the observed failure mode;
• the agent's actions immediately before the failure;
• the website response;
• the failing signal — and, since every signal maps to a maturity level, where in the model the journey broke;
• supporting technical evidence.

Common failure modes, split by how we know them

The diagnosis is based on observed evidence. Where the exact underlying cause cannot be confirmed, it is labelled as a probable cause rather than a definitive finding.

Optimization

Every investment into improving the agent experience of your website should connect directly to an observed problem. Last Mile presents optimizations using a simple causal structure with failure mode, corresponding fix, and expected outcome.

Fixes are grouped by maturity level so organisations can see whether they primarily need to improve discovery, understanding or interaction. They are sequenced by dependency, and the tier is not a matter of taste — it follows from which signal broke the task:

Some fixes unlock task completion directly; others reduce friction and increase reliability without changing an outcome on their own. Either way the sequence holds: a Tier 2 investment on a site that fails at Tier 0 is a way of spending money without moving the Index.

Retesting and confirmation

A fix counts as confirmed only when the same task, with the same success check, is run again against the changed website and completes. We keep task ids and their checks stable for exactly this reason: renaming a journey step or relabelling a signal never changes a task id, so a before-and-after comparison stays meaningful.

We are explicit about how much evidence a confirmation carries. Today each task is attempted once per website per sweep, so a successful re-run shows that the fix works, not yet that it works reliably. Repeated runs of the same task are the obvious next step; until they exist we do not make reliability claims, and the website grade compensates by requiring at least two completed measurable tasks before a website is called "Completes".

The controlled version of this loop already runs on the playground (opens in a new tab), where we own the site: the same tasks are driven against the same journeys with agent-friendly practices switched on and off, and the completion rate moves. That is where we can show the effect of a specific practice directly, rather than inferring it from a live site that changed several things at once.

Succeeding in an evolving ecosystem

The ecosystem around agentic AI is evolving quickly, and new standards, protocols, and frameworks keep emerging. You can use the Agent Readiness profile in the AI Radar to track relevant standards, protocols and technical practices. However, as emphasized before, their presence should not be treated as an end in itself. It is important to distinguish between:

In our optimization guidance, standards are classified according to their maturity into three layers:

Our current catalogue holds twenty standards — eight established, six emerging, six experimental — alongside eight best practices. Both are maintained as versioned data rather than editorial prose, so every classification is auditable and dated.

Experimental standards may be useful as low-cost strategic bets, but they are not treated as universal requirements. A website should not receive a higher outcome score merely for implementing a new protocol, and it should not automatically be penalised for omitting one if agents can already complete the relevant task.

Summary