// Airline Agent Experience

Scans pass.
Agents still fail.

Today, AI agents are browsing, comparing, and buying for your customers. We send real browser agents on real customer journeys and check the only thing that matters: can they actually complete their tasks?

See the benchmark → How we test

// Benchmark — top 15 of 31 airlines tested

Who completes the task

# Airline Tasks completed Pass rate Blocked Top failing signal
1 Delta 22 of 22 100% 2 —
2 United Airlines 22 of 22 100% 2 —
3 Lufthansa 14 of 15 93% 9 Unattributed non-completion
4 American Airlines 11 of 12 92% 11 Unattributed non-completion
5 Air Canada 20 of 22 91% 2 Consent wall
6 IndiGo 10 of 11 91% 13 Unattributed non-completion
7 Air India Express 19 of 21 90% 2 Failure transparency
8 Qatar Airways 19 of 22 86% 2 Unattributed non-completion
9 Ryanair 19 of 22 86% 2 Unattributed non-completion
10 Emirates 16 of 19 84% 5 Unattributed non-completion
11 Ethiopian Airlines 18 of 22 82% 1 Unattributed non-completion
12 JetBlue 18 of 22 82% 2 Unattributed non-completion
13 KLM 18 of 22 82% 2 Unattributed non-completion
14 Porter Airlines 18 of 22 82% 2 Unattributed non-completion
15 Southwest 12 of 15 80% — Consent wall

Showing the top 15 of the 31 airlines tested. Ranked by completion rate (tasks completed ÷ measurable tasks); an airline needs at least 5 measurable tasks to be ranked, so a single lucky run cannot top the board. Blocked runs (access walls, including bot protection) are not failures and are excluded from the rate. The headline numbers below cover all 31 airlines, not just this list: 480 of 609 measurable tasks completed (79%). 10 additional runs failed on infrastructure grounds and are excluded, not counted as fails.

Methodological information
Measurable completion rate
79%

480 of 609 completed

Measurable runs
609

22 distinct tasks across 31 airlines

Blocked
119

119 of 728 runs

Airlines tested
31

airlines in this sweep

Not a full-matrix sweep. 31 of the 81 airlines in our target list have persisted runs, covering 22 distinct measurable tasks. Coverage is uneven per airline, which is why the board requires 5+ measurable tasks. Agent model: claude-opus-5-5 (744); run 2026-09 to 2026-10. Runs were driven in Claude's own browser (Claude Cowork). Each outcome is decided by the task's deterministic success check against the agent's final answer; the step-by-step browsing trace is reported by the agent itself.

Top failure signals across the measurable sample: taskCompletion (75) · failureTransparency (23) · consentWall (17) · inputAffordances (14)

Blocked is not failure. A booking-reference lookup that correctly asks for a booking reference is a site working as designed. 119 of 728 airline runs hit an access wall — a login, PNR or payment gate, or bot protection (67 of them bot walls) that stopped the agent before the task could be attempted. Blocked runs are excluded from the Index — never counted as a completion, never counted against the site.

// The loop

Assess. Diagnose. Optimize.

01 — Assess Run the benchmark Run the test suite against your own website and see where an agent actually completes the job — the full Index, task by task. Open the full Index
02 — Diagnose Read the failure signals Per-run failure signals name what broke — discoverability, understanding, interaction — not just complete or fail.
03 — Optimize Apply the playbook Fixes grouped by the dimension they repair — discoverability, understanding, interaction — and sequenced by what has to be fixed first. Read the playbook
Playground Try it in the playground — Friction Airways, a test airline we own. Open the playground

Takeaway_01

480 of 609 measurable airline tasks completed — 79% across 31 airlines.

Takeaway_02

Failure transparency is the leading attributed failure signal — 23 of 609 measurable runs failed silently, with no legible error or empty-result state.

Takeaway_03

The controlled site we own goes from 0% to 100% — hostile to ready, same tasks, same check.

// WebMCP case study

Judge WebMCP by the outcome it produces

Friction Airways is a controlled airline site we own, published in three WebMCP-readiness profiles. Same site, same 21-task measurable slice — only the profile changes.

Hostile
0% · 0/21
Median
71% · 15/21
Ready
100% · 21/21

Hostile's fails cluster on one signal: 18 of 21 measurable fails returned no machine-readable answer at all; ready has zero measurable fails. This is the assess → diagnose → optimize → retest loop in miniature, run on one controlled site, not an airline-industry claim.