smell Status: ready

Scanner Theater

Status: ready

Symptom

The team says accessibility is fine because the scanner passed. Nobody walked through the task with VoiceOver, Larger Text, Switch Control or Voice Control.

Accessibility:  automated scan passed
Decision:       ship

Nobody knows whether a user can finish checkout, recover an account or contact support.

Where it shows up

Release gates, PR templates, CI dashboards, vendor reports, App Store label preparation, QA summaries, audit decks.

Why it blocks the task

A scanner catches real baseline issues: missing labels, low contrast, small hit areas, some traits. That's useful. It can't tell whether the declined card error is reachable, whether focus lands on a stale element after a sheet closes, whether Switch Control can reach a row action, or whether checkout survives AX5 in German. Those are the bugs that block tasks.

A scanner pass is a guardrail, not proof.

Root cause

The tool's output became the goal.

A CI score replaces the walkthrough.
The report has no user profile and no task.
Findings are fixed one by one without re-running the flow.
App Store labels are filled in from a tool pass.
Compliance review accepts a scanner report as evidence.

Before

Tool:      passed
Task:      —
Profile:   —
Evidence:  —
Decision:  ship

This says the scanner passed. It doesn't say the app can be used.

Fix

Verdict: the scanner runs on every PR and blocks merges. The release decision is made on task evidence. A task is never marked "verified" by a scanner alone.

Why: the scanner is cheap and catches regressions early. The walkthrough is expensive and catches what decides whether users can complete the task. Each one does the job the other can't.

The scanner in CI:

1. Fails the PR on missing labels and hit region issues on critical screens.
2. Posts findings to the PR.
3. Doesn't mark any task as verified.

The release record:

Task:      checkout after a declined card
Profile:   VoiceOver
Result:    fail
Evidence:  transcript + screenshot
Impact:    user can't continue checkout
Decision:  blocker
Owner:     checkout team

An accepted finding is a written decision, not a silenced rule:

Finding:   low contrast on a disabled label
Status:    accepted
Reason:    the reason for the disabled state is shown in full contrast below
Owner:     design systems
Review:    2027-03-01

If the scanner report is used for ADA, EAA or procurement, see Compliance Theater and Regulatory Evidence Map.

Check

Profile:   release review
Task:      approve checkout accessibility
Expected:  scanner result attached
           VoiceOver walkthrough evidence attached
           Larger Text AX5 screenshot attached
           blocker decision written

Profile:   PR review
Task:      review scanner findings
Expected:  each finding has a status
           no task marked verified by the scanner alone

WCAG mapping

Scanners cover some criteria well and others not at all.

4.1.2 Name, Role, Value: partly automatable.
1.4.3 Contrast (Minimum): partly automatable.
2.4.3 Focus Order: needs a walkthrough.
3.3.1 Error Identification: needs a walkthrough of the error path.

App Store and product risk

Accessibility Nutrition Labels are claims about common tasks. A label claimed from a scanner pass and contradicted by a real user's review is a trust problem.