# Playwright MCP Acceptance Testing: Agents Run the Journeys

Agent-driven acceptance testing is a verification loop where an AI agent drives each user journey in a real browser and reports pass or fail against frozen acceptance criteria. We run acceptance testing with Playwright MCP on ConvOps: the agent that tests never fixes, and a failure goes back to the build until the whole run passes again.

By David Marsa, Founder & CEO. Updated 2026-10-12.

## What you will be able to do

- Write observable acceptance criteria and freeze them before the build starts.
- Turn each criterion into one journey with preconditions and observable steps.
- Prepare and prove the test environment in a separate step before the tester runs.
- Run the journeys with Playwright MCP and read the report by its verdicts.
- Wire a fix-and-rerun loop where nobody certifies their own fix.

## Key takeaways

- Agent-driven acceptance testing has an AI agent drive each user journey against the running app and issue a verdict per frozen acceptance criterion.
- Freeze the criteria before code exists and give each one exactly one journey that quotes it word for word.
- The agent that tests never fixes: a failure goes back to the build, and only a full re-run counts as proof.
- BLOCKED is a verdict, not a gap to hide; read it before FAIL.
- This layer does not replace CI; it catches the defects that live between layers that each passed on their own.

## What acceptance testing with Playwright MCP does here

**Agent-driven acceptance testing** is a verification loop where an AI agent drives each user journey in a real browser and reports pass or fail against frozen acceptance criteria. API and MCP journeys run the same way, as real calls against the running app.

A test the builder writes for itself proves nothing. We want a verdict from an agent that only reads the criteria and drives the product. After Claude Code builds a feature, a second agent walks every journey with Playwright MCP and screenshots each deciding observation. ConvOps runs the steps, the gates and the fix loop. It is one of the [loops we run our company on](/learn/guides/run-a-company-with-ai-agents).

Every example uses Quillbay, a fictional class-booking app for pottery studios, and its new waitlist for full classes.

| It is not | Because |
|---|---|
| User acceptance testing | Business users sign off there; here an agent verifies approved criteria |
| Testing an AI agent | The agent is the tester, not the subject |
| Criteria turned into test code | That ends in a `.spec.ts` file. This ends in a verdict per journey, with evidence |

| Part | What it is |
|---|---|
| Trigger | The build steps of a plan finish. No schedule |
| Inputs | Frozen acceptance criteria, the journey plan, the running app on its own branch environment |
| Outputs | A report with one verdict per journey, screenshots, and a recommendation: PROCEED or FIX REQUIRED |

![Criteria are frozen first; a failure goes back through a fix and the whole run repeats.](https://storage.googleapis.com/neomanex-public-assets/neomanex/guides/playwright-mcp-acceptance-testing-loop-diagram.png)

## Why we built it: green layers, broken journeys

Green unit tests tell you the parts work. They say nothing about whether a person can finish the job on screen.

Before this loop, every lower layer passed: unit tests, phase reviews, the regression suite on the merge pipeline. Then the first agent-driven run of the journeys against the real app failed a large share of them anyway. The defects lived between layers: a link built by one service and read by another, a redirect that dropped a parameter. No layer owned the seam, so none caught it.

Two convictions came out of it. The builder cannot be the judge, so the tester never writes or fixes code. And only a run against the real app proves a criterion.

If your agents' features pass every check and still break on screen, [see how ConvOps runs a plan from frozen criteria to a verified build](https://convops.app).

## How it runs: six steps and four agents

The order of the steps is the safety. Criteria are frozen before code exists, the environment is proven before the tester arrives, and the decision is taken by someone who did not write the fix.

| # | Step | Who runs it | Gate |
|---|---|---|---|
| 1 | Write the criteria | Drafted with the plan, approved by a person | Approval freezes the list |
| 2 | Plan the journeys | Test planner | None |
| 3 | Prepare the environment | Infrastructure agent | Green only, then a person approves |
| 4 | Run the journeys | Acceptance tester | A person approves the report |
| 5 | Decide | Decision step | Pass, or fail into the fix |
| 6 | Fix the failures | Developer agent | None |

Claude Code builds the feature between steps 2 and 3.

**Write the criteria.** Each names an actor, a behavior and something you could watch: a screen, a message, a response. "Code exists" is never a criterion, because what you cannot watch cannot be accepted. Approval freezes the list: later steps never add, drop or reinterpret one.

**Plan the journeys.** One journey per criterion, quoted word for word, its id cited, so every criterion has exactly one place where it is proven.

**Prepare the environment.** A dedicated infrastructure agent brings up the plan's own branch environment, provisions data and credentials by the plan's mechanism, and records them. The gate is on green, not on effort. The tester prepares nothing, so it cannot pass by building its own easy case.

**Run, decide, fix.** The acceptance tester drives the journeys and never fixes. A fail sends the report to the developer agent, which commits fixes on the same branch, and the whole plan runs again.

The loop has no timer: it runs after every plan's build, and on a task whenever a journey changed. A task-level run derives 3 to 8 journeys: the happy path plus the highest-risk error path per changed behavior. Skipping is allowed only when there is no journey to drive (a pure refactor, tooling, docs, config or test-only changes), with the ground stated. When in doubt, run.

Microsoft's [Playwright Test Agents](https://playwright.dev/docs/test-agents) (planner, generator, healer) produce and repair test code. Ours produce a plan from frozen requirements and a verdict report. The gates work as in [how we build a self-improving AI system](/learn/guides/build-self-improving-ai-system), and the approvals are [human-in-the-loop gates](/blog/human-in-the-loop-ai-systems).

![Four agents plan, prepare, test and fix; a person approves the criteria and reads every report.](https://storage.googleapis.com/neomanex-public-assets/neomanex/guides/playwright-mcp-acceptance-testing-who-runs-each-step-v2.png)

## A run, step by step: the Quillbay waitlist

The interesting moment is not the first FAIL. It is the re-run after the fix, because the decision step accepts no other proof.

**Trigger.** The build steps of `plan/waitlist` finish. Criteria FR-1 to FR-4 were frozen before any code existed.

**The plan.** Journey U2 cites FR-2 and quotes it: "When a seat frees, the first learner in line receives an email whose link opens the class with the seat held for 2 hours." Each precondition says how to confirm it: class `acc-wl-r7-class-wheel-basics` is full at 8 of 8 (the page shows "Full" and Join waitlist), both learners can sign in, and outgoing mail lands in the capture inbox.

![Each journey quotes its criterion word for word and cites its id.](https://storage.googleapis.com/neomanex-public-assets/neomanex/guides/playwright-mcp-acceptance-testing-criterion-journey-v2.png)

**Prepare.** The infrastructure agent proves `wl-waitlist.app.quillbay.example.com` and its API green, and records the learner fixtures and the studio owner's key for reuse.

**Run 1.** The acceptance tester drives four journeys at 1440x900 and re-checks layout screens at 390x844.

| Journey | Verdict | What it saw |
|---|---|---|
| U1 | PASS | "You are #1 on the waitlist" |
| U2 | FAIL | The email link opens "Class not found": it carries the waitlist entry id, not the class id |
| U3 | BLOCKED | P3 missing: no mail capture in the branch environment, named in the header |
| A1 | PASS | `GET /v1/classes/{id}/waitlist` as the owner returns 200 with learner A at position 1 |

Recommendation: FIX REQUIRED. U2 is a seam: API and app each pass their unit tests, and only the journey crosses both.

**Decide.** A person approves the report, and the decision is fail. U2 is code this plan shipped, so it is fixed on the branch, not filed as a ticket.

**Fix.** The developer agent reproduces U2 on the same environment, changes the link, runs the unit test for the touched file, and commits `fix(waitlist): email link uses the class id` (`4c1e9a7`) on `plan/waitlist`. The infrastructure agent configures mail capture.

**Run 2.** The whole plan runs again: all four journeys, not only U2 and U3. All pass, and U2's screenshot shows "Seat held for you for 2 hours". Recommendation: PROCEED. The decision is pass, and the report feeds the merge request's verification section.

![The fix counts only after the full plan runs again and every journey passes.](https://storage.googleapis.com/neomanex-public-assets/neomanex/guides/playwright-mcp-acceptance-testing-round1-fix-round2-v2.png)

## What the report looks like, and how to read it

Read BLOCKED before FAIL. A journey nobody could drive is the one most likely to hide a real defect.

| Part | What it holds |
|---|---|
| Header | Run, previous run with counts, branch and head, hosts, viewports, verdict |
| Required header statements | Everything the run could not test, each named. Run 1: "P3 mail capture not configured: U2, U3 depend on it" |
| Journey results | One row per journey, previous beside current run, evidence images |
| Failures in detail | Step, expected, actual, evidence, recommended fix |
| Regression checklist | Booking a non-full class still works. Cancelling a booking still frees the seat |
| Recommendation | PROCEED or FIX REQUIRED |

| Verdict | Meaning |
|---|---|
| PASS | Every step observed as the criterion states |
| PARTIAL PASS | The journey works, but a stated observation does not fully hold |
| FAIL | A step's actual result differs from the expected one |
| BLOCKED | The journey could not be driven; the missing item is named |
| COVERED BY SUITE | Proven by an automated regression test the report cites, not driven again |

Our plans state one more rule: a step that needed an undocumented prerequisite, a retry or a manual fix-up is a FAIL, even if the behavior worked in the end. A pass that needed help will not pass for a user. And a PASS whose screenshot does not show the asserted element is not evidence: the person accepting the verdict opens the images.

![Read the header and BLOCKED rows first: they say what the run could not test.](https://storage.googleapis.com/neomanex-public-assets/neomanex/guides/playwright-mcp-acceptance-testing-read-the-report-v2.png)

## What goes wrong: seams no layer owned

Most of our acceptance failures were not bugs the model missed. They were seams between layers that no layer owned.

The main one has the shape of Quillbay's U2: a first acceptance run fails a large share of journeys while every lower layer is green, and every defect sits between layers. The rule it produced: run acceptance on a real environment before the review steps stop being cheap, and budget a second full run for any plan with UI, API and MCP surface.

![Each layer passed on its own; the defects lived between them, where only a journey crosses.](https://storage.googleapis.com/neomanex-public-assets/neomanex/guides/playwright-mcp-acceptance-testing-green-layers-broken-journeys.png)

| Failure | Rule it produced |
|---|---|
| Fix rounds introduced regressions the next run caught. On Quillbay, a U2 fix that also tidied the class page would break U1 | Fix only the reported failures, no tidy-ups. The full re-run catches what a fix broke, and a person decides when to stop looping |
| Two sessions shared one Playwright MCP browser. A sibling navigated the tab mid-journey and the observed state vanished | Each session drives its own browser context or a standalone Playwright script, re-finds its page on every call, and checks the URL before reading |
| A journey passed on the dev server and showed stale content once built | Journeys that depend on prerender or cache mode run on a built image or deployed environment: a dev server renders every request fresh |
| Missing configuration left a journey BLOCKED for several runs, hiding a separate defect behind it | BLOCKED is a verdict that names the missing item in the header. The run never fakes a pass |
| A plan held checks the environment could never satisfy: a rule already broken on main, a login the environment did not have | The test planner checks each journey against what the environment can actually do |

Finding and fixing as separate jobs is also the spine of our [self-healing website loop](/learn/guides/self-healing-website-ai-agent-qa-fix-loop).

## Build your own: the instructions each agent receives

Start with the criteria and the "never fix" rule. Playwright MCP is the easy part. The tester is for verification, not investigation: debug first, then run acceptance once you believe the build works. The steps below give the order; these are the instructions, ready to copy.

**Test planner: the journey template.**

```text
For each frozen criterion, write ONE journey:
  ID: U<n> (UI) or A<n> (API/MCP)
  Criterion: <FR id>, quoted verbatim
  Preconditions: each with "How to confirm".
    A miss is a blocker, not a workaround.
  Steps: numbered; each states what you will observe.
  Viewports: 1440x900; re-check every layout-bearing
    screen at 390x844.
  Fixtures: prefix acc-<plan>-<run>-; sweep after the run.
Check each journey against what the environment
can actually do.
```

**Infrastructure agent: prepare and verify.**

```text
Prepare the plan's branch environment, then prove it green.
1. Start every service the plan touches. All must be up.
2. Provision data and credentials ONLY by the plan's
   mechanism. No mechanism: stop and ask. Record the
   credentials for the tester.
3. Verify:
   - health returns 200
   - a REAL data endpoint returns 2xx, not just health
   - unauthenticated call: 401/403; authenticated: 2xx
     (MCP: tools/list AND one authenticated tool call)
   - 3-5 key pages via Playwright: no console errors,
     no failed requests, no blank screens
   - the main flow works end to end once
4. Fix what is red, re-verify, advance only on green.
   Report product defects with location and evidence.
Do not run the journeys. Do not tear down.
```

**Acceptance tester: run and report.**

```text
The environment is prepared and running. Reuse the recorded
credentials. Do not start or tear down anything.
Run every UI journey with Playwright MCP, every API/MCP
journey with real calls, then the regression checklist.
Follow the plan in order. Never fix code. Never skip a
failure. No improvised tests.
Screenshot each observation that decides a step.
Log API response bodies.
Verdict per journey: PASS, PARTIAL PASS, FAIL, BLOCKED,
COVERED BY SUITE.
Report: Header; Required header statements (what you
could not test, each named); Journey results; Failures in
detail (step, expected, actual, evidence, recommended
fix); Regression checklist; Recommendation: PROCEED or
FIX REQUIRED.
```

**Decision rule.**

```text
PASS only when every journey is PASS or an accepted
exception: a failure in code this change did not ship,
tracked as its own ticket and cited by id.
Anything else is FAIL: send the report to the fix, then
run the WHOLE plan again.
A failure in code this change shipped is fixed on the
branch, never filed as a separate ticket.
Unclear whether a behavior is intended? A person decides.
```

**Developer agent: fix.**

```text
For each failed journey: reproduce it on the same
environment, find the cause, fix it, re-drive the
journey live.
Fix only what the failures require. No nits, no tidy-ups.
Run unit and lint checks for the touched files only.
Commit on the same branch. Record what changed and why.
Never certify your own fix: the re-run is the proof.
```

This layer does not replace CI: unit, integration and regression tests stay the deterministic gate in the merge pipeline.

[Copy these instructions, then add the acceptance steps to your workflow in ConvOps](https://convops.app).

## Steps

1. **Write observable criteria and freeze them**: Give each criterion an actor, a behavior and something you could watch on screen or in a response. A person's approval freezes the list; a scope change goes back to this step.
2. **Write one journey per criterion**: Quote the criterion verbatim, cite its id, list preconditions with how to confirm each, and state an observation for every step.
3. **Prepare the environment in its own step**: An infrastructure agent provisions data and credentials by the plan's mechanism and proves health, a real data endpoint, auth both ways and key pages green before anyone tests.
4. **Run the journeys with a separate tester**: Give the acceptance tester Playwright MCP in its own browser context, real calls for API journeys, and two rules: never fix, never skip.
5. **Read the report and decide**: Read BLOCKED rows first. Pass only when every journey is PASS or an accepted exception tracked as its own ticket.
6. **Fix on the branch, then re-run everything**: The developer agent fixes only the reported failures and commits on the same branch. The whole plan runs again; nobody certifies their own fix.

## FAQ

### What is agent-driven acceptance testing?

Agent-driven acceptance testing is a verification loop where an AI agent drives each user journey in a real browser and reports pass or fail against frozen acceptance criteria. ConvOps runs it as workflow steps: a person freezes the criteria, a test planner writes one journey per criterion, a separate acceptance tester drives them with Playwright MCP, and any failure is fixed on the branch before the whole plan runs again.

### Who is agent-driven acceptance testing for?

Agent-driven acceptance testing is for engineering leads and founders who ship features with coding agents such as Claude Code. ConvOps runs it after the build of every plan, so each criterion a person approved is proven on the running app, with a screenshot per deciding observation, before the merge request is reviewed.

### Can AI agents be used in QA testing?

Yes, as the journey layer. In ConvOps, an acceptance tester agent drives each user journey with Playwright MCP and each API journey with real calls, then reports a verdict per journey with evidence. It never fixes code and never skips a failure. The regression suite in CI stays the deterministic gate, and a person reads the report before the decision.

### Is Playwright MCP reliable for acceptance testing?

Yes, as the vehicle, under two conditions. ConvOps proves the environment green before the tester arrives, and each session drives its own browser context, re-finds its page on every call and checks the URL before reading. A shared browser loses state when another session navigates the tab. Every deciding observation gets a screenshot that shows the asserted element.

### How do I make an AI coding agent test its own code in a real browser?

Add the ConvOps MCP server at https://mcp.convops.app/ to your AI client and sign in once. Then write observable acceptance criteria and freeze them, have a test planner write one journey per criterion, and give a second agent Playwright MCP to drive them. The builder never grades its own work: failures are fixed on the branch and the whole plan runs again.

### Does agent-driven acceptance testing replace human UAT?

No. In ConvOps, the agent verifies against criteria a person approved before the build, and a person reads and approves every report. A finding where nobody knows whether the behavior is intended goes to a person before the pass or fail decision. The agent rules on what it observed, never on intent.

### Does acceptance testing with an AI agent replace CI tests?

No. In ConvOps plans, unit tests, integration tests and the regression suite run in the merge pipeline as the deterministic gate. Agent-driven acceptance is a fourth layer: it drives journeys on the running app and catches defects that live between layers that each passed on their own. It does not gate the merge alone.

### What happens when the agent cannot test a journey?

The journey is BLOCKED, and the report header names the missing item, such as a mail capture inbox the environment lacked. ConvOps never counts BLOCKED as a pass. The infrastructure agent supplies what was missing, and the whole plan runs again. Read BLOCKED rows first: a journey nobody could drive can hide a real defect.

Canonical: https://neomanex.com/learn/guides/playwright-mcp-acceptance-testing