SEO + GEO Experiments With Claude Code and a Control Group

What the SEO experiments loop does

SEO experiments only tell you something when the pages you changed moved more than a group of untouched pages. We run our site's SEO fixes through that rule. Claude Code writes the experiment record before it touches a page, changes a randomised treated group, leaves a matched control group alone, and a script compares the two at day 15 and day 31. ConvOps runs it all as scheduled tasks with workflows, and a human merges every change.

A pre-registered SEO experiment is a page change whose hypothesis, metric, page groups and kill rule are committed before the change ships, and never edited after. A page family is one page plus its locale variants (en, es, de). Families are the unit the loop assigns to treated or control.

Every screen and number below uses example data for Tidewell, a fictional invoicing app at tidewell.example.com.

How the loop runs: monthly research and human-approved strategy feed a weekly planner that creates experiment tasks, a daily pulse wakes the ones that are due, everything reads and writes one ledger repo, and a human merges every change before it goes live.

Four schedules drive it: a daily pulse, a weekly planner, and monthly research and strategy.

Four schedules drive the loop: a daily pulse, a weekly planner, monthly research and monthly strategy, each with its rule and what it does.

Why we built it: one page is noise, and agents invent numbers

Most small-site SEO tests lie to you: one page swings by a third with nothing changed. Google updates, seasonality, crawl timing and site-wide dips all move a page on their own. A before and after chart of one page cannot tell a fix from a quiet week.

The answer is a control group from the same site, assigned at random at the same moment. Both groups face the same conditions, so the gap between them is the effect. Google's testing guidance says the time a reliable test needs depends on how much traffic the site gets (Google Search Central, 2025(opens in new tab)). This loop never changes a URL or adds a redirect, and a small site gets 28 measured days plus one extension.

One page swings by a third week to week with nothing changed, while treated and control groups move together through a site-wide dip, so the gap between them is what you measure.

The second problem is the agent. Ask an agent to "check the data" and it computes ratios, compares them with benchmarks it made up, and recommends changes. Ours did exactly that. So a script computes every metric and verdict, and Claude Code runs the steps and reads the results.

If you want a scheduled loop like this, with steps, gates and human approvals, see how ConvOps runs it(opens in new tab).

How it runs: four schedules, five workflows

An experiment loop is only as honest as its clock, so nothing here waits for a person to remember it.

The planner only uses a strategy memo whose approval task is completed. Without one, it keeps its defaults. A memo steers weeks of experiments, so a person signs it.

A pool is the set of families where one metric is measurable: AI crawler fetches, Google impressions, position or click-through rate. An arm is one fix type applied to one group of page families. A wave is the set of arms started together on one Monday, sharing one control group.

Each arm is its own task with a 10-step workflow, a graph of steps and gates, the shape we describe in What is graph engineering?.

Every merge waits for a person, the human gate described in Building human-in-the-loop AI systems.

The experiment workflow as a graph: register, implement and review, a Ship step that needs human approval, then Go live (day 0), Interim (day 15) and Final (day 31), with gate outcomes jumping to Rollback or Act before Learn.

Between looks, the task parks on its due date and the daily pulse wakes it.

A run, step by step (example data)

Here is one full Tidewell run, from the Monday planner to the day 31 verdict.

Monday: the planner opens a wave

The weekly planner's report: eligible families and arm capacity per pool, the fix types drawn for each wave, the seeded assignment, and four experiment tasks left for the next pulse to dispatch.

The planner counts eligible families and arm capacity per pool (callout 1). Capacity is the eligible count divided by 15, rounded down, minus one: Tidewell's 68 AI-pool families allow 3 arms. The Google pool is capped at 1 arm, and each pool runs one live wave at a time.

A pool gets a wave only when its historical A/A baseline is ok. That baseline replays past data with fake splits and counts how often noise alone looks like a win (2.1% in the example). A pool that fails it gets no wave, because any verdict there would be noise.

Fix types that have won more get picked more often (callout 2), with about 20% of arms kept for fix types with no verdict (Thompson sampling, Beta(1 + wins, 1 + losses + inconclusives)).

Assignment uses the date as the random seed, shuffles within each diagnosis group, and deals families round-robin with control first (callout 3). Each group gets 17 families. The next pulse, not the planner, dispatches the four tasks.

Register: the record comes first

The experiment record at registered: frontmatter, one falsifiable hypothesis, the rule that kills it, families and baseline, all committed before the agent touches a page.

The record holds one fix type (callout 1), one falsifiable sentence (3), the rule that kills it (4) and a 28-day baseline (5). After the commit, only the status and the dates move. We pre-register because an agent, like a person, can find a story in any number after the fact; a committed kill rule leaves nothing to reinterpret.

Implement, review, ship, go live

On arm A, Claude Code adds a question-led summary and FAQ block to the 17 treated invoice-template families. That is the one variable. It never touches URLs, slugs, redirects, shared layout or facts, and it moves the date on every changed page. Static review passes; a human merges. Here is arm B of the same wave, parked after go-live.

A sibling experiment task after go-live: each step's note on the task, the live check highlighted, Interim (day 15) next, and the due date set to day 15 so the task sleeps until then.

Go live checks the treatment in production HTML on all 17 pages (callout 1). Google says crawling can take from a few days to a few weeks (Google Search Central, 2025(opens in new tab)), so day 0 is the day the change is verified live, not the merge day. Go live sets day 0 and both due dates, then parks the task until day 15 (callouts 2 and 3).

Day 15 and day 31: a script decides

The Final (day 31) step carries the decision rules in its instructions: run the verdict script, then win, loss, or inconclusive with one extension to day 59.

The decision rules live in the step instructions (callout 2), so the agent cannot argue with them. It runs the verdict script and follows the matching branch.

The decision note on the same task records each look as a row; here the final look is inconclusive, so the task is extended once to day 59.

At day 15 the script reads ratio 1.08, p 0.31: no harm, no early win, continue. At day 31 it reads ratio 1.11, p 0.19 (callout 1): inconclusive, so the task extends once to day 59 and parks again.

What the output looks like: the ledger

The ledger, not a traffic chart, is what this loop produces.

The ledger keeps one row per experiment with the real column names, and the status column shows where each one is in its run.

An experiment moves through registered, awaiting-merge, live, interim-done and final, and can end as extended, rolled-back or void. Each arm is its own task with its own verdict.

The verdict script runs a family-level permutation diff-in-diff over 10,000 shuffles. The ratio is the treated group's change divided by the control group's change over the same window, so 1.5 means treated moved 50% more than control. For AI fetches (each metric has its own thresholds), a win needs p ≤ 0.0477 at day 31, a ratio of 1.5 or more, and at least 60% of treated families above the control median. A day 15 AI-fetch result at p ≤ 0.0074 that also meets the ratio and share rules stops the arm early: it is final and goes straight to Act. A loss is the mirror image (ratio 0.67 or less, 60% below the median), and rollback follows. Anything else, or fewer than 15 families per arm, is inconclusive.

The two p thresholds split one 5% false-win budget across the two looks, so stopping early at day 15 costs no extra false wins. The ratio and the 60% share stop a result that is statistically real but tiny, or carried by one big page.

How a look becomes a decision: at the day 15 interim an AI-fetch result at p ≤ 0.0074 with ratio ≥ 1.5 is an early win that stops the arm and goes to Act, harm rolls back, a Google update voids the look, anything else continues; the day 31 final returns win, loss or inconclusive (thresholds from verdict.py) with one extension to day 59; a win only files a replication idea.

Inconclusive is not a loss. It says the effect, if there is one, is smaller than this site can detect. The task extends once to day 59 and then closes, because a test that runs until it wins eventually wins on noise. A win does not spawn more experiments either. It files a replication idea for a human, because a 5% false-win budget means some wins are noise, and only a second run on fresh families tells them apart.

What goes wrong: the agent that invented a crisis

Our worst morning came from an agent trying to be helpful. Our daily check skipped its check list and wrote its own report. It computed a click-through rate, compared it with an "industry standard" nobody gave it, labelled it CRITICAL, recommended changes and cited tracking bugs that had never been filed. Every number looked plausible. None came from a defined check. We archived the report and rewrote the step the same day. Here it is retold on Tidewell data.

Before: the agent judged CTR against a made-up benchmark and recommended changes; after: a script builds the snapshot and only four fixed checks decide, catching one real anomaly, a sitemap returning 500.

The rule it produced has two parts. First, a script builds the snapshot, and the agent never computes a snapshot number or writes the file by hand. Second, only four fixed checks count as anomalies:

1. Impressions fall below half of their 7-day mean.
2. AI crawler fetches fall below 50% or rise above 300% of their 7-day mean.
3. A family in a live wave is missing from the data for 7 days.
4. The homepage or sitemap does not return HTTP 200 to an AI crawler.

No CTR judgements, rankings, benchmarks or recommendations. A bug is cited only by the id the tracker returned.

The second failure was quieter, and worse. A package release for one of our sites carried an unreleased product change, and it rewrote copy on one of our control pages before day 0. A control that changes is no longer a control: any gap it shows is our own edit, not the fix. We caught it before go-live. The rule: a family touched after assignment moves to "Excluded after assignment" and leaves the comparison.

Build your own: the minimal version

You do not need five workflows to start. You need one repo, two scripts, one record template and one daily scheduler.

The minimal version you can copy: a daily scheduler runs a snapshot script into one ledger repo, records come from one template, and a verdict script on due dates returns keep, extend or revert.

The record template, one file per experiment:

---
experiment: exp-0001
site: example
wave: example-2026-01-05
arm: A
fix_type: internal-links
primary_metric: impressions
status: registered
branch: exp/exp-0001
day0: null
interim_due: null
final_due: null
---
Hypothesis:
One falsifiable sentence: what changes, on which treated families, against which control, by day 31.

Falsified if:
The day 31 verdict is not a win under the thresholds fixed in the verdict script.

Families:
treated (15+): ...
control (15+): ...

Baseline (28 days):
treated ... | control ...

Verdicts:
| Look | Date | n treated / control | Ratio | p | Verdict |
|---|---|---|---|---|---|

Lesson:

Copy the record template, then schedule the daily pulse in ConvOps(opens in new tab) so every experiment gets woken on its due date.