This is the part of growth marketing most agencies describe in one line. It is where results are won or lost, so we set out exactly how we do it.
1. Build the growth model before the backlog
We agree one North Star Metric, the single measure that best reflects the value customers get from you, and break it into the input metrics a team can influence week to week. Every experiment must name the input it expects to move.
2. Score every idea the same way
Ideas go into one backlog and are scored on expected impact, confidence in the evidence and ease of delivery. The scoring is visible to you. It stops the loudest opinion in the room from setting the plan.
3. Design the test before it starts
Each experiment has a one-page brief written before launch. It is the discipline set out in Trustworthy Online Controlled Experiments by Kohavi, Tang and Xu, the standard reference on A/B testing.
- A written hypothesis: because we observed X, we believe changing Y will move metric Z.
- One primary metric, plus guardrail metrics that must not get worse, such as refund rate or lead quality.
- The smallest effect worth detecting, and the sample size needed to detect it at 95% confidence and 80% power.
- A fixed duration that covers complete weeks, so weekday and weekend behavior are both included.
- A check that traffic was split as planned. A broken split invalidates the result.
- A decision rule agreed in advance: ship, iterate or stop.
The reason for this rigour is that intuition is a poor guide. Harvard Business Review reports that a headline change at Bing, shelved for months as low priority, raised revenue by 12% once an engineer tested it.
An experiment brief, as you would receive it
| Hypothesis |
Session recordings show visitors scrolling past the demo form to look for pricing. Showing a starting price above the form will increase demo requests. |
| Primary metric |
Demo request rate |
| Guardrail metrics |
Share of requests that sales accept as qualified; page load time |
| Baseline and target |
3.0% today; the smallest lift worth acting on is 15% relative, to 3.45% |
| Sample needed |
About 24,000 visitors per version at 95% confidence and 80% power |
| Duration |
Four full weeks at current traffic |
| Decision rule |
Ship if the lift is significant and accepted-lead share holds. Otherwise stop and log the learning. |
Illustrative example to show the format. It is not a client result.
4. Measure incrementality when an A/B test is not possible
Many B2B companies and most paid channels cannot be split cleanly. In those cases we use holdout groups, geography-based tests, or marketing mix modeling with open tools such as Google's Meridian. We also read analytics for what it is. Google Analytics currently offers three attribution models: data-driven, paid and organic last click, and Google paid channels last click. Each one assigns credit. None of them proves that a sale would not have happened anyway, which is why a control group matters.
5. Protect organic search while testing
Page experiments follow Google's guidance on website testing. Search engines see what users see, canonical tags point to the original URL, split-URL tests use temporary redirects, and each test ends when its planned sample is reached.
6. Keep every result, including the losses
A large share of well-run tests show no measurable effect. We report those as plainly as the wins, because a result that stops you spending on the wrong thing has value. Every experiment is logged in a library you own, so the same idea is not paid for twice.