Wrong ad A/B tests: statistics and duration mistakes
Calling winners early locks budget on wrong creative.
Calling an A/B winner in two days or testing five variables at once — wrong tests lock budget on wrong creative. Ad A/B needs statistics and duration discipline. Test protocol must be written.
Single variable rule
Changing headline + image + audience together hides the winner. One test = one hypothesis.
Control vs variant; multivariate needs enough traffic.
- Single variable
- Control group
- Written hypothesis
- Traffic minimum
Duration and sample
At least 7 days, preferably 14 or 100+ conversions per goal. Weekday/weekend and season day mix.
Early stopping (peeking) biases stats — write stop rules upfront.
Google Experiments
Campaign experiment or ad variation. 50-50 split; don't open during learning.
RSA asset test: pinning breaks test — unpin during test.
Meta split test
A/B tool at campaign level; same budget and duration auto. Scale winner, kill loser.
Separate fatigue from A/B — diagnose fatigue first.
Metric choice
Upper funnel: CTR, CPC. Lower: CPA, ROAS. Don't decide early on one metric.
Statistical significance: simple calc or tool; 95% confidence.
Documentation
Test log: hypothesis, start, end, result, action. Coordinate with creative fatigue.
Archive losers — don't repeat mistakes.
Bad tests produce bad decisions — duration and single variable are non-negotiable.
Build A/B test protocol with paid ads. Contact us.
