How long should an A/B test run?

A test should run until it reaches statistical significance, which depends on your traffic volume and the size of the effect you are measuring. As a general rule, run tests for at least one to two full business cycles (typically 2-4 weeks) to account for day-of-week and seasonal variations. Never stop a test early because it looks like one version is winning.

What is statistical significance in A/B testing?

Statistical significance indicates the probability that the observed difference between variants is real and not due to random chance. A 95% confidence level means there is only a 5% chance the result is due to randomness. Most businesses use 95% as their threshold, though higher-stakes decisions may warrant 99%.

Can you A/B test with low traffic?

Low-traffic sites can still A/B test, but tests will take longer to reach significance. Strategies for low traffic include testing larger changes (which produce bigger effect sizes), focusing on high-impact pages, using Bayesian methods that work better with smaller samples, and being patient with test duration.

Book Free Assessment Calculator

A/B Testing

Also known as:split testingbucket testingcontrolled experiment

A method of comparing two versions of a web page, email, or other asset to determine which performs better by randomly splitting traffic between the two variants and measuring outcomes.

In-Depth Explanation

A/B testing (also called split testing) is a controlled experiment where two versions of a variable are compared against each other to identify which produces better results. It is a fundamental technique in data-driven decision-making.

How A/B testing works:

Hypothesis: Form a clear hypothesis about what change will improve performance
Variants: Create version A (control) and version B (treatment) with a single variable changed
Randomisation: Randomly assign users to see either version A or B
Measurement: Track the key metric (conversion rate, click-through rate, revenue, etc.)
Analysis: Use statistical methods to determine if the difference is significant
Decision: Implement the winning variant or iterate further

Statistical considerations:

Sample size: Ensure sufficient data for statistically significant results
Statistical significance: Typically aim for 95% confidence level
Duration: Run tests long enough to account for daily and weekly patterns
Multiple testing: Avoid the pitfall of testing too many variables simultaneously
Novelty effect: Account for initial changes in behaviour that may not persist

Advanced testing approaches:

Multivariate testing: Testing multiple variables simultaneously
Bandit algorithms: Dynamically allocating traffic to better-performing variants
Sequential testing: Analysing results as data accumulates rather than at a fixed endpoint
Bayesian A/B testing: Using Bayesian statistics for more intuitive probability statements

Common A/B testing applications:

Website layout, copy, and design elements
Email subject lines, content, and send times
Pricing strategies and offers
Onboarding flows and user journeys
Ad creative and targeting

Business Context

A/B testing removes guesswork from business decisions by using data to validate changes before full implementation, improving conversion rates, customer experience, and revenue.

How Clever Ops Uses This

Clever Ops implements A/B testing frameworks for Australian businesses, integrating testing tools with analytics platforms to create a continuous optimisation cycle. We help clients move from opinion-based decisions to evidence-based improvements across their digital touchpoints.

Example Use Case

"An e-commerce business tests two different checkout page layouts, discovering that the simplified version increases completion rates by 12% with 95% statistical confidence."