What does 95% confidence actually mean?

A 95% confidence level means that if you repeated the experiment 100 times, you would expect a similar result at least 95 times if the effect is real. It does NOT mean there is a 95% chance the result is correct. The 95% level is a convention - some contexts warrant higher (99%) or lower (90%) thresholds.

How many visitors do I need for a statistically significant A/B test?

The sample size depends on your baseline conversion rate and the minimum detectable effect. For a site with a 3% conversion rate wanting to detect a 10% relative improvement, you need approximately 30,000 visitors per variant. Online sample size calculators can determine the exact number for your situation.

Can I check significance before my test is complete?

Checking early inflates false positive rates because you are effectively running multiple tests. If you check at 50%, 75%, and 100% of target, your actual false positive rate may be 15-20% instead of 5%. Use sequential testing methods if you need to monitor during the test, or commit to checking only at completion.

Book Free Assessment Calculator

Statistical Significance

A determination that the result of an analysis or experiment is unlikely to have occurred by random chance alone, typically measured at a 95% confidence level.

In-Depth Explanation

Statistical significance measures confidence that an observed result reflects a genuine effect rather than random variation. It is fundamental to reliable data-driven decisions, particularly in A/B testing and performance analysis.

Key concepts:

P-value: The probability of observing results as extreme as the data, assuming no real effect. A p-value below 0.05 is the conventional threshold
Confidence level: The complement of alpha (1 - alpha). At alpha = 0.05, confidence level = 95%
Alpha (significance level): The threshold for rejecting the null hypothesis, typically 0.05
Null hypothesis: The default assumption that there is no real effect or difference
Type I error (false positive): Concluding there is an effect when there is not
Type II error (false negative): Failing to detect a real effect

Practical considerations:

Sample size: Larger samples increase the ability to detect smaller effects
Effect size: Statistical significance does not mean practical importance
Multiple comparisons: Testing many things increases false positive risk
Practical significance: A result can be statistically significant but too small to matter
Confidence intervals: Provide more information than p-values alone

Common misinterpretations:

A p-value is NOT the probability that the result is due to chance
Significance does NOT mean the effect is large or important
Lack of significance does NOT prove there is no effect - it may indicate insufficient data
Very large samples can make trivial differences appear "significant"

Business Context

Understanding statistical significance prevents businesses from acting on random noise, ensuring that changes are genuinely effective rather than coincidental.

How Clever Ops Uses This

Clever Ops integrates statistical significance testing into the A/B testing and experiment frameworks we build for Australian businesses. We ensure results are reported with proper statistical context, preventing premature decisions based on inconclusive data.

Example Use Case

"A marketing team runs an A/B test on a landing page and sees a 5% improvement in conversion rate. Statistical significance testing confirms the result at 98% confidence, giving confidence to implement the change."