What is statistical significance?
A result is statistically significant when the difference you observed is larger than random chance would plausibly produce if there were really no difference. It tells you an effect is probably real, not that it is large or worth acting on.
The p-value explained
The p-value is the probability of seeing a difference at least as large as the one observed if the two versions actually performed the same. A common threshold is 0.05: below it, the result is called significant. A p-value of 0.03 does not mean a 97% chance the new version is better; it means the data would be unusual if there were no difference.
Statistical vs practical significance
With enough traffic, a tiny difference becomes significant. A 0.2% lift can be real and still not worth the cost of shipping and maintaining a change. Decide in advance the smallest effect that would matter to the business, and judge results against it as well as against the p-value.
How long to run an A/B test
Work out the sample size for an A/B test before you start, from your baseline rate, the smallest effect you care about and the confidence you need, then run until you reach it. Run for whole weeks so weekday and weekend behaviour are both included, and do not stop the moment the result first looks significant: checking early and often inflates false positives.
Significance in price tests
Price tests are noisy, because revenue varies far more than clicks. Dynamic Pricing Engine gives each pricing experiment a clear verdict rather than leaving you to read meaning into a week of sales.
Last reviewed 2026-11-01.