Sample size

A Sample size is the amount of data that’s required to get a statistically significant result. It can be reduced by changing the winning threshold.


Other things being equal, reducing the sample size decreases the amount of time it takes to get a result in an experience. The trade-off is that it also reduces the confidence we have in the result. As the sample size decreases, the confidence in our estimates of uplift also decreases.