11/20/2023 0 Comments What does high variance in data meanThe pre-experiment information we use is referred to as a "control variate". CUPED ĬUPED is a technique in which we use information about an experiment's users from before an experiment started to reduce the variance in their experiment metrics. Refer to our launch post for CUPED for more details. This observably leads to significant variance reduction in the large majority of metrics where CUPED can be applied. This is automatically applied to experiments at Statsig, and is run for the topline results on key metrics in Pulse. Because of this, techniques have been developed to reduce the variance in experiment results in order to reduce run times and increase confidence.Īt Statsig, we use a form of CUPED based on a 2013 Microsoft paper (Deng, Xu, Kohavi, & Walker). Reducing variance can lead to shorter experiment run times due to the lower sample required. Higher variance is associated with larger confidence intervals, and leads to experiments requiring more sample size to consistently observe a statistically significant result on the same effect size. Variance is a measurement of dispersion which measures the amount of "noise" in a metric or experiment results.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |