QPRC 2016

Online Versus Offline Experimentation

Roger Longbotham

Microsoft

Abstract

Online and offline experiments rely on the same statistical principles but there are many differences in how they are carried out. Typically, online experiments have much larger sample sizes and have a software platform that facilitates and adds structure to these experiments. This allows non-technical people to run online experiments and to scale the number of experiments to thousands per site per year. This talk will discuss some of the details of how online experiments are carried out and go into additional detail on how online and offline experiments differ in practice and how a practicing statistician may make the transition to running online experiments.