Online media and social-advertising necessitate new ways to measure and drive data-based decision-making among customers. They are also creating a new field of experimental learning techniques and tools that are replacing classic, randomized market testing practices in many cases.
Dean Eckles, a social scientist, statistician and assistant professor in the MIT Sloan School of Management, explained how tools for designing, deploying and analyzing online field experiments can encourage good statistical and methodological practices as well as better understanding of online customer behavior. As MIT Sloan professor Glen Urban and IDE researcher, Sinan Aral, and others, have discussed, different types of ads and messaging are being tested all the time to determine what motivates online marketing and social activities.
Eckles, a former member of the Core Data Science team at Facebook who also worked at Yahoo, knows first-hand that “the Internet industry has distinct advantages in how organizations can use data to make decisions. Firms can cheaply introduce numerous variations on the service and observe how a large random sample responds when randomly assigned to these variations.”
At the same time, he told a recent MIT IDE seminar, “rapid, iterative, and organizationally distributed experimentation” also introduces important challenges-- such as understanding the effects of a change intervention.
Challenges arise because many experiments are being run -- often by different teams -- requiring tools to support rapid experimentation. For example, "How can multiple different teams experiment with the design of a single page at the same time?"
PlanOut: An App for Experimental Design
Facebook was seeking alternatives to standard A/B tests to answer questions like these about its users. A/B tests work well when minor tweaks to a system are needed, but not when more complicated or nuanced change has to be measured. Eckles and his team team built an open-source app called PlanOut, a language for describing complex experimental designs for behavioral science experiments. It uses standard code script to assign value to specific procedures and also can help manage multiple testing that takes place simultaneously on a site. Less technical users can program it via a GUI.
Eckles is interested in other applications for these types of tools and analytics that might, for example, show the best way to motivate voting or civic participation.
Feedback is important on social media content, he said, and experiments can focus on straightforward items such as comment boxes to measure user engagement or they can analyze more subtle factors such as how comments affect users and influence others.