Abstract: Getting A/B experimentation to work for you is a science that goes beyond purchasing a tool. It's about creating a culture of learning and celebrating failed hypotheses alongside the winning ones. It's about putting processes in place to make sure you're following a rigorous testing regimen. And along the way, it's about making a lot of mistakes.
Through this interactive workshop, we'll go through a case study of a recent experiment my team ran at SpotHero and you'll have a chance to set the variables up yourself. We'll learn about how complicated these experiments can get and how to discover if you've made an error in your test setup. We'll look at what you should do to build on the momentum of a failed experiment. You'll learn why the only way an experiment can go wrong is bad data, not bad ideas.
A/B testing is complicated and even the best teams make mistakes. This session is aimed at experienced practitioners who want to improve team processes, learn how to design tests so that they provide accurate data, and recover from those errors when they do happen. It's about finding the joy in a failed experiment and making sure each test is set up so you're guaranteed to learn something. A strong experimentation regimen can take you from amateur tinkerer to data-driven guru, where every losing variation makes you smile.
Learning Outcomes: - Why you should test everything: The power of measuring the success and failure of everything you release.
- Hypothesis framing: How to use qualitative research to set up your hypotheses for A/B experimentation
- Different kinds of tests: When to use multivariate versus single-variation experiment structures and what they are
- Iterative experimentation: Why set up iterative experiments where you run one test and follow it immediately with a new variation, and how that should change the way you design your tests
- All the things that can go wrong: Different, unexpected ways your experiment can be ruined due to poor data, and why the only way an experiment can really fail is inconclusive data, not a losing variation.
Attachments:
