Agile2018 has ended
Back To Schedule
Tuesday, August 7 • 10:45 - 12:00
The Joy of Proving Yourself Wrong: A Case Study in A/B Experimentation (Jenny Wanger)

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Limited Capacity seats available

Getting A/B experimentation to work for you is a science that goes beyond purchasing a tool. It's about creating a culture of learning and celebrating failed hypotheses alongside the winning ones. It's about putting processes in place to make sure you're following a rigorous testing regimen. And along the way, it's about making a lot of mistakes.
Through this interactive workshop, we'll go through a case study of a recent experiment my team ran at SpotHero and you'll have a chance to set the variables up yourself. We'll learn about how complicated these experiments can get and how to discover if you've made an error in your test setup. We'll look at what you should do to build on the momentum of a failed experiment. You'll learn why the only way an experiment can go wrong is bad data, not bad ideas.
A/B testing is complicated and even the best teams make mistakes. This session is aimed at experienced practitioners who want to improve team processes, learn how to design tests so that they provide accurate data, and recover from those errors when they do happen. It's about finding the joy in a failed experiment and making sure each test is set up so you're guaranteed to learn something. A strong experimentation regimen can take you from amateur tinkerer to data-driven guru, where every losing variation makes you smile.

Learning Outcomes:
  • Why you should test everything: The power of measuring the success and failure of everything you release.
  • Hypothesis framing: How to use qualitative research to set up your hypotheses for A/B experimentation
  • Different kinds of tests: When to use multivariate versus single-variation experiment structures and what they are
  • Iterative experimentation: Why set up iterative experiments where you run one test and follow it immediately with a new variation, and how that should change the way you design your tests
  • All the things that can go wrong: Different, unexpected ways your experiment can be ruined due to poor data, and why the only way an experiment can really fail is inconclusive data, not a losing variation.


avatar for Jenny Wanger

Jenny Wanger

Product Manager, SpotHero
Jenny Wanger is a senior product manager for SpotHero, a top-ranking app that helps people park easier. Her years of product management experience prior to that include running the developer experience team at Arity, a startup founded by Allstate in downtown Chicago, and consulting... Read More →

Tuesday August 7, 2018 10:45 - 12:00 PDT
Marina F