What other A/B testing experts do when there is no clear winner

When there is no clear winner (results are very close, significance reached), what do you do? We asked A/B testing experts what they thought and how they operated at their companies.

Will Kurt, Data Scientist at Kissmetrics

The big thing to keep in mind with A/B testing is you’ll get better results in the long run if you stick with only decisions that are clear wins. I would recommend sticking with the original since the uncertainty in the results means you always risk that the other variant is worse. It’s a much better use of your time to wait until you’ve found a really great variant, and not risk too much on variants that are likely to bring you little if any gain.

Jake Peterson, Growth at Segment.io

After 2 weeks I stop the experiment, then I wait another 2 weeks to see the long term effects on metrics like revenue or what plans they are on. Trials for software as a service businesses are anywhere from 2 weeks to 30 days, and I can see what happened as a result of a particular test. I’ll also ask qualitative questions to my team like “what resonated with customers more?” Our sales team has a good beat on this as they are talking with customers every day. The bottom line, I think that when there is no clear winner that you look at how your long term metrics are affected, not just what the experiment was trying to optimize.