A/B Testing 101 For Product Managers

A/B testing is a method that involves the comparison of two versions of a webpage or app to determine which one leads to better results. In product management, an A/B test would involve testing a webpage design change against the current design by showing both versions to site visitors at the same time. Half of the visitors will see version A while the other half would see version B, and the version that drives better results wins.

Why Does a Product Need to Undergo A/B Testing?

While there are other methods for getting customer feedback, such as focus group discussions and surveys, A/B testing is an effective way for product managers to validate their ideas and consequently build the right product. Customer feedback from focus groups and surveys, on the other hand, are great for generating ideas for product features and improvements.

How to Run an A/B Test

While the usual ratio for splitting your users is 50-50, with 50% being shown the current webpage design and the other 50% sees the new layout, the ratio needed to make the test achieve significant results will depend on the number of your users. If you are Facebook, you might only need to test on 1% of your users. But for other companies, the recommended percentage is 50% for a minimum period of two weeks. Testing would involve hypotheses, and this process might make you feel like it slows down the development of your product. But really, A/B testing enables a faster advancement of your product by eliminating wastes and complexities from implementing product features that would eventually prove to be ineffective and unsuccessful.

Key Factors in A/B Testing

For product managers to succeed in their A/B testing, there are four key factors that they need to have for this process. For one, product managers should begin by formulating well-informed hypotheses. Also, everything users should be logged so that there is a thorough understanding of the product – what happened, what went wrong in the test, and why some of the users did not react as expected. The third factor necessary in an A/B test is having a solid technical framework. And lastly, statistical significance need to be considered when analyzing the results of the A/B test. This ensures that conclusions are derived from true positives.

Avoiding the Local Maximum

One major challenge when running A/B tests and other test methods is encountering the local maximum. This happens when a product has reached the point where small changes can no longer make significant improvements to it. This can be avoided by intermittently taking working on major features. For these, the project can be broken down into smaller pieces and executed incrementally.

For product managers to succeed in launching a great new product, the new product needs to address the users’ needs and effectively convince them to make the purchase. A product management training teaches the importance of A/B tests in ensuring that the next product that is launched is exactly what the users want and need.

Image Source : Creative Commons

About Author:

Michelle Rubio has been writing for SMEs across the United States, Canada, Australia and the UK for the last five years. She is a highly-experienced blogger and SEO copywriter, writing business blogs for various industries such as marketing, law, health and wellness, beauty, and education, particularly on product management training such as those offered by ProductSchool.com.