Why?

A minor change in a design may alter user behaviour in ways that are hard to detect in a usability test. An A/B test allows you to compare real-world user behaviour across different versions of the product.

How?

In an A/B test the user is confronted with one or two versions of an interactive product. Remote software records metrics for user behaviour. These metrics are compared between the two versions to see which alternative is better. Sometimes more than two alternatives are tested: A/B/X testing.

Ingredients

• Specialized software.
• Reliable metrics.
• A large amount of users.

In practice

Google does a lot of A/B testing on its search services. The tiniest details can count for Google because they have so many users. Some companies also use the A/B experimental design for testing concepts, rather than finished products.

Useful resources

• Universal Methods of Design (Hanington & Martin, 2012). Spends a couple of pages about A/B tests. In the Dutch version chapter 1. P8-9
• A good background read for doing an A/B test can be chapter 9 of Researching Information Systems and Computing (Oates, 2006) on experimental design p126-140