A minor change in a design may alter user behaviour in ways that are hard to detect in a usability test. An A/B test allows you to compare real-world user behaviour across different versions of the product.
In an A/B test the user is confronted with one or two versions of an interactive product. Remote software records metrics for user behaviour. These metrics are compared between the two versions to see which alternative is better. Sometimes more than two alternatives are tested: A/B/X testing.
Google does a lot of A/B testing on its search services. The tiniest details can count for Google because they have so many users. Some companies also use the A/B experimental design for testing concepts, rather than finished products.