Last month the Wall Street Journal “Bookshelf” (3/15/20, David A. Shaywitz) featured a review of a book about The Surprising Power of Business Experiments.
“Tests at Microsoft in 2012 revealed that a tiny adjustment in the way its Bing search engine displayed ad headlines resulted in a 12% increase in revenue, translating into an extra $100 million annually for the company in the U.S. alone.”
Stefan Thomke, author of Experimentation Works: The Surprising Power of Business Experiments.
It’s great to see attention paid to the huge advantages gained from statistically rigorous experiments. However, vastly greater returns await those willing to go beyond simple-comparative one-factor A/B testing to multifactor design of experiments. The reason is obvious: Only by testing more than one factor at a time, can interactions be discovered.
A case in point is provided by an experiment I did on postcard advertisements. It produced a non-intuitive finding that, unlike marketers, our engineering clients preferred less colorful layouts. Knowing this, we succeeded in increasing our response at a far lower printing cost. See the proof in the interaction plot at the conclusion of this white paper on That Voodoo We Do – Marketers Are Embracing Statistical Design of Experiments.
Another compelling example of the value of multifactor testing is illustrated by website-conversion results* shown here—produced from a replicated, full, two-level factorial design.
The key to a more than 5-fold increase in clicks turned out to be the combination of going to a modern font (factor A) with a more compelling button label (C). A third factor (B), background being white versus blue, did not create a significant effect, which also provided valuable insights on the drivers for conversion.
Why settle for testing only one factor when, without investing much more time, if any, you can investigate many factors and, as a huge bonus, detect possible interactions?
*From Pochiraju & Seshadri, Essentials of Business Analytics, 2019, Springer, p 737.