Business community discovers that “Experimentation Works”
Posted by mark in design of experiments on April 21, 2020
Last month the Wall Street Journal “Bookshelf” (3/15/20, David A. Shaywitz) featured a review of a book about The Surprising Power of Business Experiments.
“Tests at Microsoft in 2012 revealed that a tiny adjustment in the way its Bing search engine displayed ad headlines resulted in a 12% increase in revenue, translating into an extra $100 million annually for the company in the U.S. alone.”
Stefan Thomke, author of Experimentation Works: The Surprising Power of Business Experiments.
It’s great to see attention paid to the huge advantages gained from statistically rigorous experiments. However, vastly greater returns await those willing to go beyond simple-comparative one-factor A/B testing to multifactor design of experiments. The reason is obvious: Only by testing more than one factor at a time, can interactions be discovered.
A case in point is provided by an experiment I did on postcard advertisements. It produced a non-intuitive finding that, unlike marketers, our engineering clients preferred less colorful layouts. Knowing this, we succeeded in increasing our response at a far lower printing cost. See the proof in the interaction plot at the conclusion of this white paper on That Voodoo We Do – Marketers Are Embracing Statistical Design of Experiments.
Another compelling example of the value of multifactor testing is illustrated by website-conversion results* shown here—produced from a replicated, full, two-level factorial design.
The key to a more than 5-fold increase in clicks turned out to be the combination of going to a modern font (factor A) with a more compelling button label (C). A third factor (B), background being white versus blue, did not create a significant effect, which also provided valuable insights on the drivers for conversion.
Why settle for testing only one factor when, without investing much more time, if any, you can investigate many factors and, as a huge bonus, detect possible interactions?
*From Pochiraju & Seshadri, Essentials of Business Analytics, 2019, Springer, p 737.
Beware of birds making stick bombs in your backyard
Posted by mark in Nature, Uncategorized on April 13, 2020
One advantage of being home-bound during this COVID-19 pandemic is spending more time watching the birds. I especially like the cardinals who flock to my feeder in a variety of plumage being male, female or juvenile. However, the finches are fun to see as well, particularly the yellow ones. The other day a blue jay came by to provide a bit of blue for the color mix.
While greatly appreciating birds, I failed to recognize their superior engineering skills until reading this New York Times article about avian nest construction. They report how researchers at University of Akron used a ‘smushing’ chamber to measure how bird-homes bounce back after being compressed.
“We hypothesize that a bird nest might effectively be a disordered stick bomb, with just enough stored energy to keep it rigid.”
Hunter King, experimental soft-matter physicist, University of Akron*
I now feel a lot smarter saying “smushing”, it being a scientific term used by world-class physicists. However, I’m more interested in the stick-bomb bit. This is explained best by Popular Mechanics in their report (stemming from the same scientific study by Hunter et al) on Why the Humble Bird Nest Is an Engineering Marvel.
As a fun project to while away the time indoors, build your own stick bombs using popsicle or jumbo sticks such as those available here.
The stick bomb illustrated in this video by Brain Coach Don offers a great deal of excitement, but I do not recommend it for building bird nests—ha ha. The difference is them making ones that are disordered and thus nonexplosive.
*(Mechanics of randomly packed filaments—The “bird nest” as meta-material, Journal of Applied Physics 127, 050902 (2020))
Pi day—tau be or not tau be, that is the question
Posted by mark in pop, Uncategorized on March 14, 2020
Math-savvy pizza and pie shops around the world will be celebrating this afternoon of 3/14 at 3:14 pm to honor the mathematical constant pi.
Rounding pi to 3.14 suffices for most rational people, but those of you who are trained matheletes might like to carry this never-ending irrational number out to 100 or a 1000 decimal places. If so, knock yourself out at this post by math.com. You might as well quit at this point because the record is now 50 trillion digits, held by cybersecurity analyst Timothy Mullican who used 303 days of computation to complete this calculation, which he detailed here.
A good way to build up your chops on pi is to memorize a ‘piem’, that is a poem in which the length of each word represents a number, for example, “Now I need a drink, alcoholic of course, after the heavy lectures involving quantum mechanics.” See a much longer (101 digits!) piem sung by musician Andrew Huang and many other amazing feats related to pi in this article by Andrew Whalen posted today by Newsweek.
Sadly, some mathematicians are reigning in the pi parade by insisting it be doubled to the constant tau.
“To describe 3/4 of a circle in trigonometry, you would say 3/4 tau radians. But in the pi world, that’s 3/2 pi radians. ‘Blegh!’ says Prof. [Bob] Palais [Utah Valley University]. ‘People are so ingrained that they don’t even see how stupid it is.’”
For Math Fans, Nothing Can Spoil Pi Day—Except Maybe Tau Day
Wall Street Journal, 3/14/20
You’d best circle (ha ha-math joke) June 28 to celebrate Tau Day, even though that’s no reason to eat pizza or any other kind of pie.
Permanent calendar proposed to eliminate leap day
Posted by mark in pop, Uncategorized on February 29, 2020
“Those born on Feb. 29 know they exist, but the computer at the DMV is skeptical.”
Subheadline for the Wall Street Journal article today about how Leap-Year Babies Fight a Lonely, Quadrennial Fight for Recognition
Today being a day that comes only every four years is special—even more so now that my niece delivered a leap-year baby girl. Unfortunately, this precious little leapling (“LL”) faces a lifetime of calendar kerfuffles with computer systems that do not compute February 29th birthdays. However, a solution is at hand: the Hanke-Henry Permanent Calendar (HHPC). By HHPC’s reckoning this special day is March 2nd and comes again on Saturday next year and every year thereafter—no need for LL’s parents to wait four years to celebrate her first birthday.
Being that my birthday would always fall on a Monday, I cannot build much enthusiasm for the HHPC feature of any given date always falling on the same day (LL lucked out, though). Other off-putting days are Independence Day being on a Wednesday and Halloween being eliminated due to October ending at 30 days. But the weirdest aspect of HHPC is the “Xtr” week every 6 years. This year of 2020 features an Xtr, for example. Minnesotans do not need 7 more days of winter!
PS. Watch the video to see what would happen, if we did not add a day every 4 years: Eventually summer would become winter!
Major League Baseball experimenting with robot umpires
Posted by mark in sports, Uncategorized on February 19, 2020
After a somewhat successful* year-long trial of automated balls and strikes (ABS) in the Atlantic League, MLB will bring in these ‘robots’ to second-guess their human umpires at nine of Florida’s spring-training games. The ABS system makes use of TrackMan radar technology, already in play for StatCast.
After MLB’s tech-team improves ABS’s reliability and accuracy, it might be worth using, but only if it speeds up the game. Using ABS simply to challenge calls will just make things worse, while eliminating the spectacle of on-field blow-ups by volatile managers like Billy Martin (former Minnesota Twin). When the calls are made by invisible radar, who do you throw the dirt at?
“You turn back (to the umpire and say), ‘TrackMan?’ They say, ‘Yeah,'” “‘Well, I’m not going to argue with you.’ Because it’s the robots.”
Southern Maryland Blue Crabs outfielder Tony Thomas commenting on the experimental use of ABS in the Atlantic League
PS. When the baseball robots get smart enough to call balks, then we’d all best bunker down for them taking over the world.
*Baseball America reporting Imperfections And All, Robo Umps Make Significant Impact
Assessing the threat from Wuhan Coronavirus (2019-nCoV) via quantitative analytics
“In God we trust. All others must provide data.”
Dr. William Schaffner, a preventive medicine specialist at Vanderbilt University Medical Center, quoted by NY Times on 2/2/20 in their report that “Wuhan Coronavirus Looks Increasingly Like a Pandemic, Experts Say”.
Stay calm and carry on is a good mantra for these unsettled times. But it pays to keep an eye on developments with a critical view on the facts so far as they can be ascertained. Here are some interesting statistics and data-driven observations on the impact thus far and in future of 2019-nCoV from the New York Times:
- Fatalities now come to 425 in mainland China as of Tuesday morning according to this latest news, eclipsing Chinese deaths from SARS (349). For a view of its spread throughout Asia, see the Coronavirus Map via the link above.
- For a very enlightening interactive-graphical view graphics see this in-depth report.
Keep in mind that much remains to be determined. For example, it may turn out that 2019-nCoV for better (less fatal) or worse (more infectious) may turn out to be more like 2009 H1N1 Swine flu than SARS.
You, like me, may be curious about the effectiveness of wearing a mask. From this NPR investigation, I surmise that N95 respirators, such as those made by 3M, work best, but only if fitted properly and worn with great discipline in not touching one’s face. In any case during this flu season, we all need to wash our hands with soap and water often (hand sanitizers don’t work nearly so well by my reckoning) and stay home if we get sick.
For what it’s worth, that’s my take for now. You all must make your own judgments. All I suggest is that you not let fear rule—assess the data and adjust your thinking as more accumulates. But best you be conservative on the safe side.
PS. Those of you who fly frequently are well-advised to check out this advice from yesterday’s Wall Street Journal. I am comforted by the statistic of 99.9% particulate removal and thus “the cabin air environment is not conducive to the spread of most infectious diseases” per the CDC. However, as always, it’s best to be careful during times when the flu and/or other viruses run rampant.
Experiment provides fetching evidence about wolf puppies
Posted by mark in Nature, pop, Uncategorized on January 21, 2020
Swedish zoologists reported this month in iScience that Intrinsic Ball Retrieving in Wolf Puppies Suggests Standing Ancestral Variation for Human-Directed Play Behavior. This counteracted widely-held beliefs that wolves do not socialize with humans. It may suggest that ancestors of dogs were primed for domestication.
Given the small sample size—only 13 wolves—I am not so sure. But I always feel better after encountering puppies like Flea pictured in Gizmodo’s engaging report on the Swedish study. I look forward to more rigorous research on wolf puppies and hope to be picked as a tennis-ball tosser.
“When I saw the first wolf puppy retrieving the ball, I literally got goosebumps.”
Christina Hansen Wheat, a co-author of the study and a researcher from Stockholm University.
Achieve New Year’s resolution to reduce stress at your work-desk
With the page turned over to 2020, office-workers worldwide must bear down again and do even better than ever before. If you did really well in last year, that just raises expectations for a similar improvement over the next 12 months. Naturally stress levels rise and performance drops off into a downward spiral. But, no worries, a solution is at hand: Simply pick up a small plant, preferably fool-proof such as these fine fifteen recommended by House Beautiful. Then, per a study by Japanese scientists*, whenever you feel worn down, take a 3 minute break to gaze at your plant and, when needed, care for it. Based on the results from experiments on over 60 participants, you will become more relaxed (as measured by pulse rate) and relieved of anxiety (based on scoring).
“Nature can serve as an antidote to over-stimulation or “attention fatigue,” as well as boost cognitive performance.”
Cory Steig, Health and Wellness Reporter, Make It, CNBC, 3/3/20, “This 3-minute, $3 habit could lower your stress and anxiety at work”.
It works for me. In any case, happy New Year! Don’t let the stress get to you.
* Toyoda, et al, “Potential of a Small Indoor Plant on the Desk for Reducing Office Workers’ Stress”, 12/19/19, HortTechnology.
Testing the adage that if you drink beer before wine then you will feel fine
Just in time for the partying hearty for Christmas today, my son-in-law Ryan, a chemist with 3M, alerted me to a statistical study published after last year’s holiday season by the The American Journal of Clinical Nutrition that questioned the advice of grape or grain but never the twain. Naturally, being a drinker of these undistilled alcoholic beverages, I wondered if my tendency to drink beer before dinner and wine for the meal would pass the test. But being a wonk for design of experiments, I was most curious to see a randomized controlled multiarm matched-triplet crossover trial—pictured below for this experiment on the order of addition for beer and/or wine.
Based on results from 90 participants, including a control group, “neither type nor order of consumed alcoholic beverages significantly affected hangover intensity (P > 0.05)”. What really mattered was the total consumption, although, interestingly, hangover intensity did not correlate to breath alcohol concentration (BrAC). However, the authors warn that
“The fact that we did not find a direct correlation between maximal BrAC and hangover intensity should not be misinterpreted as an invitation to drink until the cows come home. Likely, this correlation overall does exist but is not directly apparent in the narrow range of peak alcohol levels studied here.”
It’s disclosed at the end that Carlsberg provided the beer (premium Pilsner lager recipe from 1847) free of charge “for the sole purpose of utilization in this study”. Although I trust the author’s disclaimer of any bias, perhaps further study is warranted with stronger beers such as a Belgian trippel. Maybe wine would then be best drunk first. To be continued…
Business people taking notice of pushback on p-value
Posted by mark in Basic stats & math on December 15, 2019
As the headline article for their November 17 Business section, my hometown newspaper, the St. Paul Pioneer Press, picked up an alarming report on p-values by Associated Press (AP). That week I gave a talk to the Minnesota Reliability Consortium*, after which one of the engineers told me that he also read this article and lost some of his faith in the value of statistics.
“One investment analyst reacted by reducing his forecast for peak sales of the drug — by $1 billion. What happened? The number that caused the gasps was 0.059. The audience was looking for something under 0.05.”
– Malcom Ritter, AP, relaying the reaction to results from a “huge” heart drug study presented this fall by Dr. Scott Solomon of Harvard’s Brigham and Women’s Hospital.
As I noted in this May 1st blog/, rather than abandoning p-values, it would pay to simply be far more conservative by reducing the critical value for significance from 0.05 to 0.005. Furthermore, as pointed out by Solomon (the scientist noted in the quote), failing to meet whatever p-value one sets a priori as the threshold, may not refute a real benefit—perhaps more data might generate sufficient power to achieve statistical significance.
Rather than using p-values to arbitrarily make a binary pass/fail decision, analysts should use this statistic as a continuous measure of calculated risk for investment. Of course, the amount of risk that can be accepted depends on the rewards that will come if the experimental results turn out to be true.
It is a huge mistake to abandon statistics because of p being hacked to come out below 0.05, or p being used to kill projects due to it coming out barely above 0.05. Come on people, we can be smarter than that.
* “Know the SCOR for Multifactor Strategy of Experimentation”