Archive for category design of experiments
Gambling with the devil
Posted by mark in Basic stats & math, design of experiments on November 15, 2009
In today’s “AskMarilyn” column by Marilyn vos Savant for Parade magazine she addresses a question about the game of Scrabble: Is it fair at the outset for one player to pick all seven letter-tiles rather than awaiting his turn to take one at a time? The fellow’s mother doesn’t like this. She claims that he might grab the valuable “X” before others have the chance. Follow the link for Marilyn’s answer to this issue of random (or not) sampling.
This week I did my day on DOE (design of experiments) for a biannual workshop on Lean Six Sigma sponsored by Ohio State University’s Fisher College of Business (blended with training by www.MoreSteam.com.) Early on I present a case study* on a training experiment done by a software publisher. The goal is to increase the productivity of programmers by sending them to workshop. The manager asks for volunteers from his staff of 30. Half agree to go. Upon their return from the class his annual performance rating, done subjectively on a ten-point scale, reveals a statistically significant increase due to the training. I ask you (the same as I ask my lean six sigma students): Is this fair?
“Designing an experiment is like gambling with the devil: only a random strategy can defeat all his betting systems.”
– RA Fisher
PS. I put my class to the test of whether they really “get” how to design and analyze a two-level factorial experiment by asking them to develop a long-flying and accurate paper helicopter. They use Design-Ease software, which lays out a randomized plan. However, the student tasked with dropping the ‘copters of one of the teams just grabbed all eight of their designs and jumped up the chair. I asked her if she planned to drop them all at once, or what. She told me that only one at a time would be flown – selected by intuition as the trials progressed. What an interesting sampling strategy!
PPS. Check out this paper “hella copter” developed for another statistics class (not mine).
*(Source: “Design of Experiments, A Powerful Analytical Tool” by Christopher Nachtsheim and Bradley Jones, Six Sigma Forum Magazine, August 2003.)
Small sample sizes produce yawning results from sleep studies
Posted by mark in Basic stats & math, design of experiments on July 15, 2009
“Too little attention has been paid to the statistical challenges in estimating small effects.”
— Andrew Gelman and David Weakliem, “Of Beauty, Sex and Power,” American Scientist, Volume 97, July-August 2009 .
In last week’s “In the Lab” column of the Wall Street Journal (WSJ)*, Sarah Rubinstein reported an intriguing study by the “light and health” program of the Rensselaer Polytechnic Institute (RPI). The director, Mariana Figueiro, is trying to establish a lighting scheme for older people that will facilitate their natural rhythms of wakefulness and sleep. In one 2002 experiment (according to WSJ), Dr. Figueiro subjected four Alzheimer patients to two hours of blue, red or no light-emitting diodes (LEDs). After then putting the individuals to bed, their nurses made observations every two hours and found that the “blue-light special” out-did the red by 66% versus 54% on how often they caught patients napping.
Over the years we’ve accumulated many electrical devices in our bedroom – television, cable box, clocks, smoke and carbon monoxide monitors, etc., which all feature red lights. They don’t bother me, but they keep my wife awake. So it would be interesting, I think, if blues would promote snooze. Unfortunately the WSJ report does not provide confidence intervals on the two percentages – nor do they detail the sample size so one could determine statistical significance on the difference of 0.12 (0.66 minus 0.54). (I assume that each of the 4 subjects were repeatedly tested some number of times.) According to this simple calculator posted by the Southwest Oncology Group (a national clinical research group), it would take a sample size of 554 to provide 80% power for achieving statistical significance at 0.05 for this difference!
So, although whether blue light really does facilitate sleep remains questionable, I am comforted by the testimonial of one of the study participants (a 100 years old!) – “It’s a beautiful light,” she says.
PS. Fyi, for more sophisticated multifactor experimentation (such as for screening studies), Stat-Ease posted a power calculator for binomial responses and provided explanation in its June 2009 Stat-Teaser newsletter .
* “Seeking a Light Approach to Elderly Sleep Troubles,” p. D2, 7/7/09
Does good experimental design require changing only one factor at a time (OFAT)?
Posted by mark in design of experiments, science on June 23, 2009
“Good experimental design usually requires that we change only one factor at a time” according to an article I read recently in The Scientist magazine (“Why Don’t We Share Data,” page 33, Issue 4, Volume 23). This guide for science fairs tells students that “you conduct a fair test by making sure that you change only one factor at a time while keeping all other conditions the same.”
Obviously changing two variables together makes no sense, such as the time that as science project one of my kids asked me to do a blind taste test on Coke versus Pepsi, but to keep them straight in their mind, she poured one cola in blue plastic cup and the other in white Styrofoam! Needless to say I was completely confounded.
The OFAT method is so engrained that it’s literally become the law according to scientist who told me that, when as an expert witness he presented statistically significant evidence, it was thrown out of court due to the experiment design having changed multiple factors simultaneously. What a crime!
Multifactor testing is far more effective for statistical power, screening efficiency and detection of interactions. Industrial experimenters are well-advised to forget their indoctrination in OFAT and make use of multifactorial designs. For reasons why, see my two-part series on Trimming the FAT out of Experimental Methods and No-FAT Multifactor Design of Experiments.
Good experimental design does NOT require changing only one factor at a time!
Awesome demonstration of design of experiments
Posted by mark in design of experiments on April 27, 2009
Team Awesome
The engineering students at South Dakota School of Mines and Technology really do rock. Where else could one present a class on statistics until 8:30 pm on a Friday night and continue it less than 12 hours later – early on a Saturday morning?
Our workshop on design of experiments (DOE) finished with a spirited competition of paper helicopters.* The winner was Team Awesome: Kayla Rithmiller, MacKenzie Trask and Samantha Johnson (pictured from left to right). They scored highest on the basis of flight time and accuracy. You can see their ‘copter spinning to another precise landing in their confirmation run.
Congratulations to Team Awesome and all the SDSM&T students who devoted their free time to learning DOE and demonstrating this newly-gained knowledge via well-planned experiments on the helicopter exercise. I predict that they all will go far!
*See details on this DOE exercise in the September 2004 Stat-Teaser article on Playing with Paper Helicopters.
