Archive for category design of experiments
A helpful hierarchy for statistical analysis spells out how deep to drill on the statistics
Posted by mark in design of experiments on April 26, 2015
Fred Dombrose, a force for use of statistical design of experiment in biomedical research, alerted me to an enlightening article on statistics asking “What is the question?” in the March 20 issue of Science magazine. It lays out these 6 types of data analyses laid out by biostatistician Jeffrey Leak:
- Descriptive
- Exploratory
- Inferential
- Predictive
- Causal
- Mechanistic
For the distinguishing details going up this ladder see this Data Scientist Insight blog. However, the easiest way to determine where your study ranks is via the flowchart provided in the Science publication. There you also see four common mistakes that stem from trying to get too much information from too little data.
“Poor design of experiments is one of, if not the most, common reason for an experiment to fail.”
– Jeff Leek, “Great scientist – statistics = lots of failed experiments”, simplystats blog of 4/12/13
Design of experiments (DOE) most important for optimizing products, processes and analytical technologies
Posted by mark in design of experiments, Uncategorized on March 1, 2014
According to this February 2014 Special Report on Enabling Technologies two-thirds of BioProcess readers say that DOE makes the most impact on their analytical work.
“The promise of effective DOE is that the route of product and process development will speed up through more cost-effective experimentation, product improvement, and process optimization. Your ‘batting average’ will increase, and you will develop a competitive advantage in the process.”
–Ronald Snee
A lot to love in new release of software–v9 is mighty fine!
Posted by mark in design of experiments on February 7, 2014
Here’s a shout-out for Valentine’s Day that there’s a lot to love in the new release of Stat-Ease software-see the major improvements here http://www.statease.com/dx9.html#description. Consultant Wayne Adams put v9 to fun use by developing equations that produced the 3D response surface renderings of the heart and number 9. Geeks rule! (No offense, Wayne, I am one.)
Must we randomize our experiment?
Posted by mark in design of experiments on December 31, 2013
In the early 1990s I spoke at an applied statistics conference attended by DOE gurus George Box and Stu Hunter. This was a time when Taguchi methods had taken hold, which engineers liked because the designs eschewed randomization in favor of ordering by convenience–with hardest-to-control factors changed only once during the experiment. I might have fallen for this as well, but in my early days in R&D I worked on a high-pressure hydrogenation unit that, due to risks of catastrophic explosion, had to be operated outdoors and well away from any other employees. (Being only a summer engineer it seemed that I was disposable.) Naturally the ambient conditions varied quite dramatically at times, particularly in the Fall season when I was under pressure (ha ha) to wrap up my project. Randomization of my experiment designs provided me insurance against the time-related lurking variables of temperate, humidity and wind.
I was trained to make runs at random and never questioned its importance. Thus I was really surprised when Taguchi disciples attending my talk picked on me for bothering to do so. But, thank goodness, Box had already addressed this in his 1989 report Must We Randomize Our Experiment. He advised that experimenters:
- Always randomize in those cases where it creates little inconvenience.
- When an experiment becomes impossible being subjected to randomization
- and you can safely assume your process is stable, that is, any chance-variations will be small compared to factor effects, then run it as you can in non-random order;
- but, if due to process variation, the results would be “useless and misleading” without randomization, abandon it and first work on stabilizing the process;
- or consider a split-plot design.
I am happy to say that Stat-Ease with the release of version 9 of its DOE programs now provides the tool for the compromise, as Box deems it, between randomizing or not, that is—split plots. For now it is geared to factorial designs, but that covers a lot of ground for dealing with hard-to-change factors such as oven temperature in a baking experiment.* Details on v9 Design-Expert® software can be found here http://www.statease.com/dx9.html along with a link to a 45-day free trial. Check it out!
*For a case study on a split-plot experiment that can be easily designed, assessed for power and readily analyzed with the newest version of Stat-Ease software, see this report by Bisgaard, et al (colleagues of Box).
Educational system turned upside down by distance-based learning
Posted by mark in design of experiments, pop on April 14, 2013
I’ve been watching with interest the trend for ‘flipping’ classrooms; that is, using time together for working on homework and leaving the teaching to web-based and other materials (books, still!) for students to teach themselves on their own time. At the college level this new educational approach for is gaining momentum via massive open online courses, called MOOCS.
For example University of Minnesota chemistry professor Chris Cramer will teach this 9-week MOOC on Statistical Molecular Thermodynamics starting next month. Follow the link and watch him demonstrate a thermite reaction. If anyone can make statistical molecular thermodynamics interesting, it will be him, I think, so I enrolled. It’s free, thus there’s nothing to lose. Also, I still feel guilty about getting an A grade in the stat thermo class I took 30 years ago—the reason being it was graded on a curve and thus my abysmal final score of 15 out of 100 got rated highly as the second highest in my class. As you can infer, it was not taught very well!
P.S. I recently unveiled a distance-based lecture series on design of experiments called the DOE Launch Pad. It augments my book (co-authored by Pat Whitcomb) on DOE Simplified. Contact me at mark @ statease .com to sign up. It’s free for now while in pilot stage.
George Box–a giant in the field of industrial design of experiments (DOE)
Posted by mark in design of experiments, Uncategorized on March 30, 2013
George Box passed away this week at 94. Having a rare combination of numerical and communication skills along with an abundance of common sense, this fellow made incredible contributions to the cause of industrial experimenters. For more about George, see this wonderful tribute by John Hunter.
My memorable stories about Box both relate to his way with words that cut directly to a point:
- In 1989 at the Annual Quality Congress in Toronto seeing him open his debate with competing guru Genichi Taguchi by throwing two words on an overhead projector–“Obscurity” and “Profundity”, and then after a dramatic pause, adding the not-equal sign between them. This caused Taguchi’s son Shin to leap up from the front row and defend his father. This cause the largest crowd I have ever seen at a technical conference to produce a collective gasp that one only rarely experiences.
- In 1996 at a DOE workshop in Madison, Wisconsin enjoying his comeback to a very irritating disciple of Taguchi who kept interrupting the lecture: “If you are going to do something, you may as well do it right.”
Lest this give the impression that Box was mean-spirited see this well-reasoned white paper that provides a fair balance of praise and criticism of Taguchi, who created a huge push forward for the cause of planned experimentation for quality improvement.
The body of work by George Box in his field is monumental. It provides the foundation for all that we do at Stat-Ease. Thank you George, may you rest in peace!
Random thoughts
Posted by mark in Basic stats & math, design of experiments, Uncategorized on August 26, 2012
The latest issue of Wired magazine provides a great heads-up on random numbers by Jonathan Keats. Scrambling the order of runs is a key to good design of experiments (DOE)—this counteracts the influence of lurking variables, such as changing ambient conditions.
Designing an experiment is like gambling with the devil: only a random strategy can defeat all his betting systems.
— R.A. Fisher
Along those lines, I watched with interest when weather forecasts put Tampa at the bulls-eye of the projected track for Hurricane Isaac. My perverse thought was this might the best place to be, at least early on when the cone of uncertainty is widest.
In any case, one does best by expecting the unexpected. That gets me back to the topic of randomization, which turns out to be surprisingly hard to do considering the natural capriciousness of weather and life in general. When I first got going on DOE, I pulled numbered slips of paper out of my hard hat. Then a statistician suggested I go to a phone book and cull numbers from the last 4 digits from whatever page opened up haphazardly. Later I graduated to a table of random numbers (an oxymoron?). Nowadays I let my DOE software lay out the run order.
Check out how Conjuring Truly Random Numbers Just Got Easier, including the background by Keats on pioneering work in this field by British (1927) and American (1947) statisticians. Now the Australians have leap-frogged (kangarooed?) everyone, evidently, with a method that produces 5.7 billion “truly random” (how do they know?) values per second. Rad mon!
Strategy of experimentation: Break it into a series of smaller stages
Posted by mark in design of experiments on June 20, 2011
Tia Ghose of The Scientist provides a thought-provoking “Q&A” with biostatistician Peter Bacchetti on “Why small is beautiful” in her June 15th column seen here. Peter’s message is that you can learn from a small study even though it may not provide the holy grail of at least 80 percent power.* The rule-of-thumb I worked from as a process development engineer is not to put more than 25% of your budget into the first experiment, thus allowing the chance to adapt as you work through the project (or abandon it altogether). Furthermore, a good strategy of experimentation is to proceed in three stages:
- Screening the vital few factors (typically 20%) from the trivial many (80%)
- Characterizing main effects and interactions
- Optimizing (typically via response surface methods).
For a great overview of this “SCO” path for successful design of experiments (DOE) see this detailing on “Implementing Quality by Design” by Ronald D. Snee in Pharm Pro Magazine, March 23, 2010.
Of course, at the very end, one must not overlook one last step: confirmation and/or verification.
* I am loathe to abandon the 80% power “rule”** but, rather, increase the size of effect that you screen for in the first stage, that is, do not use too fine a mesh.
** For a primer on power in the context of industrial experimentation via two-level factorial design, see these webinar slides posted by Stat-Ease.
Video of paper-helicopter fly-offs at South Dakota School of Mines & Technology
Posted by mark in design of experiments on May 10, 2011
Stat-Ease Consultant Brooks Henderson produced this video — it’s quite impressive!
For background on the paper helicopter experiment, see this previous StatsMadeEasy post.
Brain-bending thoughts on a coffee experiment
Posted by mark in design of experiments, Uncategorized, Wellness on October 24, 2010
The Stat-Ease training center here at our world headquarters in Minneapolis features a wonderful single-cup brewing system that you can see demoed here. When we are not holding a workshop, I sometimes sneak in to steal a cup late in the day. By then I am reaching my limit, so I brew a “half-calf” at the half-cup setting. Being a chemical engineer, I calculate that, in this case, half of half makes a whole, that is, coffee with the normal concentration of caffeine. Does that make sense?
Making a tasty and effective cup of coffee is a huge deal for knowledge workers who need to keep their heads in gear from start to finish of every single day. One of our workshop students, a PhD, has been picking my brain about testing coffee blends on her staff of scientists. She proposes to do a mixture design such as I did on varying types of beers (see Mixture Design Brews Up New Beer Cocktail—Black & Blue Moon).
Obviously overall liking on a sensory basis should be first and foremost for such an experiment on coffee – a 5 to 9-point scale works well for this.* However, the tricky part is assessing the impact of coffee for accelerating information processing and general problem-solving, which I hypothesize depends on level of caffeine. I wonder if an online “brain training” service, such as this one developed by neuroscientists at Stanford and UCSF, might provide a valid measure.
The down side of doing a proper test on whether coffee improves cognitive skills will be the necessity of reverting to the base line, that is, every morning getting up and trying to function without the first cup.
“A mathematician is a machine for turning coffee into theorems.”
— Alfréd Rényi
*Turn your volume down (to not hear the advert) and see this primer on sensory evaluation by S-Cool– a UK educational site for teenagers.