• Screening the vital few factors (typically 20%) from the trivial many (80%)
• Characterizing main effects and interactions
• Optimizing (typically via response surface methods).

For a great overview of this “SCO” path for successful design of experiments (DOE) see this detailing on “Implementing Quality by Design” by Ronald D. Snee in Pharm Pro Magazine, March 23, 2010.

Of course, at the very end, one must not overlook one last step: confirmation and/or verification.

* I am loathe to abandon the 80% power “rule”** but, rather, increase the size of effect that you screen for in the first stage, that is, do not use too fine a mesh.

** For a primer on power in the context of industrial experimentation via two-level factorial design, see these webinar slides posted by Stat-Ease.

For background on the paper helicopter experiment, see this previous StatsMadeEasy post.

Making a tasty and effective cup of coffee is a huge deal for knowledge workers who need to keep their heads in gear from start to finish of every single day.  One of our workshop students, a PhD, has been picking my brain about testing coffee blends on her staff of scientists.  She proposes to do a mixture design such as I did on varying types of beers (see Mixture Design Brews Up New Beer Cocktail—Black & Blue Moon).

Obviously overall liking on a sensory basis should be first and foremost for such an experiment on coffee – a 5 to 9-point scale works well for this.*  However, the tricky part is assessing the impact of coffee for accelerating information processing and general problem-solving, which I hypothesize depends on level of caffeine.  I wonder if an online “brain training” service, such as this one developed by neuroscientists at Stanford and UCSF, might provide a valid measure.

The down side of doing a proper test on whether coffee improves cognitive skills will be the necessity of reverting to the base line, that is, every morning getting up and trying to function without the first cup.

“A mathematician is a machine for turning coffee into theorems.”

– Alfréd Rényi

*Turn your volume down (to not hear the advert) and see this primer on sensory evaluation by S-Cool– a UK educational site for teenagers.

*Click to enlarge the plots so you can see the legend, etc.

1. Speaker set: A vs B
2. Speaker wire: Thin vs Thick.
3. Source: CD vs FM-Radio
4. Speaker: Left vs Right.

It’s very possible that an interaction of two or more factors may be causing the problem, so to cover all bases I need to do all 16 possible combinations (2^4).  But, aside from the work this involves for all the switching around of parts and settings, I am stymied by the failure being so sporadic.

Anyways, I feel better now having vented this to my blog while listening to some soothing Sunday choir music by the Dale Warland Singers on the local classical radio station.  I’m taking no chances: It’s playing on my backup Panasonic SA-EN25 bookshelf system.

*Vastly over-rated according to this report by the Skeptic’s Dictionary.

It’s always interesting to see how intensely these teams from industry compete to develop the ‘copter that flies longest while landing most accurately.  However, this year one group stood out as being less competitive than the others.  Therefore, I was very surprised that they handily won our final fly-off.  It turns out that one of their factors was dropping the helicopter either wings-up or wings-down – the latter configuration being completely non-intuitive.  It turns out that going upside down makes it easier to drop, the flight time suffers only slightly and the flight becomes far more accurate – a premium in my overall scoring.

“The chief enemy of creativity is ‘good’ sense.”
- Pablo Picasso

Ironically, another team who benefited from having an expert in aeronautical engineering and a very impressive work ethic all around – they did more runs by far than anyone else – never thought of flying the ‘copters upside down.  In fact, their team leader objected very vigorously that this orientation must not be allowed, it being clearly unfair.  Fortunately, other executives in this black-belt class hooted this down.

I thought this provided a good lesson for process and product improvement – never assume that something cannot work when it can be easily tested.  That’s the beauty of DOE – it enables one to screen unknown (and summarily dismissed) factors to uncover a vital few that often prove to be the key for beating the competition.

*I also do this experiment for a class on DOE that I teach every Spring at South Dakota School of Mines and Technology.  In fact, I am writing this blog from their campus in Rapid City where I’ll be teaching class tonight.  For details, pictures and results of prior experiments here and at OSU, see this 2004 Stat-Teaser article on “Playing with Paper Helicopters”.

With the aid of Design-Expert® software, I reproduced in color the contour plot in Figure 1.3 from the book on EVOP by Box and Draper (see figure at the right).  To illustrate the basic principle of evolution, Box and Draper supposed that a series of mutations induced variation in length of lobster claws as well as the pressure the creatures could apply.  The contours display the percentage of lobsters at any given combination of length and pressure who survive long enough to reproduce.  Naturally this species then evolves toward the optimum of these two attributes as I’ve shown in the middle graph (black and white contours with lobsters crawling all over them).

In this way, Box and Draper present the two key components of natural selection:

1. Variation
2. An environment that favors select variants.

The strategy of EVOP mimics this process for improvement, but in a controlled fashion.  As illustrated here in the left-most plot, a two-level factorial,** with ranges restricted so as not to upset manufacturing, is run repeatedly – often enough to detect a significant improvement.  In this case, three cycles suffices to power up the signal-to-noise ratio.  This case illustrates a big manufacturing-yield improvement over the course of an EVOP.  However, any number of system attributes can be accounted for via multiple-response optimization tools provided by Design-Expert or the like.  This ensures that an EVOP will produce more desirable operating conditions overall for process efficiency and product quality.

It pays to pay attention to nature!

*Box, G. E. P. and N. R. Draper, Evolutionary Operation, Wiley New York, 1969.  (Wiley Classics Library, paperback edition, 1998.)

**(We show designs with center points as a check for curvature.)

Justin Hunter is the founder of Hexawise, a SaaS tool that aids in setting up tests for software using statistical methods. This also happens to be the subject of his blog – no doubt influenced in part by his father, William Hunter, author of the classic text Statistics for Experimenters.  Justin started the blog mid-way through ’09, so the pickings are a little slim, but there is still plenty of good stuff.

Some highlights from 2009:

• 10/6 The Stackoverflow.com for Software Testers marks the release of a beta version of testing.stackexchange.com.  This is a community driven Q and A site that uses the same technology as Stack Overflow, a popular site for coders looking for help.  Hunter’s version is aimed at testers, and already has an impressive database of answers and discussion.
• 8/25 What Else Can Software Development and Testing Learn from Manufacturing? Don’t Forget Design of Experiments (DoE) links to a Tony Baer post comparing software development to the manufacturing industry. Hunter further focuses on the application of Design of Experiments, pointing out the extensive use of DoE in quality improvement initiatives in Toyota and Six Sigma.  These initiatives have yet to really penetrate the software development industry, despite some high profile successes (Google’s Website Optimizer and Youtube are mentioned).
• 12/9 Defect Seen >10 Million Times and Still not Corrected has some interesting trivia about the grammatical error in Lands’ End – something I hadn’t even noticed, and apparently the company hadn’t either until it was too late.  The real point of the post, however, is to point out another much more fixable grammatical error in Google’s Blogger software.  If there is only 1 comment on a post, it still says “1 comments”, instead of dropping the s.  A trivial defect, perhaps, but a very visible and easily fixed one.  It reminds me of something Mark always says about taking a break from work to sweep the dirt off the shop floor.  That is, you shouldn’t let the little inconsequential bugs pile up while you’re focused on the big ones.

On a lighter note, in Famous Quotes that Make Just as Much Sense When You Substitute PowerPoint for Power Justin linked to a post by Jerry Brito about substituting PowerPoint for Power in famous quotes, adding a few of his own.  I’d also like to add:

Kirk: “Spock, where the hell’s the PowerPoint you promised?”
Spock: “One damn minute, Admiral.” –Star Trek IV