Does good experimental design require changing only one factor at a time (OFAT)?
Posted by mark in design of experiments, science on June 23, 2009
“Good experimental design usually requires that we change only one factor at a time” according to an article I read recently in The Scientist magazine (“Why Don’t We Share Data,” page 33, Issue 4, Volume 23). This guide for science fairs tells students that “you conduct a fair test by making sure that you change only one factor at a time while keeping all other conditions the same.”
Obviously changing two variables together makes no sense, such as the time that as science project one of my kids asked me to do a blind taste test on Coke versus Pepsi, but to keep them straight in their mind, she poured one cola in blue plastic cup and the other in white Styrofoam! Needless to say I was completely confounded.
The OFAT method is so engrained that it’s literally become the law according to scientist who told me that, when as an expert witness he presented statistically significant evidence, it was thrown out of court due to the experiment design having changed multiple factors simultaneously. What a crime!
Multifactor testing is far more effective for statistical power, screening efficiency and detection of interactions. Industrial experimenters are well-advised to forget their indoctrination in OFAT and make use of multifactorial designs. For reasons why, see my two-part series on Trimming the FAT out of Experimental Methods and No-FAT Multifactor Design of Experiments.
Good experimental design does NOT require changing only one factor at a time!
“Decisions taken by statistical professionals are final”
I’m just catching up on the Wall Street Journal issues that accumulated while I attended a statistical conference and then co-taught a workshop on Designed Experiments for Life Sciences. A June 3rd article by WSJs “Numbers Guy” Carl Bialik caught my eye with a graphic showing that most UK citizens distrust official statistics. This caused their government to create a Statistics Authority that will police other agencies on the numbers they release to the public. Here some key points as reported at this UK government web site:
So it seems that the number nerds will rule after all — just like they always dreamed when being belittled by the bullies who thought math and stats were simply a waste of time. Statisticians rule!
Rabid for numbered bones
I am absorbing a great deal of information from the 2009 American Statistical Association’s Quality & Productivity Research Conference at IBM’s Watson Research Center in Yorktown Heights, New York. However, since IBM sold off their PC business to the Chinese manufacturer Lenovo, I am not quite sure what’s being researched at this facility. The official word is that IBM now provides “solutions.” See if you can puzzle things out from this 2009 newsletter . But, for those who are hard-core ‘techies’, check out this impressive list of IBM R&D projects, which include such things as quantum mirages and blue genes.
IBM presents an impressive collection of calculating devices in the lobby of this R&D center. For example, see pictured an actual 1617 set of Napier’s bones made by the Scottish inventor of logarithms. Via a process called rabdology (from the Greek “rabid” for rod, and “logos” for calculating), these numbered rods of skeletal origin facilitated multiplication and the computation of square and cube roots.
PS. Coincidentally, I just saw the latest Star Trek movie. Being an engineer by profession, I am naturally drawn to the character Scotty. Having seen what his ancestor Napier did with bones (not to be confused with the Star Trek doctor “Bones”), I now understand why the Enterprise engineer is such a wizard. Given enough time, these Scots will solve any technical problem. “I canna change the laws of physics! I’ve got to have thirty minutes.“
Political science (?) based on happenstance regression
My daughter Carrie, a junior at University of Minnesota — majoring in political science, asked me to look over a paper she wrote last week for her quantitative-analysis class. Her assignment was to test “the theory that Christian religiosity, measured through church attendance, affected the outcome of the 2004 presidential election” (Bush over Kerry). Carrie considered many other variables that could logically have influenced voting decisions before settling on two alternative factors – per-capita income, and level of education.
As I’d expected, her regression analysis (using the SPSS software) showed a positive correlation of “frequent church goers” voting for Bush (0.166 R2) and negative for “population with college degree or higher” (0.293). However, the highest correlation was seen with per-capita income, which surprised me by being negative – the more the voter earned, the more likely they were to NOT vote for Bush. I always thought that the Republicans were the party of the rich. But from this data one must conclude that they mainly appeal to poor, less-educated church-goers! (Please do not take the previous two ‘tongue-in-cheek’ statements seriously, I am only making a humorous point about how misleading statistics can be!)
I don’t give too much credence to any of this – mainly due to my great skepticism of using statistics to dissect historical data and generate inferences on cause and effect relationships. However, it makes me curious as to the driving forces of today’s party politics in the USA. That’s about all I figure that regression of happenstance data really offers – some food for thought that may lead to more rigorous investigation.
Awesome demonstration of design of experiments
Posted by mark in design of experiments on April 27, 2009
Team Awesome
The engineering students at South Dakota School of Mines and Technology really do rock. Where else could one present a class on statistics until 8:30 pm on a Friday night and continue it less than 12 hours later – early on a Saturday morning?
Our workshop on design of experiments (DOE) finished with a spirited competition of paper helicopters.* The winner was Team Awesome: Kayla Rithmiller, MacKenzie Trask and Samantha Johnson (pictured from left to right). They scored highest on the basis of flight time and accuracy. You can see their ‘copter spinning to another precise landing in their confirmation run.
Congratulations to Team Awesome and all the SDSM&T students who devoted their free time to learning DOE and demonstrating this newly-gained knowledge via well-planned experiments on the helicopter exercise. I predict that they all will go far!
*See details on this DOE exercise in the September 2004 Stat-Teaser article on Playing with Paper Helicopters.
Technology facilitates building a stronger database on blood pressure and other medical measurements
Posted by mark in Uncategorized on April 20, 2009
Some years ago my wife was diagnosed with high blood pressure (hypertension). This necessitated regular measurements with an instrument called a sphygmomanometer, which took me a long while to master for spelling and pronunciation. Being a chemical engineer helped – we used manometers to track barometric pressure. The hard part is the “sphygmo” – a Greek word meaning to throb or pulse. However, it works nicely for blood pressure!
Blood pressure measurements via the mercury gravity sphygmomanometer are still considered to be “gold standard.” Nevertheless, electronic devices are far easier to use and affordable for home use. To help my wife keep track of blood pressure, I bought one made by Panasonic. This came in handy when I developed heart problems of my own – chronicled in my article “How DOE Saved My Life and Made it Worth Living” in the June 2008, Stat-Teaser.
This week’s CRNtech brought news of a Digital Blood Pressure Check via an inexpensive (less than $100) device that connects via USB to a PC for capturing results. This data can then be uploaded to Microsoft’s HeathVault. From there you can enable care givers to watch for statistical trends.
My guess is that by repeated measurements over time, facilitated by this do-it-yourself system, medical professionals would get a far more precise assessment of hypertension. This may be the answer to Blood Pressure Variability: The Challenge of Variation – an issue recognized in this recent publication of the American Journal of Hypertension (2008, 21 3–4).
“It is therefore practically impossible for a clinician to know whether he is changing a drug or dose in response to chance variation in blood pressure or true changes in the underlying mean blood pressure.”
— Tom P Marshall, Department of Public Health and Epidemiology, University of Birmingham
TV detectives stumble over odds of matching birthdays
Posted by mark in Uncategorized on April 12, 2009
PS. The photo is one of my all-time favorites from the family album — it’s my son Hank, who helps me with this blog. The Anderson clan now is up to 9 counting those who’ve married in. So far none of us share a birthday.
The science of “guesstimation”
Posted by mark in Uncategorized on April 5, 2009
The latest National Geographic Science column on Mind Games shows a jar of jelly beans (presumably provided by the Easter bunny) and it offers a formula for estimating the number:
1. Count the jars radius (r) in beans. (This is hard to see due to the angle of the picture, but let’s say r equals 5.)
2. Estimate the height (h) in beans. (I can count this fairly easily from the photo – h equals 35.)
3. The volume (V) in beans is: V = 3 h r^2, where the constant 3 is a round-off on the circular constant pi. (So I estimate the beans in the National Geographic jar number 3x35x5^2, or 3x35x25 – the product of which is 2,625.)
The scientific, calculated estimate I made (2,625) for the count of jelly beans came a lot closer than my initial guess of ten thousand: The answer is 4,466. Going to all this effort might be worth it if you come across a bean-counting contest with a prize worth taxing your math skills.
Meanwhile, two professors at Old Dominion University in Virginia, one a mathematician (John Adam) and the other a physicist (Lawrence Weinstein), have teamed up to provide a primer on Guesstimation: Solving the World’s Problems on the Back of a Cocktail Napkin. As the publisher Princeton University Press says: “The ability to estimate is an important skill in daily life.”
As the father of five, I frequently was asked to help with math problems. First I’d ask that the student (my kid) work out a bottom-line number. Then I’d suggest they do a “reality check” by estimating the answer to at least the order of magnitude. That often sent them back to the beginning of the problem due to their first answer being so obviously wrong. The way facts and figures get thrown around the airwaves and internet nowadays it’s more important than ever to do reality checks.
I’ll bet this new book will be very helpful to equip reality checkers with the tools they need to achieve more accuracy. I learned about Guesstimation from its review in the March 31st New York Times. The Times article provides an interesting test of estimating ability: How many times does the American teenager say “like”? I heard this much more from my three daughters than my two sons, thus I hypothesize that there’s a gender bias. I’d hear this so word so over-used –- at least, like, once per sentence –- that I’d start counting them aloud, thus creating a great deal of aggravation for my teenager. I suppose the work “like” might come out ten times a minute and one hundred times per conversation. So I’m going to say a thousand “likes” per day could be in the realm of possibility. However, some teenagers are not afflicted by this word termite. My guess is ten thousand “likes” per year per teenager. To learn the answer, take this eight-question test of your estimation abilities.
Phenology — the study of the timing of natural events
Posted by mark in Uncategorized on March 27, 2009
Anyways, all this is an excuse for me to upload a photo I took last week along the Natchez Trace in Mississippi while on spring break last week. I do not know the identity of the plant in the foreground, but it caught my attention — especially with the wonderful profusion of blooming azealas as a backdrop.
Nearly 90% of cardiologist-approved heart therapies are not supported by high-quality scientific testing
Posted by mark in Uncategorized on March 21, 2009
Recently the Wall Street Journal reported that a Study Questions Evidence Behind Heart Therapies — specifically by this alarming statistic: “Just 11% of more than 2,700 recommendations approved by cardiologists for treating heart patients are supported by high-quality scientific testing.” It seems that the vast majority of prescribed treatments remain unratified by multiple randomized clinical trials – the highest level of evidence according to guidelines issued jointly by the American College of Cardiology and the American Heart Association.
I am caught up in this personally due to having had one heart attack some years ago. Ever since then I’ve been working hard to avoid a second one. My daily aspirin is strongly supported by scientific study, but it’s not very sure that I should be keeping on with the platelet inhibitor Clodiprogel (Plavix™, Bristol-Meyers Squibb/Sanofi Pharmaceuticals) prescribed after getting my clogged artery stented. I have to credit my cardiologist though – he is utterly impartial on the Clodiprogel – I cannot get any signal – pro or con. What can he say? As pointed out in this related article by US News & World Report no clinical trials exist beyond about one year (even that time is a bit vague!) of the heart surgery.
So as not to let all this cause me too much stress (possibly bad for the heart, but not strongly supported by solid scientific study) I picked up on this promising therapy – waltzing as a form of cardio-exercise. Evidently this works as well as trudging the treadmill and the dancing leads to a better quality of life as measured by the Minnesota Living With Heart Failure Questionnaire.* I note that the subjects were selected at random – that’s good, but “the study was not blinded, neither to the investigators nor to the patients.” Obviously it would not do to waltz blindly along, no matter how blissful that might be – until one hits the wall!
*I’d be skeptical if this weren’t based on Minnesota standards, that is, bitter cold, biting insects and so forth. 😉