Last week The Scientist reported that “Bad Stats Plague Neuroscience”. According to researchers who dissected 730 studies published in 2011, neuroscientists pressed ahead with findings on the basis of only 8 percent median statistical power. This falls woefully short of the 80 percent power that statistician advise for experimental work. It seems that the pressure to publish overwhelms the need to run enough tests for detecting important effects.
“In many cases, we’re more incentivized to be productive than to be right.”
– Marcus Munafo, University of Bristol, UK
#1 by jem777dy on May 17, 2013 - 9:40 am
I spent about 6 hours with one of my friends working on a thesis proposal. She needed to test the effect of 4 different factors on drug absorption to the brain. She had 100 rats to work with total. She used every trick she could think of to create t-tests that gave her the power she needed. I used my copy of DX to create an optimal FFD with 60 rats that had the power she wanted. We then discussed what to do with the 40 other rats she “had” to use.
In any research that involves human subjects, getting adequate power is a challenge. You might only have 60-70 people to work with, or less. If only more scientists knew about DOE, they would be able to get better power and better results. Then you would have to worry about “data cleansing”.
What many scientists I have dealt with, learned that data that does not conform to your expectations is wrong/bad data. Even if there is overwhelming evidence that you are wrong, blame the data first. Blame the technicians second. Blame the instruments. Then rerun everything you can (repeated measures). When the replicates show the same results as the first experiment, you might consider collecting more samples. If they still don’t conform to your expectations, then, you might be willing, at least a little bit, to admit, that you might not know everything there is to know, maybe.