Archive for category science
American Statistical Association (ASA) defends itself against P-shooters
With the fundamental statistic of P value coming under severe attack, e.g., it being banned in early 2015 by the Basic and Applied Social Psychology (BASP) journal, the ASA has been compelled to issue an unprecedented press release with guidelines for avoiding misuse of hypothesis testing by scientists claiming significant experimental results.* “The ASA statement is intended to steer research into a ‘post p<0.05 era,’” said Ron Wasserstein, the ASA’s executive director.
“To pounce on tiny P values and ignore the larger question is to fall prey to the ‘seductive certainty of significance.’”
– Geoff Cumming, emeritus psychologist, La Trobe University, Melbourne, Australia
The ASA statement on “Statistical Significance and P-Values” can be seen here. It includes 6 guidelines on proper use of this essential tool for assessing research data, beginning with the assertion that “P-values can indicate how incompatible the data are with a specified statistical model.”
*See, for example, this Nature article that claims P values, the ‘gold standard’ of statistical validity, are not as reliable as many scientists assume.
Technology advanced beyond any hope for healthy curiosity
I am watching the Syfy’s series “Childhood’s End” this week. It is based on a science fiction novel by British author Arthur C. Clarke, one of my favorites growing up. One of the main characters is a very bright boy who at the end of the premier episode last night becomes an astrophysicist, despite this scientific profession being made entirely superfluous by the advanced technology of the alien Overlords.
This morning Robert Scherrer, the chairman of the department of physics and astronomy at Vanderbilt University, lamented in an editorial* for Wall Street Journal that children no longer have any reason to be interested in science, mainly because most of our household gadgets fall into the category of magic—alluding to Clarke’s observation that “any sufficiently advanced technology is indistinguishable from magic.”
“The world’s now placid, featureless, and culturally dead: nothing really new has been created since the Overlords came. The reason’s obvious. There’s nothing left to struggle for, and there are too many distractions and entertainments.”
― Arthur C. Clarke, Childhood’s End
Crater Experiment Makes a Big Impact
Posted by mark in design of experiments, science on June 25, 2015
Craters are crazy and cool. One that is quite amazing was created by the Barringer Meteorite that crashed into Arizona about 50,000 years ago with an explosion equal to 2.5 megatons of TNT. Based on this detailing of what a 2 MT bomb would do I figure that Barringer would have completely wiped out my home town of Stillwater, Minnesota and its 20,000 or so residents, plus far more beyond us. The picture my son Hank took of the 1 mile wide 570 foot deep crater does not do justice to its scale. You really need to go see it for yourself
as the two of us did.
Because of my enthusiasm for craters, making these rates number on my list of fun science projects in DOE It Yourself. As noted there, members the Salt Lake Astronomical Society wanted to drop bowling balls from very high altitudes onto the salt flats of Utah, but workers in the target area from the U.S. Bureau of Land Management objected to the experiment.
Kudos to science educator Andrew Temme for leading students through a far more manageable experiment shown in this video. In reply to me asking for permission for providing a link to his fantastic impact movies Andrew gave me this heads-up. “I attended a NASA workshop to get certified to handle real moon rocks and meteorites at the NJ State Museum in Trenton. This lab in the educator guide suggested mixing up your own lunar powder and throwing objects to simulate impact craters. When I got home I ran the lab with a few of my classes and then made the video. I used a Sony handheld camera that had a slow motion setting (300 fps).” Awesome!
The other day I went up to the 9th floor of my condo building in Florida and tossed a football down on to the parking lot. I am warming up to heaving a 15 pound mushroom anchor over on the beach side from atop one of the far pricier high rises along the Gulf. However, I have to wait until the turtle nesting season is over.
Believe it or not–sweet statistics prove that you can lose weight by eating chocolate
A very happy lady munching on a huge candy bar caught my eye in The Times of India on Friday, May 25. Not the lady—the chocolate.
After tasting a variety of delectable darks from a chocolatier in Belgium many years ago, I became hooked. However, I never imagined this addiction would provide a side benefit of weight loss. It turns out that a clinical trial set up by journalist John Bohannon and two colleagues came up with this finding and showed it to be statistically significant. This made headlines worldwide.
Unfortunately, at least so far as I’m concerned, the whole study was a hoax based on deliberate application of junk science done to expose phony claims made by the diet industry.
It turns out to be very easy to generate false positive results that favor a dietary supplement. Simply measure a large number of things on a small group of people. Something surely will emerge that out of this context tests significantly significant. What this will be, whether a reduction in blood pressure, or loss in weight, etc., is completely random.
Read the whole amazing story here.
My thinking is while Bohannan’s study did not prove that eating chocolate leads to weight loss, the subjects did in fact shed pounds faster than the controls. That is good enough for me. Any other studies showing just the opposite results have become irrelevant now—I will pay no attention to them.
Now, having returned from my travel to India, I am going back to dip into my horde of dark chocolate.
Null hypothesis significance testing procedure (NHTSP) psyched out
My colleague Brooks Henderson alerted me to this new policy by the editors of the Basic and Applied Psychology (BASP) journal to ban the NHSTP. According to the editorial in their Feb 2015 issue, authors must remove all p-values and the like and not refer to “significant” differences. They also banned confidence intervals, which really makes this new policy onerous, in my opinion.
I do see the sense of focusing on effect sizes and allowing the readers, presumably subject matter experts, to judge their importance. However, although they do “encourage the the use of larger sample sizes”, it makes no sense, I feel, to disregard the impact of small studies on the uncertainty of the results.
Blaming the misuse of NHTSP and p-values in particular for bad science is like letting a bad guy go by saying the gun is at fault.
Picking on P in these times of measles
Posted by mark in science, Uncategorized on February 3, 2015
Randall Munroe takes a poke at over-valuers of p in this XKCD cartoon
Nature weighed in with their shots against scientists who misuse P values in this February 2014 article by statistics professor Regina Nuzzo. She bemoans the data dredgers who come up with attention-getting counterintuitive results using the widely-accepted 0.05 P filter on long-shot hypotheses. A prime example is the finding by three University of Virginia finding that moderates literally perceived the shades of gray more accurately than extremists on the left and right (P=0.01). As they admirably admitted in this follow up report on Restructuring Incentives and Practices to Promote Truth Over Publishability, this controversial effect evaporated upon replication. This chart on probable cause reveals that these significance chasers produce results with a false-positive rate of near 90%!
Nuzzo lays out a number of proposals to put a damper on overly-confident reports on purported scientific studies. I like the preregistered replication standard developed by Andrew Gelman of Columbia University, which he noted in this article on The Statistical Crisis in Science in the November-December issue of American Scientist. This leaves scientists free to pursue potential breakthroughs at early stages when data remain sketchy, while subjecting them to rigorous standards further on—prior to publication.
“The irony is that when UK statistician Ronald Fisher introduced the P value in the 1920s, he did not mean it to be a definitive test. He intended it simply as an informal way to judge whether evidence was significant in the old-fashioned sense: worthy of a second look.”
The best accidental inventions of all time
I learned from my latest issue of Chemical and Engineering News that Stanley Stookey of Corning Glass Works died last month at age 99. In 1952 he mistakenly heated an alumino-silicate glass to 900 degrees C meaning only to top out at 600. After much cursing, according to the CEN story, Stookey found that instead of the molten mess expected, the material crystallized into a new type of material called a glass ceramic that proved to be “harder than carbon steel yet lighter than aluminum—shatterproof.”
Being in the business of planned experimentation it always amazes me to come across stories like this of serendipitous science. Obviously chance favors the prepared mind because most of the momentous discoveries are made by world-class chemists such as Stookey and others of his kind in the fields of physics and so forth.
I am a huge fan of 3M Post-It® Notes, not only due to their incredible usefulness, but also because it delights me to think of my fellow Minnesotan Art Fry coming up these by accident. For a list including him and a dozen other experts in their field who made the most of mishaps into inventions see 13 Accidental Inventions That Changed The World by Drake Baer of Business Insider. The one I like best is George Crum (great surname for a chef!) who reacted to customer complaining about his French fries by slicing them into ridiculously thin and hard-backed pieces. Never mind that it probably was his sister Katie who made the accidental discovery according to this Snopes investigation. Either way this works out to be a delicious story.
My advice to our clients is to keep a close watch for any strange results that crop up as statistically deviant in the course of a designed experiment. They may turn out to be really Crummy!
The rarest of birds—a reproducible result from a scientific study
Posted by mark in science, Uncategorized on February 3, 2014
In The New York Times new column Raw Data, science writer George Johnson laments experimenters
“ways of unknowingly smuggling one’s expectations into the results, like a message coaxed from a Ouija board.”
– Science Times, 1/21/14
This, of course, leads to irreproducible findings.
As a case in point, only 6 of 53 landmark papers about cancer found support in follow up studies, even with the help of the original scientists working in their own labs, according to an article in the Challenges in Irreproducible Research archive of Nature cited by Johnson.
That is discouraging but I am not surprised. I feel fairly sure that the any assertions of import get filtered very rigorously until only ones that reproduce reliably make it through.
The trick is to remain extremely skeptical of initial reports, especially those that get trumpeted and reverberate around the popular press and the internet. Evidently it is human nature to then presume that when an assertion is repeated often enough then it must be true, even though it has not yet been reproduced. Saying it’s so does not make it so.
Kids & Science
Posted by mark in pop, science, Uncategorized on June 23, 2013
I am heartened to hear of great work being done by current and former colleagues to get K-12 kids involved in Science, Technology, Engineering, and Mathematics (STEM). For example, Columbia Academy, a middle school (grades 6-8) in Columbia Heights (just north of Minneapolis), held an Engineering and Science Fair last month where two of our consultants, Pat Whitcomb and Brooks Henderson, joined a score of other professional engineers who reviewed student projects. Winners will present their projects this summer at the University of Minnesota’s STEM Colloquium.
Also, I ran across a fellow I worked with at General Mills years ago who volunteers his time to teach middle-schoolers around the Twin Cities an appreciation for chemistry. He makes use of the American Chemical Society (ACS) “Kids & Chemistry” program, which offers complete instructions and worksheets for many great experiments at middle-school level. Follow this link to discover:
– Chemistry’s Rainbow: “Interpret color changes like a scientist as you create acid and base solutions, neutralize them, and observe a colorful chemical reaction.”
– Jiggle Gels: “Measure with purpose and cause exciting physical changes as you investigate the baby diaper polymer,* place a super-absorbing dinosaur toy in water, and make slime.”
– What’s New, CO2? “Combine chemicals and explore the invisible gas produced to discover how self-inflating balloons work.”
– Several other intriguing activities contributed by ACS members.
Kudos to all scientists, engineers, mathematician/statisticians who are engaging kids in STEM!
*(The super-slurpers invented by the diaper chemists really are quite amazing as I’ve learned from semi-quantitative measurements of weight before and after soakings by my grandson. Thank goodness! Check out this video by “Professor Bunsen”, which includes a trick to recover the liquids that I am not going to try.)
Brain scientists flunk statistical standards for power
Last week The Scientist reported that “Bad Stats Plague Neuroscience”. According to researchers who dissected 730 studies published in 2011, neuroscientists pressed ahead with findings on the basis of only 8 percent median statistical power. This falls woefully short of the 80 percent power that statistician advise for experimental work. It seems that the pressure to publish overwhelms the need to run enough tests for detecting important effects.
“In many cases, we’re more incentivized to be productive than to be right.”
– Marcus Munafo, University of Bristol, UK