Check out this blog by statistician William Briggs that gives the heads up on how Hurricane Predictors Admit They Can’t Predict Hurricanes. Years ago as a chemical engineer working on process development I would be encouraged by plant personnel to crank old data through a regression analysis to model the operation, thus avoiding any work on their part to run designed experiments. The joke was that we got very good at predicting what would happen last month.
In this case the issue is hurricanes. As Briggs explains, the top experts can fit past data very well (r = 0.79 for 50-year period the last half of 20th century). He refers to this as a ‘hindcast’. But, as the hurricane forecasters themselves admit, these models predict so poorly (r = 0.04) that you may as well just use an average — what I call the ‘mean’ model as a double meaning (ha ha) because it is so disappointing for the analyst.
What it boils down to is that any forecasts on hurricanes this early before the coming season will really just be a lot of hot air, despite impressive statistics from models fit to prior years. The same goes for long-term outlooks on other natural phenomena.
#1 by Eric Kvaalen on February 8, 2012 - 2:58 am
I am rather surprised by this result. It means either that the climate mechanisms have changed drastically in the last 20 years or that these top experts (Klotzbach and Gray) had developed a complicated model that was not supported by statistically significant effects! How else can one explain that the hindcasts are so good and the forecasts in the last couple decades so bad?
They should be able to make a new model which hindcasts equally well both the older data and the newer data, unless of course the climate mechanisms really have changed.
#2 by ronwroc on February 13, 2012 - 10:27 am
Seems to me that the hindsight prediction vs foresight prediction may be a case of initial state determing the behavior. The hindsight data may not contain any pertinent initial state data but could be very good with fitting and predicting the results.