Saturday, January 28, 2017

Hypothesis Testing Using (Non-) Overlapping Confidence Intervals

Here's something (else!) that annoys the heck out of me. I've seen it come up time and again in economics seminars over the years.

It usually goes something like this:

There are two estimates of some parameter, based on two different models.

Question from Audience: "I know that the two point estimates are numerically pretty similar, but is the difference statistically significant?"

Speaker's Response: "Well, if you look at the two standard errors and mentally compute separate 95% confidence intervals, these intervals overlap, so there's no significant difference, at least at the 5% level."

My Reaction: "What utter crap!  (Eye roll!)

So, what's going on here?