Star Wars

Today, Ryan MacDonald, a UVic Economics grad. who works with Statistics Canada, sent me an interesting paper by Abel Brodeur et al.: "Star Wars: The Empirics Strike Back". Who can resist a title like that!

The "stars" that are being referred to in the title are those single, double (triple!) asterisks that authors just love to put against the parameter estimates in their tables of results, to signal statistical significance at the 10%, 5% (1%!) levels. A table without stars is like champagne without bubbles!

Basically, the paper is about selection bias in the refereeing and publishing of empirical papers in economics. It's a topic that has received attention previously, but in this paper come up with some compelling evidence.  In particular, they provide a new way of measuring "star inflation".

Here's the abstract of their paper:

"Journals favor rejection of the null hypothesis. This selection upon tests may distort the behavior of researchers. Using 50,000 tests published between 2005 and 2011 in the AER, JPE, and QJE, we identify a residual in the distribution of tests that cannot be explained by selection. The distribution of p-values exhibits a camel shape with abundant p-values above 0.25, a valley between 0.25 and 0.10 and a bump slightly below 0.05. The missing tests (with p-values between 0.25 and 0.10) can be retrieved just after the 0.05 threshold and represent 10% to 20% of marginally rejected tests. Our interpretation is that researchers might be tempted to inflate the value of those almost-rejected tests by choosing a “significant” specification. We propose a method to measure inflation and decompose it along articles’ and authors’ characteristics."

 I have to share a couple of quotes that the authors include in their paper:

If the stars were mine
I'd give them all to you
I'd pluck them down right from the sky
And leave it only blue. 

("If The Stars Were Mine" by Melody Gardot)

and,

 He who is fi xed to a star does not change his mind.

(Leonardo Da Vinci)

It's an interesting paper.

© 2013, David E. Giles