Different Views on Significance Testing

Econ Journal Watch is an online resource that provides "scholarly comments on academic economics". If you don't follow it, or at least browse it from time to time, then I urge you to do so. Yes, that includes econometricians!

 

In the September 2012 issue you'll find a piece by Thomas Mayer. It's titled, "Ziliak & McCloskey's Criticisms of Significance Tests: An Assessment", and you can download a pdf version of the full article for free.

 Here's the abstract of his article:

"Stephen Ziliak and D. N. McCloskey have sharply criticized the prevailing use of significance tests. Their work has, in turn, come under vigorous attack. The vehemence of the debate may induce readers to wrongly dismiss it as a “he said-she said” debate, or else to take sides in an unbending way that does not do justice to valid points raised by the other side. This paper aims at a more balanced reading. While Ziliak and McCloskey claim that a substantial majority of economists who use significance tests confuse statistical with substantive significance, or commit the logical error of the transposed conditional, I argue that such errors are much less frequent than they claim, though still much too pervasive. They also argue that since significance tests focus on the existence of an effect rather than on its size, the tests do not answer scientific questions. I respond with counter-examples. Ziliak and McCloskey also complain that significance tests ignore loss functions. I argue that loss functions should be introduced only at a later stage. Ziliak and McCloskey are correct, however, that confidence intervals deserve much more emphasis. The most valuable message of their work is that significance tests should be treated less mechanically."

Deidre McCloskey and Stephen Ziliak have responded. Here's the abstract of their rebuttal:

"Econometricians have been claiming proudly since World War II that significance testing is the empirical side of economics. In fact today most young economists think that the word “empirical” simply means “collect enough data to do a significance test”. Tjalling Koopmans’s influential book of 1957, Three Essays on the State of Economic Science, solidified the claim. A century of evidence after Student’s t-test points strongly to the opposite conclusion. Against conventional econometrics we argue that statistical significance is neither necessary nor sufficient for proving commercial, human, or scientific importance. A recent comment by Thomas Mayer, though in parts insightful, does nothing to alter conclusions about the logic and evidence which we and others have assembled against significance testing. Let’s bury it, and get on to empirical work that actually changes minds."

A presentation with the same (abbreviated) title was given by Ziliak and McCloskey at the 2009 Joint Statistical Meetings, and it will give you a quick overview of their position on this matter.

I'm going to go back and re-read these authors',  The Cult of Statistical Significance. I have a feeling it will be good for me!


Reference

Ziliak, S. T. and D. N. McCloskey, 2008. The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. University of Michigan Press, Ann Arbor, MI..

© 2012, David E. Giles