What do we mean by a "significant result" in econometrics?
The distinction between "statistical significance" and "economic significance" has received a good deal of attention in the literature. And rightly so.
Think about the estimated coefficients in a regression model, for example. Putting aside the important issue of the choice of a significance level when considering statistical significance, we all know that results that are significant in the latter sense may or may not be 'significant' when their economic impact is considered.
Marc Bellemare provided a great discussion of this in his blog a while back.
Here, I want to draw attention to a somewhat related issue - distinguishing between the statistical and economic overall goodness-of-fit of an economic model.
Almost 30 years ago, Hal Varian published a really nice paper on this topic. I'm surprised at how little attention it's received in recent times.
To set the scene, here's the abstract from Hal's paper (with my emphasis):
Back in 1999, Lindsay Tedds and I played around a bit with Hal's approach to economic goodness-of-fit, using demand systems based on the (constrained) maximization of various utility functions, and several data-sets. We never brought that work fully to fruition, but we came up with some interesting empirical results, and compared them with those for conventional (statistical) goodness-of-fit. Some slides for a talk that I gave on that work can be found here
The papers listed in the references below provide some additional insights into the debate over statistical vs. economic significance.
The distinction between "statistical significance" and "economic significance" has received a good deal of attention in the literature. And rightly so.
Think about the estimated coefficients in a regression model, for example. Putting aside the important issue of the choice of a significance level when considering statistical significance, we all know that results that are significant in the latter sense may or may not be 'significant' when their economic impact is considered.
Marc Bellemare provided a great discussion of this in his blog a while back.
Here, I want to draw attention to a somewhat related issue - distinguishing between the statistical and economic overall goodness-of-fit of an economic model.
Almost 30 years ago, Hal Varian published a really nice paper on this topic. I'm surprised at how little attention it's received in recent times.
To set the scene, here's the abstract from Hal's paper (with my emphasis):
"Conventional econometric tests of optimizing models typically involve embedding the optimizing model in a parametric specification and then examining the parametric restrictions imposed by the optimization hypothesis. The optimization hypothesis is rejected if the estimated parameters are significantly different, in the statistical sense, from the values implied by optimization. I argue that a more fruitful approach to testing optimizing behavior is to measure the departure from optimization using the estimated objective function, and see whether this departure is significant in an economic sense. I discuss procedures for doing this that can be used in several sorts of optimizing models, and give a detailed illustration in the case of aggregate demand estimation."Contrary to a lot of what I see in the literature these days, microeconometrics should be all about the empirical analysis of optimization models. This makes Hal's work absolutely relevant!
Back in 1999, Lindsay Tedds and I played around a bit with Hal's approach to economic goodness-of-fit, using demand systems based on the (constrained) maximization of various utility functions, and several data-sets. We never brought that work fully to fruition, but we came up with some interesting empirical results, and compared them with those for conventional (statistical) goodness-of-fit. Some slides for a talk that I gave on that work can be found here
The papers listed in the references below provide some additional insights into the debate over statistical vs. economic significance.
Akerlof, G. and J. Yellen, 1985. Can small deviations from rationality make significant differences to economic equilibria? American Economic Review, 75, 708-720.
Cochrane, J., 1989. The sensitivity of tests of intertemporal allocation of consumption to near-rational alternatives. American Economic Review, 79, 319-337.
Cowell, F. A., E. Flachaire, & S. Bandopadhyay, 2009. Goodness-of-fit: An economic approach. Discussion Paper No. 444, Department of Economics, University of Oxford.
McCloskey, D. N., 1985. The loss function has been mislaid: The rhetoric of significance tests. American Economic Review, 75, 201-205.
McCloskey, D. N. & S. T. Ziliak, 1996. The standard error of regressions. Journal of Economic Literature, 34, 97-114.
Cochrane, J., 1989. The sensitivity of tests of intertemporal allocation of consumption to near-rational alternatives. American Economic Review, 79, 319-337.
Cowell, F. A., E. Flachaire, & S. Bandopadhyay, 2009. Goodness-of-fit: An economic approach. Discussion Paper No. 444, Department of Economics, University of Oxford.
McCloskey, D. N., 1985. The loss function has been mislaid: The rhetoric of significance tests. American Economic Review, 75, 201-205.
McCloskey, D. N. & S. T. Ziliak, 1996. The standard error of regressions. Journal of Economic Literature, 34, 97-114.
Varian, H. R. , 1990. Goodness of fit in optimizing models. Journal of Econometrics, 46, 125-140.
Ziliak, S. T. & D. N. McCloskey, 2004. Size matters: The standard error of regressions in the American Economic Review. Journal of Socio-Economics, 33, 527-546
© 2018, David E. Giles
Hi Dave: This is extremely relevant in intraday modelling in finance because, in that case, one has tons ( multiple thousands ) of observations, so the chances of accepting any null hypothesis ( such as say beta = 1 in a simple regression ) is essentially zero. ( because the t-test wasn't designed for 1000's of observations ). But, there are bootstrapping approaches that one can use to check whether the estimated coefficient really does improve forecasts compared to coefficient under the null. Clark and McFadden ( Federal Reserve of St. Louis, IIRC ) have a ton of papers regarding the approach. Great post. Thanks.
ReplyDelete