A few years ago - twelve, to be specific - an interesting paper appeared in the Journal of the Royal Statistical Society. That paper, "Bayesian measures of model complexity and fit", by Spiegelhalter et al., stirred up a good deal of controversy within the statistical community. That much is apparent even from the "discussion" that accompanied its publication. More than 4,600 Google Scholar citations later, it continues to attract widespread attention - though not that much among econometricians, as far as I can tell. An exception is the paper by Berg et al. (2004).
In their 2002 paper, Spiegelhalter et al. introduced a new measure of model fit. They termed it the "Deviance Information Criterion" (DIC). Briefly, here's how it's defined:
First, we need to define"deviance" itself. This is a concept that arises in both "frequentist" statistics and Bayesian statistics. It's usually defined as
D(θ) = -2loge[p(y | θ)] ,
where p(y | θ) is the joint data density (which equals the likelihood function if we view it as a function of the parameter vector, θ, rather than as a function of the data vector, y. This quantity is used, of course in the construction of a Likelihood Ratio Test statistic in the case of competing models or hypotheses that are "nested".
The deviance information criterion is a Bayesian measure that takes account of both the "goodness of fit" and the "complexity" of a fitted model. It can be used for comparing and ranking competing models. "Complexity" is measured as the "effective number of parameters" defined as
pD = Eθ|y[D] - D[Eθ|y(θ)] .
That is, the effective number of parameters is the posterior mean deviance minus the deviance measured at the posterior mean of the parameters.
Then the deviance information criterion itself is defined as
DIC = D[Eθ|y(θ)] + 2pD = Eθ|y[D] + pD .
The smaller the value of DIC, the better the model.
You'll find a very concise, but readable account of the logic behind the DIC here; and these slides from David Spiegelhalter are excellent!
Obviously, the DIC is closely related to Akaike's Information Criterion (AIC) and Schwarz's Bayesian Information Criterion (SIC/BIC) that econometricians use a lot. I discussed these, and some other related model selection criteria, in an earlier post here. For instance,
AIC = D(θ*) + 2k,
and
SIC = D(θ*) + kloge(n) ,
where θ* is the MLE of θ; k is the number of parameters (i.e., the dimension of θ), and n is the sample size.
First, we need to define"deviance" itself. This is a concept that arises in both "frequentist" statistics and Bayesian statistics. It's usually defined as
D(θ) = -2loge[p(y | θ)] ,
where p(y | θ) is the joint data density (which equals the likelihood function if we view it as a function of the parameter vector, θ, rather than as a function of the data vector, y. This quantity is used, of course in the construction of a Likelihood Ratio Test statistic in the case of competing models or hypotheses that are "nested".
The deviance information criterion is a Bayesian measure that takes account of both the "goodness of fit" and the "complexity" of a fitted model. It can be used for comparing and ranking competing models. "Complexity" is measured as the "effective number of parameters" defined as
pD = Eθ|y[D] - D[Eθ|y(θ)] .
That is, the effective number of parameters is the posterior mean deviance minus the deviance measured at the posterior mean of the parameters.
Then the deviance information criterion itself is defined as
DIC = D[Eθ|y(θ)] + 2pD = Eθ|y[D] + pD .
The smaller the value of DIC, the better the model.
You'll find a very concise, but readable account of the logic behind the DIC here; and these slides from David Spiegelhalter are excellent!
Obviously, the DIC is closely related to Akaike's Information Criterion (AIC) and Schwarz's Bayesian Information Criterion (SIC/BIC) that econometricians use a lot. I discussed these, and some other related model selection criteria, in an earlier post here. For instance,
AIC = D(θ*) + 2k,
and
SIC = D(θ*) + kloge(n) ,
where θ* is the MLE of θ; k is the number of parameters (i.e., the dimension of θ), and n is the sample size.
In the latest issue of the JRSS (B), Spiegelhalter et al. revisit their 2002 paper. If you're going to be using DIC, you should read their discussion of some of its strengths and weaknesses,
Among other things, we also learn a little about some of the hurdles that they faced to get the 2002 paper published. Here's one passage that I just loved:
References
Among other things, we also learn a little about some of the hurdles that they faced to get the 2002 paper published. Here's one passage that I just loved:
"... the first draft of the paper on the deviance information criterion DIC [was] submitted to the Royal Statistical Society (authors Spiegelhalter, Best and Carlin). After rather negative referees' reports had been received, the first author want into a massive sulk for a year, but recruiting van der Linde into the team in 200 led to a new vitality and a new draft was submitted in 2001, and finally read to the Society in 2002." (p.486)I guess a lot of us can associate with the first author's reaction!
References
Berg, A. R. Meyer, and J. Yu, 2004. Deviance information criterion for comparing stochastic volatility models. Journal of Business and Economic Statistics, 22, 107-120.
Spiegelhalter, D. J., N. G. Best, B. P. Carlin, and A. van der Linde, 2002. Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society, B, 64, 583-639.
Spiegelhalter, D. J., N. G. Best, B. P. Carlin, and A. van der Linde, 2014. The deviance information criterion: 12 years on. Journal of the Royal Statistical Society, B, 76, 485-493.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.