tag:blogger.com,1999:blog-2198942534740642384.post7148559924231662710..comments2023-10-24T03:16:41.009-07:00Comments on Econometrics Beat: Dave Giles' Blog: Information Criteria UnveiledDave Gileshttp://www.blogger.com/profile/05389606956062019445noreply@blogger.comBlogger18125tag:blogger.com,1999:blog-2198942534740642384.post-49796968339897784062018-09-21T11:13:51.806-07:002018-09-21T11:13:51.806-07:00Very good points.Very good points.Daumantasnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-72860628654495732742016-02-08T07:16:27.915-08:002016-02-08T07:16:27.915-08:00Raluca - no, not at all. They can be positive or n...Raluca - no, not at all. They can be positive or negative, depending on the problem and the sample values.Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-47042624116414436592016-02-08T07:05:37.936-08:002016-02-08T07:05:37.936-08:00Hello Professor!
Is it something meaningful or wr...Hello Professor!<br /><br />Is it something meaningful or wrong if one gets negative values for the information criteria like AIC, SIC/BIC or HQ?<br /><br />Regards,<br /><br />RalucaAdminhttps://www.blogger.com/profile/09008629911393135175noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-48932314611991588742013-08-13T09:31:49.772-07:002013-08-13T09:31:49.772-07:00No - it is correct as stated. I Say clearly, &...No - it is correct as stated. I Say clearly, & correctly, in the post (3 paragraphs above the numbered table), that AICc is inconsistent.Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-52488605036815651772013-08-13T09:00:15.425-07:002013-08-13T09:00:15.425-07:00"The probability of over-fitting when using A..."The probability of over-fitting when using AIC decreases as n increases.<br />The probability of over-fitting when using AICc increases as n increases.<br />The probabilities of over-fitting when using SIC/BIC or HQ decreases as n increases, and decreases faster for SIC/BIC than for the HQ criterion."<br /><br />Is it not the other way round with the first two criteria? Isn't AICC consistent?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-7050598407131545292013-08-05T16:24:27.411-07:002013-08-05T16:24:27.411-07:00Nice post. For my part, I consider the fact that A...Nice post. For my part, I consider the fact that AIC is inconsistent a feature rather than a bug. What is and is not desirable in a criterion very much depends on how you plan to use it. But I would argue that in economics we're seldom interested in determining which of our candidate models is the truth. For one thing, as you point out above, it's pretty unlikely that any of our models is entirely correctly specified. But even more fundamentally, estimating the correct specification does not necessarily provide the best forecast or estimate of a parameter of interest: there's a bias-variance trade-off. If you're interested in estimators (or forecasts) with low risk, which I think is a much more typical situation in applied work, consistent model selection is a particularly bad idea. BIC, for example, has *unbounded* minimax risk. In contrast, efficient/conservative criteria such as AIC are much better-behaved in this regard. There's a good discussion of this in Chapter 4 of of ``Model Selection and Model Averaging'' by Claeskens and Hjort (2008, Cambridge University Press). Larry Wasserman also has some recent discussion on this point:<br /><br />https://normaldeviate.wordpress.com/2013/07/27/the-steep-price-of-sparsity/<br /><br />As you point out, it's important to remember that AIC (like all other non-trivial information criteria) has its own sampling distribution. Among other things, this means that estimators associated with the selected model will *not* share the asymptotic risk properties of the selection criterion itself. Indeed, the post-selection estimator is a *randomly weighted average* of all the candidate estimators. Model selection certainly is a challenging econometric problem!<br /><br /><br />Frank DiTragliahttp://www.ditraglia.comnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-32060645888843873412013-07-29T08:15:55.542-07:002013-07-29T08:15:55.542-07:00Ι juѕt like the helpful information yοu рroviԁe in...Ι juѕt like the helpful information yοu рroviԁe in your <br />artiсles. I will bοokmагk your blog and teѕt once more here <br />frequently. I аm mοderately sure I will be tοld lots <br />οf new stuff right here! Best of lucκ for the next!<br /><br /><br />My webpage :: <a href="http://Orgumodelleri.ws/ornekleri/battaniye/" rel="nofollow">çeşitli örgü battaniye</a>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-6248301709421820932013-07-28T10:08:12.046-07:002013-07-28T10:08:12.046-07:00Thank you, Professor. I was afraid my persistence ...Thank you, Professor. I was afraid my persistence would have become annoying to you... <br />This is one of those jargons that you don't quite get unless you see the formal definitions. Anyway, it appears that, perhaps just like estimation unbiasedness, this unbiasedness is a rather weak standard. I don't know if this is why I haven't seen this being studied in recent literature at all (or maybe I haven't looked hard enough).<br /><br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-19189320848410261942013-07-28T09:36:56.837-07:002013-07-28T09:36:56.837-07:00Fair enough! Here is what we usually mean. For sim...Fair enough! Here is what we usually mean. For simplicity, suppose we are using a criterion that involves calculating and ranking the values a of a statistic, S, for two versions of the model, using the same random sample. Call these statistics (random variables) S and S2. Suppose, further, that the model-selection criterion is to select Model 1 if S1 < S2. Otherwise we select Model 2. <br /><br />If E[S1] < E[S2], for all values of the parameters of the models, we say that the model selection criterion is "unbiased".<br /><br />Note that, as with the concept of "unbiasedness" in the context of estimation or hypothesis testing, the concept involves the POPULATION expectations of the statistics we're using.<br /><br />In the econometrics literature, I believe Theil was the first to use this notion in the following context. Note that if we select between two models by maximizing adjusted R-squared (with the same sample), then this is equivalent to choosing the model with the smaller estimator of the error variance, si^2=(e'e)/(n-ki); for i = 1, 2. It's easy to show that if Model 1 is the correct model, then E[s1^2]<E[s2^2]. So, using this criterion we select the correct model "on average", and we say that this model-selection criterion is "unbiased". <br /><br />Maybe I should do a really short blog post on this.<br /><br />Thanks for persisting!Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-60068458804305389392013-07-26T18:01:51.763-07:002013-07-26T18:01:51.763-07:00Thanks, Professor. But could you please give a mat...Thanks, Professor. But could you please give a mathematical definition of "selecting the correct model on average"? (For example, a criterion is (weakly) consistent if Pr(correct model is selected) converges to 1) If any of the reference has such a definition, could you please point me to it? I am genuinely not getting what it means and eager to learn its formal definition. Thank you again. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-1459609583859814722013-07-26T10:48:48.198-07:002013-07-26T10:48:48.198-07:00AISc uses a bias-corrected estimator of the KL div...AISc uses a bias-corrected estimator of the KL divergence, in the sense you say, but still not fully unbiased. YOu'll see that I say in the post that AICc reduces the tendency to over-fit the model (relative to AIC). It doesn't fully solve the problem.Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-24202370248557652402013-07-26T10:28:49.240-07:002013-07-26T10:28:49.240-07:00Thanks, Professor. The bias they (Hurvich and Tsai...Thanks, Professor. The bias they (Hurvich and Tsai's biometrika paper) talked about is the bias in AIC as an estimator of the KL information. Are you saying that since AICC is an unbiased (or bias-corrected) estimator of the KL information, it will "select the correct model on average"? Thanks again and please feel free to ignore my reply if this is too trivial. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-61373976083072431142013-07-26T08:47:26.488-07:002013-07-26T08:47:26.488-07:00I think you should take a look at the McQuarrie an...I think you should take a look at the McQuarrie and Tsai book, or the paper by Hurvich and Tsai.Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-55171342557500616132013-07-26T08:41:44.671-07:002013-07-26T08:41:44.671-07:00Thanks, Professor. But can you be more explicit ab...Thanks, Professor. But can you be more explicit about "selecting the correct model on average"? Suppose I do 1000 MC experiments, if it selects an underfitting model 50% of the time, and an overfitting model 50% of the time, does it mean the criterion is unbiased? Or if it selects the correct model 50% of the time, and an underfitting model and an overfitting model each 25% of the time, does it mean it is unbiased? Or if it selects the correct model 90% of the time, and an overfitting model 10% of the time, does it mean it is biased?<br />Thanks, Professor, for the clarification.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-34277975845230937992013-07-26T08:30:28.535-07:002013-07-26T08:30:28.535-07:00The term "biased" is used in various way...The term "biased" is used in various ways. Apart from it's usual meaning in estimation, we say a test is biased if, at any point in the parameter space its power falls below the assigned significance level.<br /><br />Here, a model-selection procedure is unbiased if, when used repeatedly, it selects the correct model on average. For example, using maximum adjusted R-squared as a model selection procedure is an unbiased selection process, in this sense. Of course, the correct model has to be among those being chosen between,Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-18747591560390621092013-07-26T07:24:22.668-07:002013-07-26T07:24:22.668-07:00Professor, can you explain what you mean when you ...Professor, can you explain what you mean when you say a model selection criterion is "biased"? What does it mean "it selects the correct model on average"? I cannot relate the concept of estimation bias to this. Any reference would be very helpful.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-42056419042976349712013-07-25T15:25:50.022-07:002013-07-25T15:25:50.022-07:00Dave,
you are getting lazy during your holidays.
...Dave,<br /><br />you are getting lazy during your holidays.<br /><br />please note Canada is no longer a province of the USA.<br /><br />Keep it upnottrampishttp://nottrampis.blogspot.com.aunoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-73117674120725203932013-07-25T01:56:40.260-07:002013-07-25T01:56:40.260-07:00This comment has been removed by a blog administrator.Anonymoushttps://www.blogger.com/profile/10388655445078044264noreply@blogger.com