tag:blogger.com,1999:blog-2198942534740642384.post1020023711294809855..comments2017-12-11T22:09:03.980-08:00Comments on Econometrics Beat: Dave Giles' Blog: An Overly Confident (Future) Nobel LaureateDave Gileshttp://www.blogger.com/profile/05389606956062019445noreply@blogger.comBlogger27125tag:blogger.com,1999:blog-2198942534740642384.post-41743060664519727512017-08-04T04:51:59.500-07:002017-08-04T04:51:59.500-07:00Thanks Mark!Thanks Mark!Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-51001525291139300232017-08-03T20:50:57.450-07:002017-08-03T20:50:57.450-07:00The best discussion on Andrew Gelman's blog is...The best discussion on Andrew Gelman's blog is in connection with this entry:<br /><br />http://andrewgelman.com/2017/03/04/interpret-confidence-intervals/<br /><br />Some good contributions there, esp. by Carlos Ungil and Daniel Lakeland.Mark Schafferhttp://ideas.repec.org/e/psc51.htmnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-15394184929803638722017-08-03T08:04:00.171-07:002017-08-03T08:04:00.171-07:00«if you were a Bayesian, then the whole idea of a ...«if you were a Bayesian, then the whole idea of a confidence interval will be meanngless, regardless of the sample size - because you'd have no interest in "repeated sampling", or the associated idea of the "sampling distribution".»<br /><br />I think that is an bizarre misdescription of bayesian approaches, as if they were "well one sample is plenty and then we bet the farm, because priors!".Blissexnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-90397949726258628452017-08-03T07:58:55.564-07:002017-08-03T07:58:55.564-07:00«I like to get students doing MC simulations nice ...«I like to get students doing MC simulations nice and early.»<br /><br />That is a really really good point. For example I found that myself and others only understand ("somewhat") the dreaded p-value if it is computed from a MC simulation, because the classic definition is in effect a double negative, not a constructive one.Blissexnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-23932731852124072422017-08-03T07:42:53.771-07:002017-08-03T07:42:53.771-07:00«The true value of the coefficient in this regress...«The true value of the coefficient in this regression model is a parameter. It's a constant, whose value we just don't happen to know. On the other hand, the (point) estimate of 1.1 is just one particular "realized" value of a random variable - generated using this one particular sample of data. An estimator is a formula - like the OLS formula in our example. Except in rather silly cases, this formula involves using the (random) sample data. So, an estimator is a function of the sample data - in othere words, what we call a statistic. When we apply this formula using a particular sample of data, we generate a number - a point estimate. Because an estimator is a function of the random data, it's random itself. Being a random variable, an estimator has a distribution function.»<br /><br />To me this looks like extremely loose and obfuscating terminology that gets so many people in trouble, for example can a "formula" be a "random variable" and have a "distribution function"? That's simply ridiculous. The way I learned it from some very clear thinking definettian subjectivists (but it is not a subjectivist point of view) is:<br /><br />* There is an algebra of arithmetic number and an algebra of stochastic numbers, and they are fundamentally different.<br /><br />* A "statistic" is a measure over a set of numbers, whether they be arithmetic or stochastic. The same formula for a measure can portend two different functions, one over arithmetic numbers, one over stochastic numbers.<br /><br />* Arithmetic numbers arise from populations, stochastic numbers from samples (under the hypothesis that the sampling process is ergodic, but I am not sure that is what a definettian subjectivist would say).<br /><br />* A measure on a sample is at the same time an arithmetic number with respect to the sample, and a stochastic number if *interpreted* as an estimate of the same measure on the population, while a measure on a population is always and only an arithmetic number.<br /><br />* Bonus point: it fantastically important (especially in studies of the political economy) to always ask what is the population from which a sample has been drawn, and whether the sampling process was indeed ergodic. And if you consider those two questions deeply enough, you end up a definettian subjectivist I guess :-).<br /><br />I do hope that I was not that loose conceptually or in terminology in the above, and that it reflects the insights I got from those clear thinking people.Blissexnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-57622927931892628002017-08-03T06:47:27.450-07:002017-08-03T06:47:27.450-07:00I will definitely be looking into this - thanks ag...I will definitely be looking into this - thanks again for alerting me (and other readers).Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-84356096868555581222017-08-03T06:45:02.702-07:002017-08-03T06:45:02.702-07:00Ah... hadn't noticed that! In 2011 I wasn'...Ah... hadn't noticed that! In 2011 I wasn't aware of "bet-proofness" either - I only learned about it from the M-N 2016 paper. But the concept has been around for decades, apparently. It's curious that it isn't more widely known.Mark Schafferhttp://ideas.repec.org/e/psc51.htmnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-74668767068748671652017-08-03T06:21:33.091-07:002017-08-03T06:21:33.091-07:00Mark - thanks for pointing this out! I'll chec...Mark - thanks for pointing this out! I'll check it out.(Note that my blog post was from 2011 - I promoted it recently because it was the 100'th anniversary of Friedman's birth.)Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-77331569994474404032017-08-03T06:13:23.347-07:002017-08-03T06:13:23.347-07:00Actually, there IS a way to interpret realized CIs...Actually, there IS a way to interpret realized CIs. The concept is “bet-proofness”. We had quite a good discussion about it over at Andrew Gelman's blog several months ago. I learned about the concept from a recent paper by Mueller-Norets (Econometrica 2016).<br /><br />Mueller-Norets (2016, published version, p. 2185):<br /><br />“Following Buehler (1959) and Robinson (1977), we consider a formalization of “reasonableness” of a confidence set by a betting scheme: Suppose an inspector does not know the true value of θ either, but sees the data and the confidence set of level 1−α. For any realization, the inspector can choose to object to the confidence set by claiming that she does not believe that the true value of θ is contained in the set. Suppose a correct objection yields her a payoff of unity, while she loses α/(1−α) for a mistaken objection, so that the odds correspond to the level of the confidence interval. Is it possible for the inspector to be right on average with her objections no matter what the true parameter is, that is, can she generate positive expected payoffs uniformly over the parameter space? … The possibility of uniformly positive expected winnings may thus usefully serve as a formal indicator for the “reasonableness” of confidence sets.”<br /><br />“The analysis of set estimators via betting schemes, and the closely related notion of a relevant or recognizable subset, goes back to Fisher (1956), Buehler (1959), Wallace (1959), Cornfield (1969), Pierce (1973), and Robinson (1977). The main result of this literature is that a set is “reasonable” or bet-proof (uniformly positive expected winnings are impossible) if and only if it is a superset of a Bayesian credible set with respect to some prior. In the standard problem of inference about an unrestricted mean of a normal variate with known variance, which arises as the limiting problem in well behaved parametric models, the usual [realized confidence] interval can hence be shown to be bet-proof."<br /><br />Full reference:<br /><br />Credibility of Confidence Sets in Nonstandard Econometric Problems<br />Ulrich K. Mueller and Andriy Norets (2016)<br />https://www.princeton.edu/~umueller/cred.pdf<br />http://onlinelibrary.wiley.com/doi/10.3982/ECTA14023/abstract<br /><br />Interesting stuff!<br /><br />--Mark<br />Mark Schafferhttp://ideas.repec.org/e/psc51.htmnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-21813538768676256242017-08-02T08:59:57.770-07:002017-08-02T08:59:57.770-07:00Excellent explanation... but sorry, but you're...Excellent explanation... but sorry, but you're really just parsing words here. If 95% of the intervals would cover the true value, then IMO it's not illogical at all to say that there's a 95% chance that any particular interval selected contains the true value. Yes, I get that the specific one we estimated either does or does not, but on average 95% is the best estimate we have of whether it does or does not.Billhttps://www.blogger.com/profile/15982661420006351208noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-24521652093782250742017-08-01T21:45:00.668-07:002017-08-01T21:45:00.668-07:00I am grateful to all of you (Dave Giles and Richar...I am grateful to all of you (Dave Giles and Richard Morey et al) for explaining this so clearly. Thank you!Alan T.noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-15839597439871534102015-11-22T21:30:20.985-08:002015-11-22T21:30:20.985-08:00Anonymous: What you can say about the single inter...Anonymous: What you can say about the single interval is, barring other available information that would make such a statement absurd, that you believe the true value is in it. People run into the most trouble trying to wrap such statements in probabilities not understanding that after the interval is calculated there aren't any known ones for the interval (without additional work). But consider the confidence in the procedure. If you perform a procedure that is correct 95% of the time it's perfectly rational to then just act as if the procedure gave you the correct answer even if you have no idea the exact probability that you're correct this particular time. I always find it fascinating that people have no problem acting as if their decision following a typical test is correct even though they may be wrong at a much higher rate than for a CI but can't do the same with a CI. The difference is that you're not stating sig./non-sig. but instead saying that mu is here.Unknownhttps://www.blogger.com/profile/00227235335343168838noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-62389218372739658862015-08-08T01:24:45.612-07:002015-08-08T01:24:45.612-07:00Anonymous: You can't say anything about a sing...Anonymous: You can't say anything about a single interval. As Neyman (1952) said, "[all the CI] does assert is that the probability of success in estimation using [any] formula[] is equal to [95\%]." You can read our paper on this topic in our upcoming paper, "The Fallacy of Placing Confidence in Confidence Intervals" (<a href="http://learnbayes.org/papers/confidenceIntervalsFallacy/index.html" rel="nofollow">http://learnbayes.org/papers/confidenceIntervalsFallacy/index.html</a>.Richard Moreyhttps://www.blogger.com/profile/11319149283079163004noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-22868198226871415732014-03-09T16:08:50.235-07:002014-03-09T16:08:50.235-07:00Thanks professor. So what can we say about a singl...Thanks professor. So what can we say about a single interval? In your example, how would you interpret the confidence interval of [0.9,1.3]?. Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-63313472280394601422013-08-14T10:42:22.452-07:002013-08-14T10:42:22.452-07:00Ok, I just wanted to convince myself this is the o...Ok, I just wanted to convince myself this is the only reason. <br />All doubts clarified. Thanks for that.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-74307717073302251292013-08-14T10:33:27.561-07:002013-08-14T10:33:27.561-07:00The interval is random; the parameter is constant....The interval is random; the parameter is constant.Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-84810647129937367292013-08-14T10:25:15.584-07:002013-08-14T10:25:15.584-07:00Brilliant! Now, I just can't link the two thin...Brilliant! Now, I just can't link the two things:<br />(1) the probability of a single interval covering the unknown parameter is 95%.<br />(2) the probability of the unknown parameter be within a single interval is either zero or 1.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-52545904219014631752013-08-14T10:15:18.139-07:002013-08-14T10:15:18.139-07:00Yes, but only in the sense that if we repeated the...Yes, but only in the sense that if we repeated the exercise again and again, with randomly drawn samples of the same size, then 95% of all of the intervals that we constructed would cover the parameter. In practice, we're not (usually) able to do this.Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-5007905201631970402013-08-14T10:11:03.312-07:002013-08-14T10:11:03.312-07:00Many thanks for your reply.
But in this case, the...Many thanks for your reply. <br />But in this case, the probability of the single interval covering the unknown parameter would be 95%, wouldn't it?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-5714942539336522042013-08-14T10:03:50.924-07:002013-08-14T10:03:50.924-07:00Yes we can. And of course given that the confidenc...Yes we can. And of course given that the confidence interval is constructed using the sampling distribution of the point estimator, the notion of "probability" in this context (whether we like it or not), is based on "repeated sampling". We'll never know if our single interval covers the unknown parameter or not.Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-8268890847564042312013-08-14T09:43:13.062-07:002013-08-14T09:43:13.062-07:00Very nice post.
About Mr. F's question: I am ...Very nice post. <br />About Mr. F's question: I am ok with the fact that θ10 is a constant, and as such the probabilities are zero or 1. But the intervals are random variables, and we can ask about the probability of one of those intervals covering θ10 or not. Can't we?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-19218451941623350972012-01-07T14:29:25.537-08:002012-01-07T14:29:25.537-08:00Rasmus: Thanks for the comment. No, the story (int...Rasmus: Thanks for the comment. No, the story (interpretation of the confidence interval) doesn't change in the asymptotic case.<br /><br />Of course, if you were a Bayesian, then the whole idea of a confidence interval will be meanngless, regardless of the sample size - because you'd have no interest in "repeated sampling", or the associated idea of the "sampling distribution".<br /><br />Glad you are enjoying the blog.Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-88419421344952799642012-01-07T13:26:25.916-08:002012-01-07T13:26:25.916-08:00Great post and a great blog in general.
Does the ...Great post and a great blog in general. <br />Does the story (i.e. the interpretation of the confidence intervals) change if we are considering an estimator where only the asymptotic distribution is known? E.g T^0.5 (b-beta) is asymptotically normal.Rasmus S. Pedersenhttps://www.blogger.com/profile/06250771677228438050noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-83673716605917755182011-08-31T10:00:59.062-07:002011-08-31T10:00:59.062-07:00Jeremy: Thanks. I agree about the medics. There ar...Jeremy: Thanks. I agree about the medics. There are some gems in the med. journals! You're also right about the key role the sampling distribution plays in understanding what follows. I like to get students doing MC simulations nice and early.Dave Gileshttps://www.blogger.com/profile/05389606956062019445noreply@blogger.comtag:blogger.com,1999:blog-2198942534740642384.post-66781639875062832582011-08-30T19:03:24.628-07:002011-08-30T19:03:24.628-07:00Great post and great story! In addition to judges,...Great post and great story! In addition to judges, doctors also have a great deal of trouble properly interpreting the confidence intervals presented in medical literature.<br /><br />I think students' difficulties (my own, anyway) stem from an inadequate understanding of the sampling distribution before diving into moments and OLS regression. Peter Kennedy's Guide to Econometrics opens with an excellent, intuitive treatment of sampling distributions that helped me immensely.Jeremy Austinhttps://www.blogger.com/profile/02090728665357884319noreply@blogger.com