An Overly Confident (Future) Nobel Laureate

For some reason, students often have trouble interpreting confidence intervals correctly. Suppose they're presented with an OLS estimate of 1.1 for a regression coefficient, and an associated 95% confidence interval of [0.9,1.3]. Unfortunately, you sometimes see interpretations along the following lines: 

• There's a 95% probability that the true value of the regression coefficient lies in the interval [0.9,1.3].
• This interval includes the true value of the regression coefficient 95% of the time.

So, what's wrong with these statements?

Well, something pretty fundamental, actually, but if you keep the following three things straight in your mind, you'll be fine:

1. The distinction between a parameter and point estimate of that parameter.
2. The distinction between an estimator and an estimate.
3. What is meant by the sampling distribution of an estimator.

Let's get this over with, and then I'll tell you a rather good story. ☺

The true value of the coefficient in this regression model is a parameter. It's a constant, whose value we just don't happen to know. On the other hand, the (point) estimate of 1.1 is just one particular "realized" value of a random variable - generated using this one particular sample of data.

An estimator is a formula - like the OLS formula in our example. Except in rather silly cases, this formula involves using the (random) sample data. So, an estimator is a function of the sample data - in othere words, what we call a statistic. When we apply this formula using a particular sample of data, we generate a number - a point estimate.

Because an estimator is a function of the random data, it's random itself. Being a random variable, an estimator has a distribution function. We reserve a special name for distributions of statistics (including estimators and test statistics): we call them "sampling distributions".

In our OLS example, under standard assumptions (including normally distributed errors) the estimator of the coefficient has a sampling distribution which is Normal. Why is this? Well, linear functions of normal random variables are themselves normal. (This doesn't work for lots of other distributions, so be careful.) In the linear regression model, y = Xβ + ε, if the values of ε are normally distributed, so are the values of y. The OLS estimator, b = (X 'X)^-1X 'y, is again a linear function of the y values, so b is also (multivariate) normal in its distribution.

So, every time we use the estimator (apply the OLS formula) with a different sample of data, we're likely to get a different result (estimate). Just what result we get depends not only on the values of the input data (the X values and the random y values), but also on the fact that the estimates are being generated by a random process - a normal distribution in this case.

But exactly what is a "sampling distribution"? In essence, it's a hypothetical construct. We have to imagine the following experiment:

1. We fix the sample size, n.
2. We take a random sample of n values of y.
3. We estimate the regession model by OLS (in our example).
4. We keep the point estimate of the coefficient vector.
5. We repeat steps 2 to 4, again, and again.........an infinite number of times.
6. We look at all of the estimates that we have obtained. If we were to plot them as a histogram, the latter would represent the "sampling distribution" of the OLS estimator.

Of course, in practice, we can't undetake an infinity of repetitions of this experiment. But we can always approximate the end result by using a simple Monte Carlo exepriment with a large number of replications.

Using this idea, we can get the correct interpretation of a confidence interval. Modify the above experiment as follows:

1. We fix the sample size, n.
2. We take a random sample of n values of y.
3. We estimate the regession model by OLS (in our example).
4. We construct a (say) 95% confidence interval for the true coefficient of interest to us.
5. We repeat steps 2 to 4, again, and again.........an infinite number of times.
6. We look at all of the confidence intervals (interval estimates) that we have obtained. Out of all of these many confidence intervals, 95% of them will cover the true (unknown) value of the coefficient.

However, any single one of these intervals may, or may not, cover it. We'll never know!

So, the first interpretation I gave for the confidence interval in the opening paragraph above is clearly wrong. The correct probability there is not 95% - it's either zero or 100%! The second interpretation is also wrong. "This interval" doesn't include the true value 95% of the time. Instead, 95% of such intervals will cover the true value.

And notice that we really should talk about the (random) intervals "covering" the (fixed) value of the parameter. If, as some people do, we talk about the parameter "falling in the interval", it sounds as if it's the parameter that's random and the interval that's fixed. Not so!

In a recent post (here) I talked about Wolfram's new cdf file format. They have a nice app. to illustrate confidence intervals here.

Incidentally, anyone who has served in court as an expert witness in statistics will tell you that judges have a lot of trouble understanding the concept of a confidence interval. Students may find some comfort in this!

So, with all of this under our belts, it's time for that story I promised you.

My colleague, Malcolm Rutherford, who specializes in HET, has a forthcoming paper about the history of the U.S. Department of Agriculture Graduate School. A year or two ago, he kindly lent me some archival material he had acquired. This related to the eminent statistician, Jerzy Neyman's, visiting lectures and "conferences" (we'd now call them "seminars") at the Graduate School in April 1937.

This material makes fascinating reading. Pertinent to this post is the transcript of one of Neyman's conferences, titled "An Outline of the Theory of Confidence Intervals". It's a gem! As with the other conferences, not only is Neyman's presentation itself recorded in full detail, but so are the questions from the attendees, and Neyman's responses.

Keep the date in mind, as well as the fact that Neyman was Polish, and did not move to Berkeley until 1938. Confidence intervals were at the cutting edge at this time, leading to the following interchange (p.157):

Mrs. Kantor:

"Is there anything in English literature on confidence intervals?"

Dr. Neyman:

"It is a relatively new subject. Apart from my paper in the Phil Trans. already cited on page 28, where I give the theory as discussed here, the references are as follows......."

Neyman then proceeded to reference 5 papers published in 1934 and 1935, and one to be published in 1937.

For me, the most interesting interchange is following one (pp.144-145), where Neyman summarizes the steps associated with constructing a confidence interval:

Dr. Neyman:

"The practical statistician is able to observe the x_i, and he wishes to know how these should be used for making some statement concerning the value of θ₁. We may advise him to perform the following three steps which, together, are equivalent to a single random experiment:

(i) to observe the values of the x_i, called E.

          (ii) to calculate the corresponding values of the functions θ_L (E) and θ_U (E).

(iii) to state that θ_L (E) ≤ θ₁ ≤ θ_U (E).

You will notice that in this statement he may be correct or he may be wrong. But, owing to the properties of the functions θ_L and θ_U as expressed by Eq.(1) [not shown here; DG], the probability of his being correct will be equal to α (e.g. 0.99). It follows that, if the experiment is so arranged that the x_i do follow the elementary law that served for constructing the functions θ_L and θ_U, then the empirical law of big numbers will guarantee that the practical statistician following the above advice will be correct in his statements concerning the value of θ₁ in 99 percent of all cases."

(immediately followed by)

Mr. F:

"Your statement of probability that he will be correct in 99 per cent of the cases is also equivalent to the statement, is it not, that the probability is 99 out of 100 that θ₁⁰ [the true value of the parameter; DG] lies between the limits θ_L and θ_U?"

Dr. Neyman:

"No. This is the point I tried to emphasize in my first two lectures both in theoretical discussions and in examples. [OUCH!@#!; DG] θ₁⁰ is not a random variable. It is an unknown constant. In consequence, if you consider the probability of θ₁⁰ falling within any limits, this may be either zero or unity, according to whether the actual value of θ₁ happens to outside of those limits or within."

Ex post, W. Edwards Deming (who assisted Neyman in the editing of the transcript) added a footnote (p.145):

"...The point is that we must not speak of the probability of θ₁ lying within fixed limits, nor limits that are not random variables. Ed."

Succinctly put, Dr. Deming! But who was the mysterious "Mr. F." who just couldn't get it straight, and got his knuckles rapped by Neyman?

Actually, it was no other than Milton Friedman!

Students - take note. And take heart!

References

Neyman, J. (1937). An outline of the theory of confidence intervals. (A conference with Dr. Neyman in the auditorium of the Department of Agriculture, 9th April 1937, 10 a.m., Dr. Frederick V. Waugh presiding). In Lectures and Conferences in Mathematical Statistics, Delivered by J. Neyman at the United States Department of Agriculture in April 1937, Graduate School of the United States Department of Agriculture, Washington D.C., pp. 143-160.

Rutherford, M. A. (2011). "The USDA Graduate School: Government training in statistics and economics, 1921-1945."   Journal of the History of Economic Thought, forthcoming.

© 2011, David E. Giles