What's the Variance of a Sample Variance?

This post is really pitched at students who are taking a course or two in introductory economic statistics. It relates to a couple of estimators of the variance of a population that we all meet in such courses - plus another one that you might not have met. In addition, I'll be emphasising the fact that some "standard" results depend crucially on certain assumptions. Not surprisingly - but  not always made clear by instructors and text books.

To begin with, let's consider a standard problem. We have a population that is Normal, with a mean of μ and a variance of σ². We take a sample of size n, using simple random sampling. Then we form the simple arithmetic mean of the sample values: x* = (1/n)Σx_i , where the range of summation (here and everywhere below) is from 1 to n.

Under my assumptions, we know that the sampling distribution of x* is N[μ , (σ² / n)]. The normality of the sampling distribution follows from the Normality of the population, and the fact that x* is a linear function of the data. The variance of the sampling distribution stated above is correct only because simple random sampling has been used.

Now, let's get to what I'm really interested in here - estimating σ². We all learn that the mean squared deviation of the sample, σ^*2 = (1 / n)Σ[(x_i - x*)²], is a (downward-) biased estimator of σ². If we allow for the fact that we've actually lost one degree of freedom by estimating μ using x*, then an unbiased estimator of σ² is s² = (1 / (n - 1))Σ[(x_i - x*)²].

O.K., now what does the sampling distribution of s² look like?

Well, under the assumptions I've made, including the Normality of the population, s² has a sampling distribution that is proportional to a Chi-square distribution. More specifically, the statistic, c = [(n - 1)s² / σ²] is Chi-square with (n - 1) degrees of freedom.

[As an aside, s² and x* are independently distributed if and only if the population is Normal. The "only if part" of the latter statement is due to the Irish statistician, Geary - see here.]

So, we now know something, indirectly, about the sampling distribution of s², and we know that E[s²] = σ². What is the variance of σ²?

Because we're assuming a Normal population, implying that the statistic I've called "c" follows a Chi-square distribution, we can use the result that the variance of a Chi-square random variable equals twice its degrees of freedom.

Re-arranging the formula for "c", we can write: s² = cσ² / (n - 1).

Then, Var.(s²) = {[σ² / (n - 1)]² Var.(c)} = {[σ⁴ / (n - 1)²]2(n - 1)} = 2σ⁴ / (n -1).

[As another aside, the mean of a Chi-square random variable equals its degrees of freedom, so applying this result to "c" and re-arranging, we immediately get the result that E[s²] = σ². However, we know this already, and this result holds even if the data are non-Normal.]

Now, this is as far as things usually go in an introductory economic statistics course. To sum up:

• E[s²] = σ²
• c = [(n - 1)s² / σ²] ~ χ²_{(n - 1)}
• Var.[s²] = 2σ⁴ / (n -1)

Notice that Var.(s²) vanishes when n grows very large. This, together with the first above, implies that s² is a (mean-square) consistent estimator of σ².

Unfortunately, students often don't realize that the second and third of these results rely on both simple random sampling and the Normality of the population.

A thoughtful student will notice that the first result holds even if the data are non-Normal, and will ask, "what's the variance of s² if the population isn't Normal?" That's a good question!

To answer it, let's introduce an important concept - the "moments" of a probability distribution. Let X be a random variable. Then E[X^k] is called the k^th "raw moment" (or, moment about zero) of the distribution of X.  (Here, "k" is a positive integer, but more generally we can allow k to be negative, or a fraction.) Let's denote the k^th such moment by μ'_k. So, the first raw moment is just the population mean. That is, μ'₁ = μ.

Then consider the quantities, μ_k = E[(X - μ)^k], for k = 1, 2, 3,.......... We call these the "centered moments" of the distribution of X. You'll notice that μ₂ is just the population variance. The third and fourth centered moments are used (together with μ₂) to construct measures of skewness and kurtosis, but that's another story. 

By the way, there's an important detail. The expectations involved in the construction of the moments require forming an integral. If that integral diverges, the corresponding moment isn't defined. or instance, the k^th moment for a Student's-t distribution with v degrees of freedom exists only if v > k. In the case of the Cauchy distribution (which is just a Student's-t distribution with v = 1), none of the moments exist!

Alright - back to the question in hand! What is the variance of s² if the population is non-Normal? The answer, in the case of simple random sampling, is:

           Var.(s²) = (1 / n)[μ₄ - μ₂²(n - 3) / (n -1)] .

If the population is Normal, then μ₄ = 3σ⁴, and μ₂² = σ⁴. So, we get  Var.(s²) = 2σ⁴ / (n - 1), in this case.

Notice that this more general expression for Var.(s²) also vanishes as n grows. So, a pair of sufficient conditions for the mean-square consistency of s² (as an estimator of σ²) is:

1. The data are obtained using simple random sampling; 
2. At least the first 4 moments of the population distribution exist.

We can easily work out the expressions for Var.(s²) in the case where the population follows some other distributions that you may have heard about. Here are just a few illustrative results:

Uniform, continuous on [a , b]

μ₂ = (b - a)² / 12  ;  μ₄ =  (b - a)⁴ / 80
Var.(s²) = (2n + 3)(b - a)⁴ / [380n(n - 1)]

Standard Student's-t, with v degrees of freedom

μ₂ =   v / (v - 2)  ;  μ₄ =   3v² / [(v - 2)(v - 4)]
Var.(s²) = [2v²(nv - 3  - n)] / [n(n - 1)(v - 2)²(v - 4)]                       ;   for v > 4

χ², with v degrees of freedom


μ₂ = 2v  ;  μ₄ = 12v(v + 4)
Var.(s²) = [8v(nv + 6n - 6))] / [n(n - 1)]

Exponential, with mean θ


μ₂ = θ²  ;  μ₄ = 9θ⁴  
Var.(s²) = [2(4n - 3)θ⁴] / [n(n - 1)]

Poisson, with parameter λ

μ₂ = λ  ;  μ₄ = λ(3λ + 1)
Var.(s²) = 2λ² / (n - 1) + (λ / n)

Keep in mind that in each of the cases, the sampling distribution of  c = [(n - 1)s² / σ²] will no longer be a χ² distribution! Given our assumption of simple random sampling, you should be able to convince yourself that the asymptotic sampling distribution of "c" will be Normal.

References

Cho, E. & M. J. Cho (2008). Variance of sample variance. Proceedings of the 2008 Joint Statistical Meetings, Section on Survey Research Methods, American Statistical Association, Washington DC,1291-1293.

Geary, R. C. (1936). The distribution of the Student's ratio for the non-normal samples. Supplement to the Journal of the Royal Statistical Society, 3, 178-184.

© 2013, David E. Giles