This post follows on from my recent one, Recursions for the Moments of Some Discrete Distributions. I'm going to assume that you've read the previous post, so this one will be shorter.
What I'll be discussing here are some useful recursion formulae for computing the moments of a number of continuous distributions that are widely used in econometrics. The coverage won't be exhaustive, by any means. I provide some motivation for looking at formulae such as these in the previous post, so I won't repeat it here.
When we deal with the Normal distribution, below, we'll make explicit use of Stein's Lemma. Several of the other results are derived (behind the scenes) by using a very similar approach. So, let's begin by stating this Lemma.
Stein's Lemma (Stein, 1973):
Stein's Lemma (Stein, 1973):
"If X ~ N[θ , σ2], and if g(.) is a differentiable function such that E|g'(X)| is finite, then
E[g(X)(X - θ)] = σ2 E[g'(X)]."
It's worth noting that although this lemma relates to a single Normal random variable, in the bivariate Normal case the lemma generalizes to:
In this latter form, the lemma is useful in asset pricing models.
There are extensions of Stein's Lemma to a broader class univariate and multivariate distributions. For example, see Alghalith (undated), and Landsman et al. (2013), and the references in those papers. Generally, if a distribution belongs to an exponential family, then recursions for its moments can be obtained quite easily.
"If X and Y follow a bivariate Normal distribution, and if g(.) is a differentiable function such that E|g'(Y)| is finite, then
Cov.[g(Y ), X] = Cov.(X , Y) E[g'(Y)]."
In this latter form, the lemma is useful in asset pricing models.
There are extensions of Stein's Lemma to a broader class univariate and multivariate distributions. For example, see Alghalith (undated), and Landsman et al. (2013), and the references in those papers. Generally, if a distribution belongs to an exponential family, then recursions for its moments can be obtained quite easily.
Now, let's get down to business............
Recall that the rth. "raw moment" (or "moment about zero") for the random variable X, with distribution function F, is defined as μr' = E[X r] = ∫ xrdF(x) , for r = 1, 2, .....; and the "central (centered) moments" of X are defined as μr = E[(X - μ1' )r] , for r = 1, 2, 3, .......
Also, we can express one of these sets of moments in terms of the other set by using the two relationships:
μr = Σ{[r! / (i! (r - i)!)] μi' (- μ1' ) r-i } , (1)
where the range of summation is from i = 0 to i = r ; and
μr' = Σ{[r! / (i! (r - i)!)] μi (μ1' ) r-i } , (2)
where, again, the range of summation is from i = 0 to i = r.
Normal distribution
Also, note that we can write:
μ3' = E[X 3] = E[X 2(X - θ + θ)] = E[X 2(X - θ)] + θ E[X 2] .
μ3' = E[X 3] = E[X 2(X - θ + θ)] = E[X 2(X - θ)] + θ E[X 2] .
Applying Stein's Lemma, with g(X) = X 2, we obtain:
Chi-square distribution
μ3' = 2σ2 E[X] + θ E[X 2] = θμ2' + 2σ2μ1' = θ3 + 3θσ2.
Similarly, if we apply the Lemma with g(X) = X k ; k =3, 4, ......, we obtain:
μ4' = E[X 4] = E[X 3(X - θ)] + θ E[X 3] = θμ3' + 3σ2μ2' = θ4 + 6θ2σ2 + 3σ4.
μ5' = E[X 5] = E[X 4(X - θ)] + θ E[X 4] = θμ4' + 4σ2μ3' = θ5 + 10θ3σ2 + 15θσ4;
and so on.
So, each moment is obtained recursively from all of the lower-order moments, through the repeated use of Stein's Lemma.
More particularly, you can see immediately that following recursion formula holds for the raw moments of the Normal distribution (Bain, 1969; Willink, 2005):
μr+1' = θ μr' + rσ2 μr-1' ; r = 1, 2, 3, .........Similarly, if we apply the Lemma with g(X) = X k ; k =3, 4, ......, we obtain:
μ4' = E[X 4] = E[X 3(X - θ)] + θ E[X 3] = θμ3' + 3σ2μ2' = θ4 + 6θ2σ2 + 3σ4.
μ5' = E[X 5] = E[X 4(X - θ)] + θ E[X 4] = θμ4' + 4σ2μ3' = θ5 + 10θ3σ2 + 15θσ4;
and so on.
So, each moment is obtained recursively from all of the lower-order moments, through the repeated use of Stein's Lemma.
More particularly, you can see immediately that following recursion formula holds for the raw moments of the Normal distribution (Bain, 1969; Willink, 2005):
Chi-square distribution
We'll use the following standard result for the Chi-square distribution with "v" degrees of freedom:
"For any real-valued function, h, E[h(χ2v)] = v E[h(χ2v+2) / χ2v+2], provided that the expectations exist."
"For any real-valued function, h, E[h(χ2v)] = v E[h(χ2v+2) / χ2v+2], provided that the expectations exist."
(This result can be generalized to the case of a non-central Chi-square distribution. See Appendix B of Judge and Bock (1978).
Let h(x) = xk, for some integer, k. Then, applying the above result repeatedly, we get:
- If k = 1, then μ1' = E[h(χ2v)] = v E[χ2v+2 / χ2v+2] = v .
- If k = 2, then μ2' = E[(χ2v)2] = v E[(χ2v+2)2 / χ2v+2] = vE[χ2v+2] = v(v + 2).
Immediately, we see that μ2 = Var.[χ2v] = μ2' - (μ1')2 = 2v.
- If k = 3, then μ3' = E[(χ2v)3] = v E[(χ2v+2)3 / χ2v+2] = vE[(χ2v+2)2] = v(v + 2)(v + 4) .
In terms of a genuine recurrence relationship, we see from the above that we can write:
μr' = μr-1' (v + 2(r -1)) ; r = 1, 2, 3, ........
Although we can then use equation (1) to obtain the central moments of X, there's also a separate recursion formula for these moments in the case of the Chi-square distribution This is discussed in the section on the Gamma distribution, below.
Student-t distribution
Suppose that X follows a Student-t distribution, with v degrees of freedom. Then the moments of X can be summarized as follows:
μr' = 0 ; if r is odd, and 0 < r < v
μr' = v(r/2) Π[(2i - 1) / (v - 2i)] ; if r is even, and 0 < r < v (3)
where the product is for i = 1 to i = (r / 2).
The inequality, v > r, ensures that the moment "exists" - that is, it is finite. If v = 1, then X follows a Cauchy distribution, and none of its moments "exist".
Equation (3) gives us a nice, tidy, and direct formula for obtaining any of the even-order moments of X. For instance, we see that:
μ2' = v [(2- 1) / (v - 2)] = v / (v - 2) ; if v > 2.
Obviously, μ2 = Var.[X] = μ2', because E[X] = 0.
Also,
μ4' = v2 [1 / (v - 2)][3 / (v - 4)] = 3v2 / [(v - 2)(v - 4)] ; if v > 4.
μ6' = v3 [1 / (v - 2)][3 / (v - 4)][5 / (v - 6)] = 15v3 / [(v - 2)(v - 4)(v - 6)] ; if v > 6.
etc.
This set of results isn't in the form of a recurrence relationship, but we can obtain one very easily. Note that:
and so on.
Clearly, in general, the following recurrence formula holds -
μr' = μr-2' [(r - 1) / (v - r)] ; if r is even, and 0 < r < v.
Trust me - this formula is much more appealing than integrating functions of the density, or working with the characteristic function, for this particular distribution.
Finally, note that because E[X] = μ1' = 0, μr = μr', for all (even) r.
F distribution
Suppose that X follows an F distribution, with numerator and denominator degrees of freedom, v1 and v2 respectively.
The general formula for the raw moments of X is:
μr' = (v2 / v1) r Γ[(v1 / 2) + r] Γ ([v2 / 2) - r] / (Γ [v1 / 2] Γ[v2 / 2]) . (4)
So,
μ1' = (v2 / v1)Γ[(v1 / 2) + 1] Γ ([(v2 / 2) - 1] / (Γ [v1 / 2] Γ[v2 / 2] .
Using the result that Γ[x + 1] = xΓ[x], and cancelling terms, this simplifies to
μ1' = v2 / (v2 - 2) ; if v2 > 2
Similarly,
μ2' = (v2 / v1)2 Γ[(v1 / 2) + 2] Γ ([(v2 / 2) - 2] / (Γ [v1 / 2] Γ[v2 / 2].
Again, noting that Γ[(v1 / 2) + 2] = ((v1 / 2) + 1)Γ[(v1 / 2) + 1]; that Γ[(v2 / 2) - 2] = Γ[(v2 / 2) - 1] / ((v2 / 2) - 2); and then simplifying, we get:
μ2' = μ1' (v2 /v1)[(v1 + 2) / (v2 - 4)] ; if v2 > 4
Proceeding in a similar manner, we find that:
μ3' = μ2' (v2 / v1)[(v1 + 4) / (v2 - 6)] ; if v2 > 6
μ4' = μ1' (v2 / v1)[(v1 + 6) / (v2 - 8)] ; if v2 > 8
etc.
So, in general you can see that we have the following recursion formula for the moments of X:
μr+1' = μr' (v2 / v1)[(v1 + 2r) / (v2 - 2(r + 1))] ; r = 0, 1, 2, .............; if v2 > 2(r + 1).
This recurrence relationship avoids us having to deal with the gamma functions in (4), let alone having to perform any integration, or deal with this distribution's messy characteristic function. (As with the Student-t distribution, the moment generating function isn't defined here, because of the above conditions on the existence of the moments.)
The distributions that we've considered so far are ones that you use every day in your econometrics work. Now let's consider a couple more distributions that arise a bit less frequently.
Gamma distributionThe distributions that we've considered so far are ones that you use every day in your econometrics work. Now let's consider a couple more distributions that arise a bit less frequently.
One situation where the Gamma distribution comes up in econometrics is in the context of "count data". This might seem a bit odd, because count data are non-negative integers, and we're talking about continuous random variables here.
However, the Gamma distribution comes into play when we generalize the Poisson distribution to a particular form of the Negative Binomial distribution. Both of these distributions were discussed in my previous, related, post. Looking back at that post, you'll be able to see that the variance of a Poisson random variable equals its mean. We say that the distribution is "equi-dispersed". This is very restrictive, and isn't realistic with a lot of count data in practice. On the other hand, the variance of the Negative Binomial distribution exceeds its mean. We say that this distribution is "over-dispersed", and in practice this is often more reasonable.
To construct the Negative Binomial distribution in the form that we usually use it in econometrics, we take a Poisson random variable and then add in an unobserved random effect to its (conditional) mean. If this random effect follows a Gamma distribution, we end up with the Negative Binomial distribution for the count data. (See Greene, 2012, pp.806-807 for details.)
With this by way of motivation, consider a random variable, X that follows a Gamma distribution with a shape parameter 'a', and a scale parameter 'b'. (Be careful here - there are two forms of the Gamma distribution. The other one has the shape parameter 'a' and the rate parameter, θ = 1 / b.)
Willink (2003) shows that the central moments (moments about the mean) for X can be obtained from the following recursion -
μr = (r - 1)(bμr-1 + ab2μr-2) ; r = 2, 3, ..........
There are two special cases of the Gamma distribution that we might note. First, if a = (v / 2), and b = 2, then X follows a Chi-square distribution with v degrees of freedom. So, the central moments of the Chi-square distribution follow the recursion relationship:
μr = 2(r - 1)(μr-1 + vμr-2) ; r = 2, 3, ..........
Second, if a = b = 1, then X follows an Exponential distribution, and its central moments satisfy:
μr = (r - 1)(μr-1 + μr-2) ; r = 2, 3, .........
In addition, Withers (1992) shows that for the Exponential distribution we have the following simpler recursion:
μr = rμr-1 + (-1)r ; r = 1, 2, 3, .......
Beta distribution
One of the important features of the Beta distribution is that its density's support is the unit interval (although this can be generalized). It's one of the few distributions used by econometricians that has a finite support.
This suggests that it might be useful when modelling data that are continuous in nature, and are in the form of proportions. Indeed, this is the case. I discuss this in the context of consumer demand analysis in this old post. More generally, the Beta regression model is discussed in detail by Ferrari and Cribari-Neto (2004) and Cribari-Neto and Zeileia (2010).
The density function for X, that follows a Beta distribution is:
f (x | α, β) = [Γ(α + β) / (Γ(α)Γ(β))] x α-1 (1 - x) β-1 ; 0 < x < 1 ; α , β > 0
The general formula for the rth. raw moment of X is
μr' = E[X r] = [ (α + r - 1)(α + r - 2) ...... α] / [(α + β + r - 1)(α + β + r - 2) .....(α + β)] ; r = 1, 2, 3, ......
Immediately, we have:
μ1' = α / (α + β)
μ2' = [α (α + 1)] / [(α + β + 1)(α + β)] = [(α + 1) / (α + β + 1)] μ1'
μ3' = [α (α + 1)(α + 2)] / [(α + β + 2)(α + β + 1)(α + β)] = [(α + 2) / (α + β + 2)] μ2'
etc.
So, a general recursion formula for the raw moments of the Beta distribution is:
μr' = [(α + r - 1) / (α + β + r - 1)] μr-1' ; r = 1, 2, 3, .....
One of the important features of the Beta distribution is that its density's support is the unit interval (although this can be generalized). It's one of the few distributions used by econometricians that has a finite support.
This suggests that it might be useful when modelling data that are continuous in nature, and are in the form of proportions. Indeed, this is the case. I discuss this in the context of consumer demand analysis in this old post. More generally, the Beta regression model is discussed in detail by Ferrari and Cribari-Neto (2004) and Cribari-Neto and Zeileia (2010).
The density function for X, that follows a Beta distribution is:
f (x | α, β) = [Γ(α + β) / (Γ(α)Γ(β))] x α-1 (1 - x) β-1 ; 0 < x < 1 ; α , β > 0
The general formula for the rth. raw moment of X is
μr' = E[X r] = [ (α + r - 1)(α + r - 2) ...... α] / [(α + β + r - 1)(α + β + r - 2) .....(α + β)] ; r = 1, 2, 3, ......
Immediately, we have:
μ1' = α / (α + β)
μ2' = [α (α + 1)] / [(α + β + 1)(α + β)] = [(α + 1) / (α + β + 1)] μ1'
μ3' = [α (α + 1)(α + 2)] / [(α + β + 2)(α + β + 1)(α + β)] = [(α + 2) / (α + β + 2)] μ2'
etc.
So, a general recursion formula for the raw moments of the Beta distribution is:
μr' = [(α + r - 1) / (α + β + r - 1)] μr-1' ; r = 1, 2, 3, .....
Numerical examples
Remember, the whole point abut these recursion formulae is they help us to rapidly compute all of the moments of a distribution, up to some pre-specified maximum order, using one general formula.
There is an R script file on the code page for this blog that illustrates this point, first for X ~ N[θ, σ2]; and second for X ~ F[v1,v2]. In the first case the first ten raw moments for X when θ = 1 and σ2 = 4 are: 1, 5, 13, 73, 281, 1741, 8485, 57233, 328753, 2389141, ....
In the second case, the first ten raw moments for X when v1 = 7 and v2 = 24 are: 1.09, 1.68, 3.5265, 9.82, 36.09, 175.28, 1141.85, 10276.63, 135064.30, 2894235.00, ....
References
References
Alghalith, M., undated. A note on generalizing and extending Stein's lemma.
Bain, L. J., 1969. Moments of non-central t and on-central F distribution. American Statistician, 23, 33-34.
Cobb, L., P. Koppstein, & N. H. Chen, 1983. Estimation and moment recursion relations for multimodal distributions of the exponential family. Journal of the American Statistical Association, 78, 124-130.
Bain, L. J., 1969. Moments of non-central t and on-central F distribution. American Statistician, 23, 33-34.
Cobb, L., P. Koppstein, & N. H. Chen, 1983. Estimation and moment recursion relations for multimodal distributions of the exponential family. Journal of the American Statistical Association, 78, 124-130.
Cribari-Neto F. & A. Zeileis, 2010. Beta regression in R. Journal of Statistical Software, 1–24.
Ferrari, S. L. P. & F. Cribari-Neto, 2004. Beta regression for modelling rates and proportions. Journal of Applied Statistics, 31, 799–815.
Greene, W. E., 2012. Econometric Analysis, 7th. ed.. Prentice Hall, Upper Saddle River, NJ.
Judge, G. G. & M. E. Bock, 1978. The Statistical Implications of Pre-Test and Stein-Rule Estimators in Econometrics. North-Holland, New York.
Landsman, Z., S. Vanduffel, & J. Yao, 2013. A note on Stein's lemma for multivariate elliptical distributions. Journal of Statistical Planning and Inference, 11, 2016-2022.
Stein, C. M., 1973. Estimation of the mean of a multivariate normal distribution. Proceedings of the Prague Symposium on Asymptotic Statistics, 345-381.
Willink, R., 2003. Relationships between central moments and cumulants, with formulae for the central moments of gamma distributions. Communications in Statistics - Theory and Methods, 32, 701-704.
Willink, R., 2005. Normal moments and Hermite polynomials. Statistics and Probability Letters, 73, 271-275.
Withers, S. W., 1992. A recurrence relation for the moments of the exponential. New Zealand Statistician, 27, 13-14.
Woodland, A. D., 1979. Stochastic specification and the estimation of share equations. Journal of Econometrics, 10, 361-383.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.