A Regression "Estimator" that Minimizes MSE

Let's talk about estimating the coefficients in a linear multiple regression model. We know from the Gauss-Markhov Theorem that, within the class of linear and unbiased estimators, the OLS estimator is most efficient. Because it is unbiased, it therefore has the smallest possible Mean Squared Error (MSE), within the linear and unbiased class of estimators.

However, there are many linear estimators which, although biased, have a smaller MSE than the OLS estimator. You might then think of asking: “Why don’t I try and find the linear estimator that has the smallest possible MSE?”

This certainly sounds like a sensible, well-motivated question. Unfortunately, however, attempting to pursue this line of reasoning yields an “estimator” that can’t actually be used in practice. It's non-operational.

Let's see why. I'll demonstrate this by using the simple linear regression model without an intercept, although the result generalizes to the usual multiple linear regression model. So, our model, with a non-random regressor, is:

                   y_i = βx_i + ε_i    ;     ε_i ~ i.i.d. [0 , σ²]  ;  i = 1, 2, ...., n.

Let β* be any linear estimator of β, so that we can write β* = Σa_iy_i, where the a_i's are non-random weights, and all summations are taken over i = 1 to n.

So, E[β*] = βΣ(a_ix_i) , and

                Bias[β*] = β[Σ(a_ix_i) - 1].                                                       (1)

Similarly,

               var.[β*] = Σ[a_i²var.(y_i)] = σ²Σ(a_i²).                                       (2)

From, (1) and (2),

              M = MSE[β*] = var.[β*] + (Bias[β*])² = σ²Σ(a_i²) + β²[Σ(a_ix_i) - 1]² .

Now, let's find the weights in the construction of β* that will minimize that estimator's MSE. Let M_j' be the partial derivative of M with respect to a typical a_j (for j = 1, 2, ...., n). Then:

             M_j' = 2σ²a_j + 2β²[Σ(a_ix_i) - 1]x_j    ;   j = 1, 2, ...., n.                 (3)

Setting all of the equations in (3) to zero, multiplying by y_j, and summing over j, we get:

            σ²β* + β²[Σ(a_ix_i) - 1]Σ(x_jy_j) = 0 .                                              (4)

Similarly, setting all of the equations in (3) equal to zero, multiplying by x_j, and summing over all j, we get:

             Σ(a_jx_j) = Σ(a_ix_i) = [β²Σ(x_i²)] / [σ² + β²Σ(x_i²)] .                         (5)

Substituting (5) into (4), and re-arranging the result, we finally get:

             β* = { [β²Σ(x_i²)] / [σ² + β²Σ(x_i²)] }b ,                                      (6)

where b = [Σ(x_iy_i)] / [Σ(x_i²] is the OLS estimator of β.

So, the minimum MSE linear "estimator" of β is non-operational. It can't be applied, because it is a function of β and σ², both of which are unknown. Yes, we could make the estimator operational by replacing the unknown parameters with their OLS estimators - but the resulting modified β* would then be nonlinear and, more importantly, it would no longer have any optimal MSE property.

Nice idea, but it didn't work! There's no viable linear estimator of the regression coefficient vector that minimizes MSE.
   
Finally, we can see from equation (6) that because σ² > 0, β* is also a shrinkage "estimator" - it shrinks the value of b towards the origin.

© 2013, David E. Giles