Reverse Regression Follow-up

At the end of my recent post on Reverse Regression, I posed three simple questions - homework for the students among you, if you will. 

Here they are again, with brief "solutions":

First recall the context. We fitted the following simple regression model, using OLS:

            y_i = βx_i + ε_i   .                                                              (1)

All of the data are calculated as deviations from their respective sample means.

The OLS estimator of β is, 

            b = Σ(x_iy_i) / Σ(x_i²) ,

where the summations are for i = 1 to n (the sample size).

Then we estimated the "reverse regression":

           x_i = αy_i + u_i   ,                                                              (2)

and the OLS estimator of α is,

           a = Σ(x_iy_i) / Σ(y_i²).

We showed that a ≤ (1 / b), regardless of the values of the data in the sample.

The questions I posed, with their answers, are as follows:

1.  Under what circumstances will (4) hold as an equality?

If each value of x_i is proportional to the corresponding y_i value, with the same proportionality constant, for all i, then a = b.

2.  What can you say about the relationship between the two R² values that we get when we estimate (1) and (2) by OLS?

They must be identical to each other!

For (1), the sum of the squared residuals is:     

Σ(y_i - bx_i)² = Σ(y_i²) + b²Σ(x_i²) - 2bΣ(x_iy_i)          

                 = Σ(y_i²) - [Σ(x_iy)]² / Σ(x_i²)

                          = [Σ(x_i²) Σ(y_i²) - [Σ(x_iy_i)]²] / Σ(x_i²) .                      (3)

So, the corresponding R² is:

      R_b²  = 1 - [Σ(x_i²) Σ(y_i²) - [Σ(x_iy_i)]²] / [Σ(x_i²) Σ(y_i²)].

Note that this expression is totally symmetric in the x_i's and y_i's. So, obviously, the R² associated with equation (2), say R_a², equals R_b².

Moreover, we can see from the expression for R_b² that it is just the squared (Pearson) sample correlation between the x's and the y's. So, of course it's the same in each case.

3.  What can you say about the relationship between the t-ratios for testing H₀: β = 0 in (1); and for testing H₀': α = 0 in (2)?

Once again, they must also be identical to each other! 

From equation (3) the unbiased estimator of the error variance in equation (1) is:

s_b² = (1 / (n - 1)) [Σ(x_i²) Σ(y_i²) - [Σ(x_iy_i)]²] / Σ(x_i²) ,

and so the "standard error" associated with the OLS estimator, b, is:

s.e.(b) = {(1 / (n - 1)) [Σ(x_i²) Σ(y_i²) - [Σ(x_iy_i)]²] / [Σ(x_i²)]²}^½ ,

and the t-ratio for testing H₀ : β = 0 is:

t_β = [Σ(x_iy_i) / Σ(x_i²)] / s.e.(b)    = (n - 1)Σ(x_iy_i) / [Σ(x_i²) Σ(y_i²) - [Σ(x_iy_i)]²] .

Again, this last expression is symmetric in the x_i and y_i, and so the t-ratio for testing H₀': α = 0, say t_α, is equal to the expression for t_β. 

For any non-believers, here's a little illustration using EViews with some artificial data. The latter are available on the data page for this blog. You can replicate the results with any software of your choice.

Equation (1):

Equation (2):

From equation (2), a = 2.063878 < (1 / b) = (1 / 0.412) = 2.4272. The R² values are the same, as are the two slope coefficient t-ratios.

You'll also notice that the sample means of both X and Y are non-zero, but I retained an intercept in the models. As I noted at the beginning of my previous post, this is equivalent to going through the analysis assuming that all of the data have been expressed as deviations about their respective sample means.

© 2014, David E. Giles