Large and Small Regression Coefficients

Here's a trap that newbies to regression analysis have been known to fall into. It's to do with comparing the numerical values of the point estimates of  regression coefficients, and drawing conclusions that may not actually be justified.

What I have in mind is the following sort of situation. Suppose that Betsy (name changed to protect the innocent) has estimated a regression model that looks like this:

               Y = 0.4 + 3.0X₁ - 0.7X₂ + 6.0X₃ +.....+ residual .

Betsy is really proud of her first OLS regression, and tells her friends that "X3 is two times more important  in explaining y than is X1" (or words to that effect).

Putting to one side such issues as statistical significance (I haven't reported any standard errors for the estimated coefficients), Is Betsy entitled to make such a statement - based on the earth-shattering observation that "six equals three times two"?

It's no cop-out, but the answer is "it depends"! Specifically, it depends on the units of measurement that we're dealing with here. Yes - units!

You need to remember how the coefficients are interpreted in a linear regression model. They're the partial derivatives of Y with respect to each of the X variables, in turn. It's a ceteris paribus situation, of the type beloved of economists. It's as if the other X variables are "held constant".

In other words, in a linear regression model, the coefficients are the "marginal effects". They measure how much Y changes for one-unit changes in the X variables. The  word, unit, is key here! Let's see why.

The first thing that we need to note is that we're dealing with a regression equation here. This means that unless all of the variables happen to be unit-less, the units of the term (Y) on the left side of the equation must equal those of the total right side of the equation. You can't equate apples with oranges.

Looking more closely at the right side of the estimated regression equation, we see that it's made up of a sum of terms. You can't add (or subtract) apples and oranges, either! So now we know two things about the units in question here:

1. The units of Y must be the same as the units of the terms "4.0", and " 0.7X₂ "; etc.
2. Each term on the right side of the equation must have the same units as every other term. That is, the units of the term "0.4" just be the same as the units of the terms "3.0X₁" and 6.0X₃", etc.

Let's make things a little more specific.

Suppose that Y is measured in $; X₁ is measured in Kg; and X₂ is measured in inches; and X₃ is measured in seconds. In that case the units of measurement for the estimated regression coefficients are $ (for the intercept); and then $ per Kg, $ per inch, and $ per second, for the next three coefficients. This means that every term (coefficient times variable) on the right side of the equation has the units of just $, as required.

Now we can see that there's something fundamentally wrong when Betsy says that the effect of X₃ on Y is twice as big as the effect of X₁ on Y. The two coefficients that are being "compared" in arriving at this conclusion have different units - ($/Kg.) and ($/second), respectively. That just doesn't make any sense!

Of course, this particular problem disappears if X₃ is also measured in Kg., or if X₁ is measured in seconds. It would also disappear if (for example) all of the variables in the model were unit-less.

Notice, though, that I said "this particular problem". Unfortunately, even if we don't have a units-measurement problem, there's typically going to be another reason why Betsy's numerical comparison, and her conclusion about the relative importance of different regressors, is going to be flawed.

A Digression:

To help us at this point, let's think back to two really simple things that we learn about in elementary descriptive statistics:

1. Converting the sample covariance for two random variables into a correlation coefficient.
2. Comparing the variability in one sample with that in another sample.

You'll remember that the first of these involves taking the covariance and dividing it by both of the sample standard deviations (for the the two variables). The covariance has units - they're the product of the units for each of the two variables. So, if one variable is measured in $, and the other in Kg., then their covariance has units which are ($ times Kg.). 

This isn't very easy to interpret!. Also, a covariance can take any value, positive or negative. So, is a covariance of (say) 7.634 $ times Kg. a large covariance or not

Converting the covariance to a correlation achieves two things. First, because a standard deviation has the same units as the variable itself, dividing the covariance by the standard deviations creates a measure that is unitless. Second, the resulting correlation has a finite scale. It's bounded in value between -1 and +1. Now it's much easier to interpret the value that we've computed; and we can now make meaningful comparisons between correlations for different problems.

We've essentially standardized the original covariance measure to make it more useful.

In Case 2 above, because a sample standard deviation has units that are the same as those of the original variable, we've got a similar problem if we want to compare the standard deviations (or variances) for two different samples. Unless the data in each sample happens to have the same units, it's like comparing apples with oranges, again.

There's a further problem, too. Each sample is likely to have values that differ in terms of their typical magnitude. In this case, we learn that if we divide each sample standard deviation by the associated sample mean, the resulting ratios will be unitless, and re-scaled. The ratio of a standard deviation to the mean is called the sample "coefficient of variation".

Once again, converting standard deviations into coefficients of variation gives us something that we can use to make comparative statements.

Now, if we think back to Betsy's situation, you can see that it has a lot in common with the two things we've just been thinking about. Maybe there's some insight here to help Betsy in her quest to compare the values of here estimated regression coefficients?

Well, yes, there is. It leads us to think of computing what we used to call "Beta Coefficients"  in econometrics (e.g., Goldberger, 1964). This term is still used in some disciplines, especially in some of the social sciences. I'm not talking about the original (beta) regression coefficients themselves, and neither am I referring to the "betas" that measure the risk of holding specific financial assets! 

I'm talking about "standardized" versions of the regression coefficients.

By now, you can probably guess how these measures are going to be defined and constructed. We take the sample data for the Y variable, and we standardize each of the n values. That is, we construct Y_i* = (Y_i - Ybar) / (s.d.(Y)), where Ybar = (1 /n)Σ(Y_i). Then we standardize each of the X variables (the regressors in our model) in a corresponding way.

Notice one detail. If there is an intercept variable (a column of ones) among the regressors, as in Betsy's model, this standardizing process will eliminate this intercept. Why? Well, simply because the sample average of the n "one" values for this also just "one", so when we subtract this average, the result is a zero value (at every point in the sample).

Then, using our standardized Y and X data, we fit our model using OLS. The resulting estimates are called the "standardized" regression coefficients, or the "Beta coefficients". Notice that while the terminology, "standardized coefficients", is quite common it's actually a bit of a mis-nomer. It's the data that have been standardized - not the coefficients.

What have we gained by all of this? Well, as Y_i*, and all of the standardized regressors, are unitless, so are the associated regression coefficients. It makes sense now to compare the values of the "Beta coefficients", and draw conclusions of the type that Betsy tried to come to by (wrongly) using the "regular" regression coefficients.

Here's a simple example to illustrate this. It's an OLS regression for an imports-demand function. The regressor P is  the exchange rate, so obviously its units of measurement are different from those of imports (IMP) or GDP! The estimated coefficients have the anticipated signs, and are highly significant.

WWBS (What Would Betsy Say?)

She'd wrongly conclude that (in absolute value) the impact of P on IMP is roughly 70,000 times greater than the impact of P on IMP.

To get the standardized regression coefficients we can select:

View ..... Coefficient Diagnostics ....... Scaled Coefficients

Here's what we get:

The interpretation of the standardized coefficients is as follows:

• A one sample standard deviation increase in GDP leads to an increase of 0.62 standard deviations in IMP (ceteris paribus).
• A one sample standard deviation in P leads to a decrease of 0.36 standard deviations in IMP.

In this particular sense, and contrary to what Betsy thought, the impact of P on IMP is roughly half the impact of GDP on IMP (in absolute value). Half as big, compared with 70,000 times as big!  I'm sure you get the picture!

Now let's check to see where the above results for the standardized coefficients came from.

We can create the standardized data as follows:

        series     IMP_STAR = (IMP - @MEAN(IMP)) / @STDEVS(IMP)

        series    GDP_STAR = (GDP - @MEAN(GDP)) / @STDEVS(GDP)

        series     P_STAR = (P - @MEAN(P)) / @STDEVS(P)

(Note that the S that forms part of the standard deviation function is to ensure that we get the sample standard deviations, rather than the population standard deviations. Does this actually affect the results for the standardized coefficients?)

Then, we fit the model with the standardized data (and no intercept):

Yes, we've confirmed the results for the standardized OLS regression coefficients. You can find the EViews workfile that I've used on the Code page for this blog, and the data are on the Data page.

(The standardized regression coefficients also get produced as a matter of course in a number of other packages, such as SPSS and SHAZAM, and with Stata, you can obtain them by using the "beta" option when fitting a regression model. In SAS you use the "stb" option on the "model" statement that goes with "proc reg" command for multiple regression. In R, you load the "QuantPsyc" package, and then use the command "lm.beta", in place of the usual "lm" command.)

A final, and very important comment..........

Our focus here has been on numerical differences between estimates of the regression coefficients. This is quite different from considering if the corresponding population parameters are different, and looking at the statistical significance associated with a test of that difference.

Reference

Goldberger, A. S., 1964. Econometric Theory. Wiley, New York.

© 2013, David E. Giles