A Terrific New Book on the Linear Model

Recently, it was my distinct pleasure to review a first-class book by David Harville, titled Linear Models and the Relevant Distributions and Matrix Algebra.

(Added 28 February, 2019: You can now read the published review in Statistical Papers, here.)

Here is what I had to say:

Linear Models and the Relevant Distributions and Matrix Algebra. David A. Harville, 2018. (Chapman & Hall/CRC, Boca Raton, FL, 2018; ISBN 978-1-138-57833-3; pp. xiii-524)

Sometimes you read a book, and you think: ‘Why didn’t someone write this before now?’ This is one of those books.

The linear model is one of the key “work horse” models in statistics, and it provides a stepping-stone to many other important models and associated statistical techniques. A thorough understanding of the statistical and algebraic (and/or geometrical) foundations of this model is a core requirement for anyone training or researching in statistical modeling. 

To be sure, there are many excellent books that provide either introductory or advanced discussions of the linear model. However, one of Harville’s major contributions is that this monograph covers both the requisite linear algebra and the statistical theory in a very thorough and balanced manner. It provides a one-stop source of both the statistical and algebraic information needed for a deep understanding of the linear statistical model. In addition, of course, the large range of “tools” that are introduced and described carefully are invaluable in a many other statistical settings.

For these reasons, it has to be compared with some stellar competitors. The seminal books by Rao (1965) and Searle (1971) immediately come to mind. In this reviewer’s opinion, Linear Models and the Relevant Distributions and Matrix Algebra, compares with these gems most favourably. 

The author notes in the Preface to this volume: 

“In the writing of Matrix Algebra from a Statistician’s Perspective, I adopted the philosophy that (to the greatest extent feasible) the discourse should include the theoretical underpinnings of essentially every result. In the writing of the present volume I have adopted much the same philosophy. Of course, doing so has a limiting effect on the number of topics and the number of results that can be covered.” (p. xii)

Harville has certainly met his objective - but not at the expense of what could have been a “dry” and dense tome. On the contrary, his writing is clear and welcoming at all times, and he has a talent for explaining complex concepts in a careful but accessible manner.

So, what does this book actually cover? As foreshadowed, the topics are limited, but they ae chosen carefully and dealt with thoroughly. The book comprises seven chapters, including a more than adequate 21-page introduction. The next three chapters deal with a thorough “primer” on matrix algebra; an introduction to random vectors and matrices; and the framework of the general linear model itself.

My only disappointment in the coverage to this point was with (the very brief) section 4.5, “Multivariate Data”. Although the multiple-equation case is presented allowing for different regressor matrices in different equations, this is quickly specialized to the (very restrictive) “multivariate regression” model in which all regressor matrices are equal. In other words, and regrettably, the vast and important literature than was spawned by Zellner’s (1962) “Seemingly Unrelated Regression Equations” (SURE) model rates no mention at all.

Chapter 5 is titled, “Estimation and Prediction: Classical Approach”. This chapter contains many of the book’s core results, presented from a “classical”, or non-Bayesian perspective. Harville pre-empts this choice in his Preface:

“As discussed in a 2014 paper (Harville, 2014), I regard the division of statistical inference along Bayesian-frequentist lines as unnecessary and undesirable. What in a Bayesian approach is referred to as the prior distribution can simply be regarded as part of a hierarchical model, In combination with the original model it leads to a new model ….” (p. xi).

Apart from this explanation, Harville’s decision is consistent with his desire to limit his focus, and to provide a rigorous presentation. I have no problem with this at all.

This chapter provides a very thorough account of the Aitken (GLS) estimator and the Gauss-Markhov Theorem. Importantly, the discussion stands out from other such treatments by contrasting the notions of Best Linear Unbiased estimation, and Best Linear Translation-Equivariant estimation. In addition, not only is the Maximum Likelihood estimator discussed for the standard situation where the regression errors follow a (multivariate) Normal distribution, but this is then extended to the much more general case of elliptically symmetric errors. This is an important, extension that is all too often overlooked (or not recognized).  The optimality results associated with the GLS estimator when the errors are Normal also apply under elliptical symmetry.

A thorough treatment of several standard probability distributions is to be found in Chapter 6. The distributions chosen are those needed for the remainder of the book. This chapter also includes a nice discussion of the distributions of quadratic forms and second-degree polynomials in a Normal random vector. The material in this chapter is presented in a style, and with a level of rigour, that equals that adopted by Searle (1971) in his classic coverage of these results.

The final chapter in the book covers a lot of ground in its 150 pages. In broad terms, its title is largely self-explanatory - “Confidence Intervals (or Sets) and Test of Hypotheses”. However, this masks some hidden gems. For instance, the handling of the familiar F-test is particularly strong, in at least two respects. First, the “optimality” of that test, namely being “Uniformly Most Powerful Invariant” (UMPI), is treated very carefully and in an accessible manner. The “invariance” aspect of this result is not fully understood by many students when they encounter the linear model. Harville’s excellent discussion of this is exemplary. Second, the extension of both the distributional and optimality properties of the test to the case of elliptically symmetric errors is very important. Regrettably we still encounter many practitioners (and authors of textbooks) who wrongly claim that Normal disturbances are necessary for such results to hold, even though this has been known to be false for decades. Kutos to the author for giving this issue the attention that it deserves.

Another highlight of this chapter is the special attention paid to the issues of multiple comparisons and simultaneous confidence intervals, in section 7.7. Again, the value-added here is substantial, and it sets this book aside from its competitors.

Each chapter of the book concludes with a collection of interesting and pertinent exercises. A “Solutions Manual” is available to those who adopt it as a text. 

In summary, Linear Models and the Relevant Distributions and Matrix Algebra is a first-class volume that will serve as an essential reference for graduate students and established researchers alike in statistics and other related disciplines such as econometrics, biometrics, and psychometrics. As the author discusses, it can also serve as the basis for graduate-level courses which have various emphases.  I recommend it strongly.

Sometimes you read a book, and you think: ‘I wish I had the talent to have written this.’ This is definitely one of those books.

References

Harville, D. A. (1997). Matrix Algebra from a Statistician’s Perspective. Springer-Verlag, New York.

Harville, D. A. (2014). The need for more emphasis on prediction: A ‘nondenominational’ model-based approach. American Statistician, 68, 71-92.

Rao, C. R. (1965). Linear Statistical Inference and its Applications. Wiley, New York.

Searle, S. R. (1971). Linear Models. Wiley, New York.

Zellner, A. (1962). An efficient method of estimating seemingly unrelated regression equations and tests for aggregation bias. Journal of the American Statistical Association, 57, 348–368.

I hope that you have the opportunity to take a look at David's book - it should be on the shelf of your personal library.

© 2019, David E. Giles