Questions and materials (book) can be seen here: https://drive.google.com/drive/folders/1I-EmYKSZMan1hzv1q_qwZKxuDJkpL0ib?usp=sharing
Untitled 10 The VectorGeometry ofLinear Models* A s is clear from the previous chapter, linear algebra is the algebra of linear models.Vector geometry provides a spatial representation of linear algebra and therefore fur- nishes a powerful tool for understanding linear models. The geometric understanding of linear models is venerable: R. A. Fisher’s (1915) development of the central notion of degrees of freedom in statistics was closely tied to vector geometry, for example. Few points in this book are developed exclusively in geometric terms. The reader who takes the time to master the geometry of linear models, however, will find the effort worthwhile: Certain ideas—including degrees of freedom—are most simply developed or understood from the geometric perspective.1 The chapter begins by describing the geometric vector representation of simple and multiple regression. Geometric vectors are then employed (in the spirit of Fisher’s seminal paper) to explain the connection between degrees of freedom and unbiased estimation of the error var- iance in linear models. Finally, vector geometry is used to illuminate the essential nature of overparametrized analysis-of-variance (ANOVA) models. 10.1 Simple Regression We can write the simple-regression model in vector form in the following manner: y=α1n þ βxþ " ð10:1Þ where y[ Y1; Y2; . . . ; Yn½ % 0, x [ x1; x2; . . . ; xn½ % 0, " [ ε1; ε2; . . . ; εn½ % 0, and 1n [ 1; 1; . . . ; 1½ % 0; α and β are the population regression coefficients.2 As before, we will assume that " ; Nnð0; σ2InÞ. The fitted regression equation is, similarly, y=A1n þ Bxþ e ð10:2Þ where e [ E1; E2; . . . ;En½ % 0 is the vector of residuals, and A and B are the least-squares regres- sion coefficients. From Equation 10.1, we have EðyÞ=α1n þ βx Analogously, from Equation 10.2, by=A1n þ Bx 1The basic vector geometry on which this chapter depends is developed in online Appendix B. 2Note that the X -values are treated as fixed. As in the previous chapter, the development of the vector geometry of lin- ear models is simpler for fixed X , but the results apply as well when X is random. 245 We are familiar with a seemingly natural geometric representation of fX ; Yg data—the scatter- plot—in which the axes of a two-dimensional coordinate space are defined by the variables X and Y and where the observations are represented as points in the space according to their fxi; Yig coordinates. The scatterplot is a valuable data-analytic tool as well as a device for thinking about regression analysis. I will now exchange the familiar roles of variables and observations, defining an n-dimen- sional coordinate space for which the axes are given by the observations and in which the vari- ables are plotted as vectors. Of course, because there are generally many more than three observations, it is not possible to visualize the full vector space of the observations.3 Our inter- est, however, often inheres in two- and three-dimensional subspaces of this larger n-dimen- sional vector space. In these instances, as we will see presently, visual representation is both possible and illuminating. Moreover, the geometry of higher-dimensional subspaces can be grasped by analogy to the two- and three-dimensional cases. The two-dimensional variable space (i.e., in which the variables define the axes) and the n- dimensional observation space (in which the observations define the axes) each contains a complete representation of the ðn · 2Þ data matrix ½x;y%. The formal duality of these spaces means that properties of the data, or of models meant to describe them, have equivalent repre- sentations in both spaces. Sometimes, however, the geometric representation of a property will be easier to understand in one space than in the other. The simple-regression model of Equation 10.1 is shown geometrically in Figure 10.1. The subspace depicted in this figure is of dimension 3 and is spanned by the vectors x, y, and 1n. Because y is a vector random variable that varies from sample to sample, the vector diagram necessarily represents a particular sample. The other vectors shown in the diagram clearly lie in the subspace spanned by x, y, and 1n: EðyÞ is a linear combination of x and 1n (and thus lies α1 n α1 n + βx 1 n yε x βx E(y) = E(ε) = 0 Figure 10.1 The vector geometry of the simple-regression model, showing the three-dimensional subspace spanned by the vectors x, y, and 1n. Because the expected error is 0, the expected-Yvector, E(y), lies in the plane spanned by 1n and x. 3See Exercise 10.1 for a scaled-down, two-dimensional example, however. 246 Chapter 10. The Vector Geometry of Linear Models* in the f1n; xg plane), and the error vector " is y& α1n & βx. Although " is nonzero in this sample, on average, over many samples, Eð"Þ= 0. Figure 10.2 represents the least-squares simple regression of Y on X , for the same data as shown in Figure 10.1. The peculiar geometry of Figure 10.2 requires some explanation: We know that the fitted values by are a linear combination of 1n and x and hence lie in the f1n; xg plane. The residual vector e=y& by has length jjejj= ffiffiffiffiffiffiffiffiffiffiffiP E2i p —that is, the square root of the residual sum of squares. The least-squares criterion interpreted geometrically, therefore, speci- fies that e must be as short as possible. Because the length of e is the distance between yand by, this length is minimized by taking byas the orthogonal projection of yonto the f1n; xg plane, as shown in the diagram. Variables, such as X and Y in simple regression, can be treated as vectors—x and y—in the n-dimensional space whose axes are given by the observations. Written in vector form, the simple-regression model is y=α1n þ βxþ ". The least-squares regression, y=A1n þ Bxþ e, is found by projecting yorthogonally onto the plane spanned by 1n and x, thus minimizing the sum of squared residuals, jjejj2. 10.1.1 Variables in Mean Deviation Form We can simplify the vector representation for simple regression by eliminating the constant regressor 1n and, with it, the intercept coefficient A. This simplification is worthwhile for two reasons: e y x Bx 0 A1 n A1 n + Bx 1 n y = ∧ Figure 10.2 The vector geometry of least-squares fit in simple regression. Minimizing the residual sum of squares is equivalent to making the e vector as short as possible. The by vector is, therefore, the orthogonal projection of y onto the {1n, x} plane. 10.1 Simple Regression 247 1. Our diagram is reduced from three to two dimensions. When we turn to multiple regres- sion—introducing a second explanatory variable—eliminating the constant allows us to work with a three-dimensional rather than a four-dimensional subspace. 2. The ANOVA for the regression appears in the vector diagram when the constant is eliminated, as I will shortly explain. To get rid of A, recall that Y =Aþ Bx; subtracting this equation from the fitted model Yi =Aþ Bxi þ Ei produces Yi & Y ¼ Bðxi & xÞ þ Ei Expressing the variables in mean deviation form eliminates the regression constant. Defining y( [ fYi & Yg and x( [ fxi & xg, the vector form of the fitted regression model becomes y( =Bx( þ e ð10:3Þ The vector diagram corresponding to Equation 10.3 is shown in Figure 10.3. By the same argu- ment as before,4 by( [ fbYi & Yg is a multiple of x(, and the length of e is minimized by taking by( as the orthogonal projection of y( onto x(. Thus, B= x( ) y( jjx(jj2 = P ðxi & xÞðYi & Y Þ P ðxi & xÞ 2 which is the familiar formula for the least-squares slope in simple regression.5 Sums of squares appear on the vector diagram as the squared lengths of vectors. I have already remarked that RSS = X E2i = jjejj 2 e y* 0 RSS RegSS W TSS y* = Bx* ∧ x* Figure 10.3 The vector geometry of least-squares fit in simple regression for variables in mean deviation form. The analysis of variance for the regression follows from the Pythagorean theorem. The correlation between X and Yis the cosine of the angle W separating the x( and y( vectors. 4The mean deviations for the fitted values are fbYi & Yg because the mean of the fitted values is the same as the mean of Y . See Exercise 10.2. 5See Section 5.1. 248 Chapter 10. The Vector Geometry of Linear Models* Similarly, TSS = X ðYi & Y Þ 2 = jjy(jj2 and RegSS = X ðbYi & Y Þ 2 = jjby(jj2 The ANOVA for the regression, TSS=RegSSþ RSS, follows from the Pythagorean theorem. The correlation coefficient is r= ffiffiffiffiffiffiffiffiffiffiffiffiffiffi RegSS TSS r = jjby(jj jjy(jj The vectors by( and y( are, respectively, the adjacent side and hypotenuse for the angle W in the right triangle whose vertices are given by the tips of 0, y(, and by(. Thus, r= cosW : The correlation between two variables (here, X and Y ) is the cosine of the angle separating their mean deviation vectors. When this angle is 0, one variable is a perfect linear function of the other, and r= cos 0= 1. When the vectors are orthogonal, r= cos 908 = 0. We will see shortly that when two variables are negatively correlated, 908