Chapter 2 Simple Linear Regression

2.1 Getting started

Putting text here

2.2 Foundation

Putting text here

2.3 Inference

Putting text here

2.4 Prediction

2.5 Checking conditions

2.6 Partioning variability

2.7 Derivation for slope and intercept

This document contains the mathematical details for deriving the least-squares estimates for slope (${\beta }_1$) and intercept (${\beta }_0$). We obtain the estimates, ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$ by finding the values that minimize the sum of squared residuals ().

$$SSR=\sum _{i=1}^n[y_i-{\hat{y}}_i{]}^2=[y_i-({\hat{\beta }}_0+{\hat{\beta }}_1{x}_i){]}^2=[y_i-({\hat{\beta }}_0-{\hat{\beta }}_1{x}_i{]}^2$$

Recall that we can find the values of ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$ that minimize () by taking the partial derivatives of () and setting them to 0. Thus, the values of ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$ that minimize the respective partial derivative also minimize the sum of squared residuals. The partial derivatives are

[Math Processing Error]$$\begin{array}{rl}& \frac{\mathrm{\partial }\text{SSR}}{\mathrm{\partial }{\hat{\beta }}_1}=-2\sum _{i=1}^n{x}_i(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i)\\ & \frac{\mathrm{\partial }\text{SSR}}{\mathrm{\partial }{\hat{\beta }}_0}=-2\sum _{i=1}^n(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i)\end{array}$$

Let’s begin by deriving ${\hat{\beta }}_0$.

[Math Processing Error]$$\begin{array}{rl}\frac{\mathrm{\partial }\text{SSR}}{\mathrm{\partial }{\hat{\beta }}_0}& =-2\sum _{i=1}^n(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i)=0\\ & \Rightarrow -\sum _{i=1}^n(y_i+{\hat{\beta }}_0+{\hat{\beta }}_1{x}_i)=0\\ & \Rightarrow -\sum _{i=1}^n{y}_i+n{\hat{\beta }}_0+{\hat{\beta }}_1\sum _{i=1}^n{x}_i=0\\ & \Rightarrow n{\hat{\beta }}_0=\sum _{i=1}^n{y}_i-{\hat{\beta }}_1\sum _{i=1}^n{x}_i\\ & \Rightarrow {\hat{\beta }}_0=\frac{1}{n}(\sum _{i=1}^n{y}_i-{\hat{\beta }}_1\sum _{i=1}^n{x}_i)\\ & \Rightarrow {\hat{\beta }}_0=\overline{y}-{\hat{\beta }}_1\overline{x}\end{array}$$

Now, we can derive ${\hat{\beta }}_1$ using the ${\hat{\beta }}_0$ we just derived

[Math Processing Error]$$\begin{array}{rl}& \frac{\mathrm{\partial }\text{SSR}}{\mathrm{\partial }{\hat{\beta }}_1}=-2\sum _{i=1}^n{x}_i(y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i)=0\\ & \Rightarrow -\sum _{i=1}^n{x}_i{y}_i+{\hat{\beta }}_0\sum _{i=1}^n{x}_i+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=0\\ \text{(Fill in }{\hat{\beta }}_0\text{)}& \Rightarrow -\sum _{i=1}^n{x}_i{y}_i+(\overline{y}-{\hat{\beta }}_1\overline{x})\sum _{i=1}^n{x}_i+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=0\\ & \Rightarrow (\overline{y}-{\hat{\beta }}_1\overline{x})\sum _{i=1}^n{x}_i+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=\sum _{i=1}^n{x}_i{y}_i\\ & \Rightarrow \overline{y}\sum _{i=1}^n{x}_i-{\hat{\beta }}_1\overline{x}\sum _{i=1}^n{x}_i+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=\sum _{i=1}^n{x}_i{y}_i\\ & \Rightarrow n\overline{y}\overline{x}-{\hat{\beta }}_{1}n{\overline{x}}^2+{\hat{\beta }}_1\sum _{i=1}^n{x}_i^2=\sum _{i=1}^n{x}_i{y}_i\\ & \Rightarrow {\hat{\beta }}_1\sum _{i=1}^n{x}_i^2-{\hat{\beta }}_{1}n{\overline{x}}^2=\sum _{i=1}^n{x}_i{y}_i-n\overline{y}\overline{x}\\ & \Rightarrow {\hat{\beta }}_1(\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2)=\sum _{i=1}^n{x}_i{y}_i-n\overline{y}\overline{x}\\ & {\hat{\beta }}_1=\frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{y}\overline{x}}{\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2}\end{array}$$

To write ${\hat{\beta }}_1$ in a form that’s more recognizable, we will use the following:

$$\sum x_i{y}_i-n\overline{y}\overline{x}=\sum (x-\overline{x})(y-\overline{y})=(n-1)\text{Cov}(x,y)$$

$$\sum x_i^2-n{\overline{x}}^2-\sum (x-\overline{x}{)}^2=(n-1)s_x^2$$

where $\text{Cov}(x,y)$ is the covariance of $x$ and $y$, and $s_x^2$ is the sample variance of $x$ ($s_x$ is the sample standard deviation).

Thus, applying () and (), we have

[Math Processing Error]$$\begin{array}{rl}{\hat{\beta }}_1& =\frac{\sum _{i=1}^n{x}_i{y}_i-n\overline{y}\overline{x}}{\sum _{i=1}^n{x}_i^2-n{\overline{x}}^2}\\ & =\frac{\sum _{i=1}^n(x-\overline{x})(y-\overline{y})}{\sum _{i=1}^n(x-\overline{x}{)}^2}\\ & =\frac{(n-1)\text{Cov}(x,y)}{(n-1)s_x^2}\\ & =\frac{\text{Cov}(x,y)}{s_x^2}\end{array}$$

The correlation between $x$ and $y$ is $r=\frac{\text{Cov}(x,y)}{s_x{s}_y}$. Thus, $\text{Cov}(x,y)=r{s}_x{s}_y$. Plugging this into (), we have

$${\hat{\beta }}_1=\frac{\text{Cov}(x,y)}{s_x^2}=r\frac{s_y{s}_x}{s_x^2}=r\frac{s_y}{s_x}$$