linear regression

Goal: Find the best-fit line $y=mx+c$ for a set of data points $(x_i,y_i)$. “Best-fit” is defined as minimizing the sum of squared errors.

1. Error Function (Sum of Squared Errors):

We define the error, $E$, as the sum of the squared differences between the actual $y_i$ values and the predicted values ($m{x}_i+c$) from our line:

$E={\sum }_{i=1}^n(y_i-(m{x}_i+c){)}^2$

where n is number of element.

2. Minimizing the Error:

To minimize $E$, we find where its partial derivatives with respect to $m$ (slope) and $c$ (y-intercept) are equal to zero. This is because at a minimum (or maximum) of a function, the slope of the function in any direction is zero.

3. Partial Derivatives:

First calculate partial derivative:

• Partial derivative with respect to $m$:

$\frac{\partial E}{\partial m}={\sum }_{i=1}^{n}2(y_i-(m{x}_i+c))(-x_i)=-2{\sum }_{i=1}^n{x}_i(y_i-m{x}_i-c)$

• Partial derivative with respect to $c$:

$\frac{\partial E}{\partial c}={\sum }_{i=1}^{n}2(y_i-(m{x}_i+c))(-1)=-2{\sum }_{i=1}^n(y_i-m{x}_i-c)$

4. Setting Derivatives to Zero:

To find the minimum, set both partial derivatives equal to zero:

• $\frac{\partial E}{\partial m}=0\Rightarrow {\sum }_{i=1}^n{x}_i(y_i-m{x}_i-c)=0$
• $\frac{\partial E}{\partial c}=0\Rightarrow {\sum }_{i=1}^n(y_i-m{x}_i-c)=0$

5. Simplifying the Equations:

Expand and rearrange the equations:

• $\sum x_i{y}_i-m\sum x_i^2-c\sum x_i=0$ (Equation 1)
• $\sum y_i-m\sum x_i-nc=0$ (Equation 2)

6. Solving for c:

From Equation 2, isolate $c$:

$nc=\sum y_i-m\sum x_i$

$c=\frac{\sum y_i}{n}-m\frac{\sum x_i}{n}$

Recognize that $\frac{\sum y_i}{n}=\bar{y}$ (the mean of the $y$ values) and $\frac{\sum x_i}{n}=\bar{x}$ (the mean of the $x$ values):

$c=\bar{y}-m\bar{x}$ (Equation 3)

7. Solving for m: Substitute Equation 3 into Equation 1: $\sum x_i{y}_i-m\sum x_i^2-(\bar{y}-m\bar{x})\sum x_i=0$ $\sum x_i{y}_i-m\sum x_i^2-\bar{y}\sum x_i+m\bar{x}\sum x_i=0$ $m(\bar{x}\sum x_i-\sum x_i^2)=\bar{y}\sum x_i-\sum x_i{y}_i$

Multiply and divide equation by n: $m=\frac{n(\sum x_i{y}_i)-(\sum x_i)(\sum y_i)}{n(\sum x_i^2)-(\sum x_i{)}^2}$

Divide num and den by $n^2$:

$m=\frac{\frac{1}{n}\sum x_i{y}_i-\bar{x}\bar{y}}{\frac{1}{n}\sum x_i^2-{\bar{x}}^2}$

This is equivalent to: $m=\frac{Cov(x,y)}{{\sigma }_x^2}$ or $m=r\ast \frac{{\sigma }_y}{{\sigma }_x}$

where, $Cov(x,y)=\frac{1}{n}\sum x_i{y}_i-\bar{x}\bar{y}$ ${\sigma }_x=\frac{1}{n}\sum x_i^2-{\bar{x}}^2$

$r=\frac{Cov(x,y)}{{\sigma }_x{\sigma }_y}$

8. regression of x on y: $y=b_{yx}\ast x+c$

where, $b_{yx}=r\frac{{\sigma }_y}{{\sigma }_x}$ $c=\bar{y}-b_{yx}\ast \bar{x}$

Key Takeaways (for the exam):

• Error Function: $E=\sum (y_i-(m{x}_i+c){)}^2$
• Derivatives: Set $\frac{\partial E}{\partial m}=0$ and $\frac{\partial E}{\partial c}=0$
• Slope (m): $m=\frac{Cov(x,y)}{{\sigma }_x^2}=r\frac{{\sigma }_y}{{\sigma }_x}$
• Intercept (c): $c=\bar{y}-m\bar{x}$

• By=$b_{yx}$, means y is independent and x is dependent.