0
Introduction

When it comes to stats, one of the first topics we learn is linear regression. But many people don’t realize how deep
the linear regression topic is. Below are my partial notes on Linear Regression for anyone who may find this helpful.
Linear Model
A basic statistical model with single explanatory variable has equation describing the relation between x
and the mean
$\mu$ of the conditional distribution of Y at each value of x.
$
E(Y_{i}) = \beta_{0} + \beta_{1}x_{i}
$
Alternative formulation for the model expresses $Y_{i}$
$
Y_{i} = \beta_{0} + \beta_{1}x_{i} + \epsilon_{i}
$
where $\epsilon_{i}$ is the deviation of $Y_{i}$ from $E(Y_{i}) = \beta_{0} + \beta_{1}x_{i} + \epsilon_{i}$ is called
the error
term, since it represents the error that results from using the conditional expectation of Y at $x_{i}$ to
predict the individual observation.
Least Squares Method
For the linear model $E(Y_{i}) = \beta_{0} + \beta_{1}x_{i}$, with a sample of n observations the least squares method
determines the value of $\hat{\beta_{0}}$ and $\hat{\beta_{1}}$ that minimize the sum of squared residuals.
$
\sum_{i=1}^{n}(y_{i}-\hat{\mu_{i}})^2 = \sum_{i=1}^{n}[y_{i}-(\hat{\beta_{0}} + \hat{\beta_{1}}x_{i})]^2 = \sum_{i=1}^{n}e^{2}_{i}
$
As a function of model parameters $(\beta_{0} , \beta_{1})$, the expression is quadratic in $\beta_{0},\beta_{1}$
$
Continue reading