Linear Regression Notes
Introduction
When it comes to stats, one of the first topics we learn is linear regression. But most people don’t realize how deep the linear regression topic is, and observing blind applications in day-to-day life makes me cringe. This post is not about virtue-signaling(as I know some areas I haven’t explored myself), but to share my notes which may be helpful to others.
Linear Model
A basic stastical model with single explanatory variable has equation describing the relation between x and the mean
$\mu$ of the conditional distribution of Y at each value of x.
$ E(Y_{i}) = \beta_{0} + \beta_{1}x_{i} $
Alternative formulation for the model expresses $Y_{i}$
$ Y_{i} = \beta_{0} + \beta_{1}x_{i} + \epsilon_{i} $
where $\epsilon_{i}$ is the deviation of $Y_{i}$ from $E(Y_{i}) = \beta_{0} + \beta_{1}x_{i} + \epsilon_{i}$ is called
the error term, since it represents the error that results from using the conditional expectation of Y at $x_{i}$ to
predict the individual observation.
Least Squares Method
For the linear model $E(Y_{i}) = \beta_{0} + \beta_{1}x_{i}$, with a sample of n observations the least squares method determines the value of $\hat{\beta_{0}}$ and $\hat{\beta_{1}}$ that minimize the sum of squared residuals.
$ \sum_{i=1}^{n}(y_{i}-\hat{\mu_{i}})^2 = \sum_{i=1}^{n}[y_{i}-(\hat{\beta_{0}} + Continue reading
