01-R2.Rmd

---
date: "Apr 30, 2019"
header-includes:
- \usepackage{bbm}
- \usepackage{amsmath}
- \usepackage{graphicx}
- \usepackage{caption}
- \usepackage{bm}
- \usepackage{multirow}
- \usepackage{float}
- \usepackage{color}
- \usepackage{amssymb}
- \usepackage{amsthm}
output:
  html_document: default
---
\newcommand\pr{{\rm pr}}
\newcommand\E{{\rm E}}
\newcommand\V{{\rm Var}}
\newcommand\uA{{\bf A}}
\newcommand\ua{{\bf a}}
\newcommand\uB{{\bf B}}
\newcommand\ub{{\bf b}}
\newcommand\uC{{\bf C}}
\newcommand\uc{{\bf c}}
\newcommand\uD{{\bf D}}
\newcommand\ud{{\bf d}}
\newcommand\ue{{\bf e}}
\newcommand\uE{{\bf E}}
\newcommand\uf{{\bf f}}
\newcommand\uh{{\bf h}}
\newcommand\ug{{\bf g}}
\newcommand\uI{{\bf I}}
\newcommand\uK{{\bf K}}
\newcommand\um{{\bf m}}
\newcommand\uM{{\bf M}}
\newcommand\T{{\bf T}}
\newcommand\bO{{\bf O}}
\newcommand\bP{{\bf P}}
\newcommand\uQ{{\bf Q}}
\newcommand\uU{{\bf U}}
\newcommand\uv{{\bf v}}
\newcommand\uV{{\bf V}}
\newcommand\uS{{\bf S}}
\newcommand\uu{{\bf u}}
\newcommand\uw{{\bf w}}
\newcommand\uW{{\bf W}}
\newcommand\uH{{\bf H}}
\newcommand\uX{{\bf X}}
\newcommand\ux{{\bf x}}
\newcommand\uY{{\bf Y}}
\newcommand\uy{{\bf y}}
\newcommand\uZ{{\bf Z}}
\newcommand\uz{{\bf z}}
\newcommand\0{{\boldsymbol 0}}
\newcommand\1{{\boldsymbol 1}}
\newcommand\ubeta{{\boldsymbol \beta}}
\newcommand\btau{{\boldsymbol \tau}}
\newcommand\bg{{\boldsymbol \gamma}}
\newcommand\ueta{{\boldsymbol \eta}}
\newcommand\bpi{{\boldsymbol \pi}}
\newcommand\uxi{{\boldsymbol \xi}}
\newcommand\utheta{{\boldsymbol \theta}}
\newcommand\umu{{\boldsymbol \mu}}
\newcommand\uepsilon{{\boldsymbol \epsilon}}
\newcommand\bOmega{{\boldsymbol\Omega}}
\newcommand\uSigma{{\boldsymbol \Sigma}}
\newcommand\uPsi{{\boldsymbol \Psi}}
\newcommand\bLam{{\bf \Lambda}}
\newcommand\ualpha{{\boldsymbol \alpha}}
\newcommand\usigma{{\boldsymbol \sigma}}
\newcommand\uphi{{\boldsymbol \phi}}
\newcommand\nbd{{\rm nbd}}
\newcommand\diag{{\rm diag}}

# The coefficient of determination ($R^2$) {#R2}

**Summary**: it is not a good criterion because $R^2$ increases with the size of model; in other words, it always choose biggest model. 

**Interpretation** by [wiki]("https://en.wikipedia.org/wiki/Coefficient_of_determination"):
It is a statistic used in the context of statistical models whose main purpose is either the prediction of future outcomes or the testing of hypotheses, on the basis of other related information. **It provides a measure of how well observed outcomes are replicated by the model, based on the proportion of total variation of outcomes explained by the model.**

## $R^2$

**Denifition**: 
\begin{eqnarray}
\text{R}^2 = 1-\frac{RSS}{TSS} = 1- \frac{\sum_i(y_i-\hat{f_i})^2}{\sum_i(y_i-\bar y)^2},
\end{eqnarray}
where TSS is total sum of squares, RSS is residual sum of squares. And define $\text{ESS} = \sum_i(\hat f - \bar y)^2$ is explained sum of squares, also called the regression sum of square. $R^2$ is based on the assumption that $TSS = RSS + ESS$. Under the linear regression model setting satisfies this assumption usually.

**Proof**:

\begin{eqnarray*}
&&\sum_i(y_i-\bar y)^2 = \sum(y_i-\hat{f_i}+ \hat{f_i} - \bar y)^2 \\
&=&\sum_i(y_i-\hat{f_i})^2 + \sum_i(\hat{f_i} - \bar y)^2 + 2\sum_i(y_i-\hat{f_i})(\hat{f_i} - \bar y)\\
&=& RSS + ESS + 2\sum_i\hat{e}_i(\hat{f_i} - \bar y) \,(\hat{f_i}=\hat{y_i}=\uX\hat{\ubeta}\enspace\text{in linear model}) \\
&=& RSS + ESS + 2\sum_i\hat{e}_i(\hat{y_i} - \bar y)\\
&=& RSS + ESS + 2\sum_i\hat{e}_i\hat{y_i}-2\bar y\sum_i\hat{e}_i
\end{eqnarray*}
Then, the reamining part is to prove $\sum_i\hat{e}_i(\hat{y_i} - \bar y)=0$. 

Firstly, $\sum_i\hat{e}_i\hat{y_i} = \ue^{\T}\uH\uy = \uy^{\T}(\uI-\uH)\uH\uy = 0$ due to $\uH$ idempotent.
Then if we can show $\sum_i \hat{e}_i=0$, our proof is done. 
**However, this can not be shown for a model without an intercept.**

### $R^2$ in the model with an intercept

To see this, the partial derivative of our normal equation w.r.t $\beta_0$ is:
$$
\frac{\partial ESS}{\partial\hat\beta_0} = \frac{\sum_i(y_i-\hat\beta_0-x_i\hat\beta_1)^2}{\partial\hat\beta_0} = -2\sum_i(y_i-\hat\beta_0-x_i\hat\beta_1)=0,
$$
which can be rearranged to $\sum_iy_i = \sum_i\hat\beta_0+\hat\beta_1\sum_ix_i=\sum_i\hat y_i$. Thus, $\sum\hat e_i = \sum_iy_i - \sum_i\hat y_i = 0$.

Hence in a model with intecept, we have that $TSS = RSS + ESS$ of that 
$1 = \frac{RSS}{TSS} + \frac{ESS}{TSS}$.

From this $R^2$ is defined as $R^2\overset{def}{=}1-\frac{RSS}{TSS}$.

By the above, $R^2\geq0$

### $R^2$ in the model without an intercept

$R^2\overset{def}{=}1-\frac{RSS}{TSS}  = \frac{ESS+2\sum_i(y_i-\hat{y_i})(\hat{y_i} - \bar y)}{\sum_i(y_i-\bar y)^2}$. If the second term of numerator is large positive value, then $R^2$ can be larger than 1 or it is a small negative value, then $R^2$ can be negative. 


## Ajusted $R^2$