Maximum likelihood estimates of a binomial event

Let’s suppose that we have a population of red and white balls, and that a ball is red with an unknown probability $p$. This $p$ is the parameter of a binomial probability distribution, that gives us the probability that $k$ out of $n$ balls are red as:

$$P[(n,k),p]=(\frac{n}{k})p^k{(1-p)}^{(n-k)}$$

Let’s suppose now that we take 5 balls from the population, and that two are red and three white. We are observing that $n=5$ and $k=2$, so the likelihood function for this population is:

$$L[p,(5,2)]=(\frac{5}{2})p^2{(1-p)}^3$$

The value of $p$ that maximizes $L\left(p\right)$ will be the maximum likelihood estimator of the probability of the population. Let’s represent the likelihood function:

It is likely that you are not surprised when you learn that the maximum likelihood estimator of $p$ is $2/5=0.4$.

Maximum likelihood estimates of a normal distribution

Let’s suppose now that we have a sample of $n$ independent observations $x=\{x_1,\dots ,x_n\}$ from a normal distribution with un{kown population mean $\mu$ and population variance ${\sigma }^2$. The density probability function of this variable is the Gaussian function:

$$P(x,(\mu ,\sigma ))=\frac{1}{\sigma \sqrt{2\pi }}\text{exp}[-\frac{{(x-\mu )}^2}{2{\sigma }^2}]$$

If the observations of $x$ are independent and coming from the same normal distribution $N(\mu ,\sigma )$, the probability of joint ocurrrence is equal to the product of the values of the Gaussian function for each observation. Then, we can define the likelihood function as:

$$L[(\mu ,\sigma ),x]=\prod _{i=1}^n\frac{1}{\sigma \sqrt{2\pi }}\text{exp}[-\frac{{(x_i-\mu )}^2}{2{\sigma }^2}]$$

Finding the minimum of this function can be hard. A way of making this easier is to minimize the logarithm of the likelihood function. This arises frequently when we are dealing with likelihood functions with normal distributions, and it is known as log likelihood function $l[(\mu ,\sigma ),x]$. We can use the log likelihood instead of the likelihood because the logarithm is a monotonous function.

The log of the above likelihood function is:

$$l[(\mu ,\sigma ),x]=-\frac{n}{2}ln\left(2\pi \right)-\frac{n}{2}ln\left({\sigma }^2\right)-\frac{1}{2{\sigma }^2}\sum _{i=1}^n{(x_i-\mu )}^2$$

To obtain the maximum likelihood estimates of mean and variance $\widehat{mu}$ we equal to zero the partial derivative:

$$\frac{\partial l}{\partial \mu }=-\frac{1}{{\sigma }^2}\sum _{i=1}^n(x_i-\mu )=0$$

$$\hat{\mu }=\frac{1}{n}\sum _{i=1}^n{x}_i$$

We proceed in a similar way to obtain the maximum likelihood estimator of the variance:

$$\frac{\partial l}{\partial {\sigma }^2}=\frac{1}{2{\sigma }^2}[\frac{1}{{\sigma }^2}\sum _{i=1}^n{(x_i-\mu )}^2-n]=0$$

$${\hat{\sigma }}^2=\frac{1}{n}\sum _{i=1}^n{(x_i-\hat{\mu })}^2$$

Maximum likelihood estimates in linear regression

Let’s move now to the linear regression model:

$$y_i={\beta }_0+{\beta }_1{x}_{i1}+\cdots +{\beta }_p{x}_{ip}+{\epsilon }_i$$

Coefficients ${\beta }_0,\dots ,{\beta }_p$ are population coefficients, that we can estimate through $b_0,\dots ,b_p$ estimators. Let’s consider the residuals obtained when using those estimators:

$$e_i=y_i-b_0-b_1{x}_{i1}-\cdots -b_p{x}_{ip}$$

Let’s make some assumptions about residuals:

• observations are independent: this means that the residuals of an observation do not depend on other observations, or on exogenous variable (e.g., time) not considered in the model.
• residuals follow a normal distribution $e_i\sim N(0,\sigma )$: a normal distribution, with population mean zero and constant variance ${\sigma }^2$.

Given these assumptions, the likelihood function of the residuals is:

$$L[(\sigma ,\mu ),e]=\prod _{i=1}^3\frac{1}{\sigma \sqrt{2\pi }}\text{exp}\left(\frac{e_i^2}{2{\sigma }^2}\right)$$

And its log likelihood:

$$l[(\sigma ,\mu ),e]=-\frac{n}{2}ln\left(2\pi \right)-\frac{n}{2}ln\left({\sigma }^2\right)-\frac{1}{2{\sigma }^2}\sum _{i=1}^i{e}_i^2$$ dd If the above assumptions about residuals are valid, maximizing the log likelihood is the same as minimizing sum of squared residuals. This means that the ordinary least squeres (OLS) estimates are the maximum likelihood estimates of the coefficients of the linear regression model if $e_i\sim N(0,\sigma )$, that is, if the residuals follow a normal distribution with constant variance.

References

• Taboga, Marco (2017). Normal distribution - Maximum Likelihood Estimation, Lectures on probability theory and mathematical statistics, Third edition. Kindle Direct Publishing. Online appendix. https://www.statlect.com/fundamentals-of-statistics/normal-distribution-maximum-likelihood