# Maximum Likehood

Document Sample

```					Connexions module: m13500                                                                                                                  1

Maximum Likelihood Estimation -
∗
Examples

Ewa Paszek
This work is produced by The Connexions Project and licensed under the
†

Abstract
This course is a short series of lectures on Introductory Statistics.                  Topics covered are listed in the

this course has been supported by NSF 0203396 grant.

1 MAXIMUM LIKELIHOOD ESTIMATION - EXAMPLES

1.1 EXPONENTIAL DISTRIBUTION

Let   X1 , X2 , ..., Xn   be a random sample from the exponential distribution with p.d.f.

1 −x/θ
f (x; θ) =        e    , 0 < x < ∞, θ ∈ Ω = {θ; 0 < θ < ∞}.
θ
The likelihood function is given by

n
1 −x1 /θ         1 −x2 /θ               1 −xn /θ            1       −     i=1   xi
L (θ) = L (θ; x1 , x2 , ..., xn ) =     e                e            ···       e            =      exp                    , 0 < θ < ∞.
θ                θ                      θ                  θn             θ

The natural logarithm of      L (θ)   is

n
1
lnL (θ) = − (n) ln (θ) −                    xi , 0 < θ < ∞.
θ    i=1

Thus,
n
d [lnL (θ)]   −n             i=1   xi
=    +                      = 0.
dθ        θ              θ2
The solution of this equation for       θ   is
n
1
θ=             xi = x.
n   i=1

∗ Version   1.3: Oct 8, 2007 3:44 pm GMT-5

http://cnx.org/content/m13500/1.3/
Connexions module: m13500                                                                                                                   2

Note that,
d [lnL (θ)]   1                    nx
=          −n +                 > 0, θ < x,
dθ       θ                     θ
d [lnL (θ)]   1                    nx
=          −n +                 = 0, θ = x,
dθ       θ                     θ
d [lnL (θ)]   1                    nx
=          −n +                 < 0, θ > x,
dθ       θ                     θ
Hence,   lnL (θ)    does have a maximum at        x,    and thus the maximum likelihood estimator for                        θ   is

n
^                1
θ= X =                     Xi .
n   i=1

This is both an unbiased estimator and the method of moments estimator for                                 θ.

1.2 GEOMETRIC DISTRIBUTION

Let   X1 , X2 , ..., Xn   be a random sample from the geometric distribution with p.d.f.

x−1
f (x; p) = (1 − p)              p, x = 1, 2, 3, ....

The likelihood function is given by

P
x1 −1             x2 −1                      xn −1                      xi −n
L (p) = (1 − p)         p(1 − p)          p · · · (1 − p)            p = pn (1 − p)              , 0 ≤ p ≤ 1.

The natural logarithm of       L (θ)   is

n
lnL (p) = nlnp +                  xi − n ln (1 − p) , 0 < p < 1.
i=1

Thus restricting p to     0<p<1         so as to be able to take the derivative, we have

n
dlnL (p)  n                   xi − n
i=1
= −                         = 0.
dp     p                  1−p
Solving for p, we obtain
n              1
p=          n          =      .
i=1   xi        x
So the maximum likelihood estimator of p is

^            n               1
p=          n           =
i=1   Xi         X

Again this estimator is the method of moments estimator, and it agrees with the intuition because, in n
n
observations of a geometric random variable, there are n successes in the
i=1   xi   trials. Thus the estimate

of p is the number of successes divided by the total number of trials.

http://cnx.org/content/m13500/1.3/
Connexions module: m13500                                                                                                                                         3

1.3 NORMAL DISTRIBUTION

Let   X1 , X2 , ..., Xn   be a random sample from             N (θ1 , θ2 ),          where

Ω = ((θ1 , θ2 ) : −∞ < θ1 < ∞, 0 < θ2 < ∞) .

That is, here let    θ1 = µ    and   θ2 = σ 2 .   Then

n                                                  2
1         (xi − θ1 )
L (θ1 , θ2 ) =              √        exp −                                    ,
i−1
2πθ2           2θ2

or equivalently,
n                        n                 2
1                        −         i=1   (xi − θ1 )
L (θ1 , θ2 ) =   √                   exp −                                           , (θ1 , θ2 ) ∈ Ω.
2πθ2                                     2θ2
The natural logarithm of the likelihood function is

n                   2
n             −                          i=1  (xi − θ1 )
lnL (θ1 , θ2 ) = − ln (2πθ2 ) −                                            .
2                                            2θ2
The partial derivatives with respect to            θ1      and   θ2      are

n
∂ (lnL)   1
=                         (xi − θ1 )
∂θ1     θ2                i=1

and
n
∂ (lnL)   −n    1                                          2
=     + 2                             (xi − θ1 ) .
∂θ2     2θ2  2θ2                      i=1
∂(lnL)                                                                ∂(lnL)
The equation
∂θ1     =0    has the solution         θ 1 = x.         Setting
∂θ2       =0    and replacing           θ1   by   x   yields

n                 2
1
θ2 =               (xi − x) .
n i=1

By considering the usual condition on the second partial derivatives, these solutions do provide a maxi-

mum. Thus the maximum likelihood estimators

µ = θ1

and

σ 2 = θ2
are
^
θ1 = X
and
n                        2
^         1
θ2 =            Xi − X .
n i=1
Where we compare the above example with the introductory one, we see that the method of moments

estimators and the maximum likelihood estimators for                            µ and σ 2        are the same. But this is not always the case.

If they are not the same, which is better? Due to the fact that the maximum likelihood estimator of                                                         θ   has

an approximate normal distribution with mean                      θ    and a variance that is equal to a certain lower bound, thus

at least approximately, it is unbiased minimum variance estimator.                                       Accordingly, most statisticians prefer

the maximum likelihood estimators than estimators found using the method of moments.

http://cnx.org/content/m13500/1.3/
Connexions module: m13500                                                                                                                             4

1.4 BINOMIAL DISTRIBUTION

Observations:k successes in n Bernoulli trials.

n!                n−x
f (x) =                   px (1 − p)
x! (n − x)!

n                  n                                                                      n
n!                  n−xi                                         n!                       n− n xi
P
L (p) =         f (xi ) =                           pxi (1 − p)                            =                           pxi (1 − p)       i=1

i=1               i=1
xi ! (n − xi )!                                                   x ! (n − xi )!
i=1 i

n                                       n
lnL (p) =            xi lnp +              n−              xi       ln (1 − p)
i=1                                  i=1
n                              n
dlnL (p)   1                                                         1
=                      xi −       n−              xi           =0
dp      p         i=1                            i=1
1−p

^         n                                 n             ^
1− p            i=1        xi − (n −              i=1     xi ) p
=0
^              ^
p       1− p

n                     n                              n
^                          ^                    ^
xi − p                xi − n p +                    xi p = 0
i=1                i=1                            i=1
n
^
i=1     xi          k
p=                         =
n               n

1.5 POISSON DISTRIBUTION

Observations:        x1 , x2 , ..., xn ,
λx e−λ
f (x) =                 , x = 0, 1, 2, ...
x!
n                                                    n
λxi e−λ                            λ     i=1 xi
L (λ) =                                       = e−λn              n
i=1
xi !                                  i=1 xi

n                                n
lnL (λ) = −λn +                          xi lnλ − ln                    xi
i=1                               i=1
n
dl               1
= −n +     xi
dλ        i=1
λ
n
1
−n +                  xi     =0
i=1
λ
n
^
i=1   xi
λ=
n

http://cnx.org/content/m13500/1.3/

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 12 posted: 9/22/2012 language: Latin pages: 4