Docstoc

Maximum Likehood

Document Sample
Maximum Likehood Powered By Docstoc
					Connexions module: m13500                                                                                                                  1




       Maximum Likelihood Estimation -
                                                                                          ∗
                                                    Examples

                                                         Ewa Paszek
                          This work is produced by The Connexions Project and licensed under the
                                                                                                  †
                                            Creative Commons Attribution License




                                                              Abstract
           This course is a short series of lectures on Introductory Statistics.                  Topics covered are listed in the

        Table of Contents.     The notes were prepared by Ewa Paszek and Marek Kimmel.                         The development of

        this course has been supported by NSF 0203396 grant.




1 MAXIMUM LIKELIHOOD ESTIMATION - EXAMPLES

1.1 EXPONENTIAL DISTRIBUTION

Let   X1 , X2 , ..., Xn   be a random sample from the exponential distribution with p.d.f.


                                                 1 −x/θ
                                 f (x; θ) =        e    , 0 < x < ∞, θ ∈ Ω = {θ; 0 < θ < ∞}.
                                                 θ
      The likelihood function is given by



                                                                                                                 n
                                       1 −x1 /θ         1 −x2 /θ               1 −xn /θ            1       −     i=1   xi
 L (θ) = L (θ; x1 , x2 , ..., xn ) =     e                e            ···       e            =      exp                    , 0 < θ < ∞.
                                       θ                θ                      θ                  θn             θ

      The natural logarithm of      L (θ)   is


                                                                             n
                                                                        1
                                       lnL (θ) = − (n) ln (θ) −                    xi , 0 < θ < ∞.
                                                                        θ    i=1

      Thus,
                                                                              n
                                                 d [lnL (θ)]   −n             i=1   xi
                                                             =    +                      = 0.
                                                      dθ        θ              θ2
The solution of this equation for       θ   is
                                                                   n
                                                              1
                                                         θ=             xi = x.
                                                              n   i=1

  ∗ Version   1.3: Oct 8, 2007 3:44 pm GMT-5
  † http://creativecommons.org/licenses/by/2.0/




http://cnx.org/content/m13500/1.3/
Connexions module: m13500                                                                                                                   2




      Note that,
                                             d [lnL (θ)]   1                    nx
                                                         =          −n +                 > 0, θ < x,
                                                  dθ       θ                     θ
                                             d [lnL (θ)]   1                    nx
                                                         =          −n +                 = 0, θ = x,
                                                  dθ       θ                     θ
                                             d [lnL (θ)]   1                    nx
                                                         =          −n +                 < 0, θ > x,
                                                  dθ       θ                     θ
      Hence,   lnL (θ)    does have a maximum at        x,    and thus the maximum likelihood estimator for                        θ   is


                                                                                n
                                                         ^                1
                                                         θ= X =                     Xi .
                                                                          n   i=1

This is both an unbiased estimator and the method of moments estimator for                                 θ.

1.2 GEOMETRIC DISTRIBUTION

Let   X1 , X2 , ..., Xn   be a random sample from the geometric distribution with p.d.f.


                                                                      x−1
                                              f (x; p) = (1 − p)              p, x = 1, 2, 3, ....

      The likelihood function is given by

                                                                                                       P
                                   x1 −1             x2 −1                      xn −1                      xi −n
                   L (p) = (1 − p)         p(1 − p)          p · · · (1 − p)            p = pn (1 − p)              , 0 ≤ p ≤ 1.

      The natural logarithm of       L (θ)   is


                                                               n
                                  lnL (p) = nlnp +                  xi − n ln (1 − p) , 0 < p < 1.
                                                              i=1

      Thus restricting p to     0<p<1         so as to be able to take the derivative, we have

                                                                              n
                                                  dlnL (p)  n                   xi − n
                                                                              i=1
                                                           = −                         = 0.
                                                     dp     p                  1−p
      Solving for p, we obtain
                                                                      n              1
                                                         p=          n          =      .
                                                                     i=1   xi        x
So the maximum likelihood estimator of p is


                                                        ^            n               1
                                                        p=          n           =
                                                                    i=1   Xi         X

      Again this estimator is the method of moments estimator, and it agrees with the intuition because, in n
                                                                                                         n
observations of a geometric random variable, there are n successes in the
                                                                                                         i=1   xi   trials. Thus the estimate

of p is the number of successes divided by the total number of trials.




http://cnx.org/content/m13500/1.3/
Connexions module: m13500                                                                                                                                         3




1.3 NORMAL DISTRIBUTION

Let   X1 , X2 , ..., Xn   be a random sample from             N (θ1 , θ2 ),          where


                                        Ω = ((θ1 , θ2 ) : −∞ < θ1 < ∞, 0 < θ2 < ∞) .

That is, here let    θ1 = µ    and   θ2 = σ 2 .   Then

                                                          n                                                  2
                                                                        1         (xi − θ1 )
                                       L (θ1 , θ2 ) =              √        exp −                                    ,
                                                         i−1
                                                                       2πθ2           2θ2

or equivalently,
                                                             n                        n                 2
                                                   1                        −         i=1   (xi − θ1 )
                            L (θ1 , θ2 ) =   √                   exp −                                           , (θ1 , θ2 ) ∈ Ω.
                                                  2πθ2                                     2θ2
The natural logarithm of the likelihood function is

                                                                                                 n                   2
                                                        n             −                          i=1  (xi − θ1 )
                                      lnL (θ1 , θ2 ) = − ln (2πθ2 ) −                                            .
                                                        2                                            2θ2
      The partial derivatives with respect to            θ1      and   θ2      are

                                                                                 n
                                                    ∂ (lnL)   1
                                                            =                         (xi − θ1 )
                                                      ∂θ1     θ2                i=1

and
                                                                                       n
                                              ∂ (lnL)   −n    1                                          2
                                                      =     + 2                             (xi − θ1 ) .
                                                ∂θ2     2θ2  2θ2                      i=1
                      ∂(lnL)                                                                ∂(lnL)
      The equation
                       ∂θ1     =0    has the solution         θ 1 = x.         Setting
                                                                                             ∂θ2       =0    and replacing           θ1   by   x   yields


                                                                           n                 2
                                                                      1
                                                         θ2 =               (xi − x) .
                                                                      n i=1

      By considering the usual condition on the second partial derivatives, these solutions do provide a maxi-

mum. Thus the maximum likelihood estimators


                                                                        µ = θ1

and

                                                                       σ 2 = θ2
are
                                                                       ^
                                                                       θ1 = X
and
                                                                        n                        2
                                                         ^         1
                                                         θ2 =            Xi − X .
                                                                   n i=1
      Where we compare the above example with the introductory one, we see that the method of moments

estimators and the maximum likelihood estimators for                            µ and σ 2        are the same. But this is not always the case.

If they are not the same, which is better? Due to the fact that the maximum likelihood estimator of                                                         θ   has

an approximate normal distribution with mean                      θ    and a variance that is equal to a certain lower bound, thus

at least approximately, it is unbiased minimum variance estimator.                                       Accordingly, most statisticians prefer

the maximum likelihood estimators than estimators found using the method of moments.




http://cnx.org/content/m13500/1.3/
Connexions module: m13500                                                                                                                             4




1.4 BINOMIAL DISTRIBUTION

Observations:k successes in n Bernoulli trials.


                                                                        n!                n−x
                                                     f (x) =                   px (1 − p)
                                                                   x! (n − x)!

                 n                  n                                                                      n
                                                  n!                  n−xi                                         n!                       n− n xi
                                                                                                                                              P
       L (p) =         f (xi ) =                           pxi (1 − p)                            =                           pxi (1 − p)       i=1

                 i=1               i=1
                                           xi ! (n − xi )!                                                   x ! (n − xi )!
                                                                                                          i=1 i

                                                          n                                       n
                                           lnL (p) =            xi lnp +              n−              xi       ln (1 − p)
                                                          i=1                                  i=1
                                                                   n                              n
                                           dlnL (p)   1                                                         1
                                                    =                      xi −       n−              xi           =0
                                              dp      p         i=1                            i=1
                                                                                                               1−p

                                                      ^         n                                 n             ^
                                                1− p            i=1        xi − (n −              i=1     xi ) p
                                                                                                                     =0
                                                                   ^              ^
                                                                    p       1− p

                                                 n                     n                              n
                                                               ^                          ^                    ^
                                                      xi − p                xi − n p +                    xi p = 0
                                                i=1                i=1                            i=1
                                                                              n
                                                                ^
                                                                              i=1     xi          k
                                                                   p=                         =
                                                                                  n               n

1.5 POISSON DISTRIBUTION

Observations:        x1 , x2 , ..., xn ,
                                                                       λx e−λ
                                                      f (x) =                 , x = 0, 1, 2, ...
                                                                         x!
                                                           n                                                    n
                                                                       λxi e−λ                            λ     i=1 xi
                                             L (λ) =                                       = e−λn              n
                                                          i=1
                                                                         xi !                                  i=1 xi

                                                                              n                                n
                                             lnL (λ) = −λn +                          xi lnλ − ln                    xi
                                                                             i=1                               i=1
                                                                                           n
                                                               dl               1
                                                                  = −n +     xi
                                                               dλ        i=1
                                                                                λ
                                                                              n
                                                                                           1
                                                                −n +                  xi     =0
                                                                             i=1
                                                                                           λ
                                                                                      n
                                                                       ^
                                                                                      i=1   xi
                                                                       λ=
                                                                                      n




http://cnx.org/content/m13500/1.3/

				
DOCUMENT INFO
Shared By:
Stats:
views:12
posted:9/22/2012
language:Latin
pages:4