Document Sample

page 93 110SOR201(2002) Chapter 6 Moment Generating Functions 6.1 Deﬁnition and Properties Our previous discussion of probability generating functions was in the context of discrete r.v.s. Now we introduce a more general form of generating function which can be used (though not exclusively so) for continuous r.v.s. The moment generating function (MGF) of a random variable X is deﬁned as eθx P(X = x) if X is discrete θX x MX (θ) = E(e )= ∞ (6.1) eθx fX (x)dx if X is continuous −∞ for all real θ for which the sum or integral converges absolutely. In some cases the existence of MX (θ) can be a problem for non-zero θ: henceforth we assume that M X (θ) exists in some neighbourhood of the origin, |θ| < θ 0 . In this case the following can be proved: (i) There is a unique distribution with MGF M X (θ). (ii) Moments about the origin may be found by power series expansion: thus we may write MX (θ) = E(eθX ) ∞ (θX)r = E r=0 r! ∞ θr = E(X r ) [i.e. interchange of E and valid] r=0 r! i.e. ∞ θr MX (θ) = µr where µr = E(X r ). (6.2) r=0 r! So, given a function which is known to be the MGF of a r.v. X, expansion of this function in a power series of θ gives µr , the r th moment about the origin, as the coeﬃcient of θ r /r!. (iii) Moments about the origin may also be found by diﬀerentiation: thus dr dr {MX (θ)} = E(eθX ) dθ r dθ r r d = E (eθX ) dθ r (i.e. interchange of E and diﬀerentiation valid) = E X r eθX . page 94 110SOR201(2002) So dr {MX (θ)} = E(X r ) = µr . (6.3) dθ r θ=0 (iv) If we require moments about the mean, µ r = E[(X − µ)r ], we consider MX−µ (θ), which can be obtained from MX (θ) as follows: MX−µ (θ) = E eθ(X−µ) = e−µθ E(eθX ) = e−µθ MX (θ). (6.4) θr Then µr can be obtained as the coeﬃcient of r! in the expansion ∞ θr MX−µ (θ) = µr (6.5) r=0 r! or by diﬀerentiation: dr µr = {MX−µ (θ)} . (6.6) dθ r θ=0 (v) More generally: Ma+bX (θ) = E eθ(a+bX) = eaθ MX (bθ). (6.7) Example Find the MGF of the N (0, 1) distribution and hence of N (µ, σ 2 ). Find the moments about the mean of N (µ, σ 2 ). Solution If Z ∼ N (0, 1), MZ (θ) = E(eθZ ) ∞ 1 1 2 = eθz √ e− 2 z dz −∞ 2π ∞ 1 1 = √1 2π −∞ exp{− (z 2 − 2θz + θ 2 ) + θ 2 }dz 2 2 1 2 √1 ∞ 1 2 = exp( 2 θ ) 2π exp{− (z − θ) }dz. ∞ 2 But here √1 exp{...} is the p.d.f. of N (θ, 1)), so 2π 1 MZ (θ) = exp( θ 2 ) (6.8) 2 If X = µ + σZ, X ∼ N (µ, σ 2 ), and MX (θ) = Mµ+σZ (θ) = eµθ MZ (σθ) by (6.7) 1 = exp(µθ + 2 σ 2 θ 2 ). page 95 110SOR201(2002) Then MX−µ (θ) = e−µθ MX (θ) = exp( 1 σ 2 θ 2 ) 2 1 ∞ ( 2 σ 2 θ 2 )r ∞ σ 2r 2r = = θ r=0 r! r=0 2r r! ∞ σ 2r (2r)! θ 2r = . . . r=0 2r r! (2r)! Using property (iv) above, we obtain µ2r+1 = 0, r = 1, 2, ... σ 2r (2r)! (6.9) µ2r = , r = 0, 1, 2, ... 2r r! e.g. µ2 = σ 2 ; µ4 = 3σ 4 . ♦ 6.2 Sum of independent variables Theorem Let X, Y be independent r.v.s with MGFs M X (θ), MY (θ) respectively. Then MX+Y (θ) = MX (θ).MY (θ). (6.10) Proof MX+Y (θ) = E eθ(X+Y ) = E eθX .eθY = E(eθX ).E(eθY ) [independence] = MX (θ).MY (θ). Corollary If X1 , X2 , ..., Xn are independent r.v.s, MX1 +X2 +···+Xn (θ) = MX1 (θ).MX2 (θ)...MXn (θ). (6.11) Note: If X is a count r.v. with PGF GX (s) and MGF MX (θ), MX (θ) = GX (eθ ) : GX (s) = MX (log s). (6.12) Here the PGF is generally preferred, so we shall concentrate on the MGF applied to continuous r.v.s. Example Let Z1 , ..., Zn be independent N (0, 1) r.v.s. Show that 2 2 V = Z 1 + · · · + Z n ∼ χ2 . n (6.13) Solution Let Z ∼ N (0, 1). Then 2 ∞ 2 1 1 2 MZ 2 (θ) = E eθZ = eθz √ e− 2 z dz −∞ 2π ∞ 1 1 = √ exp{− (1 − 2θ)z 2 }dz. −∞ 2π 2 1 √ Assuming θ < 2 , substitute y = 1 − 2θz. Then ∞ 1 1 2 1 1 1 MZ 2 (θ) = √ e− 2 y . √ dy = (1 − 2θ)− 2 , θ< . (6.14) −∞ 2π 1 − 2θ 2 page 96 110SOR201(2002) Hence 1 1 1 MV (θ) = (1 − 2θ)− 2 .(1 − 2θ)− 2 ...(1 − 2θ)− 2 1 = (1 − 2θ)−n/2 , θ < 2 . Now χ2 has the p.d.f. n 1 n 1 n n w 2 −1 e− 2 w , w ≥ 0; n a positive integer. 2 Γ( 2 ) 2 Its MGF is ∞ 1 n 1 eθw n n w 2 −1 e− 2 w dw 0 2 Γ( 2 ) 2 ∞ 1 n 1 = n n w 2 −1 exp{− w(1 − 2θ)}dw 0 2 2 Γ( 2 ) 2 1 1 (t = 2 (1 − 2θ) (θ < 2 )) n 1 ∞ n = (1 − 2θ)− 2 n t 2 −1 e−t dt Γ( 2 ) 0 n = (1 − 2θ)− 2 , θ< 1 2 = MV (θ). So we deduce that V ∼ χ2 . Also, from MZ 2 (θ) we deduce that Z 2 ∼ χ2 . n 1 If V1 ∼ χ2 1 , V2 ∼ χ2 2 and V1 , V2 are independent, then n n n1 n2 1 MV1 +V2 (θ) = MV1 (θ).MV2 (θ) = (1 − 2θ)− 2 (1 − 2θ)− 2 (θ < 2 ) = (1 − 2θ)−(n1 +n2 )/2 . So V1 + V2 ∼ χ2 1 +n2 . [This was also shown in Example 3, §5.8.2.] n 6.3 Bivariate MGF The bivariate MGF (or joint MGF) of the continuous r.v.s (X, Y ) with joint p.d.f. f (x, y), −∞ < x, y < ∞ is deﬁned as ∞ ∞ MX,Y (θ1 , θ2 ) = E eθ1 X+θ2 Y = eθ1 x+θ2 y f (x, y)dxdy, (6.15) −∞ −∞ provided the integral converges absolutely (there is a similar deﬁnition for the discrete case). If MX,Y (θ1 , θ2 ) exists near the origin, for |θ1 | < θ10 , |θ2 | < θ20 say, then it can be shown that ∂ r+s MX,Y (θ1 , θ2 ) r s = E(X r Y s ). (6.16) ∂θ1 ∂θ2 θ1 =θ2 =0 The bivariate MGF can also be used to ﬁnd the MGF of aX + bY , since MaX+bY (θ) = E e(aX+bY )θ = E e(aθ)X+(bθ)Y = MX+Y (aθ, bθ). (6.17) page 97 110SOR201(2002) Example Bivariate Normal distribution Using MGFs: (i) show that if (U, V ) ∼ N (0, 0; 1, 1; ρ), then ρ(U, V ) = ρ, and deduce ρ(X, Y ), 2 2 where (X, Y ) ∼ N (µx , µy ; σx , σy ; ρ); (ii) for the variables (X, Y ) in (i), ﬁnd the distribution of a linear combination aX + bY , and generalise the result obtained to the multivariate Normal case. Solution (i) We have MU,V (θ1 , θ2 ) = E(eθ1 U +θ2 V ) ∞ ∞ 1 1 = eθ1 u+θ2 v exp − [u2 − 2ρuv + v 2 ] dudv −∞ −∞ 2π 1 − ρ 2 2(1 − ρ2 ) ∞ ∞ = √1 exp{......}dudv 2π 1−ρ2 −∞ −∞ 1 2 2 = ......... = exp{ 2 (θ1 + 2ρθ1 θ2 + θ2 )}. Then ∂MU,V (θ1 , θ2 ) = exp{.....}(θ1 + ρθ2 ) ∂θ1 ∂ 2 MU,V (θ1 , θ2 ) = exp{....}(ρθ1 + θ2 )(θ1 + ρθ2 ) + exp{....}ρ. ∂θ1 ∂θ2 So ∂ 2 MU,V (θ1 , θ2 ) E(U V ) = = ρ. ∂θ1 ∂θ2 θ1 =θ2 =0 Since E(U ) = E(V ) = 0 and Var(U ) = Var(V ) = 1, we have that the correlation coeﬃcient of U, V is Cov(U, V ) E(U V ) − E(U )E(V ) ρ(U, V ) = = = ρ. Var(U ).Var(V ) 1 Now let X = µx + σx U, Y = µy + σy V. Then, as we have seen in Example 1, §5.8.2, 2 2 (U, V ) ∼ N (0, 0; 1, 1; ρ) ⇐⇒ (X, Y ) ∼ N (µx , µy ; σx , σy ; ρ). It is readily shown that a correlation coeﬃcient remains unchanged under a linear transforma- tion of variables, so ρ(X, Y ) = ρ(U, V ) = ρ. (ii) We have that MX,Y (θ1 , θ2 ) = E eθ1 (µx +σx U )+θ2 (µy +σy V ) = e(θ1 µx +θ2 µy ) MU,V (θ1 σx , θ2 σy ) 1 2 2 2 2 = exp{(θ1 µx + θ2 µy ) + 2 (θ1 σx + 2θ1 θ2 ρσx σy + θ2 σy )]. So, for a linear combination of X and Y , MaX+bY (θ) = MX,Y (aθ, bθ) = exp{(aµx + bµy )θ + 1 (a2 σx + 2abCov(X, Y ) + b2 σy )θ 2 } 2 2 2 2 σ 2 + 2abCov(X, Y ) + b2 σ 2 )θ 2 ), = MGF of N (aµx + bµy , a x y i.e. aX + bY ∼ N (aE(X) + bE(Y ), a2 Var(X) + 2abCov(X, Y ) + b2 Var(Y )). (6.18) page 98 110SOR201(2002) More generally, let (X1 , ..., Xn ) be multivariate normally distributed. Then, by induction, n n n ai Xi ∼ N ai E(Xi ), a2 Var(Xi ) + 2 i ai aj Cov(Xi , Xj ) . (6.19) i=1 i=1 i=1 i<j (If the Xs are also independent, the covariance terms vanish – but then there is a simpler derivation (see HW 8).) ♦ 6.4 Sequences of r.v.s 6.4.1 Continuity theorem First we state (without proof) the following: Theorem Let X1 , X2 , ... be a sequence of r.v.s (discrete or continuous) with c.d.f.s F X1 (x), FX2 (x), ... and MGFs MX1 (θ), MX2 (θ), ..., and suppose that, as n → ∞, MXn (θ) → MX (θ) for all θ, where MX (θ) is the MGF of some r.v. X with c.d.f. F X (x). Then FXn (x) → FX (x) as n → ∞ at each x where FX (x) is continuous. Example Using MGFs, discuss the limit of Bin(n, p) as n → ∞, p → 0 with np = λ > 0 ﬁxed. Solution Let Xn ∼ Bin(n, p), with PGF GX (s) = (ps + q)n . Then λ θ MXn (θ) = GXn (eθ ) = (peθ + q)n = {1 + (e − 1)}n where λ = np. n Let n → ∞, p → 0 in such a way that λ remains ﬁxed. Then MXn (θ) → exp{λ(eθ − 1)} as n → ∞, since n a 1+ → ea as n → ∞, a constant, (6.20) n i.e. MXn (θ) → MGF of Poisson(λ) (6.21) (use (6.12), replacing s by eθ in the Poisson PGF (3.7)). So, invoking the above continuity theorem, Bin(n, p) → Poisson(λ) (6.22) as n → ∞, p → 0 with np = λ > 0 ﬁxed. Hence in large samples, the binomial distribution can be approximated by the Poisson distribution. As a rule of thumb: the approximation is acceptable when n is large, p small, and λ = np ≤ 5. page 99 110SOR201(2002) 6.4.2 Asymptotic normality Let {Xn } be a sequence of r.v.s (discrete or continuous). If two quantities a and b can be found such that (Xn − a) c.d.f. of → c.d.f. of N (0, 1) as n → ∞, (6.23) b Xn is said to be asymptotically normally distributed with mean a and variance b 2 , and we write Xn − a a a ∼ N (0, 1) or Xn ∼ N (a, b2 ). (6.24) b Notes: (i) a and b need not be functions of n; but often a and b 2 are the mean and variance of Xn (and so are functions of n). (ii) In large samples we use N (a, b2 ) as an approximation to the distribution of X n . 6.5 Central limit theorem A restricted form of this celebrated theorem will now be stated and proved. Theorem Let X1 , X2 , ... be a sequence of independent identically distributed r.v.s, each with mean µ and variance σ 2 . Let (Sn − nµ) Sn = X 1 + X 2 + · · · + X n , Zn = √ . nσ Then a Zn ∼ N (0, 1) or P(Zn ≤ z) → P(Z ≤ z) as n → ∞, where Z ∼ N (0, 1), a and Sn ∼ N (nµ, nσ 2 ). Proof Let Yi = Xi − µ (i = 1, 2, ...). Then Y1 , Y2 , ... are i.i.d. r.v.s, and Sn − nµ = X1 + · · · + Xn − nµ = Y1 + · · · + Yn . So MSn −nµ (θ) = MY1 (θ).MY2 (θ)....MYn (θ) = {MY (θ)}n , and S√ n −nµ MZn (θ) = M Sn −nµ (θ) = E exp √ nσ θ nσ = E exp (Sn − nµ)( √θ ) nσ n = MSn −nµ √θ = MY √θ . nσ nσ Note that E(Y ) = E(X − µ) = 0 : E(Y 2 ) = E{(X − µ)2 } = σ 2 . Then θ θ2 θ3 MY (θ) = 1 + E(Y ) + E(Y 2 ) + E(Y 3 ) + · · · 1 1! 2! 3! = 1 + 2 σ 2 θ 2 + o(θ 2 ) page 100 110SOR201(2002) g(θ) (where o(θ 2 ) denotes a function g(θ) such that θ2 → 0 as θ → 0). So 2 θ 1 1 1 1 MZn (θ) = {1 + 1 σ 2 ( nσ2 ) + o( n )}n = {1 + 2 θ 2 . n + o( n )}n 2 1 h(n) (where o( n ) denotes a function h(n) such that 1/n → 0 as n → ∞). Using the standard result (6.20), we deduce that MZn (θ) → exp( 2 θ 2 ) 1 as n → ∞ – which is the MGF of N(0,1). So Sn − nµ c.d.f. of Zn = √ → c.d.f. of N (0, 1) as n → ∞, nσ i.e. a a Zn ∼ N (0, 1) or Sn ∼ N (nµ, nσ 2 ). (6.25) Corollary n a 2 1 Let X n = n Xi . Then X n ∼ N (µ, σ ). n (6.26) i=1 1 µ Proof X n = W1 + · · · + Wn where Wi = n Xi and W1 , ..., Wn are i.i.d. with mean n and σ2 variance . So n2 a µ σ2 σ2 X n ∼ N (n. , n. 2 ) = N (µ, ). n n n (Note: The theorem can be generalised to independent r.v.s with diﬀerent means & variances dependent r.v.s –but extra conditions on the distributions are required. Example 1 Using the central limit theorem, obtain an approximation to Bin(n, p) for large n. Solution Let Sn ∼ Bin(n, p). Then Sn = X 1 + X 2 + · · · + X n , where 1, if the ith trial yields a success Xi = 0, if the ith trial yields a failure. Also, X1 , X2 , ..., Xn are independent r.v.s with E(Xi ) = p, Var(Xi ) = pq. So a Sn ∼ N (np, npq), i.e., for large n, the binomial c.d.f. is approximated by the c.d.f. of N (np, npq). 1 [As a rule of thumb: the approximation is acceptable when n is large and p ≤ 2 such that np > 5.] page 101 110SOR201(2002) Example 2 As Example 1, but for the χ2 distribution. n Solution Let Vn ∼ χ2 . Then we can write n 2 2 Vn = Z 1 + · · · + Z n , 2 2 where Z1 , ..., Zn are independent r.v.s and Zi ∼ N (0, 1), Zi2 ∼ χ2 ; 1 E(Zi2 ) = 1, Var(Zi2 ) = 2. So a Vn ∼ N (n, 2n). Note: These are not necessarily the ‘best’ approximations for large n. Thus (i) 1 s+ 2 −np P(Sn ≤ s) ≈ P Z ≤ √ npq where Z ∼ N (0, 1) 1 s+ 2 −np = FS √ npq . The 1 is a ‘continuity correction’, to take account of the fact that we are approximating a 2 discrete distribution by a continuous one. (ii) approx √ 2Vn ∼ N ( 2n − 1, 1). 6.6 Characteristic function The MGF does not exist unless all the moments of the distribution are ﬁnite. So many distri- butions (e.g. t,F ) do not have MGFs. So another GF is often used. The characteristic function of a continuous r.v. X is ∞ CX (θ) = E(eiθX ) = eiθx f (x)dx, (6.27) −∞ √ where θ is real and i = −1. CX (θ) always exists, and has similar properties to M X (θ). The CF uniquely determines the p.d.f.: 1 ∞ f (x) = CX (θ)e−ixθ dθ (6.28) 2π −∞ (cf. Fourier transform). The CF is particularly useful in studying limiting distributions. How- ever, we do not consider the CF further in this module.

DOCUMENT INFO

Shared By:

Categories:

Tags:
Moment, Generating, Functions

Stats:

views: | 34 |

posted: | 3/8/2010 |

language: | English |

pages: | 9 |

Description:
Moment Generating Functions

OTHER DOCS BY etssetcf

How are you planning on using Docstoc?
BUSINESS
PERSONAL

By registering with docstoc.com you agree to our
privacy policy and
terms of service, and to receive content and offer notifications.

Docstoc is the premier online destination to start and grow small businesses. It hosts the best quality and widest selection of professional documents (over 20 million) and resources including expert videos, articles and productivity tools to make every small business better.

Search or Browse for any specific document or resource you need for your business. Or explore our curated resources for Starting a Business, Growing a Business or for Professional Development.

Feel free to Contact Us with any questions you might have.