advertisement

16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde Lecture 4 Last time: Left off with characteristic function. 4. Prove φx (t ) = Πφx (t ) where X = X 1 + X 2 + ... + X n (Xi independent) i Let S = X 1 + X 2 + ...X n where the Xi are independent. jtS ⎡ jt ( X + X φs (t ) = E ⎡e ⎣ ⎤⎦ = E ⎣ e 1 = E ⎡⎣ e ⎤⎦ E ⎣⎡ e = ∏ φX i (t ) jtX 1 jtX 2 2 +... + X n ⎤⎦ ...E ⎡⎣ e ) ⎤ jtX n ⎦ ⎤ ⎦ This is the main reason why use of the characteristic function is convenient. This would also follow from the more devious reasoning of the density function for the sum of n independent random variables being the nth order convolution of the individual density functions – and the knowledge that convolution in the direct variable domain becomes multiplication in the transform domain. 5. MacLaurin series expansion of φ (t ) ∞ Because f(x) is non-negative and ∫ ∞ f ( x )dx = 1 (or, even better, −∞ ∫ f ( x ) dx = 1 ), it −∞ ∞ follows that ∫ f ( x ) dx = 1 converges so that f(x) is Fourier transformable. Thus −∞ the characteristic function φ (t ) exists for all distributions and the inverse relation φ (t ) → f (x) holds for all distributions. This implies that φ (t ) is analytic for all real values of t. Then it can be expanded in a power series, which converges for all finite values of t. φ (t) = φ (0) + φ (1) ( 0 ) t + φ (t ) = ∞ ∫ 1 (2) 1 φ ( 0 ) t 2 + ... + φ ( n ) ( 0 ) t n + ... n! 2! f (x )e jtx dx , φ (0) = 1 −∞ Page 1 of 6 16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde ∞ d nφ (t ) = ∫ f ( x )( jx) n e jtx dx dt n −∞ φ (n) ∞ ( 0 ) = j n ∫ x n f (x )dx = j n X n −∞ 1 2 22 1 j ) X t + ... + ( j n ) X n t n + ... ( 2! n! The coefficients of the expansion are given by the moments of the distribution. Thus the characteristic function can be determined from the moments. Similarly, the moments can be determined from the characteristic function directly by 1 d nφ (t) Xn = n j dt n t =0 φ (t) = 1 + jXt + or by expanding φ (t ) into its power series in some other way and identifying the coefficients of the various powers of t. The Generating Function The generating function has its most useful application to random variables which take integer values only. Examples of such would be the number of telephone calls into a switchboard in a certain time interval, the number of cars entering a toll station in a certain time interval, the number of times a 7 is thrown in n tosses of 2 dice, etc. For integer-valued random variables, the Generating Function yields the same advantages as the Characteristic Function and is of simpler form. Consider a random variable which takes the integer values k: P( X = k ) = pk (k=0,1,2,…) For a discrete distribution you can sum in lieu of integration. The Characteristic Function for this random variable is ∞ φ (t ) = E ⎡⎣ e jtX ⎤⎦ = ∑ e jtk pk k =0 ∞ = ∑ pk ( e jt ) k k = 0 If we define a new variable s = e jt , we have ∞ G ( s ) = ∑ pk s k k =0 Page 2 of 6 16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde which is called the Generating Function. It has all the interesting properties of the characteristic function. Note that t → 0 corresponds to s → 1 . Let’s establish the connection between moments of a distribution and the generating function: dG ∞ = ∑ kpk s k −1 ds k =0 d 2G ∞ = ∑ k ( k − 1) pk s k −2 ds 2 k =0 ∞ ∞ k =0 k =0 = ∑ k 2 pk s k −2 − ∑ kpk s k −2 Just calculate dG ds d 2G ds 2 dG ds and s=1 d 2G ds 2 and reorganize them in terms of X and X 2 : s=1 ∞ s=1 = ∑ kpk = X , ← 1st moment expression k =0 ∞ ∞ k =0 k =0 = ∑ k 2 pk − ∑ kpk s=1 X2 = 2 d G ds 2 + s=1 dG ds ← 2 nd moment expression s=1 Each moment is a linear combination of its order derivative and lower order derivatives. The generating function for the sum of independent integer-valued variables is the product of their generating functions. This is harder to prove than the same property of the characteristic function, but it does, in fact, hold true. Multiple Random Variables Characterizing a joint set of random variables, define a probability distribution function F ( x) = P( X 1 ≤ x1 , X 2 ≤ x2 ,..., X n ≤ xn ) This is called the joint probability distribution function. Properties: If any of the arguments xi goes to −∞ , then F ( x ) → 0 . lim F ( x) = 0 any xi →−∞ Page 3 of 6 16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde If all of the xi go to ∞ , then F ( x ) → 1 . lim F ( x) = 1 all xi →∞ F ( x ) is monotonically non-decreasing in each xi. Define joint density function by differentiation: ∂n f ( x) = ∂x1∂x2 ...∂xn f ( x ) ≥ 0, ∀x Fx1 ... xn ( x1...xn ) = xn x1 ∫ du ... ∫ du 1 −∞ n f x1 ... xn (u1...un ) −∞ Setting each xi → ∞ , ∞ ∞ ∫ du ... ∫ du F 1 −∞ n u1 ,...,un (u1 ,..., un ) = 1 −∞ Fx1 ,..., xk ( x1 ,..., xk ) = P( X 1 ≤ x1 ,..., X n ≤ xn ) = P( X 1 ≤ x1 ,..., X k ≤ xk , X k +1 ≤ ∞,..., X n ≤ ∞ ) = Fx1 ,..., xn ( x1 ,..., xk , ∞,..., ∞) Page 4 of 6 16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde For the density function: ∂ k f x1 ,..., xk ( x1 ,..., xk ) = Fx ,..., x ( x1 ,..., xk ) ∂x1∂x2 ...∂xk 1 k ∂ k Fx ,..., x ( x1 ,..., xk , ∞,..., ∞) = ∂x1∂x2 ...∂xk 1 n ∂k ∂x1∂x2 ...∂xk = ∞ ∫ du = ∞ ∫ = −∞ ∫ −∞ xk du1... ∫ duk −∞ ∞ ∫ −∞ ∞ duk +1... ∫ dun f x1 ,..., xn (u1 ,..., un ) −∞ ∞ k +1 −∞ x1 ... ∫ dun f x1 ,...,xn ( x1 ,..., xk , uk +1 ,..., un ) −∞ ∞ duk +1... ∫ dun f x1 ,..., xn ( x1 ,..., xn ) −∞ Marginal density If you integrate above over all variables but one, it is referred to as the marginal density. ∞ f xi ( xi ) = ∞ ∫ dx ... ∫ dx 1 −∞ n f x1 ,..., xn ( x1 ,..., xn ) −∞ 14 4244 3 n-1 terms: all except xi Mutually independent sets of random variables Definition of independence: P [ X 1 ∈ s1 , X 2 ∈ s2 ,...] = P [ X 1 ∈ s1 ] P [ X 2 ∈ s2 ]... for any sets s1, s2, … The product rule holds for joint probability distribution and density functions for independent random variables. Fx1 , x2 , x3 ,... ( x1 , x2 , x3 ,...) = Fx1 ( x1 ) Fx2 ( x2 ) Fx2 ( x2 )... f x1 , x2 , x3 ,... ( x1 , x2 , x3 ,...) = f x1 ( x1 ) f x2 ( x2 ) f x2 ( x2 )... Expectations E [ g ( x )] = ∞ ∫ −∞ ∞ dx1... ∫ dxn g ( x ) f ( x ) −∞ Page 5 of 6 16.322 Stochastic Estimation and Control, Fall 2004 Prof. Vander Velde For the sum of multiple random variables: E [ X 1 + X 2 + ... + X n ] = ∞ ∞ ∫ dx ... ∫ dx ( x 1 n −∞ ∞ = ∫ −∞ 1 ∞ ∞ −∞ −∞ dx1... ∫ dxn x1 f x1 ,..., xn ( x1 ,..., xn ) + ... + ∞ = + x2 + ... + xn ) f x1 ,..., xn ( x1 ,..., xn ) −∞ ∫xf ∫ ∞ 1 x1 ( x1 )dx1 + −∞ ∫x ∞ dx1... ∫ dxn xn f x1 ,..., xn ( x1 ,..., xn ) −∞ ∞ 2 f x2 ( x2 )dx2 + ... + −∞ = E [ X 1 ] + E [ X 2 ] + ... + E [ X n ] ∫x n f xn ( xn )dxn −∞ This relation is true whether or not the xi are independent. For the product of multiple independent random variables: E [ X 1 X 2 ... X n ] = ∞ ∫ −∞ ∞ = ∫ −∞ ∞ dx1... ∫ dxn ( x1 x2 ... xn ) f x1 ,..., xn ( x1 ,..., xn ) −∞ ∞ dx1... ∫ dxn ( x1 x2 ... xn ) f x1 ( x1 ) f x2 ( x2 )... f xn ( xn ) −∞ ∞ = ∫xf 1 x1 −∞ ∞ ∞ −∞ −∞ ( x1 )dx1 ∫ x2 f x2 ( x2 )dx2 ... ∫ xn f xn ( xn )dxn = E [ X 1 ] E [ X 2 ]... E [ X n ] Page 6 of 6