Parameter Learning and EM Le Song Machine Learning II: Advanced Topics

Parameter Learning and EM Le Song Machine Learning II: Advanced Topics CSE 8803ML, Spring 2012 Parameter Estimation and Prediction Bayesian treats the unknown parameters as a random variable: 𝑃(𝜃|𝐷) = 𝑃 𝐷 𝜃 𝑃(𝜃) 𝑃(𝐷) = 𝑃 𝐷 𝜃 𝑃(𝜃) ∫ 𝑃 𝐷 𝜃 𝑃 𝜃 𝑑𝜃 Posterior mean estimation: 𝜃𝑏𝑎𝑦𝑒𝑠 = ∫ 𝜃 𝑃 𝜃 𝐷 𝑑𝜃 𝜃 𝑋𝑛𝑒𝑤 Maximum likelihood approach 𝜃𝑀𝐿 = 𝑎𝑟𝑔𝑚𝑎𝑥𝜃 𝑃 𝐷 𝜃 , 𝜃𝑀𝐴𝑃 = 𝑎𝑟𝑔𝑚𝑎𝑥𝜃 𝑃(𝜃|𝐷) 𝑋 𝑁 Bayesian prediction, take into account all possible value of 𝜃 𝑃 𝑥𝑛𝑒𝑤 𝐷 = ∫ 𝑃 𝑥𝑛𝑒𝑤 , 𝜃 𝐷 𝑑𝜃 = ∫ 𝑃 𝑥𝑛𝑒𝑤 𝜃 𝑃 𝜃 𝐷 𝑑𝜃 A frequentist prediction: use a “plug-in” estimator 𝑃 𝑥𝑛𝑒𝑤 𝐷 = 𝑃(𝑥𝑛𝑒𝑤 | 𝜃𝑀𝐿 ) 𝑜𝑟 𝑃 𝑥𝑛𝑒𝑤 𝐷 = 𝑃(𝑥𝑛𝑒𝑤 | 𝜃𝑀𝐴𝑃 ) 2 MLE for directed model 𝑙 𝜃; 𝐷 = log 𝑃 𝐷 𝜃 = 𝑖 log 𝑃 𝑎𝑖 |𝜃𝑎 + 𝑖 log 𝑃 𝑓 𝑖 |𝜃𝑓 + 𝑖 𝑙𝑜𝑔𝑃 𝑠 𝑖 𝑎𝑖 , 𝑓 𝑖 , 𝜃𝑠 + 𝑖 𝑙𝑜𝑔𝑃(ℎ 𝑖 |𝑠 𝑖 , 𝜃ℎ ) One term for each CPT; break up MLE problem into independent subproblems Because the factorization of the distribution, we can estimate each CPT separately. 𝐴𝑙𝑙𝑒𝑟𝑔𝑦 𝐴𝑙𝑙𝑒𝑟𝑔𝑦 𝐴𝑙𝑙𝑒𝑟𝑔𝑦 𝐹𝑙𝑢 𝐹𝑙𝑢 𝑆𝑖𝑛𝑢𝑠 Learn separately 𝐹𝑙𝑢 𝑆𝑖𝑛𝑢𝑠 𝑆𝑖𝑛𝑢𝑠 𝐻𝑒𝑎𝑑𝑎𝑐ℎ𝑒 𝐻𝑒𝑎𝑑𝑎𝑐ℎ𝑒 3 MLE for BNs with tabluar CPTs Assume each CPT is represented as a table (multinomial): 𝜃𝑖𝑗𝑘 = 𝑃 𝑋𝑖 = 𝑗 𝑋𝜋𝑖 = 𝑘 Note that in case of multiple parents, 𝑋𝜋𝑖 will have a composite state, and CPT will be a high dimensional table The sufficient statistics are counts of family configurations 𝑛𝑖𝑗𝑘 = # 𝑋𝑖 = 𝑗&&𝑋𝜋𝑖 = 𝑘 The log-likelihood is 𝐿 𝜃; 𝐷 = log 𝑛𝑖𝑗𝑘 𝑖𝑗𝑘 𝜃𝑖𝑗𝑘 = 𝑖𝑗𝑘 𝑛𝑖𝑗𝑘 Using a Lagrange multiplier to enforce 𝑀𝐿 𝜃𝑖𝑗𝑘 = log 𝜃𝑖𝑗𝑘 𝑗 𝜃𝑖𝑗𝑘 = 1, we get 𝑛𝑖𝑗𝑘 𝑗′ 𝑛𝑖𝑗′ 𝑘 4 Bayesian estimator for directed models Factorization 𝑃 𝑋 = 𝑥 = 𝑖𝑃 𝑥𝑖 𝑝𝑎𝑋𝑖 , 𝜃𝑖 ) Local CPT: multinomial distribution 𝑃 𝑋𝑖 = 𝑘 𝑃𝑎𝑋𝑖 = 𝑗 = 𝜃𝑘𝑗 Factorized prior over parameters 𝑃 𝜃𝑎 𝑃 𝜃𝑏 𝑃 𝜃𝑠 𝑃(𝜃ℎ ) 𝜃𝑏 𝜃𝑎 𝐴𝑙𝑙𝑒𝑟𝑔𝑦 𝜃𝑠 𝐹𝑙𝑢 𝑆𝑖𝑛𝑢𝑠 𝜃ℎ 𝐻𝑒𝑎𝑑𝑎𝑐ℎ𝑒 5 Parameter independence Provided all variables are observed, we can perform Bayesian update for each parameter independently 𝑔𝑙𝑜𝑏𝑎𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑐𝑒 𝑙𝑜𝑐𝑎𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑐𝑒 Discrete DAG models: 𝑋𝑖 |𝑃𝑎𝑋𝑖 ∼ 𝑀𝑢𝑙𝑡𝑖 𝜃 Dirichlet prior: 𝑃 𝜃 = Γ( 𝑘 𝛼𝑘 ) 𝑘 Γ 𝛼𝑘 𝛼 𝑘 𝜃𝑘 𝜃21 𝜃1 𝑋1 𝑋2 𝑋3 𝑋4 𝜃22 6 MLE for undirected models 𝑃 𝑋1 , … , 𝑋𝑘 |𝜃 = = 1 𝑍 𝜃 1 𝑍 𝜃 exp 𝑖𝑗 exp(𝜃𝑖𝑗 𝑋𝑖 𝑋𝑗 ) 𝑍 𝜃 = 𝑙 𝜃, 𝐷 = log( 𝑥 𝑖𝑗 𝜃𝑖𝑗 𝑋𝑖 𝑋𝑗 𝑁 𝑙 ( 𝑙 𝑙 𝜃 𝑥 𝑥𝑗 + 𝑖𝑗 𝑖 𝑖𝑗 𝑐𝑎𝑛 𝑏𝑒 𝑜𝑡ℎ𝑒𝑟 𝑓𝑒𝑎𝑡𝑢𝑟𝑒 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝑓 𝑥𝑖 𝑖 exp(𝜃𝑖 𝑋𝑖 ) 𝑙 𝑙 exp (𝜃 𝑥 𝑖𝑗 𝑖 𝑥𝑗 ) 𝑖𝑗 𝑙 𝑥 𝑙 )) + = 𝑁 ( log (exp (𝜃 𝑥 𝑖𝑗 𝑖 𝑗 𝑙 𝑖𝑗 log 𝑍 𝜃 ) = 𝑖 𝜃𝑖 𝑋𝑖 𝑖 exp(𝜃𝑖 𝑋𝑖 ) 𝑖𝑗 exp(𝜃𝑖𝑗 𝑋𝑖 𝑋𝑗 ) 1 𝑁 𝑙=1 𝑍 𝜃 + 𝑋6 𝑋5 𝑋8 𝑋2 𝑋1 𝑋4 𝑋7 𝑋3 𝑋9 𝑙 exp (𝜃 𝑥 𝑖 𝑖 )) 𝑖 𝑙 log(exp (𝜃 𝑥 𝑖 𝑖 )) − 𝑖 𝑙 𝜃 𝑥 𝑖 𝑖 𝑖 − log 𝑍 𝜃 ) 𝑇𝑒𝑟𝑚 𝑙𝑜𝑔𝑍 𝜃 𝑑𝑜𝑒𝑠 𝑛𝑜𝑡 𝑑𝑒𝑐𝑜𝑚𝑝𝑜𝑠𝑒! 7 Derivatives of log likelihood 𝑙 𝜃, 𝐷 = 1 𝑁 𝜕𝑙 𝜃,𝐷 𝜕𝜃𝑖𝑗 1 𝑁 𝜕𝑙 𝜃,𝐷 𝜕𝜃𝑖𝑗 = 1 𝑁 = = 1 𝑁 𝑁 𝑙 𝑙 𝑙 𝑥𝑖 𝑥𝑗 𝑁 𝑙 𝑁 𝑙 𝑁 𝑙 − ( 𝑙𝑥 𝑙 + 𝜃 𝑥 𝑖𝑗 𝑖 𝑖𝑗 𝑗 𝑙𝑥 𝑙 𝑥 𝑖𝑗 𝑖 𝑗 𝑙𝑥 𝑙 𝑥 𝑖𝑗 𝑖 𝑗 1 Z(𝜃) 𝑥 − − 𝑙 𝜃 𝑥 𝑖 𝑖 𝑖 − log 𝑍 𝜃 ) 𝜕 log 𝑍 𝜃 𝜕𝜃𝑖𝑗 𝐴 𝑐𝑜𝑛𝑣𝑒𝑥 𝑝𝑟𝑜𝑏𝑙𝑒𝑚 Can find global optimum 1 𝜕𝑍 𝜃 Z(𝜃) 𝜕𝜃𝑖𝑗 𝑖𝑗 exp(𝜃𝑖𝑗 𝑋𝑖 𝑋𝑗 ) 𝑖 exp(𝜃𝑖 𝑋𝑖 ) 𝑋𝑖 𝑋𝑗 𝑛𝑒𝑒𝑑 𝑡𝑜 𝑑𝑜 𝑖𝑛𝑓𝑒𝑟𝑒𝑛𝑐𝑒 8 Moment matching condition 𝜕𝑙 𝜃,𝐷 = 𝜕𝜃𝑖𝑗 1 𝑁 𝑙 𝑙 𝑥 𝑥 𝑁 𝑙 𝑖 𝑗 − 1 Z 𝜃 𝑥 𝑖𝑗 exp Moment 1 𝑁 𝑖 exp 𝜃𝑖 𝑋𝑖 𝑋𝑖 𝑋𝑗 𝑐𝑜𝑣𝑎𝑟𝑖𝑎𝑛𝑐𝑒 𝑚𝑎𝑡𝑟𝑖𝑥 from model 𝑒𝑚𝑝𝑖𝑟𝑖𝑐𝑎𝑙 𝑐𝑜𝑣𝑎𝑟𝑖𝑎𝑛𝑐𝑒 𝑚𝑎𝑡𝑟𝑖𝑥 P 𝑋𝑖 , 𝑋𝑗 = 𝜃𝑖𝑗 𝑋𝑖 𝑋𝑗 𝑙 𝑙 𝑁 𝛿(𝑋 , 𝑥 ) 𝛿(𝑋 , 𝑥 𝑖 𝑗 𝑙=1 𝑖 𝑗) 𝜕𝑙 𝜃,𝐷 matching: 𝜕𝜃𝑖𝑗 = 𝐸P 𝑋𝑖 ,𝑋𝑗 𝑋𝑖 𝑋𝑗 − 𝐸𝑃(𝑋|𝜃) [𝑋𝑖 𝑋𝑗 ] 9 Optimize MLE for undirected models max 𝑙 𝜃, 𝐷 is a convex optimization problem. 𝜃 Can be solve by many methods, such as gradient descent, conjugate gradient. Initialize model parameters 𝜃 Loop until convergence Compute 𝜕𝑙 𝜃,𝐷 𝜕𝜃𝑖𝑗 = 𝐸P Update 𝜃𝑖𝑗 ← 𝜃𝑖𝑗 − 𝜂 𝑋𝑖 ,𝑋𝑗 𝑋𝑖 𝑋𝑗 − 𝐸𝑃 𝑋𝜃 𝑋𝑖 𝑋𝑗 𝜕𝑙 𝜃,𝐷 𝜕𝜃𝑖𝑗 Or use the gradient equation for fixed point iteration: iterative proportional fitting 10 Exponential Family Rand variable X, 𝑃 𝑋 𝜃 = 1 𝑍 𝜃 𝑍 𝜃 = ∫ ℎ 𝑋 exp 𝜃 ⊤ 𝑇 𝑋 ℎ 𝑋 exp(𝜃 ⊤ 𝑇 𝑋 ) 𝑑𝑋 𝑃 𝑋 𝜃 = ℎ 𝑋 exp(𝜃 ⊤ 𝑇 𝑋 − 𝐴 𝜃 ) 𝑏𝑎𝑠𝑒 𝑚𝑒𝑎𝑠𝑢𝑟𝑒 𝑐𝑎𝑛𝑜𝑛𝑖𝑐𝑎𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟 𝜃 𝑠𝑢𝑓𝑓𝑖𝑐𝑖𝑒𝑛𝑡 𝑠𝑡𝑎𝑡𝑖𝑠𝑡𝑖𝑐𝑠 l𝑜𝑔 − 𝑝𝑎𝑟𝑡𝑖𝑡𝑖𝑜𝑛 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛 𝐴 𝜃 = log 𝑍 𝜃 Example: Bernoulli, multinomial, Gaussian, Poisson, Gamma, … 11 Multivariate Gaussian 𝑃 𝑋 𝜃 = ℎ 𝑋 exp(𝜃 ⊤ 𝑇 𝑋 − 𝐴 𝜃 ) Random variable 𝑋 ∈ 𝑅𝑘 1 𝑃 𝑋 𝜇, Σ = 2𝜋 = 1 𝑘 2 1 exp − Σ2 1 2 𝑋−𝜇 1 2 ⊤ −1 Σ 1 2 (𝑋 − 𝜇) 1 2 −1 𝑋𝑋 ⊤ ) + 𝜇⊤ Σ−1 𝑋 − 𝜇⊤ Σ −1 𝜇 − log |Σ| 𝑘 exp − tr(Σ 2𝜋 2 Exponential family representation 1 2 𝜃 = (Σ −1 𝜇; − 𝑣𝑒𝑐 Σ −1 ) 𝑇 𝑋 = 𝑥; 𝑣𝑒𝑐 𝑋𝑋 ⊤ 1 2 1 2 𝐴 𝜃 = 𝜇⊤ Σ −1 𝜇 + log |Σ| ℎ 𝑋 = 2𝜋 𝑘 2 12 Multinomial distribution 𝑃 𝑋 𝜃 = ℎ 𝑋 exp(𝜃 ⊤ 𝑇 𝑋 − 𝐴 𝜃 ) multinomial distribution with k values 𝑜𝑛𝑙𝑦 𝑜𝑛𝑒 𝑒𝑛𝑡𝑟𝑦 𝑖𝑠 𝑛𝑜𝑛 − 𝑧𝑒𝑟𝑜𝑠 Binary vector 𝑋 ∈ 0,1 𝐾 , 𝑋 ∼ 𝑚𝑢𝑙𝑡𝑖(𝑋|𝜃) 𝑘 𝑋𝑘 = 1, 𝑘 𝜃𝑘 = 1 𝑃 𝑋 𝜃 = 𝜃1 𝑋1 𝜃2 𝑋2 … 𝜃𝐾 𝑋𝐾 = exp log 𝜃𝑘 + 𝑋𝐾 log 𝜃𝐾 = exp 𝐾−1 𝑘=1 𝑋𝑘 𝐾−1 𝑘=1 𝑋𝑘 = exp 𝐾−1 𝑘=1 𝑋𝑘 log = exp 𝜃 = log 𝜃𝑘 ;0 𝜃𝐾 log 𝜃𝑘 + (1 − 𝜃𝑘 1− 𝐾−1 𝜃 𝑘=1 𝑘 𝐾 𝑘=1 𝑋𝑘 ln 𝜃𝑘 𝐾−1 𝑘=1 𝑋𝑘 ) log(1 + log(1 − , 𝑇 𝑋 = 𝑋, 𝐴 𝜃 = −log(1 − − 𝐾−1 𝑘=1 𝜃𝑘 ) 𝐾−1 𝑘=1 𝜃𝑘 ) 𝐾−1 𝑘=1 𝜃𝑘 ) , ℎ 𝑋 =1 13 Why exponential family? Moment generating property: we can easily compute moments of any exponential family distribution by taking the derivatives of the log normalizer Mean: 𝑑𝐴 𝑑𝜃 = = 𝑑 log 𝑍 𝑑𝜃 1 𝑑 ∫ℎ 𝑍 𝜃 𝑑𝜃 = ∫𝑇 𝑋 𝜃 = 1 𝑑 𝑍 𝑍 𝜃 𝑑𝜃 𝑋 exp 𝜃 ⊤ 𝑇 𝑋 ℎ 𝑋 exp 𝜃 ⊤ 𝑇 𝑋 𝑍 𝜃 𝜃 𝑑𝑋 𝑑𝑋 = 𝐸𝑃(𝑋|𝜃) 𝑇 𝑋 Variance: 𝑑2𝐴 𝑑𝜃 2 = 𝐸𝑃(𝑋|𝜃) 𝑇 2 𝑋 2 − 𝐸𝑃(𝑋|𝜃) 𝑇(𝑋) = 𝑉𝑎𝑟[𝑇 𝑋 ] 14 MLE for exponential family For iid data, the log-likelihood is 𝑙 𝜃, 𝐷 = log( = 𝑁 𝑙=1 ℎ ⊤ log ℎ(𝑥 ) + 𝜃 𝑙 𝑙 𝑥𝑙 exp(𝜃 ⊤ 𝑇 𝑥𝑙 − 𝐴 𝜃 )) 𝑙 𝑇(𝑥𝑙 ) − 𝑁𝐴 𝜃 Take derivatives and set of zero: 𝜕𝑙 𝜃,𝐷 𝜕𝜃 1 𝑁 = 𝑙 𝑇(𝑥𝑙 ) − 𝑁 𝑙 𝑇(𝑥𝑙 ) = 𝜕𝐴 𝜃 𝜕𝜃 =0 𝜕𝐴 𝜃 𝜕𝜃 This is moment matching condition for exponential family 15 Partially observed graphical models Speech recognition 16 Partially observed graphical models Biological Evolution 17 Partially observed graphical models Mixture Models 𝑁(𝜇1 , Σ1 ) 𝑍𝑖 𝑋𝑖 𝑁 𝑁(𝜇2 , Σ2 ) 18 Unobserved variables A variable can be unobserved (latent,hidden,missing) because: It is an imaginary quantity meant to provide some simplified and abstract view of the data generation process Eg. Mixture models, topic modeling, image context It is a real-world object and/or phenomena, but difficult or impossible to measure Eg. Causes of disease, evolutionary ancestor It is a real-world object and/or phenomena, but sometimes wasn’t measured, because of faulty sensors etc. Discrete latent variables can be used to partition/cluster data into subgroups Continuous latent variables (factors) can be used for dimensionality reduction (factor analysis, etc.) 19 Gaussian Mixture model A density model p(X) may be multi-modal: model it as a mixture of uni-modal distributions (e.g. Gaussians) 𝑁(𝜇1 , Σ1 ) Consider a mixture of 𝐾 Gaussians 𝑝(𝑋) = 𝑘 𝜋𝑘 𝑁(𝑋|𝜇𝑘 , Σ𝑘 ) 𝑀𝑖𝑥𝑡𝑢𝑟𝑒 proportion 𝑚𝑖𝑥𝑡𝑢𝑟𝑒 Component 𝑁(𝜇2 , Σ2 ) Learn 𝜋𝑘 , 𝜇𝑘 , Σ𝑘 ; 𝑍 Correspond to 𝑝 𝑋 = 𝑧 𝑝(𝑋| 𝑧; 𝜇𝑧 , Σ𝑧 )𝑃 𝑧; 𝜋 𝑋 Can be used for unsupervised clustering 𝑁 20 Why is learning hard? In fully observed iid settings, the log-likelihood decomposes into a sum of local terms 𝑙 𝜃; 𝐷 = log 𝑝 𝑥, 𝑧 𝜃 = log 𝑃 𝑧 𝜃1 + log 𝑝(𝑥|𝑧, 𝜃2 ) With latent variables, all the parameters become coupled together via marginalization 𝑙 𝜃; 𝐷 = log 𝑧𝑝 𝑥, 𝑧 𝜃 = log 𝑧 𝑝(𝑥| 𝑧, 𝜃2 )𝑃 𝑍 𝑍 𝑋 𝑋 𝑁 𝑧|𝜃1 𝑁 21 Key questions in EM algorithm EM: Expectation-maximization for finding 𝜃 𝑙 𝜃; 𝐷 = log 𝑧𝑝 𝑥, 𝑧 𝜃 = log 𝑧 𝑝(𝑥| 𝑧, 𝜃2 )𝑃 𝑧|𝜃1 Expectation step (E-step) What distribution do we take expectation with? 𝑞 𝑧 = 𝑃(𝑧| 𝑥 , 𝜃) What do we take expectation over? 𝑓 𝜃 = 𝐸𝑞 𝑧 [log 𝑝 𝑥, 𝑧 𝜃 ] Maximization step (M-step) What do we maximize? 𝑓 𝜃 What do we maximize with respect to? 𝜃 22 Example: Gaussian mixture model A mixture of K Gaussians: 𝑍 Z is latent class indicator vector 𝑃 𝑍 𝜃 = 𝜃1 𝑍1 𝜃2 𝑍2 … 𝜃𝐾 𝑍𝐾 𝑋 X is a conditional Gaussian variable with class specific mean and covariance 𝑃 𝑋 𝑍𝑘 = 1, 𝜇, Σ = 1 2𝜋 𝑑 2 1 1 exp − 2 𝑋 − 𝜇𝑘 Σ2 ⊤ Σ −1 (𝑋 k − 𝜇𝑘 ) The likelihood of a sample: 𝑃 𝑥𝑖 𝜃, 𝜇, Σ = 𝑘 𝑃(𝑧𝑘 = 1|𝜃) 𝑃 𝑥𝑖 𝑧𝑘 = 1, 𝜇, Σ = 𝑘 𝜃𝑘 𝑁 𝑥𝑖 𝜇𝑘 , Σ𝑘 The expected complete log-likelihood < 𝑙𝑐 {𝑥, 𝑧}; 𝜃, 𝜇, Σ >𝑃 𝑍|{𝑥} = 𝑖 𝑖 < log 𝑃 𝑧 𝜃 >𝑃 𝑍|{𝑥} + = 1 2 𝑖 𝑖 𝑘 < 𝑧𝑘𝑖 >𝑃 𝑍|{𝑥} log 𝜃𝑘 − 𝑖 𝑘 < 𝑧𝑘𝑖 >𝑃 𝑍|{𝑥} ( 𝑥𝑖 − 𝜇𝑘 < log 𝑝(𝑥𝑖 |𝑧 𝑖 , 𝜇, Σ) >𝑃 ⊤ Σ −1 (𝑥 𝑖 k 𝑍|{𝑥} − 𝜇𝑘 ) + log |Σ𝑘 | + 𝐶) 23 𝑁 E-step We maximize < 𝑙𝑐 {𝑥, 𝑧}; 𝜃, 𝜇, Σ >𝑃 the following procedure: 𝑍|{𝑥} iteratively using Expectation step: computing the expected value of the sufficient statistics of the hidden variables (z) given current estimate of the parameters (𝜃, 𝜇, Σ) 𝜏𝑘𝑖 =< 𝑧𝑘𝑖 >𝑃 𝑍|{𝑥} =𝑃 𝑧𝑘𝑖 = 1 𝑥 , 𝜇, Σ = 𝜃𝑘 𝑁 𝑥𝑖 𝜇𝑘 ,Σ𝑘 𝑘 𝜃𝑘 𝑁 𝑥𝑖 𝜇𝑘 ,Σ𝑘 We are essentially doing inference 24 M-step We maximize < 𝑙𝑐 {𝑥, 𝑧}; 𝜃, 𝜇, Σ >𝑃 the following procedure: 𝑍|{𝑥} iteratively using Maximization step: compute the parameters under current results of the expected complete log-likelihood < 𝑙𝑐 {𝑥, 𝑧}; 𝜃, 𝜇, Σ >𝑃 𝑖 1 𝑖 𝑘 𝜏𝑘 log 𝜃𝑘 − 2 𝑍|{𝑥} = 𝑖 𝑖 𝑘 𝜏𝑘 ( 𝑥𝑖 − 𝜇𝑘 ⊤ Σ −1 (𝑥 𝑖 k 𝜃𝑘 = 𝑎𝑟𝑔𝑚𝑎𝑥𝜃𝑘 < 𝑙𝑐 {𝑥, 𝑧}; 𝜃, 𝜇, Σ >𝑃 ⇒ 𝜃𝑘 = 𝑘 𝜃𝑘 =1 𝑁 𝑍|{𝑥} 𝑖 𝑖 𝜏𝑘 𝑥𝑖 𝑖 𝑖 𝜏𝑘 Σ𝑘 = 𝑎𝑟𝑔𝑚𝑎𝑥Σ𝑘 < 𝑙𝑐 {𝑥, 𝑧}; 𝜃, 𝜇, Σ >𝑃 ⇒ Σ𝑘 = , 𝑠. 𝑡. 𝑖 𝑖 𝜏𝑘 𝜇𝑘 = 𝑎𝑟𝑔𝑚𝑎𝑥𝜇𝑘 < 𝑙𝑐 {𝑥, 𝑧}; 𝜃, 𝜇, Σ >𝑃 ⇒ 𝜇𝑘 = 𝑍|{𝑥} − 𝜇𝑘 ) + log |Σ𝑘 | + 𝐶) 𝑖 𝑇 𝑖 𝜏𝑘 (𝑥𝑖 −𝜇𝑘 ) (𝑥𝑖 −𝜇𝑘 ) 𝑖 𝑖 𝜏𝑘 𝑍|{𝑥} Fact: 𝜕 log |𝐴−1 | = 𝐴⊤ −1 𝜕𝐴 𝜕𝑥 ⊤ 𝐴𝑥 = 𝑥𝑥 ⊤ 𝜕𝐴 25 Expectation-Maximization Iteractions 26 K-means vs EM for Gaussian mixture The EM algorithm for mixture of Gaussian is like a soft clustering algorithm K-means: “E-step”, we do hard assignment: 𝑧 𝑖 = 𝑎𝑟𝑔𝑚𝑎𝑥𝑘 (𝑥𝑖 − 𝜇𝑘 ) Σk−1 (𝑥𝑖 − 𝜇𝑘 ) “M-step”, we update the means and covariance of cluster using maximum likelihood estimate: 𝜇𝑘 = Σ𝑘 = 𝑖𝛿 𝑖𝛿 𝑖𝛿 𝑧 𝑖 ,𝑘 𝑥𝑖 𝑧 𝑖 ,𝑘 𝑧 𝑖 ,𝑘 (𝑥𝑖 −𝜇𝑘 ) (𝑥𝑖 −𝜇𝑘 )𝑇 𝑖𝛿 𝑧 𝑖 ,𝑘 x 27 Theory underlying EM What are we doing? Recall that according to MLE, we intend to learn the model parameter that would have maximize the likelihood of the data. But we are iterating these: Expectation step (E-step) 𝑓 𝜃 = 𝐸𝑞 𝑧 log 𝑝 𝑥, 𝑧 𝜃 , 𝑤ℎ𝑒𝑟𝑒 𝑞 𝑧 = 𝑃(𝑧|𝑥, 𝜃 𝑡 ) Maximization step (M-step) 𝜃 𝑡+1 = 𝑎𝑟𝑔𝑚𝑎𝑥𝜃 𝑓 𝜃 28

Parameter Learning and EM Le Song Machine Learning II: Advanced Topics

Related documents

Products

Support

Parameter Learning and EM Le Song Machine Learning II: Advanced Topics

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib