Statistical Translation and Web Search Ranking Jianfeng Gao Natural language processing, MSR
advertisement
Statistical Translation and Web Search Ranking Jianfeng Gao Natural language processing, MSR July 22, 2011 Who should be here? • Interested in statistical machine translation and Web search ranking • Interested in modeling technologies • Look for topics for your master/PhD thesis – A difficult topic: very hard to beat a simple baseline – An easy topic: others cannot beat it either Outline • Probability • Statistical Machine Translation (SMT) • SMT for Web search ranking 3 Probability (1/2) • Probability space: π₯οπ – π(π₯)ο[0, 1] – ο₯π₯οππ(π₯) = 1 – Cannot say π π₯ > π π¦ if π₯ ∈ π but π¦οπ • Joint probability: π(π₯, π¦) – Probability that x and y are both true • Conditional probability: π(π¦|π₯) – Probability that y is true when we already know x is true • Independence: π(π₯, π¦) = π(π₯)π(π¦) – x and y are independent Probability (2/2) • π»: assumptions on which the probabilities are based • Product rule –from the def of conditional probability – π(π₯, π¦|π») = π(π₯|π¦, π»)π(π¦|π») = π(π¦|π₯, π»)π(π₯|π») • Sum rule – a rewrite of the marginal probability def – π(π₯|π») = ο₯π¦ π(π₯, π¦|π») = ο₯π¦ π(π₯|π¦, π»)π(π¦|π») • Bayes rule – from the product rule – π(π¦|π₯, π») = π π₯ π¦, π» π π¦ π» π π₯π» An example: Statistical Language Modeling Statistical Language Modeling (SLM) • Model form – capture language structure via a probabilistic model – Pr(π|π) = πο±(π|π) = π(π|π, ο±) • Model parameters – estimation of free parameters using training data – ο± = argmaxο±π(π|π, ο±) Model Form • How to incorporate language structure into a probabilistic model • Task: next word prediction – Fill in the blank: “The dog of our neighbor ___” • Starting point: word n-gram model – Very simple, yet surprisingly effective – Words are generated from left-to-right – Assumes no other structure than words themselves 8 Word N-gram Model • Word based model – Using chain rule on its history (=preceding words) π π‘βπ πππ ππ ππ’π ππππβπππ πππππ = π π‘βπ < π > × π πππ < π >, π‘βπ × π(ππ | < π >, π‘βπ, πππ) … × π(πππππ | < π >, π‘βπ, πππ, ππ, ππ’π, ππππβπππ) × π(</π > | < π >, π‘βπ, πππ, ππ, ππ’π, ππππβπππ, πππππ ) π(π€1, π€2 … π€π) = π(π€1 | < π >) × π(π€2 | < π > π€1) × π(π€3 | < π > π€1 π€2) … × π(π€π| < π > π€1 π€2 … π€π − 1) × π(</π > | < π > π€1 π€2 … π€π) 9 Word N-gram Model • How do we get probability estimates? – Get text and count! π π€2 π€1 = Count π€1 ,π€2 Count π€1 • Problem of using the whole history – Rare events: unreliable probability estimates – Assuming a vocabulary of 20,000 words, model # parameters unigram P(w1) 20,000 bigram P(w2|w1) 400M trigram P(w3|w1w2) 8 x 1012 fourgram P(w4|w1w2w3) 1.6 x 1017 From Manning and Schütze 1999: 194 Word N-gram Model • Markov independence assumption – A word depends only on N-1 preceding words – N=3 → word trigram model • Reduce the number of parameters in the model – By forming equivalence classes • Word trigram model π π€π < π > π€1π€2 … π€π−2 π€π−1 = π π€π π€π−2 π€π−1 ) π(π€1 π€2 … π€π) = π(π€1| < π >) × π(π€2| < π > π€1) × π(π€3 | π€1 π€2) ... × π(π€π |π€π−2 π€π−1 ) × π(</π > |π€π−1 π€π) 11 Model Parameters • Bayesian estimation paradigm • Maximum likelihood estimation (MLE) • Smoothing in N-gram language models 12 Bayesian Paradigm • π(πππππ|πππ‘π) = – – – – π πππ‘π πππππ π πππππ π πππ‘π π(πππππ|πππ‘π) – Posterior probability π(πππ‘π|πππππ) – Likelihood π(πππππ) – Prior probability π(πππ‘π) – Marginal probability • Likelihood versus probability π π π’, π – for fixed π’, π defines a probability over π; – for fixed π, π defines the likelihood of π’. • Never say “the likelihood of the data” • Always say “the likelihood of the parameters given the data” 13 Maximum Likelihood Estimation (MLE) • π: model; π: data • π = argmaxπ(π|π) = argmaxπ ππ π π π π – Assume a uniform prior π(π) = ππππ π‘πππ‘ – π(π) is independent of π, and is dropped • π = argmax π(π|π)ο»argmax π(π|π) – where π(π|π) is the likelihood of parameter • Key difference between MLE and Bayesian Estimation – MLE assume that π is fixed but unknown, – Bayesian estimation assumes that π itself is a random variable with a prior distribution π(π) 14 MLE for Trigram LM • πππΏπΈ (π€3|π€1 π€2) = Count π€1 π€2 π€3 Count π€1 π€2 • πππΏπΈ (π€2|π€1) = Count π€1 π€2 Count π€1 πΆππ’ππ‘ π€ π • πππΏπΈ (π€) = • It is easy – let us get some real text and start to count ο But, why is this the MLE solution? 15 Derivation of MLE for N-gram • Homework – an interview question of MSR ο • Hints – This is a constrained optimization problem – Use log likelihood as objective function – Assume a multinomial distribution of LM – Introduce Lagrange multiplier for the constraints • ∑π₯οππ π₯ = 1, and π π₯ ≥ 0 16 Sparse Data Problem • Say our vocabulary size is |V| • There are |V|3 parameters in the trigram LM – |V| = 20,000 ο 20,0003 = 8 ο΄ 1012 parameters • Most trigrams have a zero count even in a large text corpus – Count(π€1 π€2 π€3) = 0 – πππΏπΈ (π€3|π€1 π€2) = Count π€1π€2π€3 Count π€1π€2 –π π = πππΏπΈ π€1 πππΏπΈ π€2 π€1 – oops… =0 π=3…π π(π€π |π€π−2 π€π−1 ) =0 17 Smoothing: Adding One • Add one smoothing (from Bayesian paradigm) • But works very badly – do not use this • Add delta smoothing • Still very bad – do not use this 18 Smoothing: Backoff • Backoff trigram to bigram, bigram to unigram ο Dο(0,1) is a discount constant – absolute discount ο α is calculated so probabilities sum to 1 (homeworkο) • Simple and effective – use this one! 19 Outline • Probability • SMT and translation models • SMT for web search ranking 20 SMT C: ζζ΄ δΊΊε ε¨ εε‘η ζΏε± ι ε―»ζΎ ηθΏθ E: Rescue workers search for survivors in collapsed houses πΈ ∗ = argmax π(πΈ|πΆ) πΈ πΈ ∗ = argmax π(πΆ|πΈ)π(πΈ) πΈ π(πΆ|πΈ) and π(πΈ|πΆ) 1 π(πΈ|πΆ) = exp π πΆ, πΈ ππ βπ (πΆ, πΈ) π π(πΈ|πΆ) • Translation process (generative story) – C is broken into translation units – Each unit is translated into English – Glue translated units to form E • Translation models – Word-based models – Phrase-based models – Syntax-based models Generative Modeling Art Story Science Math Engineering Code Generative Modeling for π(πΈ|πΆ) • Story making – how a target sentence is generated from a source sentence step by step • Mathematical formulation – modeling each generation steps in the generative story using a probability distribution • Parameter estimation – implementing an effective way of estimating the probability distributions from training data Word-Based Models: IBM Model 1 • We first choose the length for the target sentence πΌ, according to the distribution π(πΌ|πΆ). • Then, for each position π (π = 1 … πΌ) in the target sentence, we choose a position π in the source sentence from which to generate the π-th target word ππ according to the distribution π π πΆ . • Finally, we generate the target word by translating ππ according to the distribution π(ππ |ππ ). Mathematical Formulation • Assume that the choice of the length is independent of πΆ and πΌ – π(πΌ|πΆ) = π • Assume that all positions in the source sentence are equally likely to be chosen – π(π|πΆ) = 1 π½+1 • Assuming that each target word is generated independently from πΆ – π(πΈ|πΆ) = π(πΌ|πΆ) π° π=π π(ππ |πΆ) Parameter Estimation • Model Form – π(πΈ|πΆ) = π π½+1 πΌ π° ∑π½ π=π π=0 π ππ ππ • MLE on word-aligned training data –π π π = π(π,π) ∑ ′ π(π,π′) π • Don’t forget smoothing Phrase-Based Models Mathematical Formulation • Assume a uniform probability over segmentations – π πΈπΆ ∝∑ π,π,π ∈ π π΅ πΆ,πΈ π πΆ, π ⋅ π π πΆ, π, π • Use the maximum approximation to the sum – π πΈ πΆ ≈ max π π πΆ, π ⋅ π π πΆ, π, π π,π,π ∈ π΅ πΆ,πΈ • Assume each phrase being translated independently and use distance-based reordering model – π πΈ πΆ ∝ max π,π,π ∈ π΅ πΆ,π πΎ π=1 π(ππ |ππ )π(π π‘πππ‘π − ππππ−1 − 1) Parameter Estimation MLE: π π π = π π,π ∑ ′ π π,π′ π Don’t forget smoothing Syntax-Based Models Story • Parse an input Chinese sentence into a parse tree • Translate each Chinese constituent into English – VP ο (PP ε―»ζΎ NP, search for NP PP) • Glue these English constituents into a wellformed English sentence. Other Two Tasks? • Mathematical formation – Based on synchronous context free grammar (SCFG) • Parameter estimation – Learning SCFG from data • Homework ο • Let us go thru an example (thanks to Michel Galley) – Hierarchical phrase model – Linguistically syntax-based models ζζ΄ δΊΊε ε¨ εε‘ η ζΏε± ι ε―»ζΎ ηθΏθ rescue workers search for survivors in collapsed houses εε‘ η ζΏε± collapsed houses ζζ΄ δΊΊε ε¨ εε‘ η ζΏε± ι ε―»ζΎ ηθΏθ rescue workers search for survivors in collapsed houses ε¨ εε‘ η ζΏε± ι ε―»ζΎ ηθΏθ search for survivors in collapsed houses ζζ΄ δΊΊε ε¨ εε‘ η ζΏε± ι ε―»ζΎ ηθΏθ rescue workers search for survivors in collapsed houses ε¨ εε‘ η ζΏε± ι ε―»ζΎ ηθΏθ search for survivors in collapsed houses A synchronous rule ε¨ ι ε―»ζΎ • Phrase-based translation unit • Discontinuous translation unit • Control on reordering A synchronous grammar ε¨ ι ε―»ζΎ εε‘ η ζΏε± ηθΏθ Context-free derivation: ι ε―»ζΎ search for ε¨ εε‘ η ζΏε± ι ε―»ζΎ search for ε¨ εε‘ η ζΏε± ι ε―»ζΎ ηθΏθ search for survivors in collapsed houses ε¨ in in collapsed houses A synchronous grammar ε¨ ι ε―»ζΎ εε‘ η ζΏε± ηθΏθ Recognizes: search for survivors in collapsed houses search for collapsed houses in survivors search for survivors collapsed houses in rescue staff in ζζ΄ δΊΊε ε¨ collapse of εε‘ η Rescue workers search for house in search survivors ζΏε± ι ε―»ζΎ ηθΏθ survivors IN in collapsed houses. NNS JJ NNS NNS NN NNS NP VBP PP VBP PP VP S NP PP VP NP PP rescue staff in ζζ΄ δΊΊε ε¨ collapse of εε‘ η Rescue workers search for house in search survivors ζΏε± ι ε―»ζΎ ηθΏθ survivors IN in collapsed houses. NNS JJ NNS NNS NN NNS NP VBP PP VBP PP VP S NP PP VP NP PP rescue staff in ζζ΄ δΊΊε ε¨ collapse of εε‘ η Rescue workers search for house in search survivors ζΏε± ι ε―»ζΎ ηθΏθ survivors IN in collapsed houses. NNS JJ NNS NNS NN NNS NP VBP PP VBP PP VP S NP PP VP NP PP rescue staff in ζζ΄ δΊΊε ε¨ collapse of εε‘ η Rescue workers search for house in search survivors ζΏε± ι ε―»ζΎ ηθΏθ survivors IN in collapsed houses. NNS NP VBP PP VBP PP JJ PP NNS NP PP VP VP VP VP PP ε―»ζΎ NP PP VBP IN search for NP PP rescue staff in ζζ΄ δΊΊε ε¨ collapse of εε‘ η Rescue workers search for house in search survivors ζΏε± ι ε―»ζΎ ηθΏθ survivors IN in collapsed houses. NNS NP VBP PP VBP PP JJ PP NNS NP PP VP VP SCFG rule: VP-234 PP-32 ε―»ζΎ NP-57 search for NP-57 PP-32 rescue staff in ζζ΄ δΊΊε ε¨ collapse of εε‘ η Rescue workers search for house in search survivors ζΏε± ι ε―»ζΎ ηθΏθ survivors IN in collapsed houses. NNS JJ NNS NNS NN NNS NP VBP PP VBP PP VP S NP PP VP NP PP Outline • Probability • SMT and translation models • SMT for web search ranking 47 Web Documents and Search Queries • • • • cold home remedy cold remeedy flu treatment how to deal with stuffy nose? Map Queries to Documents • Fuzzy keyword matching – Q: cold home remedy – D: best home remedies for cold and flu • Spelling correction – Q: cold remeedies – D: best home remedies for cold and flu • Query alteration – Q: flu treatment – D: best home remedies for cold and flu • Query/document rewriting – Q: how to deal with stuffy nose – D: best home remedies for cold and flu • Where are we now? Research Agenda (Gao et al. 2010, 2011) • Model documents and queries as different languages (Gao et al., 2010) • Cast mapping queries to documents as bridging the language gap via translation • Leverage statistical machine translation (SMT) technologies and infrastructures to improve search relevance Are Queries and Docs just Different Languages? • A large scale analysis, extending (Huang et al. 2010) • Divide web collection into different fields, e.g., queries, anchor text, titles, etc. • Develop a set of language models, each on one n-gram datasets from a different field • Measure language difference between different fields (queries/docs) via perplexity Microsoft Web N-gram Model Collection (cutoff = 0) • Microsoft web n-gram services. http://research.microsoft.com/web-ngram Perplexity Results • Test set – 733,147 queries from the May 2009 query log • Summary – Query LM is most predictive of test queries – Title is better than Anchor in lower order but is worse in higher order – Body is in a different league SMT for Document Ranking • Given a query (q), doc (d) can be ranked by how likely it is that q is rewritten from d, π(πͺ|π) how to deal with stuffy nose? • An example: phrasal statistical translation for Web document ranking Phrasal Statistical Translation for Ranking d: S: T: M: q: “cold home remedies” [“cold”, “home remedies”] [“stuffy nose”, “deal with”] (1 ο 2, 2ο 1) “deal with stuffy nose” title segmentation translation permutation query • Uniform probability over S: π(πͺ|π) ≈ ∑(π,π,π) π π π, π π(π|π, π, π) • Maximum approximation: π(πͺ|π) ≈ max (π,π,π)∈π΅(π,πͺ) π π π, π π(π|π, π, π) • Max probability assignment via dynamic programming: π(πͺ|π) ≈ max ∗ π π π, π and π π π, π = π=1…πΎ π(πͺπ |π°π ) (π,π,π)∈π΅(π,πͺ,π΄ ) • Model training on query-doc pairs Mine Query-Document Pairs from User Logs how to deal with stuffy nose? stuffy nose treatment cold home remedies NO CLICK NO CLICK http://www.agelessherbs.com/BestHome RemediesColdFlu.html Mine Query-Document Pairs from User Logs how to deal with stuffy nose? stuffy nose treatment cold home remedies Mine Query-Document Pairs from User Logs how to deal with stuffy nose? stuffy nose treatment cold home remedies QUERY (Q) how to deal with stuffy nose stuffy nose treatment cold home remedies …… go israel skate at wholesale at pr breastfeeding nursing blister baby thank you teacher song immigration canada lacolle Title (T) best home remedies for cold and flu best home remedies for cold and flu best home remedies for cold and flu …… forums goisrael community wholesale skates southeastern skate supply clogged milk ducts babycenter lyrics for teaching educational children s music cbsa office detailed information • 178 million pairs from 0.5 year log Evaluation Methodology • Measurement: NDCG, t-test • Test set: – 12,071 English queries sampled from 1-y log – 5-level relevance label for each query-doc pair – On a tail document sets (click field is empty) • Training data for translation models: – 82,834,648 query-title pairs Baseline: Word-Based Models (Berger&Lafferty, 99) • Basic model: • Mixture model: • Learning translation probabilities from clickthrough data – IBM Model 1 with EM Results Sample IBM-1 word translation probability after EM training on the Query-title pairs Bilingual Phrases • Notice that with context information, we have less ambiguous translations Results • Ranking results – All features – Only phrase translation features Why Do Bi-Phrases Help? • Length distribution • Good/bad examples Generative Topic Models Q: stuffy nose treatment Q: stuffy nose treatment D: cold home remedies Topic D: cold home remedies • Probabilistic latent Semantic Analysis (PLSA) – π πͺπ = π∈πͺ ∑π§ π π ππ§ π(π§|π, π½) – d is assigned a single most likely topic vector – q is generated from the topic vectors • Latent Dirichlet Allocation (LDA) generalizes PLSA – a posterior distribution over topic vectors is used – PLSA = LDA with MAP inference Bilingual Topic Model • For each topic z: ππͺπ§ , πππ§ ~ Dir(π·) • For each q-d pair: π½ ~ Dir(πΆ) • Each • Each πͺ q is generated by π§ ~ π½ and π ~ ππ§ w is generated by π§ ~ π½ and π€ ~ ππ π§ Log-likelihood of LDA Given Data • π and π½: distribution of distribution • LDA requires integral over π and π½ • This is the MAP approximation to LDA MAP Estimation via EM • Estimate (π½, ππͺ , ππ ) by maximizing joint log likelihood of q-d pairs and the parameters • E-Step: compute posterior probabilities – π π§ π, π½πͺ,π ,π π§ π€, π½πͺ,π • M-Step: update parameters using the posterior probabilities –π π πͺ ππ§ ,π π€ ππ π§ , π(π§|π½πͺ,π ) Posterior Regularization (PR) • q and its clicked d are relevant, thus they – Share same prior distribution over topics (MAP) – Weight each topic similarly (PR) • Model training via modified EM – E-step: for each q-d pair, project the posterior topic distributions onto a constrained set, where the expected fraction of each topic is equal in q and d – M-step: update parameters using the projected posterior probabilities Topic Models for Doc Ranking Evaluation Methodology • Measurement: NDCG, t-test • Test set: – 16,510 English queries sampled from 1-y log – Each query is associated with 15 docs – 5-level relevance label for each query-doc pair • Training data for translation models: – 82,834,648 query-title pairs Topic Model Results Summary • Probability – Basics – A case study of a probabilistic model: N-gram language model • Statistical Machine Translation (SMT) – Generative modeling (story ο math ο code) – Word/phrase/syntax based models • SMT for web search ranking – View query and doc as different language – Doc ranking via π(πͺ|π) – Word/phrase/topic based models • Slides/doc will be available at http://research.microsoft.com/~jfgao/ Main Reference • • • • • • • Berger, A., and Lafferty, J. 1999. Information retrieval as statistical translation. In SIGIR, pp. 222-229. Gao, J., He, X., and Nie, J-Y. 2010. Clickthrough-based translation models for web search: from word models to phrase models. In CIKM, pp. 1139-1148. Gao, J., Toutanova, K., and Yih, W-T. 2011. Clickthrough-based latent semantic models for web search. In SIGIR. Huang, J., Gao, J., Miao, J., Li, X., Wang, K., and Behr, F. 2010. Exploring web scale language models for search query processing. In Proc. WWW 2010, pp. 451-460. MacKay, David J. C. 2003. Information Theory, Inference and Learning Algorithms. Cambridge: Cambridge University Press. Manning, C., and H. Chutze. 1999. Foundations of statistical natural language processing. MIT Press. Cambridge. Philipp Koehn. Statistical Machine Translation. Cambridge University Press. 2009.
