lavrenko-tutorial-LM-SIGIR03

Language Modeling in IRTutorial, SIGIR 2003Victor LavrenkoDepartment of Computer Science,University of Massachusetts, Amherstlavrenko@cs.umass.edu© Victor Lavrenko, Jul. 27, 2003What I hope to Accomplish• General Language Modeling framework • Discuss different methods of estimation• Discuss a few applications of LMs© Victor Lavrenko, Jul. 27, 2003Outline of the Tutorial• Introduction to Language Models– what is a language model?– how can we use language models?– what are the major issues in language modeling?• Estimation of Language Models– Basic Models, – Translation Models, – Aspect Models, – Non-parametric Models• Bayesian framework for estimation of LMs• Case study: Relevance Models© Victor Lavrenko, Jul. 27, 2003What is a Language Model?• Probability distribution over strings of text– how likely is a given string (observation) in a given “language”– for example, consider probability for the following four strings– English: p > p > p > p1 2 3 4p = P(“a quick brown dog”)1p = P(“dog quick a brown”)2p = P(“быстрая brown dog”)3p = P(“быстрая собака”)4• … depends on what “language” we are modeling– in most of this tutorial we will have p == p1 2– for some applications we will want p to be highly probable 3© Victor Lavrenko, Jul. 27, 2003Language Modeling Notation• Convenient to make explicit what we are modeling:M … “language” we are trying to models … observation (string of tokens from some vocabulary)P(s|M) … probability ...