Glasgow2002-Tutorial

Topical Language ModelsAn Overview of Estimation TechniquesVictor LavrenkoDepartment of Computer ScienceUniversity of Massachusetts, Amherst©Victor Lavrenko, Aug. 2002Overview1. Introduction to Language Models 2. Estimation of Language Models3. Smoothing techniques4. Mixture models©Victor Lavrenko, Aug. 2002Part 1: Introduction• What is a Language Model?– A statistical model for generating text– Unigram and higher-order models– The fundamental problem of Language ModelingApplications of language models– Information Retrieval – Topic Detection and Tracking– Question Answering / Summarization– Speech Recognition / Machine Translation…©Victor Lavrenko, Aug. 2002What is a Language Model?A statistical model for generating text– Probability distribution over strings in a given languageMP ( | M ) = P ( | M )P ( | M, )P ( | M, )P ( | M, )©Victor Lavrenko, Aug. 2002Unigram and higher-order modelsP ( )= P ( ) P ( | ) P ( | ) P ( | )Unigram Language ModelsP ( ) P ( ) P ( ) P ( )N-gram Language ModelsP ( ) P ( | ) P ( | ) P ( | )Other Language Models– Grammar-based models, etc.©Victor Lavrenko, Aug. 2002The fundamental problem of LMsUsually we don’t know the model M– But have a sample of text representative of that modelP ( | M ( ) )Estimate a language model from a sampleThen compute the observation ...