latent-semantic-indexing-fast-track-tutorial

Latent Semantic Indexing (LSI) A Fast Track Tutorial Dr. Edel Garcia admin@miislita.com First Published on September 21, 2006; Last Update: October 21, 2006 Copyright Ó Dr. E. Garcia, 2006. All Rights Reserved. Abstract This fast track tutorial provides instructions for scoring queries and documents and for ranking results using a Singular Value Decomposition (SVD) calculator and the Term Count Model. The tutorial should be used as a quick reference for our SVD and LSI Tutorial series described at the following link: http://www.miislita.com/information-retrieval-tutorial/svd-lsi-tutorial-1-understanding.html Keywords latent semantic indexing, LSI, singular value decomposition, SVD, eigenvectors, documents, queries, cosine similarity, term count model Background: The following LSI example is taken from page 71 of Grossman and Frieder’s Information Retrieval, Algorithms and Heuristics (1) http://www.miislita.com/book-reviews/book-reviews.html A “collection” consists of the following “documents” d1: Shipment of gold damaged in a fire. d2: Delivery of silver arrived in a silver truck. d3: Shipment of gold arrived in a truck. The authors used the Term Count Model to score term weights and query weights, so local weights are defined as word occurences. The following document indexing rules were also used: • stop words were not ignored • text was tokenized and lowercased • no stemming was used • terms were sorted ...