User Tools

Site Tools


text_and_documents

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
text_and_documents [2011/12/23 00:04]
melancon [Vector space model]
text_and_documents [2012/10/11 14:36] (current)
melancon
Line 13: Line 13:
 This is where statistics and linguistics meet. Being able to decide of a relevant set of keywords to index documents, or even a set of keywords that best describe a document or set of document is difficult. There are tricks like using the tf-idf score of words to try to determine how relevant they are with respect to a given set of documents. This is where statistics and linguistics meet. Being able to decide of a relevant set of keywords to index documents, or even a set of keywords that best describe a document or set of document is difficult. There are tricks like using the tf-idf score of words to try to determine how relevant they are with respect to a given set of documents.
  
-Consider a set of keywords as given w_1, ..., w_m. Any documents may then be described with respect to how many times each of these keywords occur in the document. This information can be summarized into a n m co-occurrence matrix where each of the n lines correspond to documents and columns correspond to keywords. The c_ij entry thus computes the number of occurrences of w_j in document d_i.+Consider a set of keywords as given $w_1, \ldots, w_m$. Any documents may then be described with respect to how many times each of these keywords occur in the document. This information can be summarized into a $\times ​mco-occurrence matrix where each of the $nlines correspond to documents and columns correspond to keywords. The $c_{ij}$ ​entry thus computes the number of occurrences of $w_jin document ​$d_i$.
  
-The line vector w_i = (di1...d_im) thus describes document d_i and can be used to compute several different statistics on the dataset.+The line vector ​$w_i = (d_{i_1}\ldotsd_{i_m})thus describes document ​$d_iand can be used to compute several different statistics on the dataset.
/net/html/perso/melancon/Visual_Analytics_Course/data/pages/text_and_documents.txt · Last modified: 2012/10/11 14:36 by melancon