Method of identifying documents with similar properties utilizing principal component analysis

Abstract

The present invention generally provides methods and systems for characterizing texts, for example, for identifying textual documents by language, topic, author, or other attributes. In some embodiments, a method of the invention can include creating an n-gram frequency spectrum for a document under analysis, preferably selecting a subset of the n-gram frequency spectrum, transforming the n-gram frequency spectrum into principal component space, and identifying one or more attributes of the document according to its similarity to (or distinction from) reference documents in the principal component space.

Description

RELATED APPLICATIONS[0001]This application claims priority to a provisional application entitled “Selection of Interrogation Wavelengths in Optical Bio-detection Systems,” having a Ser. No. 60 / 916,480 and filed on May 7, 2007. This provisional application is herein incorporated by reference.[0002]The present application is also related to a commonly-owned patent application entitled “Selection of Interrogation Wavelengths in Optical Bio-Detection Systems” by Pierre C. Trepagnier, Matthew B. Campbell and Philip D. Henshaw filed concurrently herewith (Attorney Docket No. 101335-36). This concurrently filed application is also incorporated herein by reference in its entirety.BACKGROUND[0003]The present invention relates generally to methods and systems for determining characteristics of a text, such as the language or languages in which it is written, its subject matter, or its author.[0004]Traditionally, many document categorization methods have relied on high-level identifiers such a...

AI Technical Summary

Benefits of technology

[0009]As discussed in more detail below, a further advantage of PCA is that the training aspect of the algorithm (in which the principal component transformation is calculated, and which can be computationally intensive) can be done separately from the analysis of a text under study, which can be accomplished relatively quickly.

[0014]In another aspect, the invention provides a system for processing textual data, which includes a module for determining for each of a plurality of n-gram groupings occurrence frequency distribution corresponding to n-gram member of that grouping for at least two reference texts, wherein one text exhibits an attribute of interest and the other lacks that attribute. The system can further include an analysis module receiving the frequency distribution and applying a principal component transformation to that distribution so as to generate a plurality of principal component vectors corresponding to the reference texts for each n-gram grouping. The analysis module can determine, for each n-gram grouping, a minimum angle between the principal components of the texts corresponding to that grouping. Further, the analysis module can rank order the n-gram groupings based on the minimum angles corresponding thereto, e.g., by assigning a higher rank to a grouping that is associated with a larger minimum angle.

Problems solved by technology

Depending on the application, these methods, however, have several drawbacks.

Further, these methods can be sensitive to misspellings, variants, synonyms, and inflected forms, and they tend to be language specific.

This can create a very high-dimensional analysis space in which to classify the text, one which cannot be easily visualized and whose analysis can be computationally intensive.

Images