Discovering terms using statistical corpus analysis

Abstract

Software that extracts contextually relevant terms from a text sample (or corpus) by performing the following steps: (i) identifying a first term from a corpus, based, at least in part, on a set of initial contextual characteristic(s), where each initial contextual characteristic of the set of initial contextual characteristic(s) relates to the contextual use of at least one category related term of a set of category related term(s) in the corpus; (ii) adding the first term to the set of category related term(s), thereby creating a revised set of category related term(s) and a set of first term contextual characteristic(s), where each first term contextual characteristic of the set of first term contextual characteristic(s) relates to the contextual use of the first term in the corpus; and (iii) identifying a second term from the corpus, based, at least in part, on the set of first term contextual characteristic(s).

Description

BACKGROUND OF THE INVENTION[0001]The present invention relates generally to the field of natural language processing, and more particularly to “term extraction.”[0002]Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics concerned with the interactions between computers and human (natural) languages. As such, NLP is related to the area of human-computer interaction. Many challenges in NLP involve natural language understanding (that is, enabling computers to derive meaning from human or natural language input).[0003]Information Extraction (IE) is a known element of NLP. IE is the task of automatically extracting structured information from unstructured (and / or semi-structured) machine-readable documents. Term Extraction is a sub-task of IE. The goal of Term Extraction is to automatically extract relevant terms from a given text (or “corpus”). Term Extraction is used in many NLP tasks and applications, such as question answering, i...

AI Technical Summary

Benefits of technology

The present invention provides a method, computer program product, and system for identifying relevant terms in a corpus of text based on their initial context and adding them to a set of category related terms. The system then identifies new terms based on the set of initial contextual characteristics and the set of additions. The technical effect is improved efficiency and accuracy in identifying relevant terms in a text.

Problems solved by technology

Many challenges in NLP involve natural language understanding (that is, enabling computers to derive meaning from human or natural language input).

However, since domain ontologies represent concepts in very specific and often eclectic ways, they are often incompatible.

In the context of NLP, term extraction becomes difficult when the text being processed belongs to a different domain (for example, medical technology) than the domain from which the NLP software was built (for example, financial news).

Images