A New Word Discovery Method Combining Internal Aggregation Degree and External Discrete Information Entropy

Abstract

The invention discloses a new network word discovery method in combination with internal polymerization degree and external discrete information entropy. The new network word discovery method comprises the following steps: carrying out word segmentation treatment on all text sentences in network corpora, and taking all different segmented word strings as candidate word strings; and calculating the internal polymerization degree and the external discrete information entropy of the candidate word string with the frequency of occurrence in the network corpora exceeding a fixed threshold value, and further judging if a candidate target word string is a new network word according to the internal polymerization degree and the external discrete information entropy of the candidate word string. According to the new network word discovery method, two key factors for judging if one candidate word string is the new network word are proposed as follows: the internal polymerization degree and the external discrete information entropy of the candidate word string, and meanwhile, the stability, the independence and the completeness of the candidate word string are considered, so that the new network word can be effectively discovered.

Description

technical field [0001] The invention relates to a method for discovering network new words, which belongs to the field of computer natural language processing. Background technique [0002] With the rapid development of the Internet and the continuous expansion of the number of Internet users, a large number of new words appear on the Internet and quickly penetrate into people's daily life, which has become a language phenomenon. At the same time, in many Chinese information processing fields such as information retrieval, automatic word segmentation, dictionary compilation, and machine translation, the effect of new word discovery greatly affects the effect of these Chinese information processing fields, especially Chinese automatic word segmentation technology is the most important. Obviously, due to the characteristics of Chinese itself, it does not have obvious spaces between words like English. How to accurately segment the emerging new words is already a crucial step i...

AI Technical Summary

Problems solved by technology

The network new word discovery method based on statistics can make better use of statistical information to discover new words, but it lacks the consideration of the internal and external structural characteristics of words. At the same time, this method, when identifying words with low frequency of occurrence, The effect is not good, and the method based on statistics will lead to a sharp increase in time complexity when discovering longer new words, so the new word discovery method based on statistics is generally limited to identifying shorter new words

Images