Chinese domain term recognition method based on mutual information and conditional random field model

Abstract

The invention discloses a Chinese domain term recognition method based on mutual information and a conditional random field model. The Chinese domain term recognition method includes the following steps: (1) gathering domain text corpus and marking all the punctuations, spaces, numbers, ASSCII (American Standard Code for Information Interchange) characters and characters except Chinese characters in the corpus; (2) setting character strings and computing the mutual information values of the character strings, (3) computing the left comentropy and the right comentropy of every character string, (4) defining character string evaluation function, setting evaluation function threshold, computing the evaluation function values of every character string, determining that every character string is a word, comparing in sequence the evaluation function value of the former character with the evaluation function value of the latter character in the character string and segmenting character meaning character strings one by one, (5) utilizing conditional random fields to train a conditional random field model and recognizing domain terms with the conditional random field model. When the Chinese domain term recognition method is used to recognize terms, the data sparsity of legitimate terms is overcome, the amount of calculation of conditional random fields is reduced, and the accuracy of the Chinese domain term recognition is improved.

Description

technical field [0001] The invention relates to a method for recognizing Chinese domain terms based on mutual information and a conditional random field model, which belongs to the field of information technology. Background technique [0002] The definition of the national standard GB / T15237.1-2000 "Terminology Working Vocabulary" is that a term refers to a term referring to a general concept in a specific professional field, and is a word or phrase used in a subject area to express a concept or relationship in the subject area . Terminology can be divided into general terms used in daily life and domain terms used in specific fields. General terms are mostly formed according to people's living and working habits, and they are not required to be strictly accurate in the expression of concepts, and their meanings are often vague; field terms are systematic and general descriptions of a professional concept, and are not allowed Ambiguity, the concept expressed by each techn...

AI Technical Summary

Problems solved by technology

However, due to the gap between professional field terminology and common vocabulary, it is difficult to achieve accurate word segmentation of professional field corpus with general word segmentation tools

Therefore, the current mutual information and conditional random field methods have a low degree of automatic recognition in the process of field term recognition, and the recognition accuracy is not high

Images