PageRank and information entropy-based text word segmentation method for judgment document

Abstract

The invention discloses a PageRank and information entropy-based text word segmentation method for a judgment document, and belongs to a Chinese word segmentation technology in the field of natural language processing. An improved PageRank algorithm, an information entropy, mutual information and a keyword dictionary are mainly adopted to carry out word segmentation on a Chinese text. For the judgment document in the legal field, the word segmentation method is established on the basis of the PageRank algorithm; candidate words are segmented according to Rank vectors; the candidate words are corrected through the information entropy; terms are combined according to a keyword dictionary of the judgment document; and finally a word segmentation result is output. The method can carry out wordsegmentation on the judgment document more accurately. Compared with an existing method, the method has the remarkable advantages that statistics or training does not need to be carried out through alarge number of text corpora to establish a large-scale dictionary, and only the input text is subjected to statistics; the input text is used as an existing corpus to carry out statistical mining; and finally the word segmentation can be completed in combination with a keyword term dictionary of the judgment document.

Description

technical field [0001] The invention belongs to the Chinese word segmentation technology in the technical field of natural language processing, and is a technology for performing Chinese word segmentation for legal documents. Background technique [0002] Word segmentation refers to dividing the existing text into separate, independent and meaningful units. Chinese word segmentation refers to dividing a continuous sequence of Chinese characters into separate words to make them into semantically readable word sequences. Compared with English, Chinese does not have a clear separator between words as the basis for segmentation. Therefore, in terms of word segmentation, Chinese word segmentation is more difficult than other languages. The word segmentation results generated by the word segmentation algorithm will also directly affect the application effect of the upper layer, such as part-of-speech tagging and keyword extraction. Therefore, how to make the computer understand ...

AI Technical Summary

Problems solved by technology

However, there are a large number of place names, person names, organization names and other special names in legal documents, so how to identify these special words is also a difficult problem

Images