integrated automatic lexical analysis method and system for ancient Chinese texts

Abstract

The invention discloses an integrated automatic lexical analysis method for ancient Chinese texts. The method includes the following steps: pre-training the word vector of the ancient Chinese with semantic features by using the Word2Vec model; adding the information data appearing in the historical documents to the ancient name database to form a number of proper noun entries; adjusting Bi-LSTM- Each parameter of the CRF neural network model preprocesses the final training corpus into a model readable form, loads into the neural network model, continuously iteratively learns, and automaticallyevaluates the labeling result of the test corpus. According to the method, a sentence segmentation, word segmentation and part-of-speech tagging integrated tagging method is adopted, the repeated tagging process of lexical analysis of multiple sub-tasks is omitted, and multi-stage diffusion of repeated tagging errors is also avoided; According to the method, a deep learning model is adopted, richlanguage features can be learned automatically, and the work of manually customizing a feature template in traditional machine learning is omitted; The labeling model is accelerated by adopting GPU hardware, the model training time can be greatly shortened, and the efficiency is much higher than that of a traditional machine learning model.

Description

Technical field [0001] The invention relates to the technical field of text lexical analysis, in particular to an integrated automatic lexical analysis method and system of ancient Chinese texts. Background technique [0002] There are many ancient book resources. How to further extract and dig out more meaningful language knowledge from the digitized ancient book text is an important task of ancient Chinese information processing research. The basic task of ancient Chinese information processing is lexical analysis, including automatic sentence segmentation, automatic word segmentation, automatic part-of-speech tagging and automatic named entity recognition. The quality of lexical analysis will directly affect the effect of the upper task. Different from modern Chinese, the information processing of ancient Chinese is still in the exploratory stage, and the automatic processing and analysis of ancient Chinese sentences and vocabulary are less. [0003] The research results of Ch...

AI Technical Summary

Problems solved by technology

[0006] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides an integrated automatic lexical analysis method for ancient Chinese texts, which can solve the problems of large difficulty in lexical analysis, slow training speed and low accuracy in ancient Chinese, The present invention also provides an integrated automatic lexical analysis system for ancient Chinese texts

Images