Text semantic analysis method

Abstract

A text semantic analysis method and system can realize semantic analysis of text data base on lexical level and sentence level. Aiming at the semantic analysis at the lexical level, the invention firstly adopts an improved word segmentation algorithm to solve the problem that English words are segmented only by spaces. Secondly, based on word segmentation, TF-IDF modeling is performed to obtain weight value; Then the text is vectorized by weighting and summing the weight value and the word vector trained by Word2Vec, and finally the document similarity is solved. At the same time, the invention considers the contribution degree of the vocabulary to the document content and the semantic status to calculate the similarity degree of the document, the result has higher accuracy, and provide agood foundation for subsequent text clustering. The present invention extracts subject-predicate object structure based on text segmentation, part-of-speech tagging, syntactic analysis and dependencyrelation for sentence level semantic analysis. The invention realizes the extraction of subject-predicate-object structures of various sentence types in all aspects, and realizes the noun expansion function, which is more consistent with the manual extraction result.

Description

technical field [0001] The invention relates to a text semantic analysis method in natural language processing, in particular to a text data vocabulary level and sentence level semantic analysis method and system. Background technique [0002] With the continuous development of Internet technology and information technology, and the advent of the era of big data, the data in specific technical fields is constantly enriched, the total amount of data is increasing, and the relationship between data is becoming more and more complex. How to accurately and quickly Extracting valuable information from large-scale text data has become a challenge we are facing at this stage. [0003] Text segmentation is a necessary step in natural language processing, and a good word segmentation has a crucial impact on subsequent modeling and analysis. The existing English word segmentation methods are based on spaces to divide English words. Although they have been widely used, for a specific ...

AI Technical Summary

Problems solved by technology

Although Sandford Parser has done some research on the dependency syntax analysis of English text, it does not involve the analysis of different sentence types, and the word segmentation and part-of-speech tagging functions provided cannot meet the needs of specific professional fields

Images