33results about How to "Improve word segmentation efficiency" patented technology

A system for analyzing vertical public opinions based on an industry comprises an acquisition and pre-treatment module for acquiring and pre-treating Internet information relevant to the consumer electronics industry and obtaining the formative information of the consumer electronics industry based on documents; a word segmentation module for matching words by means of a character string matching algorithm, and obtaining work segmentation results by amending the matching results in a word segmentation method based on understanding and statistics; an analysis module for performing document clustering and classification on the word segmentation results according to the frequency and similarity of keywords in the word segmentation results of the documents, and for obtaining analyzed and processed information after hotspot / sensitive topic analysis, orientation analysis and trend analysis to the clustered and classified results; and a display module for pushing the analyzed and processed information to users. The invention further provides a method for analyzing vertical public opinions based on an industry.

The invention relates to a Chinese text parallel data mining method based on hierarchy, comprising the steps of: step 1: a establishing vector space model of Chinese texts: performing work segmentation regarding to the entire Chinese text set to obtain a word segmentation form and a feature term set containing all removed duplicated terms in the text set of each text, then using the feature term set to count the term frequency-inverse document frequency (TFIDF) of each text, and establishing the text vector space model according to the TFIDF; step 2: performing dimension reduction regarding to a feature item vector of the text vector space model; and step 3: clustering texts using DCURE algorithm based on hierarchy. The method is efficient in word segmentation of Chinese texts with high accuracy, requires no input of parameters like radius of neighborhood for the clustering process, can mine irregular cluster and is insensitive to noise, employs distributed calculating, has high efficiency in mining mass texts and improves calculating speed of feature weight.

The invention discloses a Chinese word segmentation method based on bidirectional LSTM, CNN and CRF, which is an improvement and optimization of traditional Chinese word segmentation based on a deep learning algorithm. The specific steps of the method are as follows: preprocessing the initial corpus, extracting the character feature information of the corpus and the corresponding pinyin feature information of the character; using the convolutional neural network to obtain the pinyin feature information vector of the character; using the word2vec model to obtain the character feature information vector of the text; Splicing the pinyin feature vector and the character feature vector to get the context information vector, put it into the bidirectional LSTM neural network; use the linear chain conditional random field to decode the output of the bidirectional LSTM to get the word segmentation tag sequence; decode the word segmentation tag sequence to get Word segmentation results. The present invention uses a deep neural network to extract text character features and pinyin features and combines conditional random fields for decoding, which can effectively extract Chinese text features and achieve good results in Chinese word segmentation tasks.

The present invention proposes a word segmentation method, computer-readable storage medium and system supporting a large number of thesaurus. The method includes the following steps: constructing a domain dictionary; constructing an offline word segmentation model based on the domain dictionary; for the original text to be segmented, by offline word segmentation The model performs word segmentation to obtain the first word segmentation result; the original text to be segmented is extracted to be searched, and then based on the word to be searched, the first-level index search and the second-level index search are performed in the domain dictionary, and finally the second-level index results are screened. Extract the candidate words; reorganize the candidate words and the first word segmentation results, construct the directed graph of the original text based on the reorganization results, and calculate the optimal word segmentation results based on the shortest path method. The present invention combines word segmentation results in a single field with big word search results, constructs a directed graph based on the combined results, and converts the problem of solving the optimal word segmentation scheme into the problem of the optimal path to quickly solve, which is very suitable for separating big words.

The invention relates to a Chinese text parallel data mining method based on hierarchy, comprising the steps of: step 1: a establishing vector space model of Chinese texts: performing work segmentation regarding to the entire Chinese text set to obtain a word segmentation form and a feature term set containing all removed duplicated terms in the text set of each text, then using the feature term set to count the term frequency-inverse document frequency (TFIDF) of each text, and establishing the text vector space model according to the TFIDF; step 2: performing dimension reduction regarding to a feature item vector of the text vector space model; and step 3: clustering texts using DCURE algorithm based on hierarchy. The method is efficient in word segmentation of Chinese texts with high accuracy, requires no input of parameters like radius of neighborhood for the clustering process, can mine irregular cluster and is insensitive to noise, employs distributed calculating, has high efficiency in mining mass texts and improves calculating speed of feature weight.

A query method for variable-length words and sentences in evaluation documents based on an inverted index, which involves an indexing method in the field of data science and a word segmentation method in the field of NLP, and solves the query problem of variable-length words and sentences in evaluation documents. The steps of the present invention are: 1. Carry out data preprocessing on the document to be queried, and use the jieba word segmentation method to carry out word segmentation processing to obtain word dictionary and word frequency information; 2. Establish an adaptive inverted table based on the inverted index principle of the complete reconstruction strategy; 3. , Combining the information of the variable-length words and sentences to be searched, through the self-adaptive inverted list indexing the position information of each word in the words and sentences, identifying the position information of the variable-length words and sentences and indexing the paragraphs where they are located, to complete the query function of variable-length words and sentences in evaluation documents. The basic idea of ​​the present invention is to segment the text data into words, establish an inverted index, and then realize fast searching for words and sentences of indefinite length, so as to realize the query function of evaluation documents. It has a wide range of application scenarios, so it has high socio-economic value.

Embodiments of the invention disclose a Chinese word segmentation method and apparatus. The method comprises the steps of dividing a text set into a plurality of short sentences and numbering the short sentences; for each Chinese character in the text set, obtaining a first short sentence number list corresponding to a current Chinese character, obtaining a second short sentence number list corresponding to an adjacent Chinese character adjacent to the right side of the current Chinese character, and calculating a degree of co-occurrence according to the first short sentence number list and the second short sentence number list; obtaining an adjacent character set corresponding to the current Chinese character, and calculating a relevant degree of adjacency according to the adjacent character set; determining whether a word consisting of the current Chinese character and the adjacent Chinese character is added into a candidate word set or not according to the degree of co-occurrence and the relevant degree of adjacency; and performing word segmentation on the text set according to the candidate word set. The method is small in calculation amount and high in accuracy when calculating the candidate word set, can effectively improve the accuracy of a word segmentation result and improve the efficiency of word segmentation, does not depend on a corpus dictionary, and can realize unsupervised candidate vocabulary extraction.

The invention relates to a personalized concurrent word segmentation processing system and a processing method of the processing system. The personalized concurrent word segmentation processing system and the processing method of the processing system comprises a word segmentation requesting module, a word segmentation module based on a personalized word segmentation dictionary, a word segmentation module based on a general word segmentation dictionary, a control module and a high speed word segmentation processing module. Word segmentation requests of a user are simultaneously sent to the word segmentation module based on the personalized word segmentation dictionary and the word segmentation module based on the general word segmentation dictionary. When the word segmentation module based on the personalized word segmentation dictionary is destined, word segmentation processing result is sent back to the word segmentation requesting module through the control module, and meanwhile word segmentation requests of the word segmentation requesting module to the word segmentation module based on the general word segmentation dictionary is interrupted; otherwise, dynamic update of the personalized word segmentation dictionary is proceeded according to an earliest and least using principle and the word segmentation processing result of the word segmentation module based on the personalized word segmentation dictionary by the control module. The personalized concurrent word segmentation processing system and the processing method of the processing system is capable of satisfying accuracy rate of the word segmentation, meanwhile improving word segmentation efficiency of the system greatly and satisfying efficient referring requirements of a mobile user.

The invention provides a multilingual-based word segmentation method and device. The method includes: receiving the text to be segmented sent by the user, carrying a sentence separator in the text to be segmented; identifying the language category of each sentence in the text to be segmented according to the sentence separator; Find the corresponding word segmentation method in the corresponding relationship between the category and the word segmentation method; use the word segmentation method corresponding to the language category to segment the sentence of the corresponding language category; output the word segmentation result of the text to be segmented to the user. The method realizes unified word segmentation for applications or texts involving multiple languages, and improves word segmentation efficiency.