Vectorization method and device of text

Abstract

The invention discloses a vectorization method and device of text, and relates to the field of text vectorization. The vectorization method of the text includes the steps of obtaining the text to be processed and determining the application type of the text to obtain a sample of the text; extracting all single-character elements of the sample to obtain a single-character set of the sample; according to the application type of the sample, extracting double-character elements of the sample to obtain a double-character set of the sample; combining the single-character set and the double-characterset to obtain a word list; according to the word list, building and obtaining text vectors of the text. By the adoption of the vectorization method and device of the text, Chinese word segmentation is omitted, and errors and subsequent cascade effects are avoided, wherein the errors are introduced by the word segmentation into colloquial sentences such as public opinions; the vectorization methodand device of the text have good fault-tolerant capabilities for wrongly written characters in the colloquial sentences such as the public opinions.

Description

technical field [0001] The present invention relates to the field of text vectorization, in particular to a text vectorization method and device. Background technique [0002] For various machine learning algorithms, their input is a vector, and the output can be a continuous value or a discrete value. Text classification or clustering is a very important application in the field of machine learning, and text vectorization is the first step in text classification or clustering, which directly determines the quality of the final result of machine learning. [0003] Existing text vectorization techniques are as follows: [0004] TF-IDF (term frequency–inverse document frequency, term frequency-inverse document frequency) is a commonly used weighting technique for information retrieval and data mining. The dimension of the sentence vector is the number of vocabulary, and the value of each dimension is the weight calculated by the TF-IDF method of the word corresponding to the...

AI Technical Summary

Problems solved by technology

[0007] 1. The existing technology is basically based on Chinese word segmentation. Chinese word segmentation works well in the application of written sentences, but the effect of dealing with colloquial sentences such as public opinion is not good, and a considerable amount of error will be introduced

Due to the existence of cascading effects, it will have a great impact on the final machine learning results

In addition, relying on Chinese word segmentation, the fault tolerance for colloquial sentences such as public opinion is poor

[0008] 2. The TF-IDF type text vectorization method finally produces a high vector dimension, ranging from tens of thousands to hundreds of thousands, resulting in large resource consumption and a long training process

Although the final vector dimension of Word2Vec or Doc2Vec is not high, it needs a powerful corpus to support, and the training is time-consuming and resource-consuming.

[0009] 3. The vectorization schemes of TF-IDF are all semantically detrimental, that is, sentences and vectors are in a many-to-one relationship, and these sentences correspond to different types, and semantic restoration cannot be performed

Word2Vec or Doc2Vec is based on statistics, similar sentence structures will have similar vectors, and semantic restoration cannot be performed

Images