Chinese commercial text preprocessing method based on machine learning

Abstract

The invention discloses a Chinese commercial text preprocessing method based on machine learning. The input Chinese commercial text is processed through the following steps that (1) performing sentence segmentation and word segmentation on the Chinese commercial text; (2) carrying out part-of-speech tagging on the segmented words by utilizing a decision tree; (3) performing word sense disambiguation by utilizing conditional probability based on a Bayesian classifier; (4) representing a word vector by using a hybrid model combining One-Hot coding and a Skip-Gram model; (5) adjusting word weights by utilizing TF-IDF, and determining corresponding word meanings of the polysemy under the current context; and (6) outputting the Chinese commercial text preprocessed based on machine learning. TheChinese commercial text preprocessing method can effectively solve the problems that a Chinese commercial question-answering system does not answer questions and is limited in response scene due to insufficient text preprocessing, can improve the text understanding accuracy of a computer, and enables extension work such as machine translation and intelligent question-answering to have implementability.

Description

technical field [0001] The invention belongs to the field of natural language processing, and in particular relates to a machine learning-based Chinese business text preprocessing method. Background technique [0002] The combination of business development and artificial intelligence has received more and more attention, and speech recognition technology is the basis of human-computer interaction. The current natural language processing usually adopts the following two methods. One is the rule-based natural language processing method. After many years of experiments using this method at home and abroad, the effect is still very unsatisfactory. Starting from syntax and other aspects, it is analyzed and processed according to the rules of the language. Because there are too many rules, there is no fixed method. At the same time, new rules are constantly added through people's production and life, so it is very difficult to implement. Another method is the natural language pr...

AI Technical Summary

Problems solved by technology

There are huge differences between Chinese data and other languages. For example, Chinese is written continuously, there is no change in voice and tense, and there are polyphonic characters, etc., which makes Chinese more flexible. Many foreign mature technologies cannot be used to process Chinese text

Images