Sensitive word detection method based on decision tree and variant recognition

Abstract

The invention belongs to the technical field of natural language processing, and provides a sensitive word detection method based on a decision tree and variant recognition. The method comprises the following steps: S1, constructing a sensitive word dictionary and updating according to a preset period, wherein Chinese character patterns and corresponding pinyin are added into the sensitive word dictionary according to an initial sequence, and constructing a decision tree according to the sensitive word dictionary; S2, inputting a text into a sensitive word detection model, wherein the sensitive word detection model detects sensitive words in the text through a matching algorithm and a matching standard on the basis of the decision tree, and positions the sensitive words. Through direct comparison and similarity comparison of character patterns and pinyin, the problems that sensitive words are not included and detection is avoided by modifying character patterns, pinyin, or English of the sensitive words, can be effectively solved.

Description

technical field [0001] The invention relates to the technical field of natural language processing, in particular to a sensitive word detection method based on decision tree and variant recognition. Background technique [0002] With the development and prosperity of the Internet age, massive network resources have made it more convenient and quick for people to obtain information, communicate in life, consume and manage money. However, while people are enjoying the convenience brought by the Internet, many people take advantage of the rapid and extensive information dissemination characteristics of the Internet to publish all kinds of illegal information such as pornography, violence, reactionary, superstition, etc. It has caused great harm and brought many adverse effects to society. [0003] In order to deal with this problem, many Internet companies and public information management departments are reviewing and filtering information published on the Internet all the ti...

AI Technical Summary

Problems solved by technology

[0004] In the existing research, there are still many problems to be solved, such as only direct comparison of Chinese characters or pinyin of sensitive words, less consideration of the variants of Chinese characters and pinyin, resulting in low filtering accuracy of sensitive words, etc. ; This application expects to solve the more prominent problems in the current research and provide a method with a higher recall rate and precision rate of sensitive words

Images