Network data automatic cleaning method and system based on webpage label distribution characteristics

Abstract

The invention discloses a method and system for automatically cleaning network news data based on webpage label distribution characteristics, and the method comprises the steps of crawling the networknews data through an offline crawler system, namely carrying out tree node analysis on the crawled offline news data, and extracting the attribute information, such as label name, attribute, text, link, etc., in a node; predicting the information of other node blocks through a current node by adopting an idea based on n-gram2vec, obtaining the word embedding information of the label through training, constructing an intelligent model discrimination system based on the pre-trained word embedding information, and determining the leaving of tiled nodes, namely dividing an intelligent model intoa text discrimination model and a picture discrimination model according to the type of an article label, training the two types of models by adopting different feature engineering, finally predicting, and combining the results of the two types of models according to the previous node sequence.

Description

technical field [0001] The invention relates to the technical field of data cleaning, and specifically relates to a method and system for automatically cleaning network data based on distribution characteristics of web page tags. Background technique [0002] Conventional data collection steps, collection business logic writing, task distribution to download webpage content through the downloader, write rules according to the style of each article, and clean out the required content. A large number of manual configuration templates are required to extract the modules that need to be cleaned. There are a lot of thumbnails, advertisement pictures, recommended reading links, promotion links, gif pictures and other noises in the network news data. Commonly used cleaning strategies use regularization or pattern matching to lose information and differ from the public. Inconsistent updating of number templates requires a lot of human resources, and the feedback of problems is very ...

AI Technical Summary

Problems solved by technology

A large number of manual configuration templates are required to extract the modules that need to be cleaned. There are a lot of thumbnails, advertisement pictures, recommended reading links, promotion links, gif pictures and other noises in the network news data. Commonly used cleaning strategies use regularization or pattern matching to lose information and differ from the public. Inconsistencies in number templates require a lot of human resources to update frequently, and the feedback of problems is very slow

The existing technical solutions are not suitable or the cleaning accuracy is not ideal, the cost is high, and the result feedback is slow, etc.

Images