A Topic Crawler Method Based on Incremental Bayesian Algorithm

Abstract

A topic crawling method based on incremental Bayesian algorithm, the steps of using incremental learning idea to train Bayesian classifier, the steps of online topic crawling based on incremental Bayesian algorithm: input initial training set and incremental training set; perform word segmentation and other preprocessing on the initial training set and incremental training set; train the initial classifier according to the initial training set and the Naive Bayesian principle, and use the initial classifier to classify the samples for the data in the incremental training set. Update the classification model according to the classification results; initialize the priority queue, visited link collection, incremental Bayesian classifier, and add the initial webpage links to the priority queue; according to whether the webpage links contain subject keywords, if they contain , the incremental Bayesian classification model is updated. Each time, the webpage with the highest priority in the priority queue is selected for webpage download, and the above steps are repeated until the conditions are met.

Description

technical field [0001] The present invention relates to a theme crawler technology based on incremental Bayesian algorithm, especially suitable for the application scenario of real-time incremental crawling of web pages Background technique [0002] The hugeness and complexity of the network lead to many challenges in obtaining web page information. Traditional web crawlers are programs or scripts that automatically grab information on the World Wide Web according to certain rules, and gradually spread from the initial web page links to the entire Internet. The main purpose is to Get a lot of Internet data within a certain period of time. [0003] In the context of the current exponential growth of network information, the use of traditional web crawlers is limited by information collection speed, value density, and professionalism. Returning web pages is usually accompanied by a lot of worthless information, which cannot meet the intelligent needs of users. [0004] The cu...

AI Technical Summary

Problems solved by technology

[0003] In the context of the current exponential growth of network information scale, the use of traditional web crawlers is limited by information collection speed, value density, and professionalism. Returning web pages is usually accompanied by a lot of worthless information, which cannot meet the intelligent needs of users.

[0004] The current topic crawler technology, when calculating the priority of links, is often obtained by weighting the correlation between anchor text and web page text. In addition, when the classification algorithm calculates the correlation between text and topics, it ignores , the distribution of the original webpage sample space will change. If the correlation degree is calculated using the same classification model, the calculation result will have a large error, and it is not suitable for the actual scene where new data comes continuously.

Images