Topic crawler system based on social labels

Abstract

The invention relates to a topic crawler system based on social labels, which comprises a page acquisition module, a page processing module, a correlation calculation module, a storage module, a link extraction module and a link analysis module. The system makes full use of the social labels of web pages, utilizes the properties that the social labels are the acknowledged description on page contents and are closer to actual contents described by the web pages to judge the correlation of the web pages, and applies the correlation to a network topic crawler to guide the crawling direction of the crawler and provide high-quality webpage data contents for a topic search engine. The system well uses the network bandwidth resources to reduce unnecessary expenditure during web page acquisition, adopts different storage modes aiming at different requirements to reduce the consumption of IO, and adopts a multi-level cache mechanism to reduce blockage and improve the efficiency of the crawler. Under the support of the social labels, the system optimizes a crawler framework, and provides an optimal webpage data set for subsequent other processing flows of the topic search engine.

Description

technical field [0001] The invention belongs to computer data mining technology, and specifically relates to a theme crawler system based on social annotations. The system proposes a new crawling strategy, and guides the crawling of crawlers according to the correlation between social annotations of web pages and predetermined topics. It enables the theme crawler to crawl relevant pages accurately and effectively, and dynamically adjusts the priority of the pages to be crawled according to the relevance of the crawled pages. Background technique [0002] With the rapid development of the Internet, people rely more and more on computer networks to find the information they need, and the Internet has become an important source of information in people's lives. The emergence of search engines enables people to use keywords to quickly query relevant web page information, avoid searching aimlessly, save time for obtaining information, and thus greatly improve work efficiency, suc...

AI Technical Summary

Problems solved by technology

The search strategy based on the link structure evaluation takes the structural characteristics of the links into consideration, and it works well when searching for websites related to the topic. However, due to ignoring the correlation between the content of the page and the topic, it is prone to the problem of "topic drift" where the search deviates from the topic. In addition, In the search process, it is necessary to iteratively calculate the PageRank value or Authority and Hub weights. When the number of pages and links continues to grow, the computational complexity also increases exponentially.

Images