Anchor Text-Based Focused Web Crawler Search Method and System

Abstract

The invention discloses a search method for focused web crawler based on an anchor text and a system thereof. The method mainly comprises the following steps of obtaining a URL (uniform resource locator) from a URL priority query and downloading from the Internet to obtain a Web page according to the URL; analyzing the downloaded Web page and extracting the URL and the anchor text thereof; screening the extracted URL and anchor text thereof; and selecting an algorithm combined by TF-IDF (term frequency-inverse document frequency) and LSI (latent semantic indexing) to calculate a topic correlativity of the URL and putting the URL matched with the condition in the priority query. The system comprises a URL priority query, a web crawler downloader, a Web page library, a URL parser, a URL filter and a topic correlativity identifier. With the adoption of the search method of focused web crawler based on the anchor text and the system thereof, the topic correlativity of the crawling result of the focused web crawler and the crawling efficiency are improved.

Description

technical field [0001] The invention relates to a crawler search method and a system thereof, in particular to a focused web crawler search method and a system thereof. Background technique [0002] At present, the Internet has increasingly become the main channel for people to obtain information, and traditional search engines can no longer fully meet people's needs. With the further maturity of artificial intelligence technology and the diversification of information services, search engine technology is developing in the direction of intelligence, personalization and domainization. [0003] Vertical search engines are professional search engines for specific fields, aiming at narrowing the total scope of search, thereby obtaining higher search accuracy and improving the search engine's ability to track network resources. As the core part of the vertical search engine, the focused web crawler takes on the important task of collecting and updating information from the Inte...

AI Technical Summary

Problems solved by technology

[0005] The present invention proposes an anchor text-based focused web crawler search method and its system to solve the following technical problems in the existing algorithm of topic relevance in the prior art: although the crawler guided by the existing breadth-first algorithm can accumulate topic relevance stably Growth, but the growth rate is slow; although the crawler guided by TD-IDF has high performance in the crawling startup stage, its accumulated topic relevance does not grow after crawling about 20 pages; although the crawler guided by LSI has the ability to traverse Ability to tunnel, but slower at the start of the crawl

Images