Distributed search engine system and implementation method thereof

Abstract

The invention discloses a hierarchical structure applied to a distributed search engine. The hierarchical structure comprises a physical layer, an abstract layer, an application layer and a presentation layer. The invention also discloses a distributed search engine system, which comprises a Web server, an agent node, a query agent pool, an abstract adapter and a plurality of working nodes, wherein the query agent pool consists of a plurality of query nodes. The invention also discloses an implement method for the distributed search engine system, which comprises the following steps of: S1, registration of a query node; S2, registration of working nodes; S3, state update of the nodes; and S4, distribution and retrieval of a query request. The hierarchical structure has the advantages of good performance, high reliability, diversification, specialization, strong applicability and the like.

Description

technical field [0001] The present invention relates to the design and implementation of a distributed search engine system, specifically referring to multiple unit search engines deployed on multiple clusters based on Web Services and RMI (Remote Method Invocation) technologies to provide unified retrieval A distributed search engine service for the service. Background technique [0002] With the rapid development of the next-generation network, the maturity of new-generation information technology such as Web2.0, and the distribution of information resources, this poses more new challenges for the architecture design of search engines. On the one hand, the geographically dispersed heterogeneous digital information contains a large number of valuable resources, and users urgently need to find the desired information from these information; Under such a large-scale condition, to retrieve such a massive amount of information, the processing power of the centralized search en...

AI Technical Summary

Problems solved by technology

On the one hand, the geographically dispersed heterogeneous digital information contains a large number of valuable resources, and users urgently need to find the desired information from these information; Under such large-scale conditions, to retrieve such a massive amount of information, the processing power of the centralized search engine is limited after all, and multiple search engines are especially needed to assist in the work

With the explosive growth of Internet information, traditional search engines began to show some limitations:

[0003] (1) Insufficient search depth: Traditional search engines can only search for superficial resources that are linked to each other on the Internet, while deep resources, such as pages that require permission to access, internal pages of organizations, and some network information island resources, cannot be indexed

[0004] (2) Poor timeliness: If the server update cycle is too long, it is easy to generate a large number of invalid links

In fact, it is impossible to avoid invalid links by using the technology of web spiders to crawl the Internet.

Dead links can only be minimized by shortening the update cycle

[0005] (3) High cost: Massive resource index information requires huge servers to maintain

Images