Data information updating system and method based on multiple verifications

Abstract

The invention discloses a data information updating system and method based on multiple verifications. The system is characterized by comprising a data obtaining module, a first verification module, a second verification module, an eliminating module and an updating module, wherein the data obtaining module is used for obtaining data information by using web crawlers; the first verification module is used for preliminarily verifying the data information obtained by the web crawlers within the preset time and obtaining a preliminary data information set; the second verification module is used for conducting multiplicity verification on the data information in the preliminary data information set and a preset data information base; the eliminating module is used for deleting the data information with the multiplicity verification failing in the preliminary data information set; the updating module is used for adding the data information with the multiplicity verification succeeding in the preliminary data information set into the preset data information base and updating the preset data information base. In this way, the similarity verification is conducted after a lot of data information is obtained by the web crawlers, the high-similarity data information which is obtained in real time is eliminated, the web crawlers are avoided from obtaining the repetitive data information, the amount of the data information is reduced, and the multiplicity verification efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of data verification, in particular to a system and method for updating data information based on multiple verifications. Background technique [0002] With the explosive growth of Internet information, the data information on the Internet is increasing geometrically every day. When users obtain the data information they need, they are often submerged in a large amount of useless repetitive information. Data information is already a convenient way recommended by most users. As one of the basic components of search engines, web crawlers need to obtain data information from the Internet to provide users with data information support. However, with the wanton reprinting and Multi-site delivery, whether the data information acquired by the web crawler is rich, whether the similarity and overlap are high, are closely related to the efficiency of the web crawler. Therefore, in order to improve the crawling speed,...

AI Technical Summary

Problems solved by technology

Therefore, in order to improve the crawling speed, the network usually adopts a parallel crawling method, which introduces new problems: repeatability, when parallel running crawlers or crawling threads run at the same time, repeated pages and quality problems are added. A crawler or crawler thread can only fetch part of the page, resulting in poor page quality

[0003] Due to the huge amount of data in the existing database, with the increase of captured data information, the workload will be larger and the efficiency of the algorithm will become lower.

Images