Massive data-oriented data exchange method

Abstract

The invention discloses a massive data-oriented data exchange method, which belongs to the technical field of information processing. The method comprises the following steps that: a data updating control module receives new data, and performs bit torrent (BT) coding on the data to form a torrent file; a distribution strategy management module performs searching to determine whether any user subscribes the data; if a certain user subscribes the data, the distribution strategy management module informs the subscribing user of downloading the torrent file, the subscribing user resolves the torrent file to obtain the address of a Tracker server, is connected with the Tracker server to obtain information about all Peers uploading or downloading the data, and uploads own information to the Tracker server, and all the Peers perform peer-to-peer (P2P) data exchange with one another; and if no user subscribes the data, the whole process is finished and the data exchange is not performed. By the method, the problems of low transmission speed, heavy server load, data updating and transmission untimeliness and the like in the transmission and exchange processes of massive data are solved.

Description

technical field [0001] The invention relates to a data exchange method for massive data, which belongs to the field of information technology processing. Background technique [0002] Massive data has emerged in recent years with the development of information technology, especially Internet technology, and is mainly used to describe huge and unprecedentedly vast data, such as various spatial data, report statistics, text, sound, images, hypertext and other environmental and cultural data information. Now, many enterprises and companies are involved in the operation and processing of massive data, such as water conservancy departments, meteorological departments, etc., handle very large amounts of data. To sum up, massive data mainly includes two types of data, one is that each piece of data itself is not large but the amount of data is huge, such as SMS service data in mobile communication; the other is that the data itself is very large, such as remote sensing images from...

AI Technical Summary

Problems solved by technology

The biggest disadvantage of this method is poor scalability. When the number of clients that need to be exchanged increases, the sending tasks on the server side will double. When one client is added, the server needs to send 400G of data. When the number of clients is increased to 10, the data to be sent is 2000G. The pressure on the server is increasing. In addition, FTP commands and data are generally transmitted in plain text , the data security is low, and the ability to identify different customers is poor, so it is difficult to achieve different sending tasks for different users

The second is that IP multicast technology can effectively reduce the server and network load by its multiplexing method, but due to many deficiencies, such as the complexity of implementation, congestion control, reliability management, etc., it is difficult to practical application

However, the existing subscription / publishing system can only handle the first type of massive data. When the data is the second type of massive data, existing subscription / publishing systems such as Google Reader, Baidu Subscription, and GreatNews cannot process a single data in several Hundreds of megabytes or even several gigabytes of data are sent to subscribers

Images