Distributed parallel database system based on Infiniband network and data processing method

Abstract

The invention discloses a distributed parallel database system based on an Infiniband network and a data processing method. The system comprises a scheduling cluster, a data cluster and a management cluster. Each cluster is composed of at least two data nodes, and the data nodes are connected through the Infiniband network. The data processing method comprises a data distribution method, a data loading method, a data query method and a data de-duplication method. Aiming at the current situation that the number of distributed parallel processing database cluster node servers is large, the characteristics of high bandwidth, low delay and low memory of an Infiniband network are fully utilized, and a mode of applying the network to data is designed, including database system configuration, data loading and storage, data query, data calculation and the like. The method is high in universality, breaks through network bandwidth bottleneck, storage space and calculation delay limitation of a current database system, guarantees high availability of the system, and provides high-performance service for users.

Description

technical field [0001] The invention relates to a distributed parallel database system and a data processing method, in particular to a distributed parallel database system and a data processing method based on an Infiniband network. Background technique [0002] Database technology came into being in the late 20th century, its theory and technology developed extremely rapidly, and its application became more and more extensive. With the wide application of microcomputers, in order to strengthen and expand the functions of microcomputers to process data, it is required to interconnect many microcomputers distributed in different places to participate in common work. In this way, the database has entered the era of distributed databases. Since the 1990s, distributed database systems have entered the stage of commercial application, and traditional relational database products have developed into distributed database products centered on computer networks and multi-tasking op...

AI Technical Summary

Problems solved by technology

[0003] Purpose of the invention: The technical problem to be solved by the present invention is to provide a distributed parallel database system and data processing method based on the Infiniband network, which solves the network bandwidth bottleneck of the existing 10 Gigabit network in a specific scenario in the distributed parallel processing database , memory loss and data processing delay, etc., through the design of database system architecture, data distribution storage, data loading mode and data query, etc., it provides a complete set of data processing methods for distributed parallel databases, breaking through the bottleneck of application scenarios, Improve the performance of database cluster data processing

Images