A cross-data center data connection method and system

Abstract

The invention provides a data connection method across a data center of an HDFS (Hadoop Distributed File System). The method comprises the following steps that: a coordination node acquires the IP (Internet Protocol) addresses of metadata nodes from a configuration file after receiving a data connection operating request, establishes connection with the metadata nodes in the configuration file one by one, and acquires metadata information about a requested metadata node when a requested table is inquired in a current metadata node; the requested node filters data of the requested node according to information about requested data, screens the requested data of the requested node, and transmits the size of a result set saved by the requested node to the coordination node; the coordination node informs small result set nodes to transmit result sets saved by the small result set nodes to all large result set nodes; all the large result set nodes perform Hash join on the result sets of the large result set nodes and the result sets transmitted by all the small result set nodes to obtain result records; and the coordination node aggregates the result records. Through the method and a system, data connection across the data center can be realized.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a data connection method and system across data centers. Background technique [0002] When faced with big data application requirements, when stand-alone performance faces bottlenecks and relational database cluster systems can no longer meet the needs of big data applications, using distributed storage systems to store data and adopting standard user interfaces based on SQL language has become a solution to large-scale structures. An effective way to simplify data retrieval problems. In the prior art, HDFS (Hadoop Distributed File System, Hadoop Distributed File System) is usually used to store data files, and HDFS files and their contents are mapped into a table structure by maintaining a piece of metadata. The application scenarios in the prior art are all oriented to the same data center, that is, to a set of distributed file systems. However, many application scenarios r...

AI Technical Summary

Problems solved by technology

The application scenarios in the prior art are all oriented to the same data center, that is, to a set of distributed file systems. However, many application scenarios require the system to perform join operations on two or more tables distributed in different data centers, that is, to The data tables of different HDFS data centers are connected, and the data connection across HDFS data centers cannot be realized in the existing technology

Images