Data transfer node, method and system and computer readable storage medium

Abstract

The invention provides a data transfer node, method, system and device and a computer readable storage medium, and the node comprises a data reading scheduler which is used for reading to-be-transferred data from a source database and putting the to-be-transferred data into a built-in queue of a memory; a queue scheduler which is used for acquiring the to-be-transferred data from the internal memory built-in queue and writing the to-be-transferred data into the target database; a memory built-in queue which comprises a reading queue and a reading buffer queue, the reading queue is used for receiving the data to be transferred from the data reading scheduler, and the reading buffer queue is used for providing the data to be transferred for the queue scheduler, wherein each time when the reading queue is fully written and the reading buffer queue is in an emptying state, the current reading queue is switched to the reading buffer queue, and the current reading buffer queue is switched to the reading queue. By means of the data transfer node, read-write separation can be achieved, high efficiency of data transfer is guaranteed, and the problem of read bottleneck is avoided.

Description

technical field [0001] The invention belongs to the field of data transfer, and in particular relates to a data transfer node, a method, a system and a computer-readable storage medium. Background technique [0002] This section is intended to provide a background or context for implementations of the invention that are recited in the claims. The descriptions herein are not admitted to be prior art by inclusion in this section. [0003] Data transfer has a strong demand in distributed systems, especially the scenario where traditional relational databases are transferred to distributed file systems (HDFS) is more complicated. However, in the process of data transfer, there are often unequal read and write performances between the source end and the destination end, which leads to the problem that the destination end cannot perform data transfer efficiently due to the system avalanche effect caused by excessive writing pressure or the source end is slow to read. At present,...

AI Technical Summary

Problems solved by technology

[0003] Data transfer has a strong demand in distributed systems, especially the scenario where traditional relational databases are transferred to distributed file systems (HDFS) is more complicated

However, in the process of data transfer, there are often unequal read and write performances between the source and destination, which leads to the problem that the destination cannot perform data transfer efficiently due to the system avalanche effect caused by excessive writing pressure or the source is slow to read.

At present, logstash is the most commonly used data transfer tool in the industry. However, logstash has problems such as high memory and CPU usage, easy message loss, and the inability to balance the read and write performance of the data source and destination.

Images