Data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO

Abstract

Provided is a data multi-level caching and high-speed transmission recording method based on MQ and asynchronous IO, multi-level caching technology and ZeroMQ technology are utilized, high-speed stable network transmission can be achieved, and network bandwidth resources are utilized to the maximum extent on the transmission level. On the data recording level, on one hand, asynchronous file writing is carried out through asynchronous IO, frequent switching between a user mode (User Mode) and a kernel mode (Kernel Mode) of a CPU is reduced, and the data writing speed is increased; and on the other hand, by sharing data receiving and asynchronous IO data caching, memory copying in the processing process is reduced, cache consumption is reduced, and data writing speed is increased. Through optimization and improvement of a transmission layer and a data recording layer, the overall transmission recording speed and stability of the system are improved.

Description

technical field [0001] The invention relates to the field of high-speed recording of the Internet of Things, in particular to a data multi-level cache and high-speed transmission recording method based on MQ and asynchronous IO. Background technique [0002] In the field of high-speed recording of the Internet of Things, it is necessary to perform high-speed transmission and high-speed recording of various massive Internet of Things data. At the transmission level, TCP or UDP is usually used for transmission, but the transmission speed of TCP is not high enough, and the transmission speed of UDP is fast, but the problem of packet loss is difficult to avoid. We need a method that can make full use of network bandwidth and stable transmission without packet loss. method. At the level of data receiving and record storage, hardware boards can be used for high-speed writing, but the flexibility is slightly insufficient and the cost is relatively high; using software for receivin...

AI Technical Summary

Problems solved by technology

At the transmission level, TCP or UDP is usually used for transmission, but the transmission speed of TCP is not high enough, and the transmission speed of UDP is fast, but the problem of packet loss is difficult to avoid. We need a method that can make full use of network bandwidth and stable transmission without packet loss. method

At the level of data receiving and record storage, hardware boards can be used for high-speed writing, but the flexibility is slightly insufficient and the cost is relatively high; using software for receiving and high-speed writing improves flexibility, but CPU and disk IO Scheduling is quite difficult, and data packet loss will be exacerbated once data congestion occurs;

Images