Data storage system and method

Abstract

The invention provides a data storage system and a data storage method, and belongs to the field of data storage. The data storage system comprises a data collection module and a time sequence database. According to the data storage method, after a time sequence database receives data and before the data is written into a disk, the data is independently stored as a log, real storage operation is carried out after the log is successfully stored, and the data is written into the disk. If the whole storage operation is interrupted due to a certain reason, when the system is restarted, the storageoperation before interruption can be recovered according to the data of the data log file, and it is guaranteed that the received data cannot be lost. When real data storage operation is carried out,multiple random writes are combined into one sequential write operation, the frequency of calling disk write operation and the moving range of a magnetic head and a mechanical arm are reduced, and the highest disk write speed of a rotating speed hard disk is achieved. One data log file is deleted after the storage operation is completed, and the occupied hard disk space can be recycled.

Description

technical field [0001] The application belongs to the technical field of data storage in time series databases, and in particular relates to a data storage system and method. Background technique [0002] Time-series database products are the basic platform of enterprise informatization and the bridge of management and control integration. A large amount of real-time data generated in the production process is one of the valuable resources of process enterprises. These data need long-term storage, fast retrieval, and serve as the basis for production data analysis, data mining, optimization control and optimization management. A typical process enterprise usually has thousands to hundreds of thousands of data collection points to be integrated. Since the data of the collection points changes dynamically in time sequence and the refresh rate is fast (millisecond level), such large-scale massive data is difficult to use. Time-series database is a new technology specially des...

AI Technical Summary

Problems solved by technology

At present, mainstream time-series databases ensure data storage efficiency by improving hardware performance and increasing software memory space, which also brings a huge hidden danger.

Whether it is the improvement of hardware performance or the use of a distributed architecture to share the pressure of storage, a large amount of money is required to purchase hardware equipment, and professional operation and maintenance personnel are required for long-term maintenance, and the investment rate has increased significantly.

However, due to the use of a large amount of memory, when the server has abnormal conditions such as power outages and crashes, it cannot be stored to the disk in time, and the resulting data loss is irreparable.

If the received data is directly stored in the disk without using the memory as the data memory, the data storage speed will also be affected by the frequent calls to the disk IO operation, which is far from reaching the highest performance.

Images