Data processing method and system supporting non-random write mode

Abstract

The invention discloses a data processing method and system supporting a non-random write mode. According to one scheme, the method includes after triggering a writing request, writing user data of the writing request into a memory buffer area, and recording into binlog files; supplying reading access of the user data written into the memory buffer area and in full data files of an SSD (solid state drive) to outside. According to another scheme, the method includes after triggering a reading request, according to user data required by the reading request, judging how to read the user data from the memory buffer area and the SSD supplying reading access to the outside. The invention further discloses a data processing system supporting the non-random write mode; after a data unit triggers a writing request, user data of the writing request can be written into the memory buffer area and are recorded in the binlog files. By the aid of the method and system, on the basis of greatly reducing calculation complication and cost, writing / reading efficiency is improved, and high concurrency data read / write service can be provided to users.

Description

technical field [0001] The invention relates to a data processing technology for storing data on a disk, in particular to a data processing method and system supporting no random write mode. Background technique [0002] User Generated Content (UGC, User Generated Content) is a new way for users to use the Internet, that is, from the original data processing that mainly focuses on downloading to data processing that focuses on both downloading and uploading. Social networking, video sharing and blogging are the main application forms of UGC. With the continuous development of the global Internet business, the UGC business is rising day by day, which has aroused widespread concern in the industry. [0003] Since data is generated by users, a large number of users will inevitably generate a large amount of data, and at the same time will bring a large amount of reading and writing. How to effectively store these user data and provide high-concurrency data read and write serv...

AI Technical Summary

Problems solved by technology

[0007] 1. Massive data will inevitably bring a large amount of read and write. Since the write request in the random write mode will erase the original data and then update it, therefore, a large number of write requests in the random write mode will frequently modify the SSD hard disk partition The update operation not only increases the time consumption of data processing, but also greatly wears out the physical storage of the SSD, resulting in a greatly shortened service life of the SSD

For example, after testing the performance of SSD, it can be seen that the time consumption of reading, writing, and erasing is 25us, 200us, and 1.5ms respectively. It can be seen that frequent erasing will bring serious time loss; The service life of SSD is 5000 times, that is to say, if a partition is repeatedly modified to reach the upper limit of 5000 times, the partition will be damaged and the entire SSD needs to be replaced

[0008] 2. The write request in the random write mode will interfere with the efficiency of data reading, thus causing very unstable service quality for high concurrent reading services

For example, after testing the performance of SSD, it can be seen that if SSD only provides read services, it can achieve a stable read quality of 4000 / S, but if the write requests of 25 / S random write mode are added at the same time, the read performance will decrease Unstable read quality up to 2500 / S, the read efficiency is greatly reduced

[0009] To sum up, the existing technology supports the data processing scheme of random write mode write request, which is not only time-consuming and affects the service life of SSD, but also has a bad effect on the data processing of read request or write request. If there is a simple, Efficient and low-cost solutions, on the basis of greatly reducing computational complexity and cost, improve read / write efficiency, so as to provide users with highly concurrent data read / write services, which will be very meaningful

Images