Distributed data deletion flow control method and device, electronic device and storage medium

Abstract

The invention discloses a distributed data deletion flow control method. The distributed data deletion flow control method comprises the following steps: adding a data deletion request to a configuredprocessing queue when receiving the data deletion request sent by a client, and returning information that the data deletion is successful to the client; obtaining a data deletion request in the processing queue every other preset time period; determining index information of the corresponding data to be deleted according to the data deletion request; storing the index information of the data tobe deleted into a configured database; obtaining a flow control threshold corresponding to the current statistical period in the deletion period; and deleting the data corresponding to the index information in the database base on the flow control threshold corresponding to the current statistical period. The invention also provides a distributed data deletion flow control device, an electronic device and a storage medium. The invention can improve the data deletion efficiency of a large-scale distributed storage system and avoid an obvious impact on the normal input and output service performance, which has a good flow control effect.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a distributed data deletion flow control method, device, electronic equipment and storage medium. Background technique [0002] With the rapid development of network applications, the amount of network information data is increasing, and mass data storage is becoming more and more important. The traditional local disk storage method can no longer meet the requirements of large capacity, high reliability, high performance, and horizontal expansion of existing applications. In order to meet these new requirements, network-based distributed storage systems have received extensive attention. [0003] Although for the client, the deletion operation on distributed data is a very lightweight operation type compared with the read and write operations, and can be completed quickly, but when it comes to the distributed file system at the back end, the deletion operation and the read and ...

AI Technical Summary

Problems solved by technology

The traditional local disk storage method can no longer meet the requirements of large capacity, high reliability, high performance, and horizontal expansion of existing applications. In order to meet these new requirements, network-based distributed storage systems have received extensive attention.

[0003] Although for the client, the deletion operation on distributed data is a very lightweight operation type compared with the read and write operations, and can be completed quickly, but when it comes to the distributed file system at the back end, the deletion operation and the read and write operation are different. The passed input / output (Input / Output, IO) paths are the same. This processing method takes a long time to wait for the response, and the delay caused by it is very obvious, especially when performing batch deletion operations of a large number of small files. It may even affect the normal business of users

[0004] In addition, the file deletion operation will generate data input and output (IO). If the IO peak of the user application happens to happen when the distributed storage system performs a large number of file deletion operations, the IO of the file deletion operation will generate IO for the user application. impact, thereby reducing the user experience of the application, and even causing system failures

Images