Cache management method and device

Abstract

The invention discloses a cache management method and device. The cache management method comprises the steps of generating a cache record file according to existing cache data of data sets of all computing nodes, and determining a cache strategy of each computing node; in response to the fact that the local computing node receives the training task, determining whether a data set required by thetraining task is stored in any computing node according to the cache record file; downloading a data set required by the training task in response to the fact that the local computing node meets the caching strategy requirement; and in response to the fact that the local computing node does not meet the caching strategy requirement, deleting the existing caching data based on the caching record file and re-determining whether the local computing node meets the caching strategy requirement or not. The cache of different training data can be managed, the training data can be selectively downloaded and deleted according to actual scene requirements, the downloading time of the training data is saved, and the availability of computing node disk storage is guaranteed.

Description

technical field [0001] The present invention relates to the field of computers, and more specifically, to a cache management method and device. Background technique [0002] In the process of deep learning model training, the larger the training data set and the more data samples, the easier it is to avoid the overfitting problem of the training model. But at the same time, large-scale data sets also bring challenges to cluster management. For example, the collected data of a video processing can reach tens of gigabytes or even hundreds of gigabytes. As a deep learning cluster shared by multiple users, different users may use the same The data set for model training, or a single user will also use a different data set for training. Due to the limited storage space of computing nodes, these training data cannot be stored in each computing node for users to use at the same time, which will also cause waste of storage space. How to store and use these training data has become ...

AI Technical Summary

Problems solved by technology

However, there are read and write bottlenecks in accessing data through shared storage, and some enterprise clusters will not build and maintain a high-performance shared file system due to technical or cost reasons; on the other hand, downloading before training and deleting after training will cost a lot Part of the time is spent on data download, if it is not deleted after training, it may cause other problems due to full disk storage space

[0004] There is no effective solution to the problem that it is difficult to deal with the training data cache in the existing technology

Images