System and method for data migration between high-performance computing architectures and data storage devices with increased data reliability and integrity

Abstract

A system for data migration between high performance computing architectures and data storage disks includes an intermediate data migration handling system which has an intermediate data storage module coupled to the computer architecture to store data received, and a data controller module which includes data management software supporting the data transfer activity between the intermediate data storage module and the disk drives in an orderly manner independent of the random I / O activity of the computer architecture. RAID calculations are performed on the data prior to storage in the intermediate storage module, as well as when reading data from it for assuring data integrity, and carrying out reconstruction of corrupted data. The data transfer to the disk drives is actuated in sequence determined by the data management software based on minimization of seeking time, tier usage, predetermined time since the previous I / O cycle, or fullness of the intermediate data storage module. The storage controller deactivates the disk drives which are not needed for the data transfer.

Description

FIELD OF THE INVENTION[0001]The present invention is directed to data migration between high performance computing cluster architectures (data generating entities) and data storage disks, and particularly, the present invention relates to a data migration technique rendering an orderly access favored by data storage disks which is decoupled from a randomized input / output (I / O) activity of data generating entities.[0002]More in particular, the present invention is directed to a data migration system employing an intermediate data migration handling system that performs high speed data caching into an intermediate data storage layer from data generating devices, aggregates and organizes the ingress and further controls migration of the cached data to a target data storage device in a manner that is most efficient for the target storage devices. When the data is stored in the intermediate data storage layer, the computing entity may return from the I / O state to the compute state withou...

AI Technical Summary

Benefits of technology

The present invention provides a data migration system that uses software to determine the best order in which to transfer data between a generating system and storage disks. This is based on factors like the fullness of the storage layer, the time elapsed from previous activity, and minimizing the usage of storage disks. After data is written in the intermediate storage module, a signal is sent to the generating system to switch to a compute state, reducing its input / output cycle. The technical effect is an improved data migration process that optimizes data transfer and reduces storage usage.

Problems solved by technology

The ratio of computer elements to File Servers is often very large and may exceed 1000 in some implementations.

Disk drives do not favor the regime of satisfying random requests since the recording heads have to be moved to various sectors of the drive (aka “seeking”), and this “seeking” movement takes a larger amount of time when compared to the actual “write” or “read” operation.

Images