A Distributed Data Organization Method for High Performance Computing

Abstract

The invention discloses a high-performance computing-oriented distributed data organization method. Data access and data management are separated in an interface layer; and at a computing node end, a client provides a simplified data access interface for an application only, and file access operation, in a POSIX form, completely transparent for the application is supported. According to the method, for a data storage demand of a high-performance computing application, POSIX file system access semantics is simplified, and the data access and the data management are decoupled, so that an efficient and lightweight programming interface is provided for the application; a distributed object storage technology is adopted for performing data organization, so that data organization limitation of a conventional file system is broken through, a more efficient data access protocol is realized, and system expandable capability is further improved; and a memory-based metadata management mode is provided, and a high-performance non-relational key-value database is introduced in a high-performance computer system for the first time, so that an efficient and expandable metadata service can be provided for outside by utilizing excellent characteristics of the high-performance computer system.

Description

technical field [0001] The present invention relates to the field of high-performance computing and the field of distributed storage systems; more specifically, the present invention relates to a distributed data organization method oriented to high-performance computing. Background technique [0002] Distributed file systems are commonly used on high-performance computer systems to build a centralized and shared storage environment to meet the needs of application data storage and access. Typical systems such as Luster, GPFS, PVFS, Panasas, etc. are general-purpose distributed shared file systems, which follow POSIX storage semantics for storage space organization and data access. The system has complete functions and complex design. There are great limitations in high-concurrency data access, mainly in the following aspects: [0003] (1) The hierarchical directory structure is inefficient when organizing massive files, the metadata operation semantics are complex, and the...

AI Technical Summary

Problems solved by technology

[0003] (1) The hierarchical directory structure is inefficient when organizing massive files, the metadata operation semantics are complex, and there is a metadata performance bottleneck;

[0004] (2) The data access is the same as the stand-alone file system. In the high-performance computing environment, the interface of the traditional file system is redundant, and its implementation is tightly coupled with the kernel VFS. In terms of interface description, data block size, cache strategy, and data consistency There are a lot of restrictions in other aspects, not flexible enough

The current object storage service is very different from high-performance computing in terms of interface and application scenarios, so it cannot be directly used to build a high-performance computer mass storage system

Images