File reading-writing method for distributed file system

Abstract

The invention discloses a file reading-writing method for a distributed file system. The file reading-writing method includes the following steps: in file reading, a file reading IO path of a client-ametadata data server-a data server-a client is adopted; in file writing, the client acquires the number of files to be written on which file writing needs to be performed; if the number of the file to be written exceeds a preset threshold, a high performance computing scene is determined, file writing is performed for mass threads in the high performance computing scene at the same time, a strategy that data is written firstly and metadata is created secondly is adopted to reduce emergency load of a metadata server; otherwise, a file write-in IO path of a client-a data server-a metadata server-a client is adopted for each target file to be written. The file reading-writing method is fast in file reading-writing speed, is high in efficiency, can reduce the number of times of interactions between the client and the metadata server, and can reduce the communication cost.

Description

technical field [0001] The invention relates to the field of distributed storage systems, in particular to a method for reading and writing files used in a distributed file system. Background technique [0002] With the popularization and deepening of big data applications, the basic computing framework poses higher challenges to the storage system in terms of scale and performance requirements. High-performance computers have higher and higher requirements for the performance of distributed file systems. In the application scenarios of frequent creation and deletion of massive small files and large-scale concurrent I / O operations, the read and write efficiency of file systems has become a key factor limiting its performance. For example, for applications such as health big data, traffic big data, and financial big data, the amount of data is usually in the order of TB, PB or even EB, so a large amount of storage resources are required to store and manage these data. In add...

AI Technical Summary

Problems solved by technology

[0004] However, after the traditional distributed file system reads and writes files, after performing step (4), the client cannot directly read and write the data server according to the file name of the file passed in by the user, and must pass steps (5) and (6) after , you must obtain the data server information of the file from the metadata server before you can read and write the data server

This way of reading and writing files reduces the efficiency of client access to files, and also increases the access pressure of Metadata server

Images