A method to replace hadoop storage module with pvfs

Abstract

A method for replacing HADOOP storage modules with PVFS, using parallel virtual file system PVFS to replace the distributed file system HDFS module of HADOOP, the present invention realizes the connection from HADOOP to PVFS, mainly includes three modules: PVFS program interface, HADOOP- PVFS module and JNI connection module. The present invention aims to select a more suitable distributed file system as the storage module of HADOOP to replace HDFS, so as to reduce the overhead of HADOOP in file operations and improve the performance of HADOOP in MapReduce calculations, especially in data-intensive calculations.

Description

technical field [0001] The invention belongs to the field of computer technology, relates to distributed computing and distributed file systems, and in particular to the connection between distributed computing architecture and distributed file systems, specifically a method of connecting PVFS distributed file systems to Hadoop to replace Hadoop The implementation method of the original storage module HDFS. Background technique [0002] Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without knowing the underlying details of the distribution. Make full use of the power of the cluster for high-speed computing and storage. Hadoop is currently the most widely used distributed computing platform. It adopts the MapReduce distributed computing model and provides a series of interfaces and frameworks to help users efficiently utilize the computing resources of distributed clusters and improve the parallelism ...

AI Technical Summary

Problems solved by technology

In the early HDFS, there was only one NameNode, which may cause a serious single point of failure. When the NameNode crashes, all unsaved data will be lost; in later versions, HDFS added multiple secondary NameNodes, and when the primary NameNode crashes, the secondary NameNode continues to run, but there is still only one NameNode running at the same time, which is likely to cause network congestion when file access is frequent

[0006] 2. HDFS uses multiple copies to access files in parallel, and nodes can only perform parallel read and write operations in units of copies

Although this can improve the overall throughput of the file system, in order to meet the access requirements of different nodes for the same data file, HDFS must create enough copies, otherwise it will lose its parallelism

[0007] 3. HDFS stores data on the corresponding computing nodes, which saves network overhead, but with the development of computer network technology, the time spent on network communication has been one to two orders of magnitude lower than that of I / O operations. Almost negligible; the price HDFS pays for saving network overhead is the complete absence of intra-file parallelism

[0010] But as far as the current technology is concerned, in order to achieve higher performance, Hadoop is very closely integrated with HDFS. The virtual methods provided by the virtual classes related to the file system in Hadoop basically correspond to the upper layer interface of HDFS. However, this It brings inconvenience and difficulty to use other file systems instead of HDFS

Images