Remote sensing data rapid concurrent read-write method based on distributed file system

Abstract

The invention provides a remote sensing data rapid concurrent read-write method based on a distributed file system. A bottom physical structure inherits the characteristics of the HDFS file system, and the method comprises the steps that a Hadoop system is installed on each data server in a computer group, and the HDFS file system is established; then a part of space is divided on each data server to serve as a physical storage space of an own file system; primary packaging is carried out on an HDFS service processing layer, access of an operating system to a file system is taken over, when the operating system only requires to read a file and file data exists, an HDFS file system interface is directly referenced, and reading of the file data is completed by an HDFS; when the operation system requires that the access to the file comprises the file writing operation, the file operation is comprehensively taken over, the own file system realizes data reading and writing, and after the data reading and writing are completed, the data are synchronized to the HDFS; and the own file system only reads and writes one server. According to the invention, rapid concurrent reading and writing of mass remote sensing data can be realized.

Description

technical field [0001] The invention relates to the field of computer application technology and remote sensing big data processing, in particular to the concurrent reading and writing technology of large data files, and in particular to the fast concurrent reading and writing technology of remote sensing data based on a distributed file system. Background technique [0002] With the rapid development of remote sensing technology, more and more remote sensing satellites are launched by various countries, and the satellite data received every day has reached PB level. The processing of massive remote sensing data puts forward higher requirements for storage technology and processing speed. On the other hand, with the rapid development of computers, especially Internet technology, data storage technology has also made a qualitative leap, especially the emergence of cloud technology in recent years, which has raised data storage technology to an unprecedented level. Cloud techn...

AI Technical Summary

Problems solved by technology

[0003] However, HDFS also has the following three disadvantages (HDFS is not applicable in these cases): 1. Low-latency data access cannot be achieved, such as reading and writing data at the millisecond level

HDFS is only suitable for high-throughput scenarios, that is, a large amount of data is written in a certain period of time, but it does not support reading data back immediately

2. Does not support a large number of small file storage

Storing a large number of small files will occupy a large amount of memory of the index service (NameNode) to store data block index information. However, the index service memory of HDFS is limited and cannot achieve massive expansion. In addition, a large number of indexes will cause the seek time to exceed the read time, which is extremely Greatly reduce access efficiency

3. Unable to concurrently write or modify files randomly

[0004] Therefore, HDFS containing the above three shortcomings cannot be applied to remote sensing big data processing. For this reason, the present invention proposes a method for improving the distributed file system to realize fast concurrent reading and writing of remote sensing big data

Images