Hadoop-based massive tile small file storage management method

Abstract

The invention provides a mass small tile file storage management method based on hadoop. According to the method, sequencing is carried out through a Hibert curve, and then, grid tile data is subjected to serialization compression storage by a Sequence File technology of the hadoop pre se; when tile compression blocks are generated, the multithreading parallel compression of a plurality of servers is realized, tile index information is generated, and the mass file library entering storage speed is accelerated; the regular naming of block file names is managed, and efficient storage, fast reading and high-performance grid data service can be provided for mass multi-source and multi-version grid small tiles; and ITMS (Improved Tile Map Service) is designed, and the problem of delay and bandwidth occupation caused by original data transmission and real-time data processing request responding, so that the project data retrieval and transmission requirements are met.

Description

Technical field [0001] The present invention relates to a method for storing and managing raster data on a Hadoop distributed platform, in particular to a method for storing and managing large, multi-source, and multi-version raster small files. Background technique [0002] With the rapid development of GIS technology, map data has grown rapidly. The corresponding tile data volume is large, data sources are multiple, and versions are multiple. How to efficiently store and manage massive map tile data becomes a problem. [0003] The emergence of cloud computing provides a new idea. In recent years, in order to solve the problem of large-scale data storage and management, many companies and organizations have proposed a series of scalable large-scale data management solutions based on "cloud computing" technology. At present, the mainstream open source project Hadoop is a distributed system architecture that includes massive data storage and calculations, and large-scale structured...

AI Technical Summary

Problems solved by technology

[0007] (1) The existing small file storage and management methods are mostly based on hierarchical indexing to manage data. The problem brought by hierarchical indexing is that it takes extra effort to develop a Hadoop cluster small file storage mechanism to ensure that the small files are merged. The block file index and the corresponding block are transmitted to the same location of the same DataNode to realize the distributed storage of the block file index after small files are merged. The NameNode in the Hadoop cluster also needs to consume additional resources to manage the index file and increase memory overhead. ;

[0008] (2) There may be multiple data sources and multiple versions of tile files in the project application, and the hierarchical index method is not suitable for fast and efficient reading of small files with multiple data sources and multiple versions;

[0009] (3) There are a large amount of raw data and frequent real-time data requests in the project. The existing solutions are not specifically optimized for data transmission and data processing, which will affect the performance of data transmission and data response when the project is running

Images