Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Massive geoscientific data parallel processing method based on distributed file system

A technology for distributed files and data, applied in the fields of ecology and geoinformatics

Active Publication Date: 2013-07-10
COMP NETWORK INFORMATION CENT CHINESE ACADEMY OF SCI
View PDF3 Cites 16 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although some traditional big data processing technologies have been applied to data processing in various fields, for geo-information calculations based on remote sensing image files, limited by the special format of data files and the single use of processing tools, traditional processing technologies It is difficult to meet the needs of geo-information computing in a short time

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Massive geoscientific data parallel processing method based on distributed file system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0026] In many distributed computing frameworks, the biggest problem in processing massive data is that it is difficult to avoid the transmission of large amounts of data on the network. For example, the commonly used open source MapReduce system Hadoop, many algorithms will generate a large amount of traffic on the network during the Reduce phase. Due to the huge difference between the CPU processing speed and the network transmission speed, this is often the biggest bottleneck in the efficiency of the entire computing process. Since geoscience data has the characteristics of large data volume and high calculation complexity, it is difficult to meet the user's requirements for calculation time by using conventional distributed computing methods. In order to solve this problem, the present invention designs a job scheduling system, combined with specific functions (The user is allowed to specify the physical location of the file copy, so that all the data blocks of a file copy ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a massive geoscientific data parallel processing method based on a distributed file system. The massive geoscientific data parallel processing method comprises the following steps: 1), taking the distributed file system as a storage system of geoscientific data, and deploying the distributed file system on a computing cluster, wherein the distributed file system has a unified name space; 2), storing received calculating tasks in a waiting queue by a task scheduling system of the computing cluster; 3), selecting one calculating task from the waiting queue by the scheduling system, and entering a running queue; 4), according to information of the calculating task, searching a computing node of a data file required by running the calculating task from metadata of the distributed file system by the scheduling system; and 5), selecting the computing node possessing the maximum data required by running the calculating task by the task scheduling system, remotely acquiring a data file, which is required by the calculating task, but not possessed by the computing node, executing the calculating task at the computing node, and returning an execution result. By the massive geoscientific data parallel processing method, computation localization is achieved to the maximum extent.

Description

technical field [0001] The invention belongs to the technical field of ecology and geographic information, and relates to the storage and parallel processing of massive remote sensing geoscience data, in particular to a method for parallel processing of massive geoscience data based on a distributed file system, which is mainly used in remote sensing ecological monitoring, species distribution prediction and Processing methods for massive data in related fields such as remote sensing geoscience data inversion. Background technique [0002] The file system is an important part of the computer system. With the development of network technology, the file system on the independent platform tends to develop to the high-speed LAN, and gradually forms a supporting technology in the distributed computer environment - Distributed file system (Distributed file system) File System). The key technology of distributed file system mainly includes virtual file system, cache technology and...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
Inventor 黎建辉沈庚周园春王学志韦远科张洋
Owner COMP NETWORK INFORMATION CENT CHINESE ACADEMY OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com