Mass data processing and visualization system and method of distributed hybrid architecture

Abstract

The invention relates to a mass data processing and visualizing system and method of distributed mixed architecture. The mass data processing and visualizing system is characterized in that a hardware system comprises a high-performance graphic work station, a high-speed disk array, a plurality of computational nodes, a kilomega or ten-thousand-mega Ethernet switchboard and a router; the corresponding data processing and visualizing method comprises a front end work station method and a computational node method, the front end work station method is adopted at a front end work station, and data loading, data processing, visualizing, interactive operating and other work are finished through the necessary cooperation of the computational nodes; the computational node method is adopted in the computational nodes and is carried out in the mode of daemon, a specific port is monitored, when it is detected that a computation task is sent, the corresponding outer storage algorithm is started to process the data according to task information, and the computing state information is sent back to the front end work station. The utilization rate of the network bandwidth can be guaranteed to the greatest extent, and meanwhile the user operation response speed of the front end work station is increased.

Description

technical field [0001] The invention relates to a mass data processing and visualization system, in particular to a mass data processing and visualization system and method of a distributed hybrid architecture. Background technique [0002] With the continuous development of hardware technology and imaging methods of various image acquisition equipment, the spatial resolution of the obtained data is getting higher and higher. The huge amount of data has brought severe challenges to traditional scalar field data processing and visualization methods. For some complex visualization methods, it is especially difficult to obtain real-time interactive processing speed. In recent years, the research on massive data processing and visualization methods mainly focuses on three aspects: 1. Design efficient external storage algorithms to process massive data. The external memory algorithm mainly adopts two computing modes: one is the "batch computing mode", which is suitable if the ac...

AI Technical Summary

Problems solved by technology

In the field of massive scalar field data processing and visualization, there is currently a lack of a mass data processing and visualization system with mature technology, clear goals, and convenient availability. Although the large-scale software represented by ParaView has relatively powerful data processing and visualization functions, due to its use of distributed The mode of storage and calculation, based on MPI to realize the collaborative work between computing nodes, leads to complex configuration of computing nodes, high requirements on the reliability of computing nodes themselves and network transmission, which is not conducive to building stable and reliable distributed massive data processing and visualization system

Images