Ocean big data master-slave index system and method based on Spark cloud network

Abstract

The invention relates to an ocean big data master-slave index system and method based on a Spark cloud network. The ocean big data master-slave index system and method are particularly suitable for ocean big data which have the advantages of space autocorrelation, high similarity, real-time performance and the like. The big data age internet of things, the cloud calculation and other technologies are applied to the field of distributed indexes, the internet of things serves as an antenna for ocean information collection, data division strategies are designed and the data are deployed and saved to all pieces of physical cloud in a distributed mode on the basis of the clustering thought, rapid and one-stop calling of the multi-source ocean data is achieved through the combination with a multi-layer master-slave index structure, and the requirements for high-performance calculation, strong real-time feedback, high-frequency inquiry and the like in various offshore pre-warning and emergency models are met. According to the system and the method, on the basis of physical cloud storage, the millisecond-grade inquiry response requirements are ensured, and therefore rapid and real-time disaster inversion and pre-warning are rapidly focused, and the digitalization and intellectualization progresses of ocean disaster prevention and reduction and the like are accelerated.

Description

technical field [0001] The present invention relates to the field of data processing, in particular to big data processing, and specifically refers to a master-slave indexing system and method for ocean big data based on Spark cloud network. Background technique [0002] my country's marine construction mainly grasps four goals: management, sharing, decision-making and service. However, in the process of building an information platform in my country, there are still problems such as imperfect integration standards of marine information resources, unsmooth information transmission channels, and poor data access and communication. . The development of big data technology and the complexity of organizational networks have increasingly increased the pressure on marine information processing, which is mainly manifested in: 1. With the large-scale layout of the marine Internet of Things, many information platforms are under increasing pressure for second-level data processing and ...

AI Technical Summary

Problems solved by technology

[0002] my country's marine construction mainly grasps four goals: management, sharing, decision-making and service. However, in the process of building an information platform in my country, there are still problems such as imperfect integration standards of marine information resources, unsmooth information transmission channels, and poor data access and communication.

The development of big data technology and the complexity of organizational networks have increasingly increased the pressure on marine information processing, which is mainly manifested in: 1. With the large-scale layout of the marine Internet of Things, many information platforms are under increasing pressure for second-level data processing and data access 2. The real-time and accuracy of maritime emergency and early warning applications are also increasing, and it is difficult to guarantee high-frequency and high-intensity data queries; 3. Traditional closed links restrict the interconnection of marine information. Access adds significant time and space overhead

Images