Multi-source heterogeneous ecological environment big data processing method and system based on data lake

Abstract

The invention provides a multi-source heterogeneous ecological environment big data processing system and method based on a data lake. The system comprises an ecological environment data acquisition layer, an ecological environment data cleaning layer, an ecological environment data storage layer, an ecological environment data processing layer and an ecological environment data management layer.The ecological environment data acquisition layer is used for acquiring original data of the ecological environment; the ecological environment data cleaning layer is used for preprocessing and standardizing the data acquired by the ecological environment data acquisition layer; the ecological environment data storage layer is used for carrying out classified and layered storage on the data transmitted by the ecological environment data cleaning layer; the ecological environment data processing layer is used for integrally processing the stream batch ecological environment data; and the ecological environment data management layer is used for monitoring the ecological environment data acquisition, cleaning, storage and processing processes. The access and analysis efficiency of the environmental data can be improved, and the storage cost is greatly reduced.

Description

technical field [0001] The invention relates to the field of computer big data processing, in particular to a data lake-based multi-source heterogeneous big data processing method and system thereof. Background technique [0002] In recent years, with the rapid development of technologies such as the Internet of Things, remote sensing, cloud computing, and mobile smart devices, ecological environment data has shown a blowout growth. On the whole, ecological environment big data can be divided into four categories: 1) basic support data: basic geography, remote sensing images, climate and meteorological data; 2) natural ecological data: farmland ecosystem, forest ecosystem, grassland ecosystem, 3) Environmental monitoring data: data on water environment, atmospheric environment, soil environment, noise environment, nuclear radiation environment, etc.; 4) Humanities and social data: economic development, infrastructure, Data on energy consumption, public participation, online...

AI Technical Summary

Problems solved by technology

The main difficulties are: 1) Data interoperability: the source of ecological environment big data covers almost all government functional departments, these departments are not connected to each other, and data often exists in the form of "data islands"

2) The problem of data standardization: data not only exists in a single structured form, but more data is presented in semi-structured and unstructured forms, lacking a unified data specification, and there are a large number of heterogeneous data

3) Data storage cost and operational performance issues: The storage of ecological environment big data in databases or data warehouses often brings high storage costs, and at the same time seriously restricts the speed of data processing

4) The problem of data openness: the total amount of ecological environment data openness is low, most of which are static data, and are concentrated in cities with developed economies, government informatization foundations, and well-developed IT industries

Images