Big data system

Abstract

The invention provides a big data system. The system comprises a big data foundation platform, a sharing resource base, a visualized editing and displaying subsystem and a flow dispatching subsystem. The system combines various existing big data processing systems; a specific big data processing system is submitted based on a data processing flow dispatching program resource generated by user edition and a data resource and is used for carrying out data processing, so that the utilization threshold of the big data system is greatly reduced, big data related sources in organizations can be conveniently shared and a data processing flow can be reused and can be taken over.

Description

technical field [0001] The invention relates to the technical field of computer information storage and processing, in particular to a big data system. Background technique [0002] After more than 60 years of development, information technology has penetrated into all aspects of various industries. A large part of political and economic activities are related to the creation, collection, transmission and use of data. With the deepening of network applications, the influence of big data applications is expanding. According to agency estimates, the total amount of data in the world is doubling every two years. In other words, the total amount of data generated in the last two years is equivalent to the sum of all the data in human history. In this context, fundamental changes will take place from corporate strategy to industrial ecology, from academic research to production practice, from urban management to even national governance. [0003] Big data refers to data sets w...

AI Technical Summary

Problems solved by technology

[0022] First, the threshold for using big data platforms is relatively high, and it is often difficult for people who are good at big data analysis to master the operation methods, which hinders the popularization of big data platforms

[0023] Second, there is no sharing mechanism for relevant codes, scripts, and data related to big data analysis and processing within the organization. Each task needs to be started from scratch, and knowledge cannot be accumulated and is not sustainable;

[0024] Third, at present, data processing based on the above-mentioned big data platform needs to write relevant programs or scripts, which is extremely inconvenient when describing complex data processing processes, which also causes great difficulties for subsequent maintenance and modification

Images