Cloud computing platform based big data processing method

Abstract

The present invention relates to the technical field of Internet information processing, and particularly to a cloud computing platform based big data processing method. The method comprises according to the same information elements contained in different data records, using a big data processing framework under a cloud computing platform to abstract associated information of target information. The big data processing framework of the cloud computing platform, such as a MapReduce, a SparkIn and the like under a Hadoop system assigns the task to a plurality of nodes to be processed by automatic segmentation of the task, and the division and autonomy manner can implement parallel processing on massive data and greatly improves data processing efficiency. The method disclosed by the present invention, through the big data processing framework, can analyze associated information hidden behind massive target information in the complex Internet information according to requirements, and provides a reliable and convenient way for deepen excavation and application of data information, and provides effective technical support for target background analysis, marketing, market segmentation, risk prediction, risk prevention and control, and the like.

Description

technical field [0001] The present invention relates to the technical field of the Internet, in particular to a big data processing method based on a cloud computing platform. Background technique [0002] With the development of society and the advancement of science and technology, the connection between individuals or groups has become closer, and the close connection has promoted the rapid dissemination and growth of information. Today's world has already entered the information age. Growth and accumulation, the era of big data is approaching. The basic characteristics of big data: large data volume, various types, low value density, fast speed and high timeliness; as the most important feature: large data volume and low value density are troublesome The difficulty of mining and utilizing such massive data information, how to accurately obtain the information that people care about in the massive data is like finding a needle in a haystack; at the same time, facing the m...

AI Technical Summary

Problems solved by technology

[0002] With the development of society and the advancement of science and technology, the connection between individuals or groups has become closer, and the close connection has promoted the rapid dissemination and growth of information. Today's world has already entered the information age. Growth and accumulation, the era of big data is approaching. The basic characteristics of big data: large data volume, various types, low value density, fast speed and high timeliness; as the most important feature: large data volume and low value density are troublesome The difficulty of mining and utilizing such massive data information, how to accurately obtain the information that people care about in the massive data is like finding a needle in a haystack; at the same time, facing the massive amount of information, how to analyze the relationship between certain types of information Correlation, and use this to analyze the hidden value behind the information, to reflect the value of data information at a higher and deeper level, but in the face of such a massive amount of data, if you want to quickly and accurately analyze the relationship between the data, very difficult

[0003] For a specific information target, how to analyze the relationship between the target and other targets has more practical significance in many fields, because the target with the relationship often has a greater impact than a single simple individual when performing various activities. In addition, targets with related relationships will be restrained or supported by internal related relationships when they establish various activities externally, which will be more complicated than the activity track of simple targets.

However, the correlations are usually hidden, requiring people to mine and analyze the complex and cumbersome data. It will become more difficult to analyze these implicit correlations in the face of massive data. If these tasks are realized one by one by individuals, it will cost Huge manpower and time costs; an analysis implementation method is urgently needed to help analysts realize this huge and cumbersome calculation process and provide such analysis results

Images