Spark streaming based big data stream processing method and system

Abstract

The invention relates to a spark streaming based big data stream processing method and system. The method includes: a step S1, receiving data sent by a data source at an appointed position, executing a step S2 if the data source is an HDFS, and executing a step S3 if the data source is an FLUME; the step S2, storing the data in a file form, and executing the step S3; the step S3, processing the received data or file through the spark streaming; and a step S4, writing the processing result of the file or the data in a result catalogue through the spark streaming according to a time interval. The method and system provide good fault-tolerant state calculation for fault-tolerant and data assurance, can support Scala programming and Java programming in the aspect of API programming; and in cluster management integration, the Spark Streaming can run on clusters thereof, and can run on a YARN and an Mesos.

Description

technical field [0001] The invention relates to the field of big data stream processing, in particular to a spark streaming-based big data stream processing method and system. Background technique [0002] In the prior art, Storm is often used to implement a data flow model. When Storm is used to implement a data flow model, data continuously flows through a transformation entity network. An abstraction of a stream of data is called a stream, which is an infinite sequence of tuples. A tuple is like a structure that uses some additional serialization code to represent standard data types (such as integers, floats, and byte arrays) or user-defined types. Each stream is defined by a unique ID, which can be used to build a topology of data sources and sinks. [0003] But Storm has its own flaws, for example: In terms of fault tolerance, data guarantee, each individual record in Storm must be tracked as it passes through the system, so Storm can at least guarantee that each rec...

AI Technical Summary

Problems solved by technology

[0003] But Storm has its own flaws, for example: In terms of fault tolerance, data guarantee, each individual record in Storm must be tracked as it passes through the system, so Storm can at least guarantee that each record will be processed once, but in recovering from errors Duplicate records are allowed when coming over, which means that the mutable state may be incorrectly updated twice; in terms of implementation and programming API, because the core of Storm is written in clojure (but most of the expansion work is written in java) , which brings some difficulties for us to understand its implementation; in terms of cluster management integration, Storm can run on its own cluster, and Storm can also run on Mesos, but when running on YARN, a third-party support component is required Storm on YARN, not natively supported

Images