Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Spark application program control method and control device

An application program and parameter technology, applied in the field of communication, can solve problems such as increasing application applicability, reducing data delay, data loss, etc., to achieve the effects of avoiding data loss, improving availability, and ensuring acceptability

Active Publication Date: 2021-02-26
KANGCHENG INVESTMENT CHINA
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The present invention provides a Spark application program control method and control device to solve the problem in the prior art that the Spark application program is prone to data loss after interruption, so as to reduce data delay and increase the applicability of the application program

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Spark application program control method and control device
  • Spark application program control method and control device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] The specific implementation of the Spark application program control method and control control provided by the present invention will be described in detail below in conjunction with the accompanying drawings.

[0039] This specific embodiment provides a kind of Spark application program control method, appended figure 1 It is a flow chart of the Spark application program control method in the specific embodiment of the present invention. Such as figure 1 As shown, the Spark application control method provided in this specific embodiment includes the following steps:

[0040] Step S11, judging whether the historical state parameters of the Spark application are valid, and if so, adding the historical state parameters to the running environment of the Spark application. Wherein, the historical state parameters refer to the state parameters saved in the process of running the Spark application before the current running of the Spark application. The historical state p...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention relates to the field of communication technology, in particular to a Spark application program control method and a control device. Described Spark application control method comprises the steps of: judging whether the history status parameter of Spark application program is effective, if so, then adding described history status parameter into the operation environment of described Spark application program; Described Spark application program; Judging whether described Spark application program appears abnormal operation, if so, then retain the current state parameter of described Spark application program; Store described current state parameter in the manager, to update described history state parameter. The invention ensures the adaptability of the running of the Spark application program, avoids problems such as data loss and repeated consumption of data caused by the discontinuity of the state, and improves the usability of the program.

Description

technical field [0001] The present invention relates to the field of communication technology, in particular to a Spark application program control method and a control device. Background technique [0002] Spark Streaming decomposes streaming computing into a series of short batch jobs. The batch processing engine here is Spark, that is, the input data of Spark Streaming is divided into pieces of data (Discretized Stream) according to the batch size (such as 1 second), and each piece of data is converted into RDD (Resilient Distributed Dataset) in Spark, and then The Transformation operation on DStream in Spark Streaming is changed to the Transformation operation on RDD in Spark, and the RDD is converted into an intermediate result and stored in memory. The entire streaming computing can superimpose the intermediate results according to the needs of the business, or store them in external devices. Kafka is a distributed publish-subscribe messaging system. It was original...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F11/14
CPCG06F11/1438
Inventor 文燕军卢春洋吴文钧
Owner KANGCHENG INVESTMENT CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com