Running environment control method based on Spark _ Streaming program

A technology of operating environment and control method, applied in the field of network communication, can solve the problems of inability to ensure the real-time performance of analysis programs and waste of resources, and achieve the effects of reducing mechanical labor, improving production efficiency, and saving computing resources

Inactive Publication Date: 2019-11-22
GUANGDONG EFLYCLOUD COMPUTING CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Let the cluster resources grow exponentially, wasting resources in vain; according to actual needs, if the Spark odd-even cluster only runs one task, it will cause a time gap in the process of switching programs, which cannot ensure the real-time performance of the analysis program

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Running environment control method based on Spark _ Streaming program
  • Running environment control method based on Spark _ Streaming program
  • Running environment control method based on Spark _ Streaming program

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0023] The present embodiment provides a kind of running environment control method based on Spark_Streaming program, comprises SprakStreaming framework, HDFS file system and configuration file, and described Sprak Streaming framework is the real-time analysis framework built on the Spark calculation engine, and described HDFS file system is a A distributed file system suitable for running on general-purpose hardware, the configuration file is a file required for configuring the Spark Streaming program operating environment, the SprakStreaming framework analyzes the Spark_Streaming program in real time, and dynamically reads the real-time data through the HDFS file system The above configuration file can modify the running environment of the Spark Streaming program in real time.

[0024] When the Spark_Streaming program is running, the present invention obtains the corresponding configuration file information through the HDFS file system, the configuration file information of t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a running environment control method based on a Spark _ Streaming program. The method comprises a Sprak Streaming framework, an HDFS file system and a configuration file. The Spark Streaming framework analyzes the Spark _ Streaming program in real time, dynamically reads the real-time configuration file through the HDFS file system, and modifies the running environment of the Spark Streaming program in real time. According to the running environment control method based on the Spark _ Streaming program, in a Spark _ Streaming real-time analysis program, a configurationfile is dynamically read to update running environment configuration in real time, so that the effects of saving deployment time, saving computing resources, not considering multiple data distributionproblems and maintaining the real-time performance of the program are achieved.

Description

technical field [0001] The invention relates to the field of network communication, in particular to an operating environment control method based on the Spark_Streaming program. Background technique [0002] Hadoop Distributed File System (HDFS) is a distributed file system suitable for running on general-purpose hardware. Spark Streaming is a framework built on Spark to process Stream data. The basic principle is to divide Stream data into small time segments (a few seconds), and process this small part of data in a manner similar to batch processing. [0003] In daily development, the development environment and the test environment are usually distinguished according to the operating scope of the program, so that the development and testing can operate in parallel without affecting each other. This is especially true for Spark Streaming applications. Due to the persistence of real-time tasks implemented by SparkStreaming, once submitted, they will continue to run in th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F8/65G06F8/41G06F16/182
CPCG06F8/41G06F8/65G06F16/182
Inventor 霍键聪闵宇胡新勇
Owner GUANGDONG EFLYCLOUD COMPUTING CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap