Automatic optimization method for configuration of big data processing platform

A big data processing and automatic optimization technology, applied in the computer field, can solve the problems of long running time and high time cost, achieve the effect of reducing time and avoiding long-term running configuration with poor parameters

Pending Publication Date: 2021-06-25
SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Due to the very long running time of the program when using poor configuration parameters (the running time of poor configuration parameters and better configuration parameters can differ by dozens of times), the time cost required to create the training set is very high

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Automatic optimization method for configuration of big data processing platform
  • Automatic optimization method for configuration of big data processing platform
  • Automatic optimization method for configuration of big data processing platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] In order to make the purpose, technical solution, design method and advantages of the present invention clearer, the present invention will be further described in detail through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0028] In all examples shown and discussed herein, any specific values ​​should be construed as exemplary only, and not as limitations. Therefore, other instances of the exemplary embodiment may have different values.

[0029] Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods and devices should be considered part of the description.

[0030] see figure 1 As shown, the method for automatically optimizing the configuration of the big data processing platform prov...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides an automatic optimization method for configuration of a big data processing platform. The method collects a configuration parameter training set according to the following steps: randomly generating a group of configuration parameters of the big data processing platform, running the group of configuration parameters, monitoring execution time t, and stopping running when the execution time t exceeds dynamically set maximum permitted time Tmax; determining whether to exit the process of collecting the configuration parameter training set or not according to the fluctuation condition of the execution time t in the configuration parameter operation process; and after determining to quit the process of collecting the configuration parameter training set, selecting a group of configuration parameters with the shortest execution time from all the successfully operated configuration parameters as optimal configuration parameters. According to the method, the data collection time can be remarkably shortened on the premise of accurately finding out the optimal configuration.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to an automatic optimization method for configuration of a big data processing platform. Background technique [0002] In recent years, with the rapid development of Internet technology, big data has been used more and more. For example, in order to quickly process big data, the AMP Lab at the University of California, Berkeley designed a general parallel framework for Spark big data systems. It uses the BSD open source license and was donated to the Apache Software Foundation in 2013. Spark covers various workloads such as batch programs, user-interactive programs, iterative algorithms, etc. It extends the MapReduce model, and greatly reduces disk read / write operations through memory cluster computing, thereby greatly improving data processing speed. [0003] The performance of the Spark framework will be affected by configuration parameters during operation. Due to the diffe...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/445
CPCG06F9/4451Y02D10/00
Inventor 陈超喻之斌
Owner SHENZHEN INST OF ADVANCED TECH CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap