A method, device and system for running mapreduce jobs

A technology of job and job description file, which is applied in the direction of multi-program device, instrument, program startup/switching, etc., can solve problems such as not supporting multi-MapReduce task jobs, and achieve the effect of strong technical versatility

Active Publication Date: 2022-06-28
CHINA MOBILE SUZHOU SOFTWARE TECH CO LTD +1
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] 1) Only a small number of fixed MapReduce functions can be matched;
[0006] 2) Only a single simple SQL can be formed instead of a SQL sequence;
[0007] 3) Jobs with multiple MapReduce tasks are not supported

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method, device and system for running mapreduce jobs
  • A method, device and system for running mapreduce jobs
  • A method, device and system for running mapreduce jobs

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0068] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, but not to limit the present invention.

[0069] figure 1 A flow chart for implementing a method for running a MapReduce job according to an embodiment of the present invention; such as figure 1 As shown, the method includes:

[0070] Step 101: Create one or more general-purpose MapReduce job description files, and parse the MapReduce job description files to generate corresponding MapReduce objects;

[0071] Here, the method mainly runs on the client server. Specifically, Mapreduce is a computing framework. When running the computing tasks of MapReduce, the task process is divided into two stages: map stage and reduce stage, and each stage uses key / value pairs (key / value) as input and output. The functions...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method for running a MapReduce job. The method includes: creating more than one general-purpose MapReduce job description file, and parsing the MapReduce job description file to generate a corresponding MapReduce object; according to the MapReduce object Generate the corresponding Structured Query Language (SQL) sequence for the MapReduce data specified in ; wherein, the SQL sequence includes at least one MapReduce task; and send the SQL sequence according to a Directed Acyclic Graph (DAG). The invention also discloses a device and system for running MapReduce jobs at the same time.

Description

technical field [0001] The invention relates to a technology for running MapReduce jobs, in particular to a method, device and system for running MapReduce jobs. Background technique [0002] At present, with the development of the Internet and mobile Internet technologies, enterprises have begun to deploy big data platforms. The MapReduce job of the Hadoop system solves the problem of big data analysis and processing very well. Here, Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without knowing the underlying details of distribution. But traditional distributed relational databases only provide Structured Query Language (SQL, Structured Query Language) to develop jobs, but programmers prefer to use MapReduce to develop jobs. [0003] At present, in the prior art, the corresponding relationship between commonly used MapReduce functions and SQL in the database is usually fixed, and used as a template...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/48
CPCG06F8/451
Inventor 叶志伟
Owner CHINA MOBILE SUZHOU SOFTWARE TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap