Eureka AIR delivers breakthrough ideas for toughest innovation challenges, trusted by R&D personnel around the world.

Efficient iterative mechanism design method based on Hadoop cloud computing framework

An iterative mechanism and design method technology, applied in the field of Hadoop cloud computing framework research, can solve problems such as reducing the performance of processing massive data or big data, reducing the iterative operation performance of the cloud computing framework, wasting network resources and CPU resources, etc.

Active Publication Date: 2017-09-01
ZHENGZHOU UNIVERSITY OF LIGHT INDUSTRY
View PDF3 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Although this can solve the iterative operation in massive data processing or big data processing, it will cause two serious problems: 1) Even if most of the data used in the slave node does not change during the entire iteration process, each iteration is also It must be re-downloaded from HDFS, which will cause frequent I / O operations on the master node, wasting a lot of network resources and CPU resources; 2) The detection of the iteration termination point after each iteration operation requires an additional MapReduce task, which will lead to Hadoop cloud computing framework wastes network resources and CPU resources every iteration
Both of these problems greatly reduce the performance of the iterative operation of the cloud computing framework, which in turn reduces the performance of processing massive or big data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Efficient iterative mechanism design method based on Hadoop cloud computing framework
  • Efficient iterative mechanism design method based on Hadoop cloud computing framework
  • Efficient iterative mechanism design method based on Hadoop cloud computing framework

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0110] like figure 1 Shown, a kind of efficient iterative mechanism design method based on Hadoop cloud computing framework of the present invention comprises the following steps:

[0111]Step S11: Create an application programming interface API for simplifying the writing of MapReduce iterative programs in the Hadoop cloud computing framework;

[0112] Step S12: call the application programming interface API, and create a loop iteration main control module at the master node;

[0113] Step S13: call the application programming interface API, and cache and index the data reused in the iteration process in the slave node;

[0114] Step S14: design task scheduling and fault tolerance mechanism for the loop iteration main control module in the master node, including:

[0115] Step S141: Design the task scheduling mechanism in the Hadoop cloud computing framework, including:

[0116] Step S1411: storing the mapping relationship between the slave nodes and the data used by the m...

Embodiment 2

[0124] like figure 2 As shown, another kind of efficient iterative mechanism design method based on Hadoop cloud computing framework of the present invention comprises the following steps:

[0125] Step S21: selecting multiple hosts or servers, and deploying the multiple hosts or servers in the same local area network;

[0126] Step S22: Build a Hadoop cloud computing platform on multiple selected hosts or servers, and install JDK and HBase, select one of the hosts or servers as the Master host or Master server, and the remaining hosts or servers as Slaver hosts or Slaver servers , and use the Master host or Master server as the master node of the Hadoop cloud computing platform, and use the Slaver host or Slaver server as the slave node of the Hadoop cloud computing platform;

[0127] Step S23: Create an application programming interface API for simplifying the writing of the MapReduce iteration program in the Hadoop cloud computing framework. The application programming in...

Embodiment 3

[0186] like image 3 Shown, a kind of efficient iterative mechanism design system based on Hadoop cloud computing framework of the present invention comprises:

[0187] The first creation module 31 is used to create an application programming interface API for simplifying the writing of MapReduce iterative programs in the Hadoop cloud computing framework;

[0188] The second creation module 32 is used to call the application programming interface API to create a loop iteration main control module at the master node;

[0189] The cache index module 33 is used to call the application programming interface API to cache and index the data reused in the iteration process from the node;

[0190] The task scheduling fault-tolerant module 34 is used to design task scheduling and fault-tolerant mechanisms for the loop iteration main control module in the master node, including:

[0191] Design the task scheduling mechanism in the Hadoop cloud computing framework, includi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention belongs to the technical field of the Hadoop cloud computing framework research, specifically relates to iterative functional research design of the Hadoop cloud computing framework, and in particular relates to an efficient iterative mechanism design method based on the Hadoop cloud computing framework. The method comprises: designing an application programming interface API in the Hadoop cloud computing framework; calling the application programming interface API to create a loop iterative main body control module in a master node; calling the application programming interface API to cache and index the reuse data in the iterative process in a slave node; and designing a task scheduling and fault tolerant mechanism for the loop iterative main body control module in the master node. The present invention provides the efficient iterative mechanism design method based on the Hadoop cloud computing framework, and designs an E-Hadoop cloud computing framework with an efficient iterative mechanism.

Description

technical field [0001] The invention belongs to the technical field of Hadoop cloud computing framework research, in particular relates to the research and design of the iterative function of the Hadoop cloud computing framework, in particular to a design method of an efficient iteration mechanism based on the Hadoop cloud computing framework. Background technique [0002] The traditional Hadoop cloud computing framework does not support efficient iterative operations, it only implements this operation through a relatively inefficient means: first, artificially formulate a series of MapReduce tasks, and then carefully write the corresponding driver to guide the iteration of these tasks implement. Although this can solve the iterative operation in massive data processing or big data processing, it will cause two serious problems: 1) Even if most of the data used in the slave node does not change during the entire iteration process, each iteration is also It must be re-downlo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/48G06F9/44G06F11/20H04L29/08
CPCG06F9/4843G06F11/203H04L67/1034
Inventor 朱颢东李红婵南姣芬张志锋赵进超
Owner ZHENGZHOU UNIVERSITY OF LIGHT INDUSTRY
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com