Big data processing framework-based task classification processing method and computer device

A technology of big data processing and processing methods, applied in software design, etc., can solve problems such as large and bloated clients, and achieve the effects of saving development resources, simplifying complexity, and improving development efficiency

Pending Publication Date: 2018-08-28
福建星瑞格软件有限公司
View PDF3 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

If you want to use different processing frameworks for the same type of tasks in an application system, you need to implement each processing framework one by one, and when submitting tasks, you need to involve a lot of information about the execution framework. The support of the data processing framework, so that the client is relatively large and bloated

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data processing framework-based task classification processing method and computer device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0021] The task classification processing method based on the big data processing framework of the present invention includes:

[0022] Step 1, classify all types of big data platform request tasks, the classification types include Query tasks, OLAP tasks, Stream tasks, Search tasks and other tasks;

[0023] Step 2. According to the type of the requested task, call the task interface to submit the requested task data to the big data platform; call the corresponding task interface of the big data framework according to the requested task data;

[0024] Step 3, the big data processing framework receives the request task data, and constructs a big data processing framework context;

[0025] Step 4. Request the source data required by the task, execute the task, encapsulate the output result according to the specified encapsulation type, and output the result.

[0026] The computer equipment of the present invention includes a memory, a processor, and a computer program stored on...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a big data processing framework-based task classification processing method. The method comprises the steps of classifying all types of big data platform request tasks; according to the types of the request tasks, submitting request task data to a big data platform by calling task interfaces; calling the corresponding task interfaces of a big data framework according to therequest task data; receiving the request task data by the big data processing framework, and constructing a context of the big data processing framework; and requesting for source data required by thetasks, executing the tasks, and outputting results. The invention provides a computer device. The development efficiency of big data platform applications is improved; and a large amount of development resources are saved.

Description

technical field [0001] The invention relates to a task classification processing method and computer equipment based on a big data processing framework. Background technique [0002] The big data processing framework is responsible for the framework system that calculates (manages and processes) the data in the big data system. Data includes data read from persistent storage or data connected to the system through message queues, etc., while computing is the process of extracting information from data. Now each big data computing framework supports a variety of big data computing types. For example, SQL operations that support Query are basically supported by every big data computing framework. Even if there is no direct support, it can also be implemented in an indirect way. Then it means that a type of big data computing request can be sent to multiple big data processing frameworks for execution, but the execution efficiency and method of each framework are different. At...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F8/20
CPCG06F8/20
Inventor 卞信铨
Owner 福建星瑞格软件有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap