HIVE task scheduling method, device and equipment based on big data platform and storage medium

A big data platform and task scheduling technology, applied in the field of data processing, can solve the problem of low task execution efficiency and achieve the effect of improving efficiency

Active Publication Date: 2019-07-30
ONE CONNECT SMART TECH CO LTD SHENZHEN
View PDF7 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Embodiments of the present invention provide a HIVE task scheduling method, device, device, and storage medium based on a big data platform to solve the problem that the current HIVE task is not associated with its predecessor HIVE task, resulting in low task execution efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • HIVE task scheduling method, device and equipment based on big data platform and storage medium
  • HIVE task scheduling method, device and equipment based on big data platform and storage medium
  • HIVE task scheduling method, device and equipment based on big data platform and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0032] According to the HIVE task scheduling method based on the big data platform provided by the embodiment of the present invention, the HIVE task scheduling method based on the big data platform can be applied as figure 1 shown in the application environment. Specifically, the HIVE task scheduling method based on the big data platform is applied in the big data platform system, and the big data platform system includes such as figure 1 As shown in...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an HIVE task scheduling method and device based on a big data platform, equipment and a storage medium. The method comprises the steps of obtaining an original HIVE task sent by a client, triggering a log program based on a starting file in the original HIVE task, and obtaining a task log table; obtaining a target HIVE task from the to-be-processed HIVE task; reading a configuration file in the target HIVE task by adopting a configuration file reading tool; if the reading is successful, obtaining a front task identifier and a self task identifier contained in a configuration file in the target HIVE task; if the front task log carries a task completion label corresponding to the front task identifier, successfully completing the front HIVE task corresponding to the front task identifier, and executing a business file in the target HIVE task; and if the business file is successfully executed, generating a task completion tag, and storing the task completion tag and the task identifier of the task completion tag in a target task log corresponding to the target HIVE task in an associated manner. According to the method, the HIVE task execution efficiency can beimproved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a HIVE task scheduling method, device, equipment and storage medium based on a big data platform. Background technique [0002] HIVE is a data warehouse tool based on Hadoop, which can map structured data files into a database table, and provides a simple sql query function, which can convert sql statements into MapReduce tasks for operation. Its advantage is that the learning cost is low, and simple MapReduce statistics can be quickly realized through SQL-like statements, without the need to develop special MapReduce applications, which is very suitable for statistical analysis of data warehouses. HIVE is a data warehouse infrastructure built on Hadoop. It provides a set of tools that can be used to extract, transform and load (ETL), a mechanism that can store, query and analyze large-scale data stored in Hadoop. HIVE defines a simple SQL-like query language ca...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/28H04L29/06H04L29/08
CPCG06F16/283H04L67/30H04L67/01
Inventor 王睿之
Owner ONE CONNECT SMART TECH CO LTD SHENZHEN
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap