Hive task scheduling method, device, equipment and storage medium based on big data platform

A big data platform and task scheduling technology, applied in the field of data processing, can solve the problem of low task execution efficiency and achieve the effect of improving efficiency

Active Publication Date: 2022-08-02
ONE CONNECT SMART TECH CO LTD SHENZHEN
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] Embodiments of the present invention provide a HIVE task scheduling method, device, device, and storage medium based on a big data platform to solve the problem that the current HIVE task is not associated with its predecessor HIVE task, resulting in low task execution efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Hive task scheduling method, device, equipment and storage medium based on big data platform
  • Hive task scheduling method, device, equipment and storage medium based on big data platform
  • Hive task scheduling method, device, equipment and storage medium based on big data platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0032] The HIVE task scheduling method based on the big data platform provided by the embodiment of the present invention, the HIVE task scheduling method based on the big data platform can be applied such as figure 1 in the application environment shown. Specifically, the HIVE task scheduling method based on the big data platform is applied in the big data platform system, and the big data platform system includes: figure 1 The shown clien...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a HIVE task scheduling method, device, equipment and storage medium based on a big data platform. The method includes: obtaining an original HIVE task sent by a client, triggering a log program based on a startup file in the original HIVE task, and obtaining a task log table; obtaining a target HIVE task from a pending HIVE task; using a configuration file reading tool to read the target The configuration file in the HIVE task; if the reading is successful, the pre-task ID and the self-task ID contained in the configuration file of the target HIVE task are obtained; if the pre-task log carries the task completion label corresponding to the pre-task ID , the pre-HIVE task corresponding to the pre-task ID is successfully completed, and the business file in the target HIVE task is executed; if the business file is successfully executed, a task completion label is generated, and the task completion label is associated with its own task ID and stored in the target HIVE in the target task log corresponding to the task. This method can improve the efficiency of HIVE task execution.

Description

technical field [0001] The invention relates to the technical field of data processing, and in particular, to a HIVE task scheduling method, device, equipment and storage medium based on a big data platform. Background technique [0002] HIVE is a data warehouse tool based on Hadoop, which can map structured data files into a database table, and provides simple SQL query functions, which can convert SQL statements into MapReduce tasks for operation. The advantage is that the learning cost is low, and simple MapReduce statistics can be quickly implemented through SQL-like statements, without the need to develop special MapReduce applications, which is very suitable for statistical analysis of data warehouses. HIVE is a data warehouse infrastructure built on Hadoop. It provides a set of tools that can be used to perform Extract Transform Load (ETL), a mechanism for storing, querying and analyzing large-scale data stored in Hadoop. HIVE defines a simple SQL-like query languag...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/28H04L67/30H04L67/01
CPCG06F16/283H04L67/30H04L67/01
Inventor 王睿之
Owner ONE CONNECT SMART TECH CO LTD SHENZHEN
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap