Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Task information analysis method and device

A task information and analysis device technology, applied in the field of data processing, can solve the problems of insufficient information, related machines cannot be accessed by external network addresses, time and quantity restrictions, etc., to achieve the effect of improving work efficiency

Inactive Publication Date: 2016-12-07
LETV HLDG BEIJING CO LTD +1
View PDF3 Cites 17 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the logs saved on the logserver are limited in time and quantity
In addition, if you want to view logs, you need to configure the local hosts mapping of all machines in the cluster. When the cluster nodes change, you need to update the relevant configuration, which will cause a lot of inconvenience to data analysts.
Moreover, in many cases, due to security considerations, Hadoop-related machines cannot be accessed by external network addresses, so that information on specific distributed computing nodes cannot be seen.
In addition, the logs printed directly on the console by hive often have insufficient information and cannot be directly used to locate the cause of the error

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Task information analysis method and device
  • Task information analysis method and device
  • Task information analysis method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0076] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0077] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are to distinguish two entities with the same name but different parameters or parameters that are not the same, see "first" and "second" It is only for the convenience of expression, and should not be construed as a limitation on the embodiments of the present invention, which will not be described one by one in the subsequent embodiments.

[0078] At the beginning of this article, first briefly introduce the general operation process of the hive task, which mainly includes the following steps:

[0079] First, submit a task request to the Resourcemanager (resource manager, RM for short, which is the centr...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a task information analysis method and device. The method comprises the following steps of: monitoring the running status of a submitted task; if report error information of the task is monitored, obtaining a log of a report error task corresponding to the report error information; carrying out text analysis on the log of the report error task to obtain report error reason information of the report error task; and comparing the report error reason information with a pre-stored report error reason analysis table to obtain a report error analysis result corresponding to the report error reason information, wherein the report error reason analysis table comprises pre-stored report error reason information and report error analysis results corresponding to the pre-stored report error reason information. The task information analysis method and device disclosed by embodiments of the invention can automatically analyze the running statuses of the tasks without manual operation.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a task information analysis method and device. Background technique [0002] Hadoop is a distributed system infrastructure. Users can develop distributed programs without knowing the underlying details of the distribution, so as to make full use of the power of the cluster for high-speed computing and storage. Hadoop implements a distributed file system (Hadoop Distributed File System), referred to as HDFS. HDFS has the characteristics of high fault tolerance and is designed to be deployed on low-cost hardware; and it provides high throughput to access application data, suitable for applications with very large data sets. HDFS relaxes the requirements of POSIX (Portable Operating System Interface, Portable Operating System Interface), and can access data in the file system in the form of streams. The core design of the Hadoop framework is: HDFS and MapReduce (can be tra...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F11/07
CPCG06F11/0766G06F11/079
Inventor 刘光华
Owner LETV HLDG BEIJING CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com