Method and device for constructing Hadoop application development frame, and electronic device

Abstract

The embodiment of the invention discloses a method and a device for constructing a Hadoop application development frame, and an electronic device. The method comprises the following steps: separating a command line to obtain task scheduling and task parameters used for setting a template configuration file, wherein the template configuration file comprises a scheduling configuration file used for packaging separated scheduler tasks and a task list configuration file used for packaging separated task parameters; setting a task scheduling sequence which carry out visual editing in the scheduling configuration file; setting a visual editing task parameter in the task list configuration file, and constructing a mapping relationship between the scheduler tasks in the scheduling configuration file and the task parameters; and setting a task parameter detection file which starts the task in the scheduling configuration file and detects the validity of the task list configuration file, and constructing the Hadoop application development frame on the basis of the template configuration file and the task parameter detection file. The method and the device can be applied to improve the construction efficiency of the Hadoop application development frame.

Description

Technical field [0001] The present invention relates to computer application technology, in particular to a method, device and electronic device for constructing a Hadoop application development framework. Background technique [0002] The Hadoop application development framework is a software framework that uses distributed file system (HDFS, HadoopDistributedFileSystem) technology to perform distributed processing of large amounts of data (tasks). By maintaining multiple copies of data (tasks), it can ensure that it can deal with failures. The data is redistributed and processed, so it has high fault tolerance and can provide high throughput for data (task) access, which is widely used in application scenarios that require very large data sets. Using the Hadoop application development framework, users can develop distributed programs without understanding the underlying details, so that they can make full use of the high throughput of the Hadoop application development framewor...

AI Technical Summary

Problems solved by technology

In this way, since the number of tasks involved in building the Hadoop application development framework is generally large, when technical developers need to implement multiple tasks, they need to independently write the command line corresponding to the task and fill in the corresponding task parameters, even for the same or similar tasks. The command line also needs to be written or copied one by one, making the work of realizing the task more repetitive and cumbersome, and each built Hadoop application development framework can only be applied to one Hadoop application development, resulting in the construction of Hadoop application development framework and Hadoop-based The efficiency of the application development framework for Hadoop application development is low, and due to the repetitive command line writing method and the need for technical developers to fill in the task parameters corresponding to multiple tasks one by one based on memory or reference data, it is easy to cause command line writing and task Parameter filling errors, for example, omission of one or more task parameters in the task, or adding task parameters that do not exist in the task, so that when the subsequent task processing is performed, the processing is terminated due to an error, which also leads to the development of the built Hadoop application. The framework is inefficient

Images