Automatic deployment method and device for big data nodes under cloud computing environment

Abstract

An embodiment of the invention discloses an automatic deployment method and device for big data nodes under a cloud computing environment. The method comprises steps as follows: a preset virtual machine template is called, and a big data operation environment is established; big data products are subjected to assembly packaging, and a big data assembly of a cloud computing management platform is obtained; the big data assembly is operated through a uniform interface in the big data operation environment. According to the automatic deployment method and device, a uniform interface protocol is defined in different big data products, a uniform deployment, monitoring, maintenance and operation page is provided for big data, the big data maintenance and deployment difficulty and workload under the cloud computing environment are reduced, version compatibility between nodes is improved, operation time is shortened, capacity requirements for operators is reduced, and manual misoperation is avoided.

Description

technical field [0001] The invention relates to the technical field of cloud computing, in particular to a method and device for automatically deploying large data nodes in a cloud computing environment. Background technique [0002] Common components of the big data product Hadoop system Zookeeper, DataNode, NameNode, Hive, Hbase. To install and deploy in a cloud computing environment, you need to download the source code, compile it manually on each virtual machine and copy it to the corresponding directory, modify the system variables and hadoop configuration files according to the environment, start it manually, execute some initialization scripts and adjust the operating parameters. The whole steps are compared Complex and cumbersome. [0003] Hadoop is an open source product, and apache does not provide the page deployment function. In the current situation, manual deployment is basically required. The most basic HDFS structure is as follows: figure 1 shown. Such as...

AI Technical Summary

Problems solved by technology

[0005] (1) There are at least dozens of nodes in the big data cluster, and all configuration files are manually modified, which is difficult to operate in batches, time-consuming, and error-prone; The operation also takes a lot of time, and the group with more nodes takes more time;

[0006] (2) The version of open source products is upgraded quickly. There are usually several major versions and more than a dozen minor versions within a year, and there are many modules involved in the entire cluster, so version management is difficult. The cluster runs for a period of time before adding nodes , if the version of the new node is different from other node versions, it will easily lead to version incompatibility;

[0007] (3) In a cluster with a large number of machines, the system often causes node abnormalities due to network, hard disk and other hardware reasons, which requires manual investigation, and hardware damage also requires personnel to be replaced. The number of personnel required is proportional to the number of nodes, because it is necessary for the operation The maintenance of the system requires a lot of manpower and has high requirements for the ability of the personnel

Images