Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Method for fast constructing Hadoop cluster based on container cloud technology

A Hadoop cluster and cloud technology, which is applied in the field of rapidly building Hadoop clusters based on container cloud technology. It can solve problems such as time-consuming, low resource utilization, and increased use costs, optimize user experience, reduce system resources, and ensure stability. Effect

Inactive Publication Date: 2017-06-13
NANJING YUNCHUANG LARGE DATA TECH CO LTD
View PDF4 Cites 62 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, due to the large number of vendors and numerous release versions, OpenStack has various technical difficulties; first, OpenStack is far from enough in terms of integration, scalability, and stability; Nova, Swift, Cinder, and Neutron use their own database storage Configuration information, installation and upgrades are time-consuming and laborious;
[0005] Secondly, OpenStack lacks integrity; OpenStack can only provide three cloud products: computing, storage, and network. If you build a Hadoop cluster based on OpenStack, you need to get through the account, security, management operation and maintenance, and monitoring systems between OpenStack and Hadoop. The process is complicated and cumbersome;
[0006] Furthermore, OpenStack cannot provide end-to-end service guarantee; users only get scattered "framework" and components, and must manually integrate the functions of multiple vendors and multiple versions;
[0007] Finally, OpenStack lacks a common basic version; there are currently more than 20 customized versions of OpenStack that can be downloaded from manufacturers, and customers do not know which version to choose, let alone how to combine, mix and match and migrate between different versions according to their own needs ;The above defects lead to the invention patent disclosed by Jiangsu Internet of Things Research and Development Center. It takes a long time to install and implement the system, the coupling is also high, and the system stability cannot be guaranteed. optimize
[0008] Using multiple PCs or physical servers to build a Hadoop cluster has relatively high cost and low resource utilization; using virtual machines to provide the entire virtualization hardware layer, but the cost of use will also increase significantly, and resource utilization is also low; In addition, After using a PC or a virtual machine to build a Hadoop cluster, if the cluster environment is damaged during use, it is difficult to find the problem immediately; after the environment is found to be damaged, it is necessary to rebuild the environment of the damaged single or multiple servers or virtual machines, and deal with the problem. timeliness cannot be guaranteed

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method for fast constructing Hadoop cluster based on container cloud technology
  • Method for fast constructing Hadoop cluster based on container cloud technology
  • Method for fast constructing Hadoop cluster based on container cloud technology

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0063] Such as Figure 2-3 As shown, a method for quickly building a Hadoop cluster based on container cloud technology provided in this embodiment includes the following specific steps:

[0064] Determine the number of physical servers to be selected according to actual needs, regard these selected physical servers as a cluster, specify the master node of the cluster, and assign the actual deployment nodes, that is, as slave nodes;

[0065] Install the CentOS7 operating system for each physical server in the cluster;

[0066] Use the yuminstall command to install and start the container orchestration service Kubernetes and the network service Flannel on each master node of the cluster;

[0067] Install the MySQL database, mirror warehouse service and management portal on the pre-allocated master node;

[0068] Use the administrator account to log in to the management portal interface and create a user; use the established user to log in to the management portal to...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method for fast constructing Hadoop cluster based on container cloud technology. The method comprises the following steps: (1) determining selected physical server number according to actual demand, regarding the entire selected physical server as a cluster, and assigning a master node of the cluster, and distributing actual deployment nodes, namely as the slave node; (2) installing a CentOS7 operation system for each physical server in the cluster; (3) installing and starting a container orchestration service Kubernetes and the network service Flannel on each master node of the cluster by use of a yuminstall command; (4) installing MySQL database, a mirror image warehouse service and management portal; (5) logging a management portal interface by use of an administrator account, establishing a user; and logging the management user by use of the established user, creating the cluster environment of the user, and constructing a Hadoop cluster.

Description

technical field [0001] The invention relates to the technical field of creating Hadoop clusters, in particular to a method for rapidly building Hadoop clusters based on container cloud technology. Background technique [0002] At present, Hadoop clusters are usually built based on virtual machines or PCs; after setting the scale of the cluster in advance (including the number of master nodes and slave nodes, etc.), it is necessary to establish a specified number of virtual machines according to the preset scale or purchase a specified Number of PCs; after these servers are established, you need to install the operating system one by one, and install the JDK and other services necessary to build the Hadoop cluster, specify the host name of each server, and configure the SSH password-free between each two servers Authentication; finally, you need to deploy Hadoop on the master node, modify the configuration, and distribute it to each of the remaining servers in the cluster; af...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): H04L29/06H04L29/08G06F9/445G06F17/30
CPCH04L67/10G06F8/61G06F8/63G06F16/252G06F16/284H04L67/1001H04L67/01
Inventor 刘鹏张真朱光耀谢超董广明吴荣荣沈大为戎新堃
Owner NANJING YUNCHUANG LARGE DATA TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products