Big data offline scheduling development platform and method of dynamic resource scheduling mechanism

Abstract

The invention relates to a big data offline scheduling development platform of a dynamic resource scheduling mechanism. The platform comprises a client, a resource distribution master node module and a plurality of task execution managers, wherein the client is respectively connected with the resource distribution main node module and the plurality of task execution managers, and each task execution manager is connected with the resource distribution main node module and is used for communicating with the resource distribution main node module at regular time, reporting the current server resource use condition and updating the resource use information of the task execution manager into a zookeeper node at regular time. By using the method, the single-point fault risk of the main node is obviously reduced, the problem that the main load of a working node in an old framework is used for monitoring the operation condition of a subprogram under a task but cannot be processed is solved. And if the task running machine has a problem, the task can be allocated to other machines and is restarted, so that the fault tolerance of the platform is greatly improved.

Description

technical field [0001] The invention relates to the technical field of resource scheduling, in particular to a large data offline scheduling development platform and method of a dynamic resource scheduling mechanism. Background technique [0002] With the rapid development of Internet companies, the computing requirements based on data-intensive applications continue to increase. The company considers factors such as server hardware resource utilization, operation and maintenance costs, and data sharing, and hopes to integrate various types of computing requirements. Deploy to a public cluster, let them share the resources of the cluster, and use the resources uniformly, and at the same time adopt a certain resource isolation mechanism to isolate each task, thus the scheduling development platform was born. However, traditional scheduling development platforms perform poorly in terms of scalability, reliability, and resource utilization. [0003] In the traditional scheduli...

AI Technical Summary

Problems solved by technology

This node has completed too many tasks, resulting in excessive resource consumption. When there are many user programs, it will cause a large memory overhead. Potentially speaking, it also increases the risk of user program distribution, which is common in the industry. It is concluded that this architecture can only support the upper limit of 4000 node hosts

[0004] On the slave node (Slave), it is too simple to simply use the number of tasks as the representation of resources, without considering the CPU and memory usage. If two tasks with large memory consumption are scheduled together, memory overflow will easily occur. question

Images