Distributed scheduling method and device based on ER relationship, equipment and storage medium

Abstract

The invention provides a distributed scheduling method and device based on an ER relationship, an electronic device and a computer storage medium. The distributed scheduling method based on the ER relationship is used for scheduling tasks in a big data platform, and comprises the following steps of S1, determining the range of calculation tasks needing to be scheduled, and generating an initial task set; S2, performing hierarchical arrangement on tasks in all the initial task sets based on an ER relationship; S3, estimating the memory and processor overhead of each task, calculating respectivescores through a resource overhead evaluation algorithm, and performing sorting according to the scores to generate an execution task set; and S4, allocating the tasks in the execution task set to aplurality of computing nodes of the big data platform to enable the computing nodes to execute respective tasks. According to the distributed scheduling method, allocation control and overhead measurement and calculation are carried out, and the problem of resource waste is solved.

Description

technical field [0001] The present invention relates to big data platform technology, in particular to a distributed scheduling method and device based on ER relationship for tasks of a big data platform, electronic equipment, and a non-transitory computer storage medium. Background technique [0002] As the scope of business development gradually expands, usually, the company's big data platform undertakes more and more data computing tasks, and gradually becomes an important pillar platform for data services. Taking the big data platform of an e-commerce company as an example, starting from the initial 30 data collection tasks and 20-30 calculation tasks of the report tasks of the e-commerce analysis center, gradually covering entrusted reports, financial statements, supply chain business, risk warning, GMV Nearly more than 400 data collection tasks and more than 700 data analysis and calculation tasks for the operation of daily newspapers and other businesses; at the same...

AI Technical Summary

Problems solved by technology

[0007] It cannot satisfy the rapid positioning and investigation of the operation of abnormal tasks in the overall range;

[0008] The fault analysis and location of the existing scheduler takes a long time, so the execution logs of all batches must be stored in order to have enough time to locate the fault information without being overwritten before the next scheduling start; the same hour level The logs of computing tasks will be stored 20+ times a day; therefore, the number of computing log files generated every day is about 15,000 or more; the daily log data volume is based on distributed storage and is greater than 1G. Therefore, there are many fragments and garbage files generated during operation. Maintenance High effort and input costs

Images