Cross-domain job flow scheduling method and system

Abstract

The invention discloses a cross-domain job stream scheduling method and system. The method comprises: selecting a job stream scheduling cluster A in a collaborative scheduling network to receive a data processing service demand sent by an application provider; arranging the job stream according to the logic of the data processing service demand, and dividing into a plurality of data service processing blocks; allocating the plurality of data service processing block a plurality of job stream scheduling clusters in the collaborative scheduling network for processing according to the arranging logic of the job stream; each of the plurality of job stream scheduling clusters processing the corresponding allocated data service processing block and generating data; and outputting the generated data to the predetermined job stream scheduling cluster through a federated data channel and storing the data in a corresponding database. The method and system achieve multi-engine parallel cross-domain data scheduling by adopting a non-master-slave cooperative scheduling network, and improve the processing speed and avoid the risk caused by single point failure through parallel processing.

Description

technical field [0001] The present invention relates to the field of communication technologies, and more particularly, to a method and system for scheduling a cross-domain job flow. Background technique [0002] With the increasing development of communication technology, the amount of data that needs to be processed in various businesses has exploded, and it has become more and more difficult to complete data processing operations only on local servers. Therefore, distributed systems emerge as the times require. The sharded data processed by the sharded server is processed in the server, so that each part of the data can be processed in parallel at the same time, which greatly speeds up the data processing capability; however, the traditional distributed data processing method still has some shortcomings: first, the traditional distributed data processing method The distributed system has an upper limit on the number of servers participating in joint scheduling, which make...

AI Technical Summary

Problems solved by technology

[0003] In order to solve the problems in the background technology that there is an upper limit on the amount of processed data, the single point of failure of the main server causes the risk of paralysis of the entire system, and the differences in development languages ​​between servers cause hidden dangers in compatibility and high maintenance costs, the present invention provides a cross- A domain job flow scheduling method and system, the method and system adopting non-main-standby mode to achieve multi-engine parallel cross-domain data scheduling; any one of the multiple scheduling clusters outputs job flows through business arrangement and distributes them to Each scheduling cluster in the collaborative scheduling network (that is, a distributed system) performs parallel processing; at the same time, the method and system adopt container technology, and there are no restrictions on the programming language and implementation of the job; the cross-domain job flow scheduling method includes :

Images