Method and system for constructing heterogeneous processing framework based on distributed

Abstract

The invention discloses a method for constructing a distributed heterogeneous processing framework, which comprises the following steps: a publishing module sends request data and blocks the current thread by subscribing to the first message; a service scheduling module receives the request data and encapsulates it into an intermediate file package And write to the cache module and block the current thread by subscribing to the second message; the calculation engine module calls the intermediate file package and loads the corresponding data according to the configuration data and processes it, and stores the processed final file package in the cache module; the calculation engine module dispatches to the service The module publishes the second message; the service scheduling module receives the second message and recognizes the second end mark, unsubscribes the second message; the service scheduling module publishes the first message to the publishing module; the publishing module receives the first message and recognizes the first end mark, Unsubscribe the first message; the publishing module reads the final file package from the cache module to obtain the data processing result; the invention improves the real-time processing efficiency on the data line and the storage capacity of the message.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a method and system for constructing a distributed-based heterogeneous processing framework. Background technique [0002] With the continuous development and growth of the Internet industry, the amount of data to be processed and stored by the website every day increases exponentially, and the scale of the website is also expanding. The traditional vertical application architecture is becoming more and more bloated, difficult to maintain, high latency, and low throughput. It is difficult to meet the underlying architecture requirements of the big data information age. Using distributed architecture for data storage and computing is an inevitable trend in the development of the Internet era. At the same time, it is necessary to have efficient real-time processing efficiency on data lines and message storage capabilities. The architecture to fill the technical need...

AI Technical Summary

Problems solved by technology

[0002] With the continuous development and growth of the Internet industry, the amount of data to be processed and stored by the website every day increases exponentially, and the scale of the website is also expanding. The traditional vertical application architecture is becoming more and more bloated, difficult to maintain, high latency, and low throughput. It is difficult to meet the underlying architecture requirements of the big data information age. Using distributed architecture for data storage and computing is an inevitable trend in the development of the Internet era. At the same time, it is necessary to have efficient real-time processing efficiency on data lines and message storage capabilities. architecture to fill today's market technology needs

Images