Distributed heterogeneous processing framework construction method and system

Abstract

The invention discloses a distributed heterogeneous processing framework construction method. The method comprises the following steps that a publishing module sends request data and blocks a currentthread by subscribing a first message; a service scheduling module receives the request data, encapsulates the request data into an intermediate file package, writes the intermediate file package intothe cache module and blocks the current thread by subscribing a second message; a calculation engine module calls the intermediate file package, loads and processes corresponding data according to the configuration data, and stores the processed final file package to the cache module; the calculation engine module issues a second message to the service scheduling module; the service scheduling module receives the second message, identifies a second end mark and cancels subscription of the second message; the service scheduling module issues a first message to the issuing module; the publishing module receives the first message, identifies a first end mark and cancels subscription of the first message; the publishing module reads the final file package from the cache module to obtain a data processing result. According to the invention, the real-time processing efficiency on the data line and the message storage capability are improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a method and system for constructing a distributed-based heterogeneous processing framework. Background technique [0002] With the continuous development and growth of the Internet industry, the amount of data to be processed and stored by the website every day increases exponentially, and the scale of the website is also expanding. The traditional vertical application architecture is becoming more and more bloated, difficult to maintain, high latency, and low throughput. It is difficult to meet the underlying architecture requirements of the big data information age. Using distributed architecture for data storage and computing is an inevitable trend in the development of the Internet era. At the same time, it is necessary to have efficient real-time processing efficiency on data lines and message storage capabilities. The architecture to fill the technical need...

AI Technical Summary

Problems solved by technology

[0002] With the continuous development and growth of the Internet industry, the amount of data to be processed and stored by the website every day increases exponentially, and the scale of the website is also expanding. The traditional vertical application architecture is becoming more and more bloated, difficult to maintain, high latency, and low throughput. It is difficult to meet the underlying architecture requirements of the big data information age. Using distributed architecture for data storage and computing is an inevitable trend in the development of the Internet era. At the same time, it is necessary to have efficient real-time processing efficiency on data lines and message storage capabilities. architecture to fill today's market technology needs

Images