Matrix calculation method of distributed large-scale matrix multiplication based on Spark

Abstract

The invention discloses a matrix calculation method of distributed large-scale matrix multiplication based on Spark. The method comprises the following steps of adopting a system framework based on a distributed computation execution engine Spark and single-machine BLAS (Basic Linear Algebra Subprograms); defining an operation interface related to a packaging matrix in a distributed system, reading a matrix file from a distributed file system, and selecting a proper scheme to execute the distributed multiplication for the quantity of resource of a distributed computing environment and the scale of a to-be-processed matrix; if the scales of the two matrixes are small, gathering to the locality to carry out single-machine multiplication; if the scale of one matrix is smaller, broadcasting the matrix execution multiplication; and if the scales of the two matrixes are large, adopting block-based distributed matrix multiplication. As for the latter two conditions, the invention respectively provides two efficient solutions, so that the problems of low performance and the poor expansibility of the existing big data processing platform about the distributed matrix operation are solved.

Description

technical field [0001] The invention relates to the technical field of parallel computing, in particular to a method capable of supporting large-scale distributed matrices for efficient distributed multiplication. Background technique [0002] With the advent of the Internet age, the scale of human data has grown tremendously. In the era of big data, matrix computing is one of the cores of many practical applications, such as social data analysis, web search, advertising computing, and recommendation systems. Especially the neural network learning that has gained attention in the field of deep learning, its core can be realized by multiplying multiple continuous matrices. However, in the face of increasing large-scale data, the traditional stand-alone matrix calculation cannot meet the basic needs due to the limitation of the hardware environment. Therefore, an efficient method that supports large-scale matrix multiplication is more needed. [0003] In recent years, with th...

AI Technical Summary

Problems solved by technology

[0006] Purpose of the invention: In order to overcome the defect that large-scale distributed matrix multiplication is poorly supported in the existing distributed data processing system, the present invention provides a matrix calculation method based on Spark's distributed large-scale matrix multiplication, which uses Users can perform efficient large-scale distributed parallel matrix multiplication calculations in an interactive or batch processing environment without knowing the specific implementation details of parallel computing, which solves the low performance of existing big data processing platforms for distributed matrix operations , the scalability is not good, the interface is not rich

Images