Mass traffic data knowledge mining and parallel processing method

Abstract

The invention discloses a mass traffic data knowledge mining and parallel processing method. The method is characterized in that an LSTM model is stored in N computing servers of a distributed cluster, a training data set is divided into N sets which are respectively input into the N computing servers, and the computing servers are trained simultaneously, so that the training time of the LSTM model can be shortened. A parameter matrix is transmitted to a parameter server after a calculation server finishes one-time forward propagation and back propagation, and the parameter matrix is decomposed into a form of multiplying two matrixes by adopting a matrix decomposition method, so that the total quantity of parameters is reduced, and the communication time consumption is reduced. A matrix compression rate is set, the matrix is compressed by using an adaptive threshold filtering method, so that the number of parameters is reduced again, and the communication time consumption is reduced again. An error matrix before and after matrix compression is calculated and is transmitted the error matrix to the next round of training, the matrix after error elimination is calculated by using the parameter matrix and the error matrix at the beginning of the next round of training, and then training is performed, so that the error caused by matrix compression can be made up, and the precision of the LSTM model is ensured.

Description

technical field [0001] The invention relates to the technical field of traffic big data, in particular to a method for knowledge mining and parallel processing of massive traffic data. Background technique [0002] In the context of the continuous improvement of the informatization level of my country's transportation system, massive traffic big data is also continuously generated. These traffic big data have the characteristics of multi-source heterogeneity and low knowledge density. How to analyze and utilize these traffic big data and carry out knowledge mining on traffic big data, so as to play a guiding role in traffic business, is a problem that needs to be solved urgently. [0003] Taking my country's civil aviation industry as an example, with the rapid development of my country's civil aviation industry, the passenger volume of airports has increased year by year. According to the statistics of the Civil Aviation Administration of my country, in 2019, the national p...

AI Technical Summary

Problems solved by technology

[0005] In view of this, the present invention provides a method for knowledge mining and parallel processing of massive traffic data, which is used to make up for the fact that the existing large-scale neural network for predicting flight track points takes too long in the training process and cannot effectively perform online incremental prediction lack of

Images