A Method for Knowledge Mining and Parallel Processing of Massive Traffic Data

Abstract

The invention discloses a massive traffic data knowledge mining and parallel processing method. The LSTM model is stored in N computing servers of a distributed cluster, and the training data set is divided into N sets and input to the N computing servers respectively. Each computing server simultaneously Training, which can reduce the training time of the LSTM model. After the calculation server completes a forward propagation and back propagation, it transmits the parameter matrix to the parameter server, and uses the matrix decomposition method to decompose it into the form of multiplying two matrices. The total amount of parameters is reduced, and the communication time is reduced. Set the matrix compression rate, and use the adaptive threshold filtering method to compress the matrix, the number of parameters is reduced again, and the communication time is reduced again. Calculate the error matrix before and after matrix compression and pass it to the next round of training. At the beginning of the next round of training, first use the parameter matrix and error matrix to calculate the matrix after eliminating the error, and then perform training to make up for the error caused by matrix compression. LSTM model accuracy.

Description

technical field [0001] The invention relates to the technical field of traffic big data, in particular to a method for knowledge mining and parallel processing of massive traffic data. Background technique [0002] In the context of the continuous improvement of the informatization level of my country's transportation system, massive traffic big data is also continuously generated. These traffic big data have the characteristics of multi-source heterogeneity and low knowledge density. How to analyze and utilize these traffic big data and carry out knowledge mining on traffic big data, so as to play a guiding role in traffic business, is a problem that needs to be solved urgently. [0003] Taking my country's civil aviation industry as an example, with the rapid development of my country's civil aviation industry, the passenger volume of airports has increased year by year. According to the statistics of the Civil Aviation Administration of my country, in 2019, the national p...

AI Technical Summary

Problems solved by technology

[0005] In view of this, the present invention provides a method for knowledge mining and parallel processing of massive traffic data, which is used to make up for the fact that the existing large-scale neural network for predicting flight track points takes too long in the training process and cannot effectively perform online incremental prediction lack of

Images