Adaptive parallel processing method aiming at variable length characteristic extraction for big data

Abstract

The invention discloses an adaptive parallel processing method aiming at variable length characteristic extraction for big data. The method aims at variable-length characteristic data, and big data are processed through a graphic processing unit (GPU) parallel computing power based on compute unified device architecture (CUDA). During processing of the big data, an adaptive parallel matrix array processing mode is used for performing multithreading concurrent execution processing on the data according to own hardware characteristics and lengths of the characteristic data, so that the characteristic extraction speed is accelerated. According to the method, the data are processed in batches according to the hardware own processing capacity and the lengths of the characteristic data, and characteristics of certain length are extracted every time, and matching results are recorded; after the whole characteristics are extracted, all matching results are processed according to the allowed error-tolerant rate of data sampling to obtain requested characteristic extraction results. Good parallelism of matrix arrays is used, and the method aims at extracting variable-length characteristics, so that data can be parallelized effectively and sufficiently, and the method is particularly applicable to big-data rapid characteristic extraction with certain fault tolerance.

Description

technical field [0001] The invention belongs to the technical field of big data processing, relates to a feature extraction method, and more particularly relates to an adaptive parallel processing method for variable-length feature extraction oriented to big data. technical background [0002] With the advent of the big data era, how to quickly process big data and extract effective information has become a cutting-edge research hotspot in the IT industry. At present, feature extraction technology is more and more widely used in image processing, pattern recognition and network intrusion detection, especially in the big data environment, the flexibility and efficiency of variable length feature extraction has become a constraint on the ability to quickly process data bottleneck. [0003] According to the retrieval of existing patent materials, there are two main methods for feature extraction of big data: one is to increase the number of CPU cores or establish a distributed...

AI Technical Summary

Problems solved by technology

However, due to the limited number of CPU cores, the high cost of establishing a distributed cluster system, and the use of GPU processing is also restricted by hardware capabilities, limited to only processing feature data of fixed length, so the processing of feature extraction in the big data environment Methods and capabilities still need to be further innovated and improved

Images