File data feature extraction method and device and computing equipment

Abstract

The invention discloses a file data feature extraction method and apparatus, and a computing device. The method comprises the steps that data volume information of file data is read, the file data isfragmented according to the preset fragmented data volume, and the starting position and the stopping position of each file data fragment are determined; starting a corresponding number of threads according to the number of the fragments, allocating a to-be-read file data fragment to each thread, so that each thread starts to read line by line from the initial position of the allocated file data fragment and performs eigenvalue calculation until the reading position exceeds the cut-off position of the file data fragment; and summarizing calculation results of all threads to obtain a feature output result of the file data. According to the method, large file data does not need to be completely loaded into a memory, and the file reading speed is greatly increased in a segmented reading mode;moreover, the reading strategy provided by the invention does not occupy too much server resources, so that the calculation efficiency is greatly improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a file data feature extraction method and device, computing equipment, and computer storage media. Background technique [0002] In actual business, it is necessary to perform feature extraction on some file data with a large amount of data (hereinafter referred to as large file data), for example: to count the eigenvalues ​​of each column of data, the eigenvalues ​​include: average value, frequency distribution, etc. When performing statistics on the data, it is also necessary to consider filtering each row of data to filter out the row data that conforms to the rules, and use it as the statistics of the characteristic values ​​of the column data. [0003] The prior art provides a large file data feature extraction method using Python Panda data processing technology. In the specific implementation process, it is necessary to load all the large file data into th...

AI Technical Summary

Problems solved by technology

In the specific implementation process, it is necessary to load all the large file data into the memory for calculation. This full loading method has the problem of slow file reading speed; in addition, in the case of limited server resources, it will lead to serious shortage of system resources. , cannot be calculated at normal speed, which greatly affects the efficiency of eigenvalue calculation

Images