Information management system and method based on big data

Abstract

The invention provides an information management method based on big data, and the method comprises the following steps: obtaining an input data stream, carrying out the association of image attribute data and text attribute data according to a data generation time sequence, and obtaining a plurality of groups of associated data streams; taking the current associated data stream as the input of a deep learning model; when the output result meets the first performance condition, starting a first video memory allocation process to allocate a first video memory space, and storing the next associated data stream in the first video memory space; when the video memory occupancy rate of the first video memory space exceeds a preset video memory capacity control threshold value, starting a second video memory allocation process to allocate a second video memory space, and storing the next associated data stream into the second video memory space; and judging whether the total capacity of the allocated video memory space exceeds a system control value or not, and if so, transferring the target associated data of the allocated video memory space to a memory component. The invention further discloses a system for implementing the method and a computer readable storage medium.

Description

technical field [0001] The invention belongs to the field of big data information management, and in particular relates to a big data-based information management method, a system for realizing the method, and a computer-readable storage medium. Background technique [0002] In today's data-expanding society, the value contained in massive data has become increasingly prominent. How to effectively mine the effective information in massive data has become a common problem faced by various fields. Technology companies represented by Internet companies have begun to apply a large number of algorithms such as machine learning, data mining, and artificial intelligence to obtain information contained in massive data according to their actual needs, and have achieved good results. Today's society has changed from the lack of information in the past to the era of information flood. Due to the continuous popularization of the network and related applications, network data is gradua...

AI Technical Summary

Problems solved by technology

The invention can solve the problems of low GPU resource utilization and poor job execution performance in the deep learning scenario of existing cluster resource scheduling methods

[0005] However, none of the existing technologies has solved the problem of how to obtain effective association data from massive big data streams as training data, and perform corresponding association of data for subsequent storage and reading.

Images