Distributed file storage system and method based on Nginx + MinIO + Redis

Abstract

The invention relates to the technical field of data storage, and particularly discloses a distributed file storage system and method based on Nginx + MinIO + Redis, and the system comprises a database and middleware; the middleware comprises a picture processing module and a picture processing module, wherein the picture processing module is used for carrying out thumbnail operation on a picture on the basis of an http - image - filter - module in Nginx; the abbreviation operation comprises zooming the picture according to the specified width and height; the storage module is used for carrying out distributed storage on the file data based on the MinIO cluster; the storage module comprises a file uploading unit, a cache compression unit and a cache replacement unit; the query module is used for executing file query operation on the distributed cached files based on the Redis cluster; and the deletion module is used for executing deletion operation on the distributed file. The invention provides a file compression caching strategy based on Thumbnalator, Base64 and Gzip compression algorithms, the problems that the length of a character string is too long and the occupied memory space is large when the Redis caches the file are solved, and the utilization rate of the memory space is improved.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to a distributed file storage system and method based on Nginx+MinIO+Redis. Background technique [0002] With the rapid development of informatization, files from all walks of life are stored in this process, and unstructured data and structured data are growing at an explosive rate. When the scale of data becomes larger and larger, the access performance drops sharply; as a lightweight open source distributed file system, MinIO is suitable for storage of small and medium files, but MinIO stores data on disk, and each access needs to The disk reads data, and multiple IOs cause query performance to decrease. In order to reduce the number of disk accesses and improve the reading speed of files, the optional solution is to introduce Redis as a file cache service, but Redis is a memory-based database. Using Redis to cache files directly will cause excessive memory space usage and...

AI Technical Summary

Problems solved by technology

When the data scale becomes larger and larger, the access performance drops sharply; as a lightweight open source distributed file system, MinIO is suitable for the storage of small and medium-sized files, but MinIO stores the data on the disk, and each access needs to be accessed from the The disk reads data, and multiple IOs cause a decrease in query performance

In order to reduce the number of disk accesses and improve the file reading speed, an optional solution is to introduce Redis as a file cache service, but Redis is a memory-based database, and directly using Redis to cache files will cause excessive memory space occupation and low query performance Therefore, it cannot be applied to large-scale file data; in addition, when using Redis as a cache service, Redis provides a cache elimination strategy, which is not suitable for specific businesses, and the cache query hit rate is not high

Images