A method and system for real-time transmission and processing of large files based on redis

Abstract

The invention relates to the field of computer technology, and discloses a method and system for real-time transmission and processing of large files based on redis. The method in the present invention mainly includes: the sending unit detects the file list to be processed in the local directory, and when a large file is detected, the file is virtual-paged, the sending unit assigns each thread unit to upload tasks, and reads the content of the specified page and sends it to the redis server Store; the receiving unit detects whether there is an update in the first ordered set of the list of large file names in the redis server, and if there is an update, then detects the newly added page in the second ordered set as the content of each page of the large file, and puts the new The content of the added page is put into the cache queue; the data in the cache queue is obtained and processed in real time until the data in the cache queue is processed; the entire file transmission status is monitored, and if the transmission status is abnormal, retransmission is performed. The method and system provided by the invention can realize real-time transmission of large files and real-time processing.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and system for realizing real-time transmission and processing of large files based on redis. Background technique [0002] There are two main methods of traditional cross-network file processing: First, use the shared network method, directly write large files into the shared directory, and both the sender and the receiver can access through the shared method, but this method has low security; Second, use the FTP method , upload the large file to the FTP server through FTP, and the receiving end determines whether there is a new file by periodically detecting the FTP directory, and detects whether the file is uploaded. The security of this method is better than that of the first method, but the processing time is relatively low, especially when the file is very large (tens of hundreds of gigabytes or even larger), it takes a long time to wait until the entire file upl...

AI Technical Summary

Problems solved by technology

[0002] There are two traditional cross-network file processing methods: 1. Use the shared network method to directly write large files into the shared directory, and both the sender and the sender can access through the shared method, but this method is low in security; 2. Use the FTP method , upload large files to the FTP server through FTP, and the receiving end judges whether there are new files by periodically checking the FTP directory, and checks whether the file is uploaded. Only when the upload is completed can the processing be started

This method is more secure than method 1, but the processing timeliness is relatively low, especially when the file is very large (tens of hundreds of gigabytes or even larger), it will take a long time to wait until the entire file is uploaded, which is very difficult for real-time processing systems Obviously it can't meet the demand, and when the regular FTP upload encounters a network interruption, it cannot resume the upload from the breakpoint

Images