Method and system for realizing real-time transmission processing of large file based on redis

Abstract

The invention relates to the technical field of computers, and discloses a redis-based large file real-time transmission processing method and system. The method mainly comprises the steps that a sending unit detects a to-be-processed file list of a local directory, after a large file is detected, the file is virtually paged, and the sending unit allocates uploading tasks of all thread units, reads specified page content and sends the specified page content to a redis server to be stored; a receiving unit detects whether a first ordered set serving as a large file name list in the redis serveris updated or not: if yes, detecting newly-added pages in a second ordered set serving as the content of each page of the large file, and putting the content of the newly-added pages into a cache queue; acquiring cache queue data, and performing real-time processing until the data of the cache queue is completely processed; monitoring the whole file transmission state, and if the transmission state is abnormal, carrying out retransmission. The method and the system provided by the invention not only can realize real-time transmission of large files, but also can carry out real-time processing.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a method and system for real-time transmission and processing of large files based on redis. Background technique [0002] There are two traditional cross-network file processing methods: 1. Use the shared network method to directly write large files into the shared directory, and both the sender and the sender can access through the shared method, but this method is low in security; 2. Use the FTP method , upload large files to the FTP server through FTP, and the receiving end judges whether there are new files by periodically checking the FTP directory, and checks whether the upload is complete. Only when the upload is completed can the processing begin. This method is more secure than method 1, but the processing timeliness is relatively low, especially when the file is very large (tens of hundreds of gigabytes or even larger), it will take a long time to wait until the enti...

AI Technical Summary

Problems solved by technology

[0002] There are two traditional cross-network file processing methods: 1. Use the shared network method to directly write large files into the shared directory, and both the sender and the sender can access through the shared method, but this method is low in security; 2. Use the FTP method , upload large files to the FTP server through FTP, and the receiving end judges whether there are new files by periodically checking the FTP directory, and checks whether the file is uploaded. Only when the upload is completed can the processing be started

This method is more secure than method 1, but the processing timeliness is relatively low, especially when the file is very large (tens of hundreds of gigabytes or even larger), it will take a long time to wait until the entire file is uploaded, which is very difficult for real-time processing systems Obviously it can't meet the demand, and when the regular FTP upload encounters a network interruption, it cannot resume the upload from the breakpoint

Images