The invention provides a fast video inference method based on a cyclic residual module, a cyclic residual module, which reduces the complexity of neural network operation, improves the sparsity of intermediate feature mapping, and synthesizes efficient inference engine, is used in the method. The process is as follows: firstly, frame similarity is utilized in cyclic residual module to reduce theredundant calculation in frame-by-frame video convolutional neural network inference; then, by improving the sparse approximation output of the intermediate feature map, the inference speed is accelerated and the inference precision is ensured by the error control mechanism. Finally, the synthesis of efficient inference engine, that is, the use of matrix vector multiplication in the dynamic sparsity of the accelerator, so that the cycle residual module is in efficient operation. Compared with the traditional method, the invention can remarkably improve the running speed of the vision processing system, and enhance the understanding ability of the network to the real-time changing video fragments, thereby realizing the acceleration of the video inference process on the premise of ensuring the identification accuracy.