The invention discloses a signal processing method of an interactive three-dimensional video system. The signal processing method comprises the following steps of: firstly, taking multiple virtual viewpoints between each two adjacent reference viewpoints as auxiliary viewpoints, obtaining the residual image of each auxiliary view point, then encoding each color image, each depth image and each residual image by using a three-dimensional video coding module of a server, transmitting each color image, each depth image and each residual image to a user side, decoding each coded image by using a viewpoint decoding module of the user side, finally generating reference viewpoint signals and auxiliary viewpoint signals by any viewpoint drawing modules, and transmitting the reference viewpoint signals and the auxiliary viewpoint signals to a video display module for display. The signal processing method has the advantages that as the server only needs to transmit the residual images containing less information content as the auxiliary viewpoint information, the coding complexity of the system server is reduced, and as virtual viewpoint signals can be generated at the server only through simple three-dimensional mapping and add operation, the requirements on user side systems are reduced, thus the complexity of the interactive three-dimensional video system is reduced.