The invention relates to an image fusion system based on a low-light and infrared sensor. The system is divided into a low-light and infrared sensor information acquisition module, an FPGA core processing module, a DSP co-processing module and an output display module; video images collected by the external low-light and infrared sensor are input to a device through an interface; the input video images are subjected to image preprocessing including image denoising, image enhancement and image registration, wherein image denoising and image enhancement are low in delay requirement and are achieved in an FPGA chip, the computation amount required for an image registration algorithm is large, and image registration is achieved in a DSP chip; the pre-processed video images are fused through the FPGA chip, wherein grayscale fusion and pseudo-color fusion are included; an FPGA control logic can select one path of images to be output through the interface, wherein one path of images include asingle low-light video, a single infrared video, a grayscale fused video and a pseudo-color fused video. The image fusion system is high in precision, low in delay and good in expansibility and has important practical significance on nighttime observation and exploration application.