The invention provides a single-pixel high-speed super-resolution imaging device and a method based on a time domain stretching. The device includes: a femtosecond pulse laser, a coupler, a first photoelectric detector, an arbitrary waveform generator, a doped fiber amplifier, a single mode fiber, a photoelectric modulator, a first collimator, a first cylindrical lens, a first virtual imaging phase array, a first diffraction grating, a first plano-convex lens, a second plano-convex lens, a first microscope objective, an observation object, a second microscope objective, a third plano-convex lens, a fourth plano-convex lens, a second diffraction grating, a second virtual imaging phase array, a second cylindrical lens, a second collimator, a second photoelectric detector, a high-speed oscilloscope, and a data processing unit. The method produces a predefined two-dimensional structure illumination pattern by a spectral coding and a specific spatial position design of an optical element realized by the time domain stretching technology, and realizes a acquisition of continuous images by the single pixel detection and data processing unit of the photoelectric detector.