High quality upscaling and denoising are required in mobile imaging devices that do not contain high quality lenses. Such is also required in order to scale up standard-definition video content for display in high-definition television screens.The disclosed method uses contextual information obtained during upscaling and / or denoising of frames. Relevant correspondences between patches within a frame and between frames, are detected, managed and exploited. The correspondence information is simultaneously used and updated while video frames are being processed. Two approaches may be used: 1. keeping, searching for and updating a database of useful patches, by adding frequently visible similar patches, aggregating high-frequency, low-noise information associated with the similar patches, and removing less-observed patches; 2. Using the high-resolution and noise-reduced information that was collected from earlier video frames, and is expressed by the output of latest processed frame, for upscaling and / or noise-reducing the next processed frame.