Encoding and decoding selectively retrievable representations of video content

Abstract

A system and method disposed to enable encoding, decoding and manipulation of digital video with substantially less processing load than would otherwise required. In particular, one disclosed method is directed to generating a compressed video data structure that is selectively decodable to a plurality of resolutions including the full resolution of the uncompressed stream. The desired number of data components and the content of the data components that make up the compressed video data, which determine the available video resolutions, are variable based upon the processing carried out and the resources available to decode and process the data components. During decoding, efficiency is substantially improved because only the data components necessary to generate a desired resolution are decoded. In variations, both temporal and spatial decoding are utilized to reduce frame rates, and hence, further reduce processor load. The system and method are particularly useful for real-time video editing applications.

Description

PRIORITY AND CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application is a continuation of and claims priority to co-pending U.S. patent application Ser. No. 14 / 340,873, entitled ENCODING AND DECODING SELECTIVELY RETRIEVABLE REPRESENTATIONS OF VIDEO CONTENT filed Jul. 25, 2014, which claims priority to U.S. patent application Ser. No. 13 / 113,950, entitled ENCODING AND DECODING SELECTIVELY RETRIEVABLE REPRESENTATIONS OF VIDEO CONTENT filed May 23, 2011, now U.S. Pat. No. 9,171,577, which claims priority to U.S. patent application Ser. No. 10 / 779,335, entitled SYSTEM AND METHOD FOR ENCODING AND DECODING SELECTIVELY RETRIEVABLE REPRESENTATIONS OF VIDEO CONTENT filed Feb. 12, 2004, now abandoned, which claims priority to U.S. Provisional Patent Application Ser. No. 60 / 465,595, entitled REAL-TIME HIGH-DEFINITION VIDEO EDITING THROUGH SELECTIVE DATA RETRIEVAL FROM A COMPRESSED REPRESENTATION filed Apr. 25, 2003.[0002]This application is also related to U.S. patent application Ser. No...

AI Technical Summary

Benefits of technology

This patent describes a method for processing videos to create frames of lower resolution. The method involves converting each frame of the video into multiple data components, which can be combined to create a resultant frame with lower resolution. This allows for flexibility in deciding the display resolution for each frame. Additionally, the patent describes a storage structure that includes a lowest resolution data component and a plurality of high-frequency data components, which can be combined to produce higher resolution frames. The resultant frames created through this process have lower file sizes and improved video quality. The patent also includes computer programs that selectively apply effects or decode video data according to different display resolutions. The technical effects of this patent include increased flexibility in deciding display resolution for video frames and improved video quality with reduced file sizes.

Problems solved by technology

At a sustained rate of 100 MBps, standard computing solutions are no longer adequate.

Individual hard drives can only sustain about 30 MBps.

Due to these combined, limitations, real-time HD editing is limited to the most expensive and custom systems on the market.

As video resolutions increase, the demands on the system architecture increase.

Although hardware compression exists for use in distribution systems (e.g., satellite or terrestrial HD broadcasts), these tools do not meet production quality or architectural requirements.

In addition the CPU load for software-based compression using existing technology is very high.

Although software decompression can be used for single-stream HD playback, it taxes the CPU, which may already be overloaded by processing video mixes and effects.

As a consequence, multi-stream HD decoding and mixing is beyond the capabilities of the standard PC.

Further, software encoding (i.e., compression) is typically much more CPU-intensive than decoding (i.e., decompression); therefore expensive hardware is required for encoding during video acquisition.

As a consequence of the difficulties associated with video compression discussed above, commercial HD-based production systems typically use uncompressed video.

These setups are expensive, and without compression, large quantities of disk storage are required for any long-form production.

Because there are no known alternatives to using uncompressed content when implementing edits, many editing applications simply do not offer a real-time preview (i.e., before edits are actually carried out) of editing results.

Video rendering performs the same mixing and effect operations as required by real-time playback; however, the results can no longer be viewed live.

If it does need to be changed, then more rendering is required; thus, editing in a rendered-only editing environment can be very time consuming.

The drawback of scaling upon capture in this manner, however, is that the video must be recaptured at full resolution before the edits are actually implemented and the final quality production can be completed.

Although this approach is intended to make use of existing real time SD equipment to assist in HD editing, the HD to- SD resizing introduces an additional processing stage after decompression, making this approach unsuitable for software-only solutions.

As a hardware solution, this approach is very costly given that it requires either expensive compression chips or a system architecture with enough bandwidth (e.g., hundreds of MBps) and with enough disk capacity to store very large uncompressed HD video files.

To reduce CPU load, some video compression technologies have limited abilities to decode to a lower resolution.

Although modifications to the decoding procedure allow some reduction in CPU usage, the results do not offer both good image quality and reduced CPU load.

Once the CPU load exceeds the system capability, however, the playback of the video will stutter or stall, preventing audio synchronization or smooth motion.

Any compromise that trades performance for artifacts, like those seen in quarter resolution DV decoding, will not be desired by the user / editor.

In such conventional systems the previews typically stutter (non-smooth motion), and although they are not considered real-time, these systems do preserve audio synchronization by computing and presenting some frames at their correct display time.

This form of preview introduces temporal artifacting, another undesirable characteristic in video production.

The resolution of most of today's high-definition frames will not fit within the window of this editing environment, however, so the image is typically scaled down by the display device as part of today's editing process.

Images