Rebinning methods and arrangements for use in compressing image-based rendering (IBR) data

Abstract

Rebinning methods and arrangements are provided that significantly improve the 3D wavelet compression performance of the image based rendering data, such as, e.g., concentric mosaic image data. Through what is essentially a selective cutting and pasting process the image data is divided into stripes that are then used to form a set of multi-perspective panoramas. The rebinning process greatly improves the performance of the cross shot filtering, and thus improves the transform and coding efficiency of 3D wavelet codecs. While the region of support after rebinning may cease to be rectangular in some cases, a padding scheme and an arbitrary shape wavelet coder can be implemented to encode the result data volume of the smart rebinning. With an arbitrary shape wavelet codec, the rebinning outperforms MPEG-2 by 3.7 dB, outperforms direct 3D wavelet coder by 4.3 dB, and outperforms a reference block coder (RBC) by 3.2 dB on certain tested concentric mosaic image scenes. Hence, the rebinning process nearly quadruples the compression ratio for selected scenes. Additional methods and arrangements are provided that include selectively dividing the image data into slits and rebinning the slits into a huge 2D array, which is then compressed using conventional still image codecs, such as, JPEG.

Description

TECHNICAL FIELD This invention relates to image-based rendering (IBR), and more particularly to certain rebinning methods and arrangements for use in compressing IBR data. BACKGROUND There is a continuing interest, within the computer graphics community, in image-based rendering (IBR) systems. These systems are fundamentally different from traditional geometry-based rendering systems, in that the underlying information (i.e., data representation) is composed of a set of photometric observations (e.g., digitized images / photographs) rather than being either mathematical descriptions of boundary regions or discretely sampled space functions. An IBR system uses the set of photometric observations to generate or render different views of the environment and / or object(s) recorded therein. There are several advantages to this approach. First, the display algorithms for IBR systems tend to be less complex and may therefore be used to support real-time rendering in certain situations. Sec...

AI Technical Summary

Benefits of technology

The above stated needs and other are met by various improved methods and arrangements that rearrange or rebin the image based rendering data prior to compression. As a result of the rebinning process, the compression ratios are significantly increased. The rebinning process may take on several forms, including, for example, a simple rebinning process, a smart rebinning process and a Rebinning Of SlitS (ROSS) process.

Thus, for example, in accordance with certain aspects of the present invention, a method is provided to support either a simple rebinning process or a smart rebinning process. The method includes selectively aligning at least two image based rendering (IBR) image data along a specific direction, and selectively rebinning the aligned IBR image data to form a multi-perspective panorama. Here, the IBR image data may include concentric mosaic (COM) image data and the specific direction may be substantially along a ho...

Problems solved by technology

Thirdly, the final rendered image may include both real photometric objects and virtual objects.

IBR systems can be complex, however, depending upon the level of detail required and the processing time constraints.

Accomplishing this in real-time, especially for a full spherical map or a large portion thereof, is typically beyond most computer's processing capability.

The COM technique, however, tends to generate and require a significant amount of data.

Moreover, transporting this amount of data, for example, over the Internet using a 56K baud modem is simply impractical.

Unfortunately, a 25 MB data file requires about one hour to download using a 56K baud modem.

However, one common problem with conventional 3D wavelet compression schemes is that the temporal wavelet filtering does not always achieve efficient energy compaction.

Unfortunately, local motion cannot be easily incorporated into the framework of conventional 3D wavelet compression schemes.

Unfortunately, each of these various proposed approaches tends to be complex, and in particular those of Ohm and Tham et al. tend to be very complex.

Their conclusion is that high dimensional wavelet coder is inferior in performance compared with a predictive based coder.

However, as indicated in this patent, it can be observed that misalignment of data is the major cause of compression inefficiency.

Images