Four-dimensional polynomial model for depth estimation based on two-picture matching

Abstract

Camera depth estimation is performed in response to picture matching based on blur difference computed between images captured at different focal positions. A blur difference model is stored in the camera based on characterization of the camera with a series of matching curves in which blur difference varies depending on the focal length, aperture, subject distance, and lens focus position. A four-dimensional polynomial model is created to fit the matching curves for use in estimating subject distance. During operation, images are captured for use in estimating subject distance. Motion compensation is applied and blur difference is determined. Blur difference is utilized in the polynomial model to estimate subject distance. Subject distance estimates can be output or utilized within an auto focus process to provide accurate focus adjustments.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]Not ApplicableSTATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT[0002]Not ApplicableINCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC[0003]Not ApplicableNOTICE OF MATERIAL SUBJECT TO COPYRIGHT PROTECTION[0004]A portion of the material in this patent document is subject to copyright protection under the copyright laws of the United States and of other countries. The owner of the copyright rights has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the United States Patent and Trademark Office publicly available file or records, but otherwise reserves all copyright rights whatsoever. The copyright owner does not hereby waive any of its rights to have this patent document maintained in secrecy, including without limitation its rights pursuant to 37 C.F.R.§1.14.BACKGROUND OF THE INVENTION[0005]1. Field of the Invention[0006]This invention perta...

AI Technical Summary

Benefits of technology

[0028]In at least one implementation, motion compensation is performed between the images, wherein at least the focus area of the first image is located as a fit within the second image, or alternatively, a portion of the second image is located as a fit within the focus area of the first image. Resultant blur difference computations are made more accurate in response to motion compensation as blur is more accurately rendered with static image content between the portions of the first and second image being compared after compensation. In at least one implementation, motion compensation is performed in response to one or more convolutions by a blur kernel to determine blur difference. In at least one implementation motion compensation is performed based on the relation(x^v,y^v)=argmin(xv,yv)fi(x,y)-fj(x-xv,y-yv),in which two images fi and fj are captured in a sequence, with i<j indicating image i is taken earlier in the sequence than image j, ∥.∥ is a norm operator that evaluates the matching error. The focus area of fi is compared with a shifted region of the same size from fj. The amount of shift (xV,yV) is up to e.g. ±5 pixels in horizontal and vertical directions. For all the shifting positions, one ({circumflex over (x)}V,ŷV) is found that yields the minimum matching error. The shifted image fjV(x,y)=fj(x-{circumflex over (x)}V,y-ŷV) represents the shifted fj with the region best matching the focus area of fi .

[0037]An element of the invention is an apparatus and method for accurately estimating subject distance in response to capturing two images at different focus settings (e.g., two picture matching).

[0038]Another element of the invention is the accurate estimation of subject distance despite motion arising between the images being compared.

[0042]Another element of the invention is a subject distance estimation apparatus and method which can properly estimate subject distance for different aperture settings (e.g., f-stop) and focal length (e.g., zoom) settings.

[0044]A still further element of the invention is that it can be implemented readily on a wide range of existing camera devices without the need of additional camera hardware.

Problems solved by technology

Passive systems often make poor focusing decisions in low contrast or low light conditions.

In addition, many focusing systems perform poorly when the subject is in motion, such as in regard to “action shots”, for example children at play, sports, animals in the wild, and so forth.

Images