System and method for generation of unseen composite data objects

Abstract

A computer implemented system for generating one or more data structures is described, the one or more data structures representing an unseen composition based on a first category and a second category observed individually in a training data set. During training of a generator, a proposed framework utilizes at least one of the following discriminators—three pixel-centric discriminators, namely, frame discriminator, gradient discriminator, video discriminator; and one object-centric relational discriminator. The three pixel-centric discriminators ensure spatial and temporal consistency across the frames, and the relational discriminator leverages spatio-temporal scene graphs to reason over the object layouts in videos ensuring the right interactions among objects.

Description

CROSS-REFERENCE[0001]This application is a non-provisional of, and claims all benefit, including priority, of U.S. Application No. 62 / 822,517, filed 22 Mar. 2019, entitled “SYSTEM AND METHOD FOR GENERATION OF UNSEEN COMPOSITE DATA OBJECTS”, incorporated herein by reference in its entirety.INTRODUCTION[0002]Recent successes in the field of image and video generation using generative models are promising. Visual imagination and prediction are components of human intelligence. Arguably, the ability to create realistic renderings from symbolic representations are considered prerequisite for broad visual understanding.[0003]While most approaches focus on the expressivity and controllability of the underlying generative models, their ability to adapt (e.g., generalize) to unseen scene compositions has not received as much attention. However, such ability to adapt is an important cornerstone of robust visual imagination as it demonstrates the capacity to reason over elements of a scene.[00...

AI Technical Summary

Benefits of technology

[0025]The zero-shot setting is particularly important for adaptation in data object (e.g., human activity video) generation, obviating the need to observe every possible characteristic (e.g., action-object) combination in training and thus avoiding the combinatorial explosion involved in modeling complex scenes. Training costs can thus be lowered and less cycles are required for training, which is useful where there is limited computational power, storage, or time available for training.

Problems solved by technology

While most approaches focus on the expressivity and controllability of the underlying generative models, their ability to adapt (e.g., generalize) to unseen scene compositions has not received as much attention.

However, image datasets cannot incorporate the dynamics of interactions with objects in videos which is a more realistic setting.

Nonetheless, even such relatively large datasets would involve a small subset of objects that humans interact with in everyday lives.

However, training instability in GANs makes it difficult to generate high resolution images and models such as WGAN and LSGAN.

Extending existing generative modeling efforts (both GANs and VAEs) to videos is not straightforward since generating a video would involve modeling of both spatial and temporal variations.

While these models predict future frames, they models have limited accuracy in the case of long duration sequences that possess high spatio-temporal variability.

As described in some embodiments herein, although an approach conditions the generation of the video on an observed frame, the problem is substantially different since the input frame is used to provide background information to the networks during video generation instead of predicting few future frames.

Additionally, these methods are designed for cases in which the missing areas are small and have limited capacity when the video has a full frame or a sequence of frames is missing.

However, these methods are heavily driven by the spatio-temporal content of the given video (with missing frames / regions).

Training for unseen compositions is an important technical problem to be solved in machine learning, as there will not always be training data that covers a particular composition.

This problem is compounded as the number of possible dimensions for a composition increases (e.g., “wash”“eggplant” in “kitchen” during “afternoon”), or where training data is expensive to generate or obtain.

Generating new composite data objects that representing an unseen composition is technically challenging for a machine learning system.

Images