Video highlight detection with pairwise deep ranking

Abstract

Video highlight detection using pairwise deep ranking neural network training is described. In some examples, highlights in a video are discovered, then used for generating summarization of videos, such as first-person videos. A pairwise deep ranking model is employed to learn the relationship between previously identified highlight and non-highlight video segments. This relationship is encapsulated in a neural network. An example two stream process generates highlight scores for each segment of a user's video. The obtained highlight scores are used to summarize highlights of the user's video.

Description

Background technique [0001] The advent of wearable devices such as camcorders and smart glasses has made it possible to document life, recorded in first-person video. For example, wearable camcorders such as Go-Pro cameras and Google Glass are now capable of capturing high-quality first-person video to document our everyday experiences. These first-person videos are often very unstructured and long-running. Browsing and editing such videos is a very tedious job. Video summarization applications can generate short summaries of full-length videos that encapsulate the most informative parts, alleviating many of the problems associated with first-person video browsing, editing, and indexing. [0002] Research on video summarization is mainly carried out along two dimensions, namely, keyframe or shot-based methods and structure-driven methods. Keyframe- or shot-based methods select a collection of keyframes or shots by optimizing the diversity or representativeness of the summar...

AI Technical Summary

Problems solved by technology

This limitation is especially severe when directly applying these methods to first-person videos, which are recorded in unconstrained settings, making them lengthy, redundant, and unstructured.

Images