Method and device for detecting human-object interaction relationship in video

Abstract

The invention discloses a method and a device for detecting a human-object interaction relationship in a video, and the method comprises the steps: extracting a space-time track of a subject and an object in the video through target track detection, recognizing a human-object interaction relationship HOI through interaction joint reasoning based on a result of target track detection, and extracting multimode features from a target space-time track through interaction joint reasoning, performing joint reasoning by using a multi-feature fusion mode according to human and object fusion semantic features, human body behavior visual description features and human and object space-time relative motion features, and predicting human and object interaction actions on a video segment to obtain a predicted interaction category label, namely a human and object interaction relationship. The invention provides a time-space domain-based human-object interaction detection method, which successfully discovers, locates and classifies human-object pairs and interaction relationships between the human-object pairs in a video by capturing advanced semantic information of a scene from a complex video vision scene.

Description

technical field [0001] The invention belongs to the video information retrieval in the field of computer technology, relates to the relationship detection between objects in the video, and is used for discovering, locating and classifying the person-object pairs in the video and the interactive relationship between them, and is a kind of person-object in the video Object interaction relationship detection method and device. Background technique [0002] Human-object interaction HOI (Ref. [1]) is crucial for understanding human-centric video content and benefits many multimedia applications, such as video captioning (Ref. [2,3]), multimodal dialogue (Ref. [4]) and visual question answering (refs [5,6]). ImgHOID has proposed many effective detection methods for human-object interaction detection on images (references [7,8]), but these methods cannot mine the temporal information and dynamic visual cues in the video. Interaction detection HOID is not effective enough. [000...

AI Technical Summary

Problems solved by technology

[0029] The problem to be solved by the present invention is: capture the high-level semantic information of the scene from the complex video visual scene, discover, locate and classify the person-object pair and the interactive relationship between them in the video

Images