A Feature Fusion Method for Multimodal Deep Neural Networks

Abstract

The invention discloses a feature fusion method of a multi-modal deep neural network. In the multi-modal deep three-dimensional CNN, by using a compression excitation (squeeze and excitation, S&E) module on the deep learning feature domain, the information about the mode can be obtained. Channel attention mask between modalities, that is, in all modalities, give greater attention to those channels that significantly help the task goal, thus explicitly establishing the weight of the multimodal 3D depth feature map on the channel distribution; then, using four-dimensional convolution and Sigmoid activation function calculation, the spatial attention mask between the modalities can be obtained, that is, in the three-dimensional feature map of each modality, which positions in space need to be given greater attention , thereby explicitly establishing the spatial correlation of the multimodal 3D depth feature map, and giving greater attention to the positions with important information in the modality, channel, and space, thereby improving the diagnosis of the multimodal intelligent diagnosis system efficacy.

Description

technical field [0001] The invention relates to the fields of medical imaging and deep learning, in particular to a feature fusion method of a multimodal deep neural network. Background technique [0002] Existing tumor detection and diagnosis methods are usually realized through medical imaging technology, including planar X-ray imaging, CT, MRI, PET / CT, ultrasound and other modalities, and tissue biopsy is performed on suspicious lesions found in the images. However, due to the heterogeneity of tumors, their properties cannot be fully characterized by a single modality image. For example, on plane X-ray and CT images, the X-ray absorption degree of tumor tissue is characterized; on MRI images, the hydrogen proton density of tumor tissue is characterized; on FDG PET / CT, the It is the activity of tumor tissue to metabolize glucose; on ultrasound images, it is characterized by the reflection degree of tumor tissue to sound waves. Therefore, more and more clinical research b...

AI Technical Summary

Problems solved by technology

However, these methods did not analyze and process the feature weight distribution of each mode, but simply added, superimposed or fused directly

Images