<span style="">Multimodal Medical Image Fusion Based on GAN and Multiscale Spatial Attention</span>

NAVIGATE

Table of Contents

STATISTICS

Viewed87

Downloads131

Multimodal Medical Image Fusion Based on GAN and Multiscale Spatial Attention

PDF下载 (131)

[1]Lin Yusong,,et al.Multimodal Medical Image Fusion Based on GAN and Multiscale Spatial Attention[J].Journal of Zhengzhou University (Engineering Science),2024,45(pre):2-.[doi:10. 13705 / j. issn. 1671-6833. 2025. 01. 001]

Copy

Journal of Zhengzhou University (Engineering Science)[ISSN 1671-6833/CN 41-1339/T] Volume: 45 Number of periods: 2024 pre Page number: 2- Column: Public date: 2024-12-31

Title:: Multimodal Medical Image Fusion Based on GAN and Multiscale Spatial Attention

Author(s):: Lin Yusong1; 2; 3 ; Li Mengya1; 2; Li Yinghao1; 2; Zhao Zhe1; 2*; (1. School of Cyber Science and Engineering, Zhengzhou University, Zhengzhou 450002, China; 2. Collaborative Innovation Center for Internet Healthcare, Zhengzhou University, Zhengzhou 450052, China; 3. Hanwei IoT Institute, Zhengzhou University, Zhengzhou 450002, China)

Keywords:: image fusion; multi-modal medical images; generative adversarial network; feature pyramid; attention mechanism

CLC:: TP391

DOI:: 10. 13705 / j. issn. 1671-6833. 2025. 01. 001

Abstract:: Aiming to address the problem of multi-scale feature and texture detail information loss in the process of multi-modal medical image fusion, a novel image fusion algorithm based on generative adversarial networks (GANs) and multi-scale spatial attention mechanism is proposed. Firstly, the generator adopts an autoencoder structure to extract, fuse, and reconstruct the input images using an encoder and a decoder, generating the fused image. Secondly, the entire GAN framework employs a dual discriminator structure, enabling the generator to preserve salient features from multiple modal images in the fused image. Lastly, a multi-scale spatial attention mechanism is constructed as a fundamental module for feature extraction in the encoder. It effectively captures and retains multi-scale features from the source images, and incorporates spatial attention mechanism to better preserve the structures an d details of the source images. Multiple sets of experimental results demonstrate that the fused images generated by the algorithm proposed in this paper not only exhibit richer texture details, which aids in human visual observation, but also achieve superior performance compared to other advanced algorithms in terms of average gradient, peak signal-to-noise ratio, mutual information, and visual fidelity objective evaluation metrics for three different types of medical image fusion tasks. The average values of these metrics are 0.3023, 20.7207, 1.4414, and 0.6498, respectively, indicating a certain advantage over other advanced algorithms

References:: [ 1 ] HUANG B, YANG F, YIN M X, et al. A review of multimodal medical image fusion techniques [ J] . Com putational and Mathematical Methods in Medicine, 2020, 2020: 8279342.

[ 2 ] AZAM M A, KHAN K B, SALAHUDDIN S, et al. A review on multimodal medical image fusion: compendi ous analysis of medical modalities, multimodal databas es, fusion techniques and quality metrics[ J] . Comput ers in Biology and Medicine, 2022, 144: 105253.

[ 3 ] PIELLA G. A general framework for multiresolution im age fusion: from pixels to regions [ J] . Information Fu sion, 2003, 4(4) : 259-280.

[ 4 ] GUO P, XIE G Q, LI R F, et al. Multimodal medical image fusion with convolution sparse representation and mutual information correlation in NSST domain[J]. Com plex & Intelligent Systems, 2023, 9(1): 317-328.

[ 5 ] YIN M, LIU X N, LIU Y, et al. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain [ J ] . IEEE Transactions on Instrumentation and Measurement, 2019, 68(1) : 49-64.

[ 6 ] ZHU Z Q, ZHENG M Y, QI G Q, et al. A phase con gruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain [ J ] . IEEE Access, 2019, 7: 20811-20824.

[ 7 ] DOGRA A, KUMAR S. Multi-modality medical image fusion based on guided filter and image statistics in mul tidirectional shearlet transform domain [ J ] . Journal of Ambient Intelligence and Humanized Computing, 2023, 14(9) : 12191-12205.

[ 8 ] ZHANG Y, LIU Y, SUN P, et al. IFCNN: a general im age fusion framework based on convolutional neural net work[ J] . Information Fusion, 2020, 54: 99-118.

[ 9 ] LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[ J] . IEEE Transactions on Image Pro cessing, 2019, 28(5) : 2614-2623.

[10] FU J, LI W S, DU J, et al. A multiscale residual pyramid attention network for medical image fusion[J]. Biomedical Signal Processing and Control, 2021, 66: 102488.

[11] 许光宇 , 陈浩宇 , 张杰 . 双路径双鉴别器生成对抗网络的红外与可见光图像融合 [ J/ OL] . 计算机辅助设计与图形学学报 , 1 - 14 [ 2024 - 04 - 07] . http:∥kns. cnki. net / kcms/ detail / 11. 2925. TP. 20240204. 1728. 06 1. html.

XU G Y, CHEN H Y, ZHANG J. Infrared and visible image fusion based on dual-path and dual-discriminator generation adversarial network [ J/ OL] . Journal of Com puter-Aided Design & Computer Graphics,1 - 14 [ 2024 - 04 - 07 ] . http: ∥kns. cnki. net / kcms/ detail / 11. 2925. TP. 20240204. 1728. 061. html.

[12] MA J Y, YU W, LIANG P W, et al. FusionGAN: a generative adversarial network for infrared and visible im age fusion[ J] . Information Fusion, 2019, 48: 11-26.

[13] ZHAO C, WANG T F, LEI B Y. Medical image fusion method based on dense block and deep convolutional gen erative adversarial network [ J ] . Neural Computing and Applications, 2021, 33(12) : 6595-6610.

[14] MA J Y, XU H, JIANG J J, et al. DDcGAN: a dual-dis criminator conditional generative adversarial network for multi-resolution image fusion [ J] . IEEE Transactions on Image Processing, 2020, 29: 4980-4995.

[15] ZHOU T, LI Q, LU H L, et al. GAN review: models and medical image fusion applications [ J ] . Information Fusion, 2023, 91: 134-148.

[16] 肖儿良 , 林化溪 , 简献忠 . 基于生成对抗网络探索潜在空间的医学图像融合算法 [ J] . 信息与控制 , 2021, 50(5) : 538-549.

XIAO E L, LIN H X, JIAN X Z. Medical image fusion algorithm adopting generative adversarial network to ex plore latent space[ J] . Information and Control, 2021, 50 (5) : 538-549.
[17] LI H C, XIONG P F, AN J, et al. Pyramid attention net work for semantic segmentation[EB / OL]. (2018-11-25) [2024-04-07]. https:∥arxiv. org / abs/ 1805. 10180.

[18] LIU Y, WANG L, LI H F, et al. Multi-focus image fu sion with deep residual learning and focus property detec tion[ J] . Information Fusion, 2022, 86: 1-16.

[19] 尹海涛 , 岳勇赢 . 基于半监督学习和生成对抗网络的医学图像融合算法 [ J] . 激光与光电子学进展 , 2022, 59(22) : 245-254.

YIN H T, YUE Y Y. Medical image fusion based on semisupervised learning and generative adversarial net work[ J] . Laser & Optoelectronics Progress, 2022, 59 (22) : 245-254.

[20] LI X X, GUO X P, HAN P F, et al. Laplacian redecom position for multimodal medical image fusion [ J] . IEEE Transactions on Instrumentation and Measurement, 2020, 69(9) : 6880-6890.

[21] XU H, MA J Y, JIANG J J, et al. U2Fusion: a unified unsupervised image fusion network [ J ] . IEEE Transac tions on Pattern Analysis and Machine Intelligence, 2022, 44(1) : 502-518.

[22] XU H, MA J Y. EMFusion: an unsupervised enhanced medical image fusion network [ J] . Information Fusion, 2021, 76: 177-186.

[23] LI J W, LIU J Y, ZHOU S H, et al. GeSeNet: a general semantic-guided network with couple mask ensemble for medical image fusion [ J] . IEEE Transactions on Neural Networks and Learning Systems, 2023,1: 14.

[24] 刘帅奇 , 王洁 , 安彦玲 , 等 . 基于 CNN 的非下采样剪切波域多聚焦图像融合 [ J] . 郑州大学学报 ( 工学

版 ) , 2019, 40(4) : 36-41.

LIU S Q, WANG J, AN Y L, et al. Multi-focus image fusion based on CNN in non-sampled shearlet domain [ J] . Journal of Zhengzhou University ( Engineering Sci ence) , 2019, 40(4) : 36-41.

Similar References:

Memo

Last Update: 2024-10-10