NAVIGATE

Table of Contents

STATISTICS

Viewed100

Downloads61

Multimodal Medical Image Fusion Based on GAN and Multiscale Spatial Attention
  PDF下载 (61)
[1]LIN Yusong,,et al.Multimodal Medical Image Fusion Based on GAN and Multiscale Spatial Attention[J].Journal of Zhengzhou University (Engineering Science),2025,46(01):1-8.[doi:10.13705/j.issn.1671-6833.2025.01.001]
Copy
Journal of Zhengzhou University (Engineering Science)[ISSN 1671-6833/CN 41-1339/T] Volume: 46 Number of periods: 2025 01 Page number: 1-8 Column: Public date: 2024-12-23
Title:
Multimodal Medical Image Fusion Based on GAN and Multiscale Spatial Attention
Author(s):
LIN Yusong1;  2;  3;  LI Mengya1;  2;  LI Yinghao1;  2;  ZHAO Zhe1;  2
1.School of Cyber Science and Engineering, Zhengzhou University, Zhengzhou 450002, China; 2.Henan Provincial Collaborative Innovation Center for Internet Medical and Health Services, Zhengzhou University, Zhengzhou 450052, China; 3. Hanwei IoT Institute, Zhengzhou University, Zhengzhou 450002, China
Keywords:
image fusion;  multimodal medical images;  generative adversarial network;  feature pyramid;  attention mechanism
CLC:
TP391
DOI:
10.13705/j.issn.1671-6833.2025.01.001
Abstract:
Aiming to address the problem of multi-scale feature and texture detail information loss in the process of multimodal medical image fusion, a novel image fusion algorithm based on generative adversarial network (GAN) and multi-scale spatial attention mechanism was proposed. Firstly, the generator adopted an autoencoder structure to extract, fuse, and reconstructed the input images using an encoder and a decoder, generating the fused image. Secondly, the entire GAN framework employed a dual discriminator structure, enabling the generator to preserve salient features from multiple modal images in the fused image. Finally, a multi-scale spatial attention mechanism was constructed as a fundamental module for feature extraction in the encoder. It could effectively capture and retain multi-scale features from the source images, and incorporate spatial attention mechanism to better preserve the structures and details of the source images. Experimental results on the Whole Brain Atlas database demonstrated that the fused images generated by the proposed algorithm could exhibit richer texture details, enhancing human visual observation. Furthermore, the algorithm outperformed other advanced algorithms in such objective evaluation metrics as average gradient, peak signal-to-noise ratio, mutual information, and visual information fidelity for three different types of medical image fusion tasks, with average values of 0.302 3, 20.720 7, 1.441 4, and 0.649 8, respectively. Thus, the proposed algorithm demonstrated a certain advantage over other advanced algorithms.
References:
[1]HUANG B, YANG F, YIN M X, et al. A review of multimodal medical image fusion techniques[J]. Computational and Mathematical Methods in Medicine, 2020, 2020: 8279342. 
[2]AZAM M A, KHAN K B, SALAHUDDIN S, et al. A review on multimodal medical image fusion: compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics[J]. Computers in Biology and Medicine, 2022, 144: 105253. 
[3]PIELLA G. A general framework for multiresolution image fusion: from pixels to regions[J]. Information Fusion, 2003, 4(4): 259-280. 
[4]GUO P, XIE G Q, LI R F, et al. Multimodal medical image fusion with convolution sparse representation and mutual information correlation in NSST domain[J]. Complex & Intelligent Systems, 2023, 9(1): 317-328. 
[5]YIN M, LIU X N, LIU Y, et al. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain[J]. IEEE Transactions on Instrumentation and Measurement, 2019, 68(1): 49-64. 
[6]ZHU Z Q, ZHENG M Y, QI G Q, et al. A phase congruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain[J]. IEEE Access, 2019, 7: 20811-20824. 
[7]DOGRA A, KUMAR S. Multi-modality medical image fusion based on guided filter and image statistics in multidirectional shearlet transform domain[J]. Journal of Ambient Intelligence and Humanized Computing, 2023, 14 (9): 12191-12205. 
[8] ZHANG Y, LIU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network [J]. Information Fusion, 2020, 54: 99-118. 
[9] LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2019, 28(5): 2614-2623. 
[10] FU J, LI W S, DU J, et al. A multiscale residual pyramid attention network for medical image fusion[J]. Biomedical Signal Processing and Control, 2021, 66: 102488. 
[11]许光宇, 陈浩宇, 张杰. 双路径双鉴别器生成对抗网络的红外与可见光图像融合[J/OL]. 计算机辅助设计与图形学学报,1-14(2024-02-06)[2024-04-07]. http: ∥kns. cnki. net/kcms/detail/11.2925. TP. 20240204.1728.061.html. 
XU G Y, CHEN H Y, ZHANG J. Infrared and visible image fusion based on dual-path and dual-discriminator generation adversarial network [J/OL]. Journal of Computer-Aided Design & Computer Graphics,1-14(202402-06)[2024-04-07].http:∥kns. cnki.net/kcms/detail/11.2925.TP.20240204.1728.061.html. 
[12] MA J Y, YU W, LIANG P W, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. 
[13] ZHAO C, WANG T F, LEI B Y. Medical image fusion method based on dense block and deep convolutional generative adversarial network[J]. Neural Computing and Applications, 2021, 33(12): 6595-6610. 
[14] MA J Y, XU H, JIANG J J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. 
[15] ZHOU T, LI Q, LU H L, et al. GAN review: models and medical image fusion applications[J]. Information Fusion, 2023, 91: 134-148. 
[16]肖儿良, 林化溪, 简献忠. 基于生成对抗网络探索潜在空间的医学图像融合算法[J]. 信息与控制, 2021, 50(5): 538-549.
XIAO E L, LIN H X, JIAN X Z. Medical image fusion algorithm adopting generative adversarial network to explore latent space[J]. Information and Control, 2021, 50 (5): 538-549. 
[17] LI H C, XIONG P F, AN J, et al. Pyramid attention network for semantic segmentation[EB/OL]. (2018-11-25) [2024-04-07].https:∥arxiv.org/abs/1805.10180. 
[18] LIU Y, WANG L, LI H F, et al. Multi-focus image fusion with deep residual learning and focus property detection[J]. Information Fusion, 2022, 86: 1-16. 
[19]尹海涛, 岳勇赢. 基于半监督学习和生成对抗网络的医学图像融合算法[J]. 激光与光电子学进展, 2022, 59(22): 245-254. 
YIN H T, YUE Y Y. Medical image fusion based on semisupervised learning and generative adversarial network[J]. Laser & Optoelectronics Progress, 2022, 59 (22): 245-254. 
[20] LI X X, GUO X P, HAN P F, et al. Laplacian redecomposition for multimodal medical image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(9): 6880-6890. 
[21] XU H, MA J Y, JIANG J J, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518. 
[22] XU H, MA J Y. EMFusion: an unsupervised enhanced medical image fusion network[J]. Information Fusion, 2021, 76: 177-186. 
[23] LI J W, LIU J Y, ZHOU S H, et al. GeSeNet: a general semantic-guided network with couple mask ensemble for medical image fusion[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023,1: 14. 
[24]刘帅奇, 王洁, 安彦玲, 等. 基于CNN的非下采样剪切波域多聚焦图像融合[J]. 郑州大学学报(工学版), 2019, 40(4): 36-41. 
LIU S Q, WANG J, AN Y L, et al. Multi-focus image fusion based on CNN in non-sampled shearlet domain [J]. Journal of Zhengzhou University (Engineering Science), 2019, 40(4): 36-41. XIAO E L, LIN H X, JIAN X Z. Medical image fusion
Similar References:
Memo

-

Last Update: 2024-12-30
Copyright © 2023 Editorial Board of Journal of Zhengzhou University (Engineering Science)