[1]林予松,李孟娅,李英豪,等.基于GAN和多尺度空间注意力的多模态医学图像融合[J].郑州大学学报(工学版),2025,46(01):1-8.[doi:10.13705/j.issn.1671-6833.2025.01.001]
　LIN Yusong,,et al.Multimodal Medical Image Fusion Based on GAN and Multiscale Spatial Attention[J].Journal of Zhengzhou University (Engineering Science),2025,46(01):1-8.[doi:10.13705/j.issn.1671-6833.2025.01.001]
点击复制

基于GAN和多尺度空间注意力的多模态医学图像融合()
分享到：

《郑州大学学报(工学版)》[ISSN:1671-6833/CN:41-1339/T]

卷:
46
期数:
2025年01期
页码:
1-8
栏目:
出版日期:
2024-12-23

文章信息/Info

Title:
Multimodal Medical Image Fusion Based on GAN and Multiscale Spatial Attention
文章编号:
1671-6833(2025)01-0001-08
作者:
林予松1; 2; 3;  李孟娅1; 2;  李英豪1; 2;  赵　哲1; 2
1.郑州大学 网络空间安全学院,河南 郑州 450002;2.郑州大学 互联网医疗与健康服务河南省协同创新中心,河南 郑州 450052;3.郑州大学 汉威物联网研究院,河南 郑州 450002
Author(s):
LIN Yusong1;  2;  3;  LI Mengya1;  2;  LI Yinghao1;  2;  ZHAO Zhe1;  2
1.School of Cyber Science and Engineering, Zhengzhou University, Zhengzhou 450002, China; 2.Henan Provincial Collaborative Innovation Center for Internet Medical and Health Services, Zhengzhou University, Zhengzhou 450052, China; 3. Hanwei IoT Institute, Zhengzhou University, Zhengzhou 450002, China
关键词:
图像融合;  多模态医学图像;  生成对抗网络;  特征金字塔;  注意力机制
Keywords:
image fusion;  multimodal medical images;  generative adversarial network;  feature pyramid;  attention mechanism
分类号:
TP391
DOI:
10.13705/j.issn.1671-6833.2025.01.001
文献标志码:
A
摘要:
针对多模态医学图像融合过程中多尺度特征和纹理细节信息丢失的问题,提出一种基于生成对抗网络和多尺度空间注意力的图像融合算法。首先,生成器采用自编码器结构,分别利用编码器和解码器对输入图像进行特征提取、融合和重建,生成融合图像;其次,整个对抗网络框架采用双鉴别器结构,使得生成器生成的融合图像同时保留多个模态图像的显著特征;最后,构建一种多尺度空间注意力作为编码器进行特征提取的基本模块,利用多尺度结构充分捕获并保留源图像的多尺度特征,并且引入空间注意力更好地保留源图像的结构和细节信息。全脑图谱数据库上的实验结果表明:所提算法生成的融合图像不仅纹理细节更为丰富,有助于人类视觉观察,而且在3种不同类型的医学图像融合任务上平均梯度、峰值信噪比、互信息、视觉信息保真度等客观评价指标的平均值分别达到0.302 3、20.720 7、1.441 4、0.649 8,与其他先进的算法相比具有一定的优势。
Abstract:
Aiming to address the problem of multi-scale feature and texture detail information loss in the process of multimodal medical image fusion, a novel image fusion algorithm based on generative adversarial network (GAN) and multi-scale spatial attention mechanism was proposed. Firstly, the generator adopted an autoencoder structure to extract, fuse, and reconstructed the input images using an encoder and a decoder, generating the fused image. Secondly, the entire GAN framework employed a dual discriminator structure, enabling the generator to preserve salient features from multiple modal images in the fused image. Finally, a multi-scale spatial attention mechanism was constructed as a fundamental module for feature extraction in the encoder. It could effectively capture and retain multi-scale features from the source images, and incorporate spatial attention mechanism to better preserve the structures and details of the source images. Experimental results on the Whole Brain Atlas database demonstrated that the fused images generated by the proposed algorithm could exhibit richer texture details, enhancing human visual observation. Furthermore, the algorithm outperformed other advanced algorithms in such objective evaluation metrics as average gradient, peak signal-to-noise ratio, mutual information, and visual information fidelity for three different types of medical image fusion tasks, with average values of 0.302 3, 20.720 7, 1.441 4, and 0.649 8, respectively. Thus, the proposed algorithm demonstrated a certain advantage over other advanced algorithms.

参考文献/References:

[1]HUANG B, YANG F, YIN M X, et al. A review of multimodal medical image fusion techniques[J]. Computational and Mathematical Methods in Medicine, 2020, 2020: 8279342. 

[2]AZAM M A, KHAN K B, SALAHUDDIN S, et al. A review on multimodal medical image fusion: compendious analysis of medical modalities, multimodal databases, fusion techniques and quality metrics[J]. Computers in Biology and Medicine, 2022, 144: 105253. 
[3]PIELLA G. A general framework for multiresolution image fusion: from pixels to regions[J]. Information Fusion, 2003, 4(4): 259-280. 
[4]GUO P, XIE G Q, LI R F, et al. Multimodal medical image fusion with convolution sparse representation and mutual information correlation in NSST domain[J]. Complex & Intelligent Systems, 2023, 9(1): 317-328. 
[5]YIN M, LIU X N, LIU Y, et al. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain[J]. IEEE Transactions on Instrumentation and Measurement, 2019, 68(1): 49-64. 
[6]ZHU Z Q, ZHENG M Y, QI G Q, et al. A phase congruency and local Laplacian energy based multi-modality medical image fusion method in NSCT domain[J]. IEEE Access, 2019, 7: 20811-20824. 
[7]DOGRA A, KUMAR S. Multi-modality medical image fusion based on guided filter and image statistics in multidirectional shearlet transform domain[J]. Journal of Ambient Intelligence and Humanized Computing, 2023, 14 (9): 12191-12205. 
[8] ZHANG Y, LIU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network [J]. Information Fusion, 2020, 54: 99-118. 
[9] LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2019, 28(5): 2614-2623. 
[10] FU J, LI W S, DU J, et al. A multiscale residual pyramid attention network for medical image fusion[J]. Biomedical Signal Processing and Control, 2021, 66: 102488. 
[11]许光宇, 陈浩宇, 张杰. 双路径双鉴别器生成对抗网络的红外与可见光图像融合[J/OL]. 计算机辅助设计与图形学学报,1-14(2024-02-06)[2024-04-07]. http: ∥kns. cnki. net/kcms/detail/11.2925. TP. 20240204.1728.061.html. 
XU G Y, CHEN H Y, ZHANG J. Infrared and visible image fusion based on dual-path and dual-discriminator generation adversarial network [J/OL]. Journal of Computer-Aided Design & Computer Graphics,1-14(202402-06)[2024-04-07].http:∥kns. cnki.net/kcms/detail/11.2925.TP.20240204.1728.061.html. 
[12] MA J Y, YU W, LIANG P W, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. 
[13] ZHAO C, WANG T F, LEI B Y. Medical image fusion method based on dense block and deep convolutional generative adversarial network[J]. Neural Computing and Applications, 2021, 33(12): 6595-6610. 
[14] MA J Y, XU H, JIANG J J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. 
[15] ZHOU T, LI Q, LU H L, et al. GAN review: models and medical image fusion applications[J]. Information Fusion, 2023, 91: 134-148. 
[16]肖儿良, 林化溪, 简献忠. 基于生成对抗网络探索潜在空间的医学图像融合算法[J]. 信息与控制, 2021, 50(5): 538-549.
XIAO E L, LIN H X, JIAN X Z. Medical image fusion algorithm adopting generative adversarial network to explore latent space[J]. Information and Control, 2021, 50 (5): 538-549. 
[17] LI H C, XIONG P F, AN J, et al. Pyramid attention network for semantic segmentation[EB/OL]. (2018-11-25) [2024-04-07].https:∥arxiv.org/abs/1805.10180. 
[18] LIU Y, WANG L, LI H F, et al. Multi-focus image fusion with deep residual learning and focus property detection[J]. Information Fusion, 2022, 86: 1-16. 
[19]尹海涛, 岳勇赢. 基于半监督学习和生成对抗网络的医学图像融合算法[J]. 激光与光电子学进展, 2022, 59(22): 245-254. 
YIN H T, YUE Y Y. Medical image fusion based on semisupervised learning and generative adversarial network[J]. Laser & Optoelectronics Progress, 2022, 59 (22): 245-254. 
[20] LI X X, GUO X P, HAN P F, et al. Laplacian redecomposition for multimodal medical image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(9): 6880-6890. 
[21] XU H, MA J Y, JIANG J J, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518. 
[22] XU H, MA J Y. EMFusion: an unsupervised enhanced medical image fusion network[J]. Information Fusion, 2021, 76: 177-186. 
[23] LI J W, LIU J Y, ZHOU S H, et al. GeSeNet: a general semantic-guided network with couple mask ensemble for medical image fusion[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023,1: 14. 
[24]刘帅奇, 王洁, 安彦玲, 等. 基于CNN的非下采样剪切波域多聚焦图像融合[J]. 郑州大学学报(工学版), 2019, 40(4): 36-41. 
LIU S Q, WANG J, AN Y L, et al. Multi-focus image fusion based on CNN in non-sampled shearlet domain [J]. Journal of Zhengzhou University (Engineering Science), 2019, 40(4): 36-41. XIAO E L, LIN H X, JIAN X Z. Medical image fusion

相似文献/References:

[1]刘帅奇,王洁,安彦玲,等.基于CNN的非下采样剪切波域多聚焦图像融合[J].郑州大学学报(工学版),2019,40(04):7.[doi:10.13705/j.issn.1671-6833.2019.04.002]
　Shuaiqi Liu,Wang Jie,An Yanling,et al.Multi- focus Image Fusion Based on Convolution Neural Network in Non-sampled Shearlet Domain[J].Journal of Zhengzhou University (Engineering Science),2019,40(01):7.[doi:10.13705/j.issn.1671-6833.2019.04.002]

更新日期/Last Update: 2024-12-30