Multimodal Glioma Segmentation with Fusion of Multiple Self-attention and Deformable Convolutions

doi:10.11938/cjmr20233059

Abstract

Abstract:

The magnetic resonance image segmentation of glioma is significant in disease diagnosis, surgical planning, and determination of treatment plans such as radiotherapy. In response to the problem of low segmentation accuracy, inaccurate edge segmentation, and prone to false positives in existing brain tumor segmentation algorithms, an improved Unet model based on multi-head self-attention and deformable convolution is proposed. The model replaces the standard convolution of the original Unet framework with residual modules to prevent vanishing gradient during model training. Multi-head self-attention modules based on Transformer are added in the bottleneck layer to extract local features and global context information for better exploration of correlations between pixels. Deformable convolution is used at cross-layer connections to enhance the model's sensitivity to shape perception and improve the ability to extract tumor edge features. Experimental results show that the segmentation evaluation metrics of the proposed algorithm are higher than those of other comparative literature and models using the same dataset, with more precise segmentation of tumor edges. This indicates that the algorithm proposed in this paper is an effective automatic glioma segmentation algorithm.

Key words: image segmentation, glioma, Unet, Transformer self-attention, deformable convolution

CLC Number:

ZHAO Xin, ZHANG Xin, LI Xinjie, WANG Hongkai. Multimodal Glioma Segmentation with Fusion of Multiple Self-attention and Deformable Convolutions[J]. Chinese Journal of Magnetic Resonance, 2023, 40(3): 280-292.

Figures/Tables 12

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Table 1

Fig. 7

Fig. 8

Table 2

Table 3

Table 4

References 25

[1]	RONNEBERGER T, FISCHER P, BROX T. U-net: Convolutional networks for biomedical image segmentation[C]// International Conference on Medical image computing and computer-assisted intervention. Springer, 2015: 234-241.
[2]	REHMAN M U, CHO S B, KIM J, et al. Brainseg-net: Brain tumor mr image segmentation via enhanced encoder-decoder network[J]. Diagnostics, 2021, 11(2): 169. doi: 10.3390/diagnostics11020169
[3]	ZHAO L, MA J, SHAO Y, et al. MM-UNet: A multimodality brain tumor segmentation network in MRI images[J]. Front Oncol, 12: 950706. doi: 10.3389/fonc.2022.950706. doi: 10.3389/fonc.2022.950706
[4]	SHENG N, LIU D, ZHANG J, et al. Second-order ResU-Net for automatic MRI brain tumor segmentation[J]. Math Biosci Eng, 2021, 18(5): 4943-4960. doi: 10.3934/mbe.2021251 pmid: 34517471
[5]	HAN Y, SONG J M, XUE A Y, et al. Triple attention segmentation network for brain tumor images[J]. Chin J Biomed Eng, 2022, 41(1): 57-63.
	韩阳, 宋金淼, 薛安懿, 等. 基于三重注意力的脑肿瘤图像分割网络[J]. 中国生物医学工程学报, 2022, 41(1): 57-63.
[6]	ZHOU Z, RAHMAN SIDDIQUEE M M, TAJBAKHSH N, et al. Unet++: A nested u-net architecture for medical image segmentation[C]// Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support:4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, Proceedings 4. Springer International Publishing, 2018: 3-11.
[7]	HUANG H, LIN L, TONG R, et al. Unet 3+: A full-scale connected unet for medical image segmentation[C]// ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 1055-1059.
[8]	QIN C, WU Y, LIAO W, et al. Improved U-Net3+ with stage residual for brain tumor segmentation[J]. BMC Medical Imaging, 2022, 22(1): 1-15. doi: 10.1186/s12880-021-00730-0
[9]	CHILD R, GRAY S, RADFORD A, et al. Generating long sequences with sparse transformers[J]. arXiv preprint arXiv:1904.10509, 2019.
[10]	YANG F, YANG H, FU J, et al. Learning texture transformer network for image super-resolution[C]// Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 5791-5800.
[11]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[J]. arXiv preprint arXiv:2010.11929, 2020.
[12]	TOUVRON H, CORD M, DOUZE M, et al. Training data-efficient image transformers & distillation through attention[C]// International conference on machine learning. PMLR, 2021: 10347-10357.
[13]	LIU Z, LIN Y, CAO Y, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]// Proceedings of the IEEE/CVF international conference on computer vision. 2021: 10012-10022.
[14]	GRAHAM B, EL-NOUBY A, TOUVRON H, et al. Levit: a vision transformer in convnet's clothing for faster inference[C]// Proceedings of the IEEE/CVF international conference on computer vision. 2021: 12259-12269.
[15]	CHEN J, LU Y, YU Q, et al. Transunet: Transformers make strong encoders for medical image segmentation[J]. arXiv preprint arXiv:2102.04306, 2021.
[16]	WANG W, CHEN C, DING M, et al. Transbts: Multimodal brain tumor segmentation using transformer[C]// Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, Proceedings, Part I 24. Springer International Publishing, 2021: 109-119.
[17]	HUANG L, CHEN L, ZHANG B, et al. A transformer-based generative adversarial network for brain tumor segmentation[J]. arXiv preprint arXiv:2207.14134, 2022.
[18]	HO J, KALCHBRENNER N, WEISSENBORN D, et al. Axial attention in multidimensional transformers[J]. arXiv preprint arXiv:1912.12180, 2019.
[19]	GUO M H, LIU Z N, MU T J, et al. Beyond self-attention: External attention using two linear layers for visual tasks[J]. IEEE Transactions on Pattern Anal, 2023, 45(1): 5436-5447.
[20]	DAI J, QI H, XIONG Y, et al. Deformable convolutional networks[C]// Proceedings of the IEEE international conference on computer vision. 2017: 764-773.
[21]	HOU A, WU L, SUN H, et al. Brain Segmentation Based on UNet++ with Weighted Parameters and Convolutional Neural Network[C]// IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA). IEEE, 2021: 644-648.
[22]	CHEN J, LU Y, YU Q, et al. Transunet: Transformers make strong encoders for medical image segmentation[J]. arXiv preprint arXiv:2102.04306, 2021.
[23]	SHAKER A, YAN W Y, LAROCQUE P E. Automatic land-water classification using multispectral airborne LiDAR data for near-shore and river environments[J]. ISPRS J Photogramm, 2019, 152: 94-108. doi: 10.1016/j.isprsjprs.2019.04.005
[24]	ZHOU Z, SIDDIQUEE M M R, TAJBAKHSH N, et al. Unet++: Redesigning skip connections to exploit multiscale features in image segmentation[J]. IEEE T Med Maging, 2019, 39(6): 1856-1867.
[25]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA, 2017: 5998-6008.

网络	Dice/%				Hausdorff_95				PPV/%				Sensitivity/%
网络	WT	TC	ET	MV	WT	TC	ET	MV	WT	TC	ET	MV	WT	TC	ET	MV
Unet	83.24	84.57	76.84	81.55	2.6167	1.6551	2.7735	2.3488	85.57	85.34	78.52	83.14	85.94	90.58	80.55	85.69
DeepResUnet	85.00	84.80	78.74	82.85	2.5781	1.6267	2.7588	2.3212	87.67	88.50	80.79	85.65	86.55	91.24	81.92	86.57
Unet++	85.76	86.44	77.88	83.36	2.5874	1.6902	2.7558	2.3445	86.70	86.30	79.75	84.25	87.13	92.51	82.16	87.27
TransUnet	86.73	86.83	79.09	84.22	2.6562	1.5829	2.7395	2.3262	86.57	88.39	80.06	85.00	87.87	92.34	83.34	87.85
本文方法	88.15	87.98	80.46	85.33	2.5637	1.5323	2.6623	2.2528	87.75	88.98	79.89	85.54	88.22	92.16	83.66	88.01

文献来源	Dice/%	Hausdorff_95/mm	PPV/%	Sensitivity/%
文献[4]	79.56	8.93	-	-
文献[5]	84.40	2.28	85	88.76
文献[8]	80.09	-	-	83.19
文献[17]	73.7	5.48	73.7	78.7
本文结果	85.53	2.2528	85.54	88.01

网络	Dice/%			Hausdorff_95/mm			PPV/%			Sensitivity/%
网络	WT	TC	ET	WT	TC	ET	WT	TC	ET	WT	TC	ET
Unet	83.24	84.57	76.84	2.6167	1.6551	2.7735	85.57	85.34	78.52	85.94	90.58	80.55
Unet+Res	84.81	85.35	77.32	2.5977	1.6414	2.7408	86.06	85.95	78.64	86.47	90.89	81.23
Unet+Res+DCM	86.67	85.57	78.23	2.5677	1.6143	2.4234	87.07	86.50	78.79	86.55	91.24	81.92
Unet+Res+MATM	86.88	87.24	79.45	2.5681	1.5667	2.7588	87.70	86.30	79.75	87.13	92.01	82.16
Unet+Res+DCM+MATM （本文方法）	88.15	87.98	80.46	2.5637	1.5323	2.6623	87.75	88.98	79.89	88.22	92.16	83.66

模态	Dice/%			Hausdorff_95/mm			PPV/%			Sensitivity/%
模态	WT	TC	ET	WT	TC	ET	WT	TC	ET	WT	TC	ET
Flair	86.63	84.71	78.46	2.5831	1.5842	2.6957	86.07	86.84	79.76	86.54	89.75	81.75
T1	84.83	86.94	77.23	2.6011	1.5548	2.7634	84.51	88.07	77.61	85.24	90.84	78.21
T1ce	84.61	87.26	79.26	2.6109	1.5424	2.6847	84.47	88.51	80.47	85.19	91.57	82.47
T2	87.05	84.09	77.48	2.5785	1.6042	2.7498	86.64	85.89	77.89	86.67	88.43	78.64
Multi modal （本文方法）	88.15	87.98	80.46	2.5637	1.5323	2.6623	87.75	88.98	81.09	88.22	92.16	83.66