Automatic Segmentation of Knee Joint Synovial Magnetic Resonance Images Based on 3D VNetTrans

doi:10.11938/cjmr20222988

Abstract

Abstract:

Knee joint is commonly hurt by rheumatoid arthritis (RA). Accurate segmentation of synovium is essential for the diagnosis and treatment of RA. This paper proposes an algorithm based on improved VNet for automatically segmenting knee joint synovial magnetic resonance images. Firstly, the knee joint magnetic resonance images of 39 patients with synovitis were preprocessed. VNetTrans was constructed by embedding Transformer at the bottom of VNet. The MemSwish activation function was used for training. The average Dice score of the final model is 0.758 5 and the HD is 24.6 mm. Compared with VNet, the proposed model increased Dice score by 0.083 6 and decreased HD by 10 mm. Experimental results demonstrated that the proposed algorithm achieved satisfying 3D segmentation of the synovial hyperplasia area in the knee magnetic resonance images. It can be utilized to facilitate the diagnosis and monitoring of RA.

Key words: magnetic resonance image, medical image segmentation, deep learning, synovitis

CLC Number:

O482.53

Ying-shan WANG, Ao-qi DENG, Jin-ling MAO, Zhong-qi ZHU, Jie SHI, Guang YANG, Wei-wei MA, Qing LU, Hong-zhi WANG. Automatic Segmentation of Knee Joint Synovial Magnetic Resonance Images Based on 3D VNetTrans[J]. Chinese Journal of Magnetic Resonance, 2022, 39(3): 303-315.

Figures/Tables 11

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Table 1

Table 2

Table 3

Fig.6

Fig.7

Table 4

References 28

1	SMOLEN J S, ALETAHA D, BARTON A, et alRheumatoid arthritis[J]. Nat Rev Dis Primers, 2018,4 (1): 18001. doi: 10.1038/nrdp.2018.1
2	JIN S Y, LI M T, FANG Y F, et alChinese registry of rheumatoid arthritis (CREDIT): II. prevalence and risk factors of major comorbidities in Chinese patients with rheumatoid arthritis[J]. Arthritis ResTher, 2017,19 (1): 251.
3	中华医学会风湿病学分会2018中国类风湿关节炎诊疗指南[J]. 中华内科杂志, 2018,57 (4): 242- 251. doi: 10.3760/cma.j.issn.0578-1426.2018.04.004
	ASSOCIATION C R2018 Chinese guideline for the diagnosis and treatment of rheumatoid arthritis[J]. Chin J Intern Med, 2018,57 (4): 242- 251. doi: 10.3760/cma.j.issn.0578-1426.2018.04.004
4	SUGIMOTO H, TAKEDA A, KANO SAssessment of disease activity in rheumatoid arthritis using magnetic resonance imaging: quantification of pannus volume in the hands[J]. Bri J Rheumatol, 1998,37 (8): 854- 861. doi: 10.1093/rheumatology/37.8.854
5	OSTERGAARD MDifferent approaches to synovial membrane volume determination by magnetic resonance imaging: manual versus automated segmentation[J]. Rheumatology, 1997,36 (11): 1166- 1177. doi: 10.1093/rheumatology/36.11.1166
6	SAKASHITA T, KAMISHIMA T, KOBAYASHI Y, et alAccurate quantitative assessment of synovitis in rheumatoid arthritis using pixel-by-pixel, time-intensity curve shape analysis[J]. Br J Radiol, 2016,89 (1061): 20151000. doi: 10.1259/bjr.20151000
7	FOTINOS-HOYER A K, GUERMAZI A, JARA H, et alAssessment of synovitis in the osteoarthritic knee: comparison between manual segmentation, semiautomated segmentation, and semiquantitative assessment using contrast-enhanced fat-suppressed T1-weighted MRI[J]. Magn Reson Med, 2010,64 (2): 604- 609. doi: 10.1002/mrm.22401
8	PERRY T A, GAIT A, O’NEILL T W, et alMeasurement of synovial tissue volume in knee osteoarthritis using a semiautomated MRI-based quantitative approach[J]. Magn Reson Med, 2019,81 (5): 3056- 3064. doi: 10.1002/mrm.27633
9	WANG A, FRANKE A, WESARG S. Semi-automatic segmentation of JIA-induced inflammation in MRI images of ankle joints[C]// Medical Imaging 2019: Image Processing, SPIE, 2019, 10949: 875-881.
10	ANDERSEN J K H, PEDERSEN J S, LAURSEN M S, et alNeural networks for automatic scoring of arthritis disease activity on ultrasound images[J]. RMD open, 2019,5 (1): e000891. doi: 10.1136/rmdopen-2018-000891
11	CHRISTENSEN A B H, JUST S A, ANDERSEN J K H, et alApplying cascaded convolutional neural network design further enhances automatic scoring of arthritis disease activity on ultrasound images from rheumatoid arthritis patients[J]. Ann Rheum Dise, 2020,79 (9): 1189- 1193. doi: 10.1136/annrheumdis-2019-216636
12	IQBAL I, SHAHZAD G, RAFIQ N, et alDeep learning-based automated detection of human knee joint's synovial fluid from magnetic resonance images with transfer learning[J]. IET Image Processing, 2020,14 (10): 1990- 1998. doi: 10.1049/iet-ipr.2019.1646
13	WONG L M, SHI L, XIAO F, et alFully automated segmentation of wrist bones on T2-weighted fat-suppressed MR images in early rheumatoid arthritis[J]. Quant Imag Med Surg, 2019,9 (4): 579. doi: 10.21037/qims.2019.04.03
14	魏小娜, 邢嘉祺, 王振宇, 等基于改进U-Net的关节滑膜磁共振图像的分割[J]. 计算机应用, 2020,40 (11): 3340- 3345.
	WEI X N, XIN J Q, WANG Z Y, et alMagnetic resonance image segmentation of articular synovium based on improved U-Net[J]. Journal of Computer Applications, 2020,40 (11): 3340- 3345.
15	王振宇, 王颖珊, 毛瑾玲, 等基于Dense-UNet++的关节滑膜磁共振图像分割[J]. 波谱学杂志, 2022,39 (2): 208- 219.
	WANG Z Y, WANG Y S, MAO J L, et alMagnetic resonance images segmentation of synovium based on Dense-UNet++[J]. Chinese J Magn Reson, 2022,39 (2): 208- 219.
16	WANG W X, CHEN C, DING M, et al. TransBTS: Multimodal brain tumor segmentation using transformer[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2021: 109-119.
17	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]// Proceedings of the IEEE conference on computer vision and pattern recognition, 2015: 3431-3440.
18	RONNEBERGER O, FISCHER P, BROX T. U-Net: Convolutional Networks for Biomedical Image Segmentation[C]// International Conference on Medical image computing and computer-assisted intervention, Cham: Springer, 2015: 234-241.
19	ÇIçEK Ö, ABDULKADIR A, LIENKAMP S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation[C]// International conference on medical image computing and computer-assisted intervention. Springer, 2016: 424-432.
20	MILLETARI F, NAVAB N, AHMADI S-A. V-net: Fully convolutional neural networks for volumetric medical image segmentation[C]// 2016 fourth international conference on 3D vision (3DV), IEEE, 2016: 565-571.
21	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]// Proceedings of the Advances in Neural Information Processing Systems, 2017: 6000-6010.
22	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: Transformers for image recognition at scale[OL]. arXiv preprint arXiv: 2010.11929, 2020.
23	NAIR V, HINTON G E. Rectified linear units improve restricted boltzmann machines[C]// Proceedings of the 27th International Conference on International Conference on Machine Learning, Haifa, Israel: 2010: 807-814.
24	RAMACHANDRAN P, ZOPH B, LE Q V. Searching for activation functions[J]. arXiv preprint arXiv: 1710.05941, 2017.
25	TAN M, LE Q. Efficientnet: Rethinking model scaling for convolutional neural networks[C]// International conference on machine learning, PMLR, 2019: 6105-6114.
26	YUAN L, CHEN Y, WANG T, et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet[C]// Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 558-567.
27	YUSHKEVICH P A, GAO Y, GERIG G. ITK-SNAP: An interactive tool for semi-automatic segmentation of multi-modality biomedical images[C]// 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), IEEE, 2016: 3342-3345.
28	HATAMIZADEH A, TANG Y, NATH V, et al. Unetr: Transformers for 3d medical image segmentation[C]// Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022: 574-584.

模型	卷积核维度	Dice	HD/mm
UNet	2D	0.6489	89.8
VNet	2D	0.6640	52.5
UNet	3D	0.6528	72.2
VNet	3D	0.6749	34.6

模型	激活函数	Dice	HD/mm	训练耗时
UNet	ReLU	0.6528	72.2	23min
	Swish	0.6613	50.8	34min
	MemSwish	0.6670	37.4	30min
VNetTrans	ReLU	0.7140	33.6	5h58min
	Swish	0.7505	29.0	6h36min
	MemSwish	0.7585	24.6	6h21min

模型	Dice	Sensitivity	Specificity	RVD/%	HD/mm
UNet	0.6528	0.7828	0.9864	33.84	72.2
VNet	0.6749	0.7262	0.9908	9.21	34.6
TransBTS	0.6996	0.7186	0.9917	4.87	36.0
UNETR	0.6355	0.6560	0.9920	?3.31	68.8
VNetTrans	0.7585	0.7633	0.9943	?1.34	24.6

网络模型	激活函数	是否加入Transformer模块	Dice	Sensitivity	Specificity	RVD/%	HD/mm
VNet	ReLU	否	0.6749	0.7262	0.9908	9.21	34.6
VNet + MemSwish	MemSwish	否	0.6962	0.7580	0.9904	14.92	35.8
VNet + Transformer	ReLU	是	0.7140	0.7527	0.9913	2.52	33.6
This work	MemSwish	是	0.7585	0.7633	0.9943	?1.34	24.6

[1]	Qin ZHOU, Yuan-jun WANG. Groupwise Registration for Magnetic Resonance Image Based on Variational Inference [J]. Chinese Journal of Magnetic Resonance, 2022, 39(3): 291-302.
[2]	Xiao CHANG,Xin CAI,Guang YANG,Sheng-dong NIE. Applications of Generative Adversarial Networks in Medical Image Translation [J]. Chinese Journal of Magnetic Resonance, 2022, 39(3): 366-380.
[3]	Meng CHEN, Chen GENG, Yu-xin LI, Dao-ying GENG, Yi-fang BAO, Ya-kang DAI. Automatic Detection for Cerebral Aneurysms in TOF-MRA Images Based on Fuzzy Label and Deep Learning [J]. Chinese Journal of Magnetic Resonance, 2022, 39(3): 267-277.
[4]	Zhen-yu WANG, Ying-shan WANG, Jin-ling MAO, Wei-wei MA, Qing LU, Jie SHI, Hong-zhi WANG. Magnetic Resonance Images Segmentation of Synovium Based on Dense-UNet++ [J]. Chinese Journal of Magnetic Resonance, 2022, 39(2): 208-219.
[5]	Jian-sheng LIN,Li-jia WANG. Reconstruction of Displacement Field of Left Ventricle Combined with Biomechanical Model [J]. Chinese Journal of Magnetic Resonance, 2022, 39(1): 72-86.
[6]	Lu HUO,Xiao-xin HU,Qin XIAO,Ya-jia GU,Xu CHU,Luan JIANG. Automatic Segmentation of Breast and Fibroglandular Tissues in DCE-MR Images Based on nnU-Net [J]. Chinese Journal of Magnetic Resonance, 2021, 38(3): 367-380.
[7]	LIU Peng, ZHONG Yu-min, WANG Li-jia. Automatic Segmentation of Right Ventricle in Cine Cardiac Magnetic Resonance Image Based on a Dense and Multi-Scale U-net Method [J]. Chinese Journal of Magnetic Resonance, 2020, 37(4): 456-468.
[8]	WANG Wan-ting, SU Shi, JIA Sen, LIANG Dong, WANG Hai-feng. Reconstruction of Simultaneous Multi-Slice MRI Data by Combining Virtual Conjugate Coil Technology and Convolutional Neural Network [J]. Chinese Journal of Magnetic Resonance, 2020, 37(4): 407-421.
[9]	ZHAO Shang-yi, WANG Yuan-jun. Classification of Alzheimer's Disease Patients Based on Magnetic Resonance Images and an Improved UNet++ Model [J]. Chinese Journal of Magnetic Resonance, 2020, 37(3): 321-331.
[10]	GONG Jin-chang, WANG Yu, WANG Yuan-jun. A Method for Segmentation of Glioma on Multimodal Magnetic Resonance Images Based on Wavelet Fusion and Deep Learning [J]. Chinese Journal of Magnetic Resonance, 2020, 37(2): 131-143.
[11]	CHENG Hui-tao, WANG Shan-shan, KE Zi-wen, JIA Sen, CHENG Jing, QIU Zhi-lang, ZHENG Hai-rong, LIANG Dong. A Deep Recursive Cascaded Convolutional Network for Parallel MRI [J]. Chinese Journal of Magnetic Resonance, 2019, 36(4): 437-445.
[12]	SU Xin-yu, WANG Li-jia, NIE Sheng-dong, HU Li-wei, ZHONG Yu-min. Progress of Right Ventricle Segmentation from Short-Axis Images Acquired with Cardiac Cine MRI [J]. Chinese Journal of Magnetic Resonance, 2019, 36(3): 377-391.
[13]	SUN Jing-wen, YAN Shi-ju, HAN Yong-sen, SONG Cheng-li. Classifying the Course of Alzheimer's Disease with Brain MR Images and a Method Based on Three-Dimensional Local Pattern Transformation [J]. Chinese Journal of Magnetic Resonance, 2019, 36(3): 268-277.
[14]	SHAO Dan-dan, WANG Xue-xue, PAN Zi-lai, CHEN Ke-min, ZHANG Zhong-shuai, YUAN Li-li, XU Zi-yue, CHEN Lei, WANG Jin-hong. Imaging Hippocampus of Mental Patients with BLADE Technique [J]. Chinese Journal of Magnetic Resonance, 2019, 36(3): 261-267.
[15]	XU Jia-wen, XU Jian, ZHOU Xiao-dong, ZHANG Cong, CHEN Qun. Multi-GPU Distributed Magnetic Resonance Image Reconstruction Based on Gadgetron [J]. Chinese Journal of Magnetic Resonance, 2018, 35(3): 303-317.