Diabetic retinopathy grading based on dual-view image feature fusion

doi:10.3969/j.issn.1000-5641.2023.06.004

Abstract

Abstract:

The diagnostic method based on dual-view fundus imaging is widely used in diabetic retinopathy (DR) screening. This method effectively solves the problems of image occlusion and limited field of view under single-view. This paper proposes a learning method of feature fusion between dual-view images based on the attention mechanism to improve the accuracy of DR classification by effectively integrating different view information. Due to the small proportion of lesions in fundus images, the self-attention mechanism was introduced to enhance the learning of local lesion features. Moreover, a cross-attention mechanism is proposed to effectively utilize information between dual-view images to improve the classification of dual-view fundus images. Experiments were performed on the internal DFiD dataset and public DeepDRiD dataset. The proposed method can effectively improve the accuracy of DR classification and can be used for large-scale DR screening to assist doctors in achieving an efficient diagnosis.

Key words: fundus image, feature fusion, dual-view image fusion, attention mechanism, diabetic retinopathy

CLC Number:

TP183

Lulu JIANG, Siqi SUN, Haidong ZOU, Lina LU, Rui FENG. Diabetic retinopathy grading based on dual-view image feature fusion[J]. Journal of East China Normal University(Natural Science), 2023, 2023(6): 39-48.

Figures/Tables 9

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Table 2

Table 3

Table 4

Fig.5

References 19

1	LI Y Z, TENG D, SHI X G, et al.. Prevalence of diabetes recorded in mainland China using 2018 diagnostic criteria from the American Diabetes Association: National cross sectional study. BMJ(British Medical Journal), 2020, 369, m997. doi: 10.1136/bmj.m997
2	VAN TULDER G, TONG Y, MARCHIORI E. Multi-view analysis of unregistered medical images using cross-view transformers [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention – MICCAI 2021, Lecture Notes in Computer Science, vol 12903. Cham: Springer, 2021: 104-113.
3	WU N, PHANG J, PARK J, et al.. Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Transactions on Medical Imaging, 2020, 39 (4): 1184- 1194.
4	WANG G T, ZHAI S W, LASIO G, et al.. Semi-supervised segmentation of radiation-induced pulmonary fibrosis from lung CT scans with multi-scale guided dense attention. IEEE Transactions on Medical Imaging, 2022, 41 (3): 531- 542.
5	WANG H Y, FENG J, ZHANG Z Z, et al.. Breast mass classification via deeply integrating the contextual information from multi-view data. Pattern Recognition, 2018, 80, 42- 52.
6	WANG Z, YIN Y X, SHI J P, et al. Zoom-in-net: Deep mining lesions for diabetic retinopathy detection [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention − MICCAI 2017, Lecture Notes in Computer Science, vol 10435. Cham: Springer, 2017: 267-275.
7	LIN Z W, GUO R Q, WANG Y J, et al. A framework for identifying diabetic retinopathy based on anti-noise detection and attention-based fusion [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention – MICCAI 2018, Lecture Notes in Computer Science, vol 11071. Cham: Springer, 2018: 74-82.
8	ZHAO Z Y, ZHANG K R, HAO X J, et al. Bira-net: Bilinear attention net for diabetic retinopathy grading [C]// 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019: 1385-1389.
9	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY, United States: Curran Associates Inc., 2017: 6000–6010.
10	WANG X L, GIRSHICK R, GUPTA A, et al. Non-local neural networks [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 7794-7803.
11	GENG Z Y, GUO M H, CHEN H X, et al. Is attention better than matrix decomposition? [C]// Proceedings of the 9th International Conference on Learning Representations. ICLR, 2021. https://openreview.net/forum?id=1FvkSpWosOl.
12	YUAN L, CHEN Y P, WANG T, et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet [C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021: 558-567.
13	WANG W H, XIE E Z, LI X, et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions [C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021: 548-558.
14	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016: 770-778.
15	PASZKE A, GROSS S, MASSA F, et al. Pytorch: An imperative style, high-performance deep learning library [C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook, NY, United States: Curran Associates Inc., 2019: 8026-8037.
16	Deep diabetic retinopathy [EB/OL]. (2022-06-01)[2022-05-01]. https://github.com/deepdrdoc/DeepDRiD.
17	HE J J, LI C, YE J, et al. Classification of ocular diseases employing attention-based unilateral and bilateral feature weighting and fusion [C]// 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020: 1258-1261.
18	LI C, YE J, HE J J, et al. Dense correlation network for automated multi-label ocular disease detection with paired color fundus photographs [C]// 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020: 1250-1253.
19	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization [C]// 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017: 618-626.

数据集	图像/张					图像总计/张
数据集	DR-0 级	DR-1 级	DR-2 级	DR-3 级	DR-4 级	图像总计/张
DFiD	2492	1072	1308	996	556	6424
DeepDR^[16]	240	160	160	160	80	800

主干网络	融合策略	二次加权kappa	F₁	AUC
ResNet-18	全局特征融合	0.656	0.908	0.927
ResNet-18	本文所提	0.685	0.932	0.940
ResNet-34	全局特征融合	0.676	0.920	0.938
ResNet-34	本文所提	0.698	0.939	0.958
ResNet-50	全局特征融合	0.690	0.930	0.947
ResNet-50	本文所提	0.712	0.950	0.959
ResNet-101	全局特征融合	0.673	0.914	0.936
ResNet-101	本文所提	0.734	0.956	0.966

数据集	方法	二次加权kappa	F₁	AUC
DFiD	AUBNet^[17]	0.733	0.953	0.964
	DCNet^[18]	0.730	0.948	0.957
	本文所提	0.734	0.956	0.966
DeepDR^[16]	AUBNet^[17]	0.692	0.894	0.876
	DCNet^[18]	0.664	0.837	0.870
	本文所提	0.695	0.894	0.916

网络模型	二次加权kappa	F₁	AUC
ResNet-101	0.661	0.894	0.915
ResNet-101 + SA	0.693	0.937	0.936
ResNet-101 + CA	0.705	0.934	0.944
RestNet-101 + SA + CA	0.734	0.956	0.966

[1]	Luping FENG, Liye SHI, Wen WU, Jun ZHENG, Wenxin HU, Wei ZHENG. Collaborative stranger review-based recommendation [J]. Journal of East China Normal University(Natural Science), 2024, 2024(2): 53-64.
[2]	Daojia CHEN, Zhiyun CHEN. Hierarchical description-aware personalized recommendation system [J]. Journal of East China Normal University(Natural Science), 2023, 2023(6): 73-84.
[3]	Zhishang DUAN, Yi RAN, Duliang LYU, Jie QI, Jiachen ZHONG, Peisen YUAN. Identifying electricity theft based on residual network and depthwise separable convolution enhanced self attention [J]. Journal of East China Normal University(Natural Science), 2023, 2023(5): 193-204.
[4]	Jin GE, Xuesong LU. Automatic generation of Web front-end code based on UI images [J]. Journal of East China Normal University(Natural Science), 2023, 2023(5): 100-109.
[5]	Zhaoyang WU, Jiali MAO. Research on travel time prediction based on neural network [J]. Journal of East China Normal University(Natural Science), 2023, 2023(2): 106-118.
[6]	Yanchun LIANG, Ailian FANG. Chinese text relation extraction based on a multi-channel convolutional neural network [J]. Journal of East China Normal University(Natural Science), 2021, 2021(3): 96-104.
[7]	LI Xiaochang, CHEN Bei, DONG Qiwen, LU Xuesong. Discovering traveling companions using autoencoders [J]. Journal of East China Normal University(Natural Science), 2020, 2020(5): 179-188.
[8]	FU Yu, LI You, LIN Yu-ming, ZHOU Ya. Self-attention based neural networks for product titles compression [J]. Journal of East China Normal University(Natural Sc, 2019, 2019(5): 113-122,167.
[9]	YANG Dong-ming, YANG Da-wei, GU Hang, HONG Dao-cheng, GAO Ming, WANG Ye. Research on knowledge point relationship extraction for elementary mathematics [J]. Journal of East China Normal University(Natural Sc, 2019, 2019(5): 53-65.