基于双视图特征融合的糖尿病视网膜病变分级

doi:10.3969/j.issn.1000-5641.2023.06.004

华东师范大学学报（自然科学版） ›› 2023, Vol. 2023 ›› Issue (6): 39-48.doi: 10.3969/j.issn.1000-5641.2023.06.004

基于双视图特征融合的糖尿病视网膜病变分级

姜璐璐^1,², 孙司琦^3,^4,*(), 邹海东^2,^5,⁶, 陆丽娜^2,⁶, 冯瑞^1,^2,^3,⁴

1. 复旦大学工程与应用技术研究院, 上海　200433
2. 上海市眼科疾病精准诊疗工程技术研究中心,上海　200080
3. 复旦大学计算机科学技术学院上海市智能信息处理重点实验室, 上海　200433
4. 复旦大学上海市智能视觉计算协同创新中心, 上海　200433
5. 上海交通大学附属第一人民医院, 上海　200080
6. 上海市眼病防治中心, 上海　200040

收稿日期:2022-06-09 出版日期:2023-11-25 发布日期:2023-11-23
通讯作者: 孙司琦 E-mail:siqi_sun@fudan.edu.cn
基金资助:
国家自然科学基金 (62172101); 上海市科委项目 (19DZ2250100, 20DZ1100205)

Diabetic retinopathy grading based on dual-view image feature fusion

Lulu JIANG^1,², Siqi SUN^3,^4,*(), Haidong ZOU^2,^5,⁶, Lina LU^2,⁶, Rui FENG^1,^2,^3,⁴

1. Academy for Engineering and Technology, Fudan University, Shanghai　200433, China
2. Shanghai Engineering Research Center of Precise Diagnosis and Treatment of Eye Diseases, Shanghai　200080, China
3. School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai　200433, China
4. Shanghai Collaborative Innovation Center of Intelligent Visual Computing, Fudan University, Shanghai　200433, China
5. Shanghai General Hospital, Shanghai　200080, China
6. Shanghai Eye Disease Prevention Center, Shanghai　200040, China

Received:2022-06-09 Online:2023-11-25 Published:2023-11-23
Contact: Siqi SUN E-mail:siqi_sun@fudan.edu.cn

摘要/Abstract

摘要：

基于双视图眼底图像的诊断方法被广泛应用于糖尿病视网膜病变 (diabetic retinopathy, DR) 的筛查, 该方法可以有效地解决单视角下图像遮挡和视场受限的问题. 针对如何有效融合不同视图信息来提高DR分级准确率, 提出了一种基于注意力机制的多视角图像之间特征融合的学习方法. 针对眼底图像中病灶占比率较小的问题, 引入了自注意力机制以加强局部病灶特征的学习; 针对双视图眼底图像分类场景, 提出了一种跨视图注意力机制, 有效地利用了双视图之间的信息. 在内部数据集DFiD和公开数据集DeepDR上进行的实验, 验证了所提方法能够有效提高DR分级精度, 可用于大规模DR筛查, 辅助医生实现高效诊断.

关键词: 眼底图像, 特征融合, 双视图融合, 注意力机制, 糖尿病视网膜病变

Abstract:

The diagnostic method based on dual-view fundus imaging is widely used in diabetic retinopathy (DR) screening. This method effectively solves the problems of image occlusion and limited field of view under single-view. This paper proposes a learning method of feature fusion between dual-view images based on the attention mechanism to improve the accuracy of DR classification by effectively integrating different view information. Due to the small proportion of lesions in fundus images, the self-attention mechanism was introduced to enhance the learning of local lesion features. Moreover, a cross-attention mechanism is proposed to effectively utilize information between dual-view images to improve the classification of dual-view fundus images. Experiments were performed on the internal DFiD dataset and public DeepDRiD dataset. The proposed method can effectively improve the accuracy of DR classification and can be used for large-scale DR screening to assist doctors in achieving an efficient diagnosis.

Key words: fundus image, feature fusion, dual-view image fusion, attention mechanism, diabetic retinopathy

中图分类号:

TP183

姜璐璐, 孙司琦, 邹海东, 陆丽娜, 冯瑞. 基于双视图特征融合的糖尿病视网膜病变分级[J]. 华东师范大学学报（自然科学版）, 2023, 2023(6): 39-48.

Lulu JIANG, Siqi SUN, Haidong ZOU, Lina LU, Rui FENG. Diabetic retinopathy grading based on dual-view image feature fusion[J]. Journal of East China Normal University(Natural Science), 2023, 2023(6): 39-48.

图/表 9

图1

图2

图3

图4

表1

表2

表3

表4

图5

参考文献 19

1	LI Y Z, TENG D, SHI X G, et al.. Prevalence of diabetes recorded in mainland China using 2018 diagnostic criteria from the American Diabetes Association: National cross sectional study. BMJ(British Medical Journal), 2020, 369, m997. doi: 10.1136/bmj.m997
2	VAN TULDER G, TONG Y, MARCHIORI E. Multi-view analysis of unregistered medical images using cross-view transformers [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention – MICCAI 2021, Lecture Notes in Computer Science, vol 12903. Cham: Springer, 2021: 104-113.
3	WU N, PHANG J, PARK J, et al.. Deep neural networks improve radiologists’ performance in breast cancer screening. IEEE Transactions on Medical Imaging, 2020, 39 (4): 1184- 1194.
4	WANG G T, ZHAI S W, LASIO G, et al.. Semi-supervised segmentation of radiation-induced pulmonary fibrosis from lung CT scans with multi-scale guided dense attention. IEEE Transactions on Medical Imaging, 2022, 41 (3): 531- 542.
5	WANG H Y, FENG J, ZHANG Z Z, et al.. Breast mass classification via deeply integrating the contextual information from multi-view data. Pattern Recognition, 2018, 80, 42- 52.
6	WANG Z, YIN Y X, SHI J P, et al. Zoom-in-net: Deep mining lesions for diabetic retinopathy detection [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention − MICCAI 2017, Lecture Notes in Computer Science, vol 10435. Cham: Springer, 2017: 267-275.
7	LIN Z W, GUO R Q, WANG Y J, et al. A framework for identifying diabetic retinopathy based on anti-noise detection and attention-based fusion [C]// International Conference on Medical Image Computing and Computer-Assisted Intervention – MICCAI 2018, Lecture Notes in Computer Science, vol 11071. Cham: Springer, 2018: 74-82.
8	ZHAO Z Y, ZHANG K R, HAO X J, et al. Bira-net: Bilinear attention net for diabetic retinopathy grading [C]// 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019: 1385-1389.
9	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY, United States: Curran Associates Inc., 2017: 6000–6010.
10	WANG X L, GIRSHICK R, GUPTA A, et al. Non-local neural networks [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 7794-7803.
11	GENG Z Y, GUO M H, CHEN H X, et al. Is attention better than matrix decomposition? [C]// Proceedings of the 9th International Conference on Learning Representations. ICLR, 2021. https://openreview.net/forum?id=1FvkSpWosOl.
12	YUAN L, CHEN Y P, WANG T, et al. Tokens-to-token vit: Training vision transformers from scratch on imagenet [C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021: 558-567.
13	WANG W H, XIE E Z, LI X, et al. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions [C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021: 548-558.
14	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition [C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016: 770-778.
15	PASZKE A, GROSS S, MASSA F, et al. Pytorch: An imperative style, high-performance deep learning library [C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook, NY, United States: Curran Associates Inc., 2019: 8026-8037.
16	Deep diabetic retinopathy [EB/OL]. (2022-06-01)[2022-05-01]. https://github.com/deepdrdoc/DeepDRiD.
17	HE J J, LI C, YE J, et al. Classification of ocular diseases employing attention-based unilateral and bilateral feature weighting and fusion [C]// 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020: 1258-1261.
18	LI C, YE J, HE J J, et al. Dense correlation network for automated multi-label ocular disease detection with paired color fundus photographs [C]// 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). IEEE, 2020: 1250-1253.
19	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization [C]// 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, 2017: 618-626.

数据集	图像/张					图像总计/张
数据集	DR-0 级	DR-1 级	DR-2 级	DR-3 级	DR-4 级	图像总计/张
DFiD	2492	1072	1308	996	556	6424
DeepDR^[16]	240	160	160	160	80	800

主干网络	融合策略	二次加权kappa	F₁	AUC
ResNet-18	全局特征融合	0.656	0.908	0.927
ResNet-18	本文所提	0.685	0.932	0.940
ResNet-34	全局特征融合	0.676	0.920	0.938
ResNet-34	本文所提	0.698	0.939	0.958
ResNet-50	全局特征融合	0.690	0.930	0.947
ResNet-50	本文所提	0.712	0.950	0.959
ResNet-101	全局特征融合	0.673	0.914	0.936
ResNet-101	本文所提	0.734	0.956	0.966

数据集	方法	二次加权kappa	F₁	AUC
DFiD	AUBNet^[17]	0.733	0.953	0.964
	DCNet^[18]	0.730	0.948	0.957
	本文所提	0.734	0.956	0.966
DeepDR^[16]	AUBNet^[17]	0.692	0.894	0.876
	DCNet^[18]	0.664	0.837	0.870
	本文所提	0.695	0.894	0.916

网络模型	二次加权kappa	F₁	AUC
ResNet-101	0.661	0.894	0.915
ResNet-101 + SA	0.693	0.937	0.936
ResNet-101 + CA	0.705	0.934	0.944
RestNet-101 + SA + CA	0.734	0.956	0.966

[1]	王畅, 马丹, 许华容, 陈攀峰, 陈梅, 李晖. SA-MGKT: 基于自注意力融合的多图知识追踪方法[J]. 华东师范大学学报（自然科学版）, 2024, 2024(5): 20-31.
[2]	冯路平, 施力业, 吴雯, 郑骏, 胡文心, 郑巍. 基于偏好级别陌生人信息辅助的推荐系统[J]. 华东师范大学学报（自然科学版）, 2024, 2024(2): 53-64.
[3]	刘家伟, 林欣. 基于解耦常识性关联的图像描述生成算法[J]. 华东师范大学学报（自然科学版）, 2024, 2024(2): 131-142.
[4]	陈道佳, 陈志云. 分层级描述感知的个性化推荐系统[J]. 华东师范大学学报（自然科学版）, 2023, 2023(6): 73-84.
[5]	段志尚, 冉懿, 吕笃良, 祁杰, 钟佳晨, 袁培森. 基于残差网络和深度可分离卷积增强自注意力机制的窃电识别[J]. 华东师范大学学报（自然科学版）, 2023, 2023(5): 193-204.
[6]	葛进, 陆雪松. 基于UI图像的Web前端代码自动生成[J]. 华东师范大学学报（自然科学版）, 2023, 2023(5): 100-109.
[7]	武朝阳, 毛嘉莉. 基于神经网络的行驶时长预测[J]. 华东师范大学学报（自然科学版）, 2023, 2023(2): 106-118.
[8]	童根梅, 朱敏. 基于双向长短记忆网络和门控注意力的文本分类网络[J]. 华东师范大学学报（自然科学版）, 2022, 2022(2): 67-75.
[9]	梁艳春, 房爱莲. 基于多通道卷积神经网络的中文文本关系抽取[J]. 华东师范大学学报（自然科学版）, 2021, 2021(3): 96-104.
[10]	李小昌, 陈贝, 董启文, 陆雪松. 基于自编码器的旅行同伴挖掘[J]. 华东师范大学学报（自然科学版）, 2020, 2020(5): 179-188.
[11]	傅裕, 李优, 林煜明, 周娅. 基于自注意力机制的冗长商品名称精简方法[J]. 华东师范大学学报(自然科学版), 2019, 2019(5): 113-122,167.
[12]	杨东明, 杨大为, 顾航, 洪道诚, 高明, 王晔. 面向初等数学的知识点关系提取研究[J]. 华东师范大学学报(自然科学版), 2019, 2019(5): 53-65.

基于双视图特征融合的糖尿病视网膜病变分级

Diabetic retinopathy grading based on dual-view image feature fusion

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献 19

相关文章 12

编辑推荐

Metrics

本文评价