基于双路径多模态交互的一阶段视觉定位模型

doi:10.3969/j.issn.1000-5641.2024.02.008

摘要/Abstract

摘要：

现有的一阶段方法分别提取视觉特征映射和文本特征, 并进行多模态推理来预测被引用对象的边界框. 这些方法存在以下两个缺点: 首先, 预先训练的视觉特征提取器在视觉特征中引入了与文本无关的视觉信号, 阻碍了多模态交互; 其次, 现有模型的推理过程缺乏对语言建模的可视化指导. 基于上述缺点, 现有的一阶段方法的推理能力是有限的. 提出了一种提取文本相关视觉特征映射的低阶交互和一种整合视觉特征的高阶交互来指导语言建模, 并进一步对视觉特征进行多步推理. 在此基础上, 提出了一种新的网络结构, 称为双路径多级交互网络. 在5种常用的视觉定位数据集上进行了实验, 结果表明该方法具有较好的性能和实时性.

关键词: 视觉定位, 多模态推理, 引用表达

Abstract:

This study explores the multimodal understanding and reasoning for one-stage visual grounding. Existing one-stage methods extract visual feature maps and textual features separately, and then, multimodal reasoning is performed to predict the bounding box of the referred object. These methods suffer from the following two weaknesses: Firstly, the pre-trained visual feature extractors introduce text-unrelated visual signals into the visual features that hinder multimodal interaction. Secondly, the reasoning process followed in these two methods lacks visual guidance for language modeling. It is clear from these shortcomings that the reasoning ability of existing one-stage methods is limited. We propose a low-level interaction to extract text-related visual feature maps, and a high-level interaction to incorporate visual features in guiding the language modeling and further performing multistep reasoning on visual features. Based on the proposed interactions, we present a novel network architecture called the dual-path multilevel interaction network (DPMIN). Furthermore, experiments on five commonly used visual grounding datasets are conducted. The results demonstrate the superior performance of the proposed method and its real-time applicability.

Key words: visual grounding, multimodal understanding, referring expressions

中图分类号:

TP389.1

王月, 叶加博, 林欣. 基于双路径多模态交互的一阶段视觉定位模型[J]. 华东师范大学学报（自然科学版）, 2024, 2024(2): 65-75.

Yue WANG, Jiabo YE, Xin LIN. Dual-path network with multilevel interaction for one-stage visual grounding[J]. Journal of East China Normal University(Natural Science), 2024, 2024(2): 65-75.

图/表 8

图1

图2

图3

表1

表2

表3

图4

图5

参考文献 38

1	NIU Y, TANG K, ZHANG H, et al. Counterfactual VQA: A cause-effect look at language bias [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 12700-12710.
2	YANG Z, GAN Z, WANG J, et al. An empirical study of GPT-3 for few-shot knowledge-based VQA [C] // Proceedings of the AAAI Conference on Artificial Intelligence. 2022: 3081-3089.
3	TU Z, WANG Y, BIRKBECK N, et al.. UGC-VQA: Benchmarking blind video quality assessment for user generated content. IEEE Transactions on Image Processing, 2021, 30, 4449- 4464.
4	SENEVIRATNE K L, MUNAWEERA I, PEIRIS S E, et al.. Recent progress in visible-light active (VLA) TiO₂ nano-structures for enhanced photocatalytic activity (PCA) and antibacterial properties: A review. Iranian Journal of Catalysis, 2021, 11 (3): 217- 245.
5	TOBIN J J, OFFNER S S R, KRATTER K M, et al.. The VLA/ALMA nascent disk and multiplicity (VANDAM) survey of Orion protostars. V. A characterization of protostellar multiplicity. The Astrophysical Journal, 2022, 925, 39.
6	WEI M, CHEN L, JI W, et al. Rethinking the two-stage framework for grounded situation recognition [C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2022: 2651-2658.
7	QIAO Y, DENG C, WU Q.. Referring expression comprehension: A survey of methods and datasets. IEEE Transactions on Multimedia, 2020, 23, 4426- 4440.
8	YANG S, LI G, YU Y. Dynamic graph attention for referring expression comprehension [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 4644-4653.
9	CHEN L, MA W, XIAO J, et al. Ref-NMS: Breaking proposal bottlenecks in two-stage referring expression grounding [C]// Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 1036-1044.
10	MU Z, TANG S, TAN J, et al. Disentangled motif-aware graph learning for phrase grounding [C]// Proceedings of the AAAI Conference on Artificial Intelligence. 2021: 13587-13594.
11	YANG Z, GONG B, WANG L, et al. A fast and accurate one-stage approach to visual grounding [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 4683-4693.
12	YANG Z, CHEN T, WANG L, et al. Improving one-stage visual grounding by recursive sub-query construction [C]// Computer Vision–ECCV 2020. 2020: 387-404.
13	DENG J, YANG Z, CHEN T, et al. TransVG: End-to-end visual grounding with transformers [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 1769-1779.
14	KAMATH A, SINGH M, LECUN Y, et al. MDETR-modulated detection for end-to-end multi-modal understanding [C]// Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 1780-1790.
15	YE J, LIN X, HE L, et al. One-stage visual grounding via semantic-aware feature filter [C]// Proceedings of the 29th ACM International Conference on Multimedia. 2021: 1702-1711.
16	KAZEMZADEH S, ORDONEZ V, MATTEN M, et al. ReferitGame: Referring to objects in photographs of natural scenes [C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. 2014: 787-798.
17	YU L, POIRSON P, YANG S, et al. Modeling context in referring expressions [C]// Computer Vision–ECCV 2016. 2016: 69-85.
18	MAO J, HUANG J, TOSHEV A, et al. Generation and comprehension of unambiguous object descriptions [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016: 11-20.
19	PLUMMER B A, WANG L, CERVANTES C M, et al. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models [C]// Proceedings of the IEEE International Conference on Computer Vision. 2015: 2641-2649.
20	YANG S, LI G, YU Y. Graph-structured referring expression reasoning in the wild [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020: 9952-9961.
21	REDMON J, FARHADI A. YOLOv3: An incremental improvement [EB/OL]. (2018-04-08) [2022-11-20]. https://doi.org/10.48550/arXiv.1804.02767.
22	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding [EB/OL]. (2019-05-24) [2022-11-22]. https://doi.org/10.48550/arXiv.1810.04805.
23	SHARMA P, DING N, GOODMAN S, et al. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018: 2556-2565.
24	WOO S, PARK J, LEE J Y, et al. Cbam: Convolutional block attention module [C]// Proceedings of the European Conference on Computer Vision. 2018: 3-19.
25	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 6000–6010.
26	TIAN Z, CHU X, WANG X, et al.. Fully convolutional one-stage 3D object detection on lidar range images. Advances in Neural Information Processing Systems, 2022, 35, 34899- 34911.
27	GIRSHICK R. Fast R-CNN [C]// Proceedings of the IEEE International Conference on Computer Vision. 2015: 1440-1448.
28	REZATOFIGHI H, TSOI N, GWAK J Y, et al. Generalized intersection over union: A metric and a loss for bounding box regression [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 658-666.
29	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft coco: Common objects in context [C]// Computer Vision–ECCV 2014. 2014: 740-755.
30	LOSHCHILOV I, HUTTER F. Decoupled weight decay regularization [EB/OL]. (2017-11-14) [2022-11-15]. https://doi.org/10.48550/arXiv.1711.05101.
31	REN S Q, HE K M , GIRSHICK R, et al. Faster R-CNN: Towards real-time object detection with region proposal networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligenc, 2017, 39(6): 1137-1149.
32	NAGARAJA V K, MORARIU V I, DAVIS L S. Modeling context between objects for referring expression understanding [C]// Computer Vision–ECCV 2016. 2016: 792-807.
33	HU R, ROHRBACH M, ANDREAS J, et al. Modeling relationships in referential expressions with compositional modular networks [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 1115-1124.
34	ZHUANG B, WU Q, SHEN C, et al. Parallel attention: A unified framework for visual object discovery through dialogs and queries [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4252-4261.
35	ZHANG H, NIU Y, CHANG S F. Grounding referring expressions in images by variational context [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018: 4158-4166.
36	WANG P, WU Q, CAO J, et al. Neighbourhood watch: Referring expression comprehension via language-guided graph attention networks [C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 1960-1968.
37	YU L, TAN H, BANSAL M, et al. A joint speaker-listener-reinforcer model for referring expressions [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017: 7282-7290.
38	CHEN X, MA L, CHEN J, et al. Real-time referring expression comprehension by single-stage grounding network [EB/OL]. (2018-12-09)[2022-11-18]. https://doi.org/10.48550/arXiv.1812.03426.

方法	模型	特征抽取器	RefCOCO上的mAP/%			RefCOCO +上的mAP/%			RefCOCOg上的mAP/%			运行时间/ms
方法	模型	特征抽取器	Val	testA	testB	Val	testA	testB	Val	testA	testB	运行时间/ms
两阶段方法	MMI^[18]	VGG16		64.90	54.51		54.03	42.81	45.85
	Neg Bag^[32]	VGG16		58.60	56.40						49.50
	CMN^[33]	VGG16		71.03	65.77		54.32	47.76	57.47
	ParallelAttn^[34]	VGG16		75.31	65.52		61.34	50.86	58.03
	VC^[35]	VGG16		73.33	67.44		58.40	53.18	62.30
	LGRAN^[36]	VGG16		76.60	66.4		64.0	53.4	61.78
	SLR^[37]	ResNet101	69.48	73.71	64.96	55.71	60.74	48.90		60.21	59.63
	MattNet^[6]	ResNet101	76.40	80.43	69.28	64.93	70.26	56.00		66.67	67.01	320
	DGA^[8]	ResNet101		78.42	65.53		69.07				63.28	341
一阶段方法	SSG^[38]	DarkNet-53		76.51	67.50		62.14	49.27	47.47	58.80		25
	FAOA^[11]	DarkNet-53	72.05	74.81	67.59	55.72	60.37	48.54	48.14	59.03	58.70	23
	ReSC^[12]	DarkNet-53	76.59	78.22	73.25	63.23	66.64	55.53	60.96	64.87	64.87	26
	DPMIN	DarkNet-53	78.21	78.23	75.90	63.33	66.71	56.74	61.49	65.27	66.10	30

方法	模型	特征抽取器	ReferItGame上的mAP/%	Flickr30k上的mAP/%
两阶段方法	CMN^[33]	VGG16	28.33
	VC^[35]	VGG16	31.13
	MattNet^[6]	ResNet101	29.04
	Similarity Net	ResNet101	34.54	50.89
一阶段方法	SSG^[38]	DarkNet-53	54.24
	FAOA^[11]	DarkNet-53	60.67	68.71
	RCCF	DLA-34	63.79
	ReSC^[12]	DarkNet-53	64.33	69.04
	DPMIN	DarkNet-53	66.88	69.47

低阶交互	高阶交互	无锚预测头	Val上的mAP/%	test上的mAP/%
			58.76	59.42
√			65.50	63.31
	√		64.35	62.16
		√	59.94	60.76
√	√		66.22	64.60
	√	√	67.66	65.17
√		√	67.42	64.89
√	√	√	68.51	66.88