Momentum-updated representation with reconstruction constraint for limited-view 3D object recognition

doi:10.3969/j.issn.1000-5641.2023.06.006

Abstract

Abstract:

We propose a neural network training framework called momentum-updated representation with reconstruction constraint for 3D (three-dimensional) object recognition using 2D (two-dimensional) images without angle labels. First, self-supervised learning is employed to address the lack of angle labels. Second, we use momentum updating based on a dynamic queue to maintain the stability of the object representation. Furthermore, the reconstruction constraint is applied to the learning process with an auto-encoder module, which enables the representation to capture more semantic information of the objects. Finally, during training, a dynamic queue reduction strategy is proposed for handling the imbalanced data distribution. Experiments on two popular multi-view datasets, ModelNet and ShapeNet, demonstrate that the proposed method outperforms existing methods.

Key words: multi-view object recognition, self-supervised learning, auto-encoder

CLC Number:

TP391.41

Ruibo CUI, Feng WANG. Momentum-updated representation with reconstruction constraint for limited-view 3D object recognition[J]. Journal of East China Normal University(Natural Science), 2023, 2023(6): 61-72.

Figures/Tables 5

Fig.1

Table 1

Table 2

Table 3

Fig.2

References 29

1	QI S H, NING X, YANG G W, et al.. Review of multi-view 3D object recognition methods based on deep learning. Displays, 2021, 69, 102053. doi: 10.1016j.displa.2021.102053
2	FENG Y F, ZHANG Z Z, ZHAO X B, et al. GVCNN: Group-view convolutional neural networks for 3D shape recognition [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 264-272. DOI: 10.1109/CVPR.2018.00035.
3	GAO Z, WANG D Y, HE X G, et al. Group-pair convolutional neural networks for multi-view based 3d object retrieval [C]// Proceedings of the AAAI Conference on Artificial Intelligence. AAAI, 2018: 2223-2231. DOI: 10.1007/s10489-021-02471-7.
4	SU H, MAJI S, KALOGERAKIS E, et al. Multi-view convolutional neural networks for 3D shape recognition [C]// Proceedings of the IEEE International Conference on Computer Vision. IEEE, 2015: 945-953. DOI:10.1109ICCV.2015.114.
5	SUN K, ZHANG J H, LIU J M, et al.. DRCNN: Dynamic routing convolutional neural network for multi-view 3D object recognition. IEEE Transactions on Image Processing, 2020, 30, 868- 877. doi: 10.1109TIP.2020.3039378
6	DAI G X, XIE J, FANG Y. Siamese CNN-BILSTM architecture for 3D shape representation learning [C]// Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI-18). AAAI, 2018: 670-676. DOI: 10.24963ijcai.201893.
7	WEI X, YU R X, SUN J. View-gcn: View-based graph convolutional network for 3d shape analysis [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 1850-1859. DOI :10.1109CVPR42600.2020.00192.
8	HO C H, MORGADO P, PERSEKIAN A, et al. PIEs: Pose Invariant embeddings [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2019: 12369-12378. DOI: 10.1109/CVPR.2019.01266.
9	AGRAWAL P, CARREIRA J, MALIK J. Learning to see by moving [C]// Proceedings of the IEEE International Conference on Computer Vision. IEEE, 2015: 37-45. DOI: 10.1109/ICCV.2015.13.
10	JAYARAMANU D, GAO R, GRAUMAN K. Unsupervised learning through one-shot image-based shape reconstruction [EB/OL]. (2018-07-31)[2022-04-27]. https://arxiv.org/abs/1709.00505v1.
11	NOROOZI M, FAVARO P. Unsupervised learning of visual representations by solving jigsaw puzzles [C]// European Conference on Computer Vision, Computer Vision – ECCV 2016 . Cham: Springer. 2016: 69-84. DOI: 10.1007/978-3-319-46466-4_5.
12	HE K M, CHEN X L, XIE S N, et al. Masked autoencoders are scalable vision learners [C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022: 15979-15988. DOI: 10.1109/CVPR52688.2022.01553.
13	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth $ 16\times16 $ words: Transformers for image recognition at scale [EB/OL]. (2021-06-03)[2022-04-28]. https://doi.org/10.48550/arXiv.2010.11929.
14	HE K M, FAN H Q, WU Y X, et al. Momentum contrast for unsupervised visual representation learning [C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 9729-9738. DOI: 10.1109/CVPR42600.2020.00975.
15	CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations [C]// Proceedings of the 37th International Conference on Machine Learning( ICML’20). The Journal of Machine Learning Research (JMLR), 2020: 1597-1607. DOI: 10.5555/3524938.3525087.
16	CHEN X L, HE K M. Exploring simple siamese representation learning [C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021: 15745-15753. DOI: 10.1109/CVPR46437.2021.01549.
17	YE M, ZHANG X, YUEN P C, et al. Unsupervised embedding learning via invariant and spreading instance feature [C]// 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019: 6203-6212. DOI: 10.1109/CVPR.2019.00637.
18	KANEZAKI A, MATSUSHITA Y, NISHIDA Y. RotationNet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints [C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018: 5010-5019. DOI:10.1109/CVPR.2018.00526.
19	SONG R, ZHANG W, ZHAO Y T, et al. Unsupervised multi-view CNN for salient view selection of 3D objects and scenes [C]//European Conference on Computer Vision, Computer Vision – ECCV 2020. Cham: Springer, 2020: 454-470. DOI: 10.1007/s11263- 022-01592-x.
20	HO C H, LIU B, WU T Y, et al. Exploit clues from views: Self-supervised and regularized learning for multiview object recognition [C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2020: 9087-9097. DOI:10.1109/CVPR42600.2020.00911.
21	KINGMA D P, WELLING M. Auto-encoding variational bayes [EB/OL]. (2014-05-01)[2022-04-28]. https://arxiv.org/abs/1312.6114v10.
22	GOOGFELLOW I, BENGIO Y, COURVILLE A, et al. Deep Learning [M]. Cambridge, MA USA: MIT Press, 2016.
23	KINGMA D P, SALIMANS T, WELLING M. Variational dropout and the local reparameterization trick [C]// Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2. Cambridge, MA USA: MIT Press. 2015: 2575-2583.
24	WU Z R, SONG S R, KHOSLA A, et al. 3D ShapeNets: A deep representation for volumetric shapes [C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2015: 1912-1920. DOI:10.1109/CVPR.2015.7298801.
25	CHANG A X, FUNKHOUSER T, Guibas L, et al. ShapeNet: An information-rich 3D model repository [EB/OL]. (2015-12-09)[2022-04-28]. https://arxiv.org/abs/1512.03012.
26	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition [EB/OL]. (2015-04-10)[2022-04-28]. https://arxiv.org/abs/1409.1556v6.
27	DENG J, DONG W, SOCHER R, et al. ImageNet: A large-scale hierarchical image database [C]// 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2009: 248-255. DOI:10.1109/CVPR.2009.5206848.
28	HOSPEDALES T, ANTONIOU A, MICAELLI P, et al.. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44 (9): 5149- 5169. doi: 10.1109/TPAMI.2021.3079209
29	VAN DER M L, HINTON G.. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008, 9, 2579- 2605.

方法	准确率(ModelNet)/%		准确率(ShapeNet) /%
方法	Seen	Unseen	Seen	Unseen
UEL^[17]	47.9	46.5	68.7	53.4
ViT^[13]	65.0	55.0	76.8	61.7
SimCLR^[15]	66.3	58.4	79.2	63.8
MoCo^[14]	69.3	59.7	78.5	60.1
MAE^[12]	67.3	56.4	72.9	54.9
VISPE^[20]	71.2	64.4	82.9	65.5
MUR +WithoutDQ+Pret	75.4	68.5	79.3	63.8
MUR +Pret	76.2	68.6	80.7	66.1
MUR+AE + Pret	78.1	71.0	81.8	67.1
MUR+AE+DQR	74.0	67.8	83.7	67.4
MUR +AE+DQR+Pret	78.0	71.8	83.0	67.2
MUR+AE+DQR+Pre(batch size 32)	76.5	69.3	84.8	67.3

方法	召回率(k-means)/%				NMI值/%
方法	$ k=1 $	$ k=2 $	$ k=4 $	$ k=8 $	NMI值/%
UEL^[17]	77.8	85.4	91.6	95.7	24.6
ViT^[13]	86.8	91.1	94.9	97.2	23.8
SimCLR^[15]	89.4	93.4	96.4	98.0	31.1
MoCo^[14]	90.7	94.3	97.1	98.4	34.2
MAE^[12]	89.7	94.0	96.8	98.3	27.6
VISPE^[20]	95.5	97.7	98.6	99.2	51.1
MUR+WithoutDQ+Pret	97.4	98.6	99.4	99.7	56.6
MUR+Pret	97.9	99.0	99.5	99.8	57.0
MUR +AE+Pret	98.5	99.2	99.7	99.9	56.6
MUR+AE+DQR	97.7	98.6	99.4	99.7	58.8
MUR+AE+DQR+Pret	98.2	99.1	99.7	99.8	57.2
MUR+AE+DQR+Pre(batch size 32)	98.4	99.1	99.6	99.8	55.1

方法	准确率(K-shot)/%
方法	$ K=1 $	$ K=3 $	$ K=5 $
UEL^[17]	23.8	30.9	34.2
ViT^[13]	38.5	39.9	43.0
SimCLR^[15]	40.3	41.8	43.9
MoCo^[14]	40.8	44.3	45.5
MAE^[12]	41.3	42.5	42.7
VISPE^[20]	43.1	52.5	55.9
MUR+WithoutDQ+Pret	43.9	57.7	63.3
MUR+Pret	44.6	58.3	64.7
MUR+AE+Pret	46.8	59.7	65.1
MUR+AE+DQR	45.1	57.6	63.8
MUR+AE+DQR+Pret	46.7	59.6	65.8
MUR+ AE+DQR+Pre(batch size 32)	44.6	59.3	64.7