Distant supervision relation extraction via the influence function

doi:10.3969/j.issn.1000-5641.2022.06.009

Abstract

Abstract:

Distant supervision relation extraction captures noisy instances while reducing the burden of manual annotation, which hinders the training and testing process. To alleviate this problem, we proposed a de-noising method based on the influence function. The influence function measures the influence of each training point; the influence of one training point is defined as the change in test loss after removing the training point. We observed that this property could be used to determine whether a training instance involves noisy data. First, we designed a scoring function based on the influence function. Then, we integrated the scoring function into a bootstrapping framework to obtain the final denoising dataset from a small clean set. Using this preprocessing method, every distantly supervised dataset could be denoised by our method. Experimental results showed that the proposed denoised dataset can achieve good performance on a public dataset.

Key words: distant supervision, relation extraction, influence function, bootstrapping

CLC Number:

TP391.1

Ziyin HUANG, Yuanbin WU. Distant supervision relation extraction via the influence function[J]. Journal of East China Normal University(Natural Science), 2022, 2022(6): 79-86.

Figures/Tables 5

Fig.1

Fig.2

Table 1

Table 2

Table 3

References 17

1	MINTZ M, BILLS S, SNOW R, et al. Distant supervision for relation extraction without labeled [C]//Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. 2009: 1003-1011.
2	DENNIS C, SANFORD W. Residuals and Influence in Regression [M]. New York: Chapman and Hall, 1982.
3	ZENG D J, LIU K, LAI S W, et al. Relation classification via convolutional deep neural network [C]//Proceedings of COLING, the 25th International Conference on Computational Linguistics. 2014: 2335-2344.
4	ZHANG S, ZHENG D Q, HU X C, et al. Bidirectional long short-term memory networks for relation classification [C]//Proceedings of the 29th Pacific Asia Conference on Language, Information and Computation. 2015: 73-78.
5	HE Z Q, CHEN W L, LI Z H, et al. See: Syntax-aware entity embedding for neural relation extraction [C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. AAAI, 2018: 5795-5802.
6	SURDEANU M, TIBSHIRANI J, NALLAPATI R, et al. Multi-instance multi-label learning for relation extraction [C]//Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. 2012: 455-465.
7	LIN Y K, SHEN S Q, LIU Z Y, et al. Neural relation extraction with selective attention over instances [C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016: 2124-2133.
8	QIN P D, XU W R, WANG W Y. Robust distant supervision relation extraction via deep reinforcement learning [C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018: 2137-2147.
9	QIN P D, XU W R, WANG W Y. DSGAN: Generative adversarial training for distant supervision relation extraction [C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018: 496-505.
10	MA R T, GUI T, LI L Y, et al. SENT: Sentence-level distant relation extraction via negative training [C]//Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. 2021: 6201-6213.
11	JIA W, DAI D, XIAO X Y, et al. ARNOR: Attention regularization based noise reduction for distant supervision relation classification [C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019: 1399-1408.
12	KOH P W, LIANG P. Understanding black-box pre dictions via influence function [C]//Proceedings of the 34th International Conference on Machine Learning. 2017: 1885-1894.
13	REN Z Z, YEH R A, SCHWING A G. Not all unlabeled data are equal: Learning to weight data in semi-supervised learning [C]//Proceedings of the 34th Conference on Neural Information Processing Systems. 2020. https://doi.org/10.48550/arXiv.2007.01293.
14	XU M J, KAZANTSEV F. Understanding goal-oriented active learning via influence functions [C]//Proceedings of the NeurIPS 2019 Workshop on Machine Learning with Guarantees. 2019. https://arxiv.org/pdf/1905.13183v3.pdf.
15	ZENG D J, LIU K, CHEN Y B, et al. Distant supervision for relation extraction via piecewise convolutional neural networks [C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (ACL), 2015: 1753-1762.
16	RIEDEL S, YAO L M, MCCALLUM A. Modeling relations and their mentions without labeled text [C]//Proceedings of the 2010 European conference on Machine Learning and Knowledge Discovery in Databases. 2010: 148-163.
17	PENNINGTON J, SOCHER R, MANNING C. Glove: Global vectors for word representation [C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics (ACL), 2014: 1532-1543.

NYT	句子数量	样本数量	正例样本数量
训练集	233038	367596	106653
开发集	1596	4567	975
测试集	1596	4484	1050

方法(模型)	验证集			测试集
方法(模型)	准确率/%	召回率/%	F₁/%	准确率/%	召回率/%	F₁/%
CNN	39.27	73.80	51.26	42.41	76.64	54.60
CNN + ATT	68.09	47.49	55.95	67.31	49..83	52.27
CNN + RL	42.50	71.62	53.34	43.70	72.34	54.49
CNN + IF	57.97	71.74	64.12	60.96	68.76	64.61
CNN + IF_10	63.15	68.71	58.42	61.75	64.28	59.33
CNN + IF_50	60.08	61.46	60.76	60.50	63.22	61.83
CNN + IF_100	58.32	69.15	63.27	62.83	61.56	62.19
CNN + IF_OT	42.48	72.28	53.51	45.63	72.30	55.95

	噪声样本	真(假)阳性	影响分数
正例	··· Leslie Moonves, the chief executive of $\underline{{\rm{CBS}}}$, ···	真阳性	4.29e–3
正例	··· Yusuf Hamied, the chairman of $\underline{{\rm{Cipla}}}$, ···	真阳性	3.41e–3
临界样本	···after $\underline{{\rm{Sony}}}$’s news conference and a speech by sir Howard Stringer, its chairman···	真阳性	6.72e–7
临界样本	··· Richard C. Noteaert, the company’s chief executive, said Qwest spent···	假阳性	–2.13e–6
负例	··· Joshua Landis, a history professor at the University of Oklahoma···	真阳性	–5.24e–3
负例	As Bruce Wasserstein left St. Regis Hotel in Manhattan on Tuesday afternoon after presenting $\underline{{\rm{Lazard}}}$’s plan···	假阳性	–1.86e–3

[1]	Xiaoqin MA, Xiaohe GUO, Yufeng XUE, Lin YANG, Yuanzhe CHEN. Data augmentation technology for named entity recognition [J]. Journal of East China Normal University(Natural Science), 2021, 2021(5): 14-23.
[2]	Yanchun LIANG, Ailian FANG. Chinese text relation extraction based on a multi-channel convolutional neural network [J]. Journal of East China Normal University(Natural Science), 2021, 2021(3): 96-104.
[3]	WANG Jianing, HE Yi, ZHU Renyu, LIU Tingting, GAO Ming. Relation extraction via distant supervision technology [J]. Journal of East China Normal University(Natural Science), 2020, 2020(5): 113-130.
[4]	SUN Chen, FU Ying-nan, CHENG Wen-liang, QIAN Wei-ning. Chinese named entity relation extraction for enterprise knowledge graph construction [J]. Journal of East China Normal University(Natural Sc, 2018, 2018(3): 55-66.