Bi-directional long short-term memory and bi-directional gated attention networks for text classification

doi:10.3969/j.issn.1000-5641.2022.02.008

Abstract

Abstract:

In this paper, we propose the construction of a bi-directional fully connected structure for better extraction of context information. We also propose the construction of a bi-directional attention structure for compressing matrices containing rich text features into a vector. The bi-directional fully connected structure and the gated structure are then combined. This research demonstrates that the proposed combined structure has a net positive effect on text classification accuracy. Finally, by combining these three structures and a bi-direction long short-term memory, we propose a new text classification model. Using this model, we obtained competitive results on seven commonly used text classification datasets and achieved state-of-the-art results on five of them. Experiments showed that the combination of these structures can significantly reduce classification errors.

Key words: text classification, attention, long short-term memory

CLC Number:

TP399

Genmei TONG, Min ZHU. Bi-directional long short-term memory and bi-directional gated attention networks for text classification[J]. Journal of East China Normal University(Natural Science), 2022, 2022(2): 67-75.

Figures/Tables 8

Fig.1

Fig.2

Table 1

Table 2

Table 3

Table 4

Table 5

Table 6

References 18

1	WANG G, LI C, WANG W, et al. Joint embedding of words and labels for text classification [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018: 2321-2331.
2	PENNINGTON J, SOCHER R, MANNING C. GloVe: Global vectors for word representation [C]// Conference on Empirical Methods in Natural Language Processing. 2014: 1532-1543.
3	PETERS M, NEUMANN M, IYYER M, et al. Deep contextualized word representations [EB/OL]. (2018-03-22)[2020-10-16]. https://arxiv.org/pdf/1802.05365v2.pdf.
4	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding [EB/OL]. (2019-03-24)[2020-10-16]. https://arxiv.org/pdf/1810.04805.pdf.
5	RADFORD A, WU J, CHILD R, et al. Language models are unsupervised multitask learners [EB/OL]. (2019-01-08)[2020-10-16]. https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf.
6	KIM Y. Convolutional neural networks for sentence classification [EB/OL]. (2014-09-03)[2020-10-16]. https://arxiv.org/pdf/1408.5882v2.pdf.
7	CONNEAU A, SCHWENK H, BARRAULT L, et al. Very deep convolutional networks for text classification [EB/OL]. (2017-01-27)[2020-10-16]. https://arxiv.org/pdf/1606.01781v2.pdf.
8	JOHNSON R, TONG Z. Deep pyramid convolutional neural networks for text categorization [C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017: 562-570.
9	HOCHREITER S, SCHMIDHUBER J. Long short-term memory [J]. Neural Computation, 1997, 9(8): 1735-1780.
10	LAI S, XU L, LIU K, et al. Recurrent convolutional neural networks for text classification [C]// Proceeding of the 25th International Joint Conference on Artificial Intelligence. 2015: 2267-2273.
11	PENG Z, QI Z, ZHENG S, et al. Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling [EB/OL]. (2016-11-21)[2020-10-22]. https://arxiv.org/pdf/1611.06639.pdf.
12	PAPPAS N, POPESCU-BELIS A. Multilingual hierarchical attention networks for document classification [EB/OL]. (2017-09-15)[2020-09-14]. https://arxiv.org/pdf/1707.00896v4.pdf.
13	ZHANG X, ZHAO J, LECUN Y. Character-level convolutional networks for text classification [EB/OL]. (2015-09-10)[2020-09-11]. https://arxiv.org/pdf/1509.01626v2.pdf.
14	TURIAN J P, RATINOV L A, BENGIO Y. Word representations: A simple and general method for semi-supervised learning [C]// Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. 2010: 384-394.
15	KINGMA D, BA J. Adam: A method for stochastic optimization [EB/OL]. (2015-07-30)[2020-10-16]. https://arxiv.org/pdf/1412.6980v8.pdf.
16	XIAO Y, CHO K. Efficient character-level document classification by combining convolution and recurrent layers [EB/OL]. (2016-02-01)[2020-10-16]. https://arxiv.org/pdf/1602.00367v1.pdf.
17	JOULIN A, GRAVE E, BOJANOWSKI P, et al. Bag of tricks for efficient text classification [EB/OL]. (2016-08-09)[2020-10-13]. https://arxiv.org/pdf/1607.01759v3.pdf.
18	QIAO C, HUANG B, NIU G, et al. A new method of region embedding for text classification [EB/OL]. (2018-01-30)[2020-10-16]. https://openreview.net/pdf?id=BkSDMA36Z.

数据集	任务	训练集(#s)/个	测试集(#s)/个	长度(#w)/个	类别
AG	news	120.0 k	7.6 k	45	4
DBP	Ontology	560.0 k	70.0 k	55	14
Yelp.P	SA	560.0 k	38.0 k	153	2
Yelp.F	SA	650.0 k	50.0 k	155	5
Yah.A	QA	1.4 M	60.0 k	112	10
Ama.F	SA	3.6 M	400.0 k	93	5
Ama.P	SA	3.0 M	650.0 k	91	2

数据集	平均单词量/个
AG	128
DBP	110
Yelp.P	350
Yelp.F	350
Yah.A	300
Ama.F	300
Ama.P	300

数据集	ATTx/%	ATTy/%	ATTxy/%
AG	9.62	7.48	7.02
Yelp.P	6.13	5.78	5.69
Yah.A	39.60	27.00	26.32

数据集	FC/%	BFC/%
AG	7.35	7.16
Yelp.P	5.70	5.32
Yah.A	26.43	26.21

数据集	BFC/%	BLSTM-ATTy/%
AG	7.04	6.43
Yelp.P	4.71	2.87
Yah.A	25.42	24.74