基于双向长短记忆网络和门控注意力的文本分类网络

doi:10.3969/j.issn.1000-5641.2022.02.008

华东师范大学学报（自然科学版） ›› 2022, Vol. 2022 ›› Issue (2): 67-75.doi: 10.3969/j.issn.1000-5641.2022.02.008

基于双向长短记忆网络和门控注意力的文本分类网络

童根梅¹, 朱敏^2,*()

1. 华东师范大学计算机科学与技术学院, 上海　200062
2. 华东师范大学数据科学与工程学院, 上海　200062

收稿日期:2020-11-09 出版日期:2022-03-25 发布日期:2022-03-28
通讯作者: 朱敏 E-mail:mzhu@cc.ecnu.edu.cn

Bi-directional long short-term memory and bi-directional gated attention networks for text classification

Genmei TONG¹, Min ZHU^2,*()

1. School of Computer Science and Technology, East China Normal University, Shanghai　200062, China
2. School of Data Science and Engineering, East China Normal University, Shanghai　200062, China

Received:2020-11-09 Online:2022-03-25 Published:2022-03-28
Contact: Min ZHU E-mail:mzhu@cc.ecnu.edu.cn

摘要/Abstract

摘要：

首先, 提出构建双向的全连接结构用于更好提取上下文的信息; 然后, 利用双向的注意力机制将包含丰富文本特征的矩阵压缩成一个向量; 最后, 将双向的全连接结构和门控制结构相结合. 通过实验验证了上述结构对于提升文本分类的准确率具有积极的作用. 将这3种结构和双向的循环网络进行结合, 组成了所提出的文本分类模型. 通过在7个常用的文本分类数据集(AG、DBP、Yelp.P、Yelp.F、Yah.A、Ama.F、Ama.P)上进行的实验, 得到了具有竞争性的结果并且在其中5个数据集(AG、DBP、Yelp.P、Ama.F、Ama.P)上获得了较好的实验效果. 通过实验表明, 所提出的文本分类模型能显著降低分类错误率.

关键词: 文本分类, 注意力机制, 长短记忆网络

Abstract:

In this paper, we propose the construction of a bi-directional fully connected structure for better extraction of context information. We also propose the construction of a bi-directional attention structure for compressing matrices containing rich text features into a vector. The bi-directional fully connected structure and the gated structure are then combined. This research demonstrates that the proposed combined structure has a net positive effect on text classification accuracy. Finally, by combining these three structures and a bi-direction long short-term memory, we propose a new text classification model. Using this model, we obtained competitive results on seven commonly used text classification datasets and achieved state-of-the-art results on five of them. Experiments showed that the combination of these structures can significantly reduce classification errors.

Key words: text classification, attention, long short-term memory

中图分类号:

TP399

童根梅, 朱敏. 基于双向长短记忆网络和门控注意力的文本分类网络[J]. 华东师范大学学报（自然科学版）, 2022, 2022(2): 67-75.

Genmei TONG, Min ZHU. Bi-directional long short-term memory and bi-directional gated attention networks for text classification[J]. Journal of East China Normal University(Natural Science), 2022, 2022(2): 67-75.

图/表 8

图1

图2

表1

表2

表3

表4

表5

表6

参考文献 18

1	WANG G, LI C, WANG W, et al. Joint embedding of words and labels for text classification [C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 2018: 2321-2331.
2	PENNINGTON J, SOCHER R, MANNING C. GloVe: Global vectors for word representation [C]// Conference on Empirical Methods in Natural Language Processing. 2014: 1532-1543.
3	PETERS M, NEUMANN M, IYYER M, et al. Deep contextualized word representations [EB/OL]. (2018-03-22)[2020-10-16]. https://arxiv.org/pdf/1802.05365v2.pdf.
4	DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding [EB/OL]. (2019-03-24)[2020-10-16]. https://arxiv.org/pdf/1810.04805.pdf.
5	RADFORD A, WU J, CHILD R, et al. Language models are unsupervised multitask learners [EB/OL]. (2019-01-08)[2020-10-16]. https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf.
6	KIM Y. Convolutional neural networks for sentence classification [EB/OL]. (2014-09-03)[2020-10-16]. https://arxiv.org/pdf/1408.5882v2.pdf.
7	CONNEAU A, SCHWENK H, BARRAULT L, et al. Very deep convolutional networks for text classification [EB/OL]. (2017-01-27)[2020-10-16]. https://arxiv.org/pdf/1606.01781v2.pdf.
8	JOHNSON R, TONG Z. Deep pyramid convolutional neural networks for text categorization [C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 2017: 562-570.
9	HOCHREITER S, SCHMIDHUBER J. Long short-term memory [J]. Neural Computation, 1997, 9(8): 1735-1780.
10	LAI S, XU L, LIU K, et al. Recurrent convolutional neural networks for text classification [C]// Proceeding of the 25th International Joint Conference on Artificial Intelligence. 2015: 2267-2273.
11	PENG Z, QI Z, ZHENG S, et al. Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling [EB/OL]. (2016-11-21)[2020-10-22]. https://arxiv.org/pdf/1611.06639.pdf.
12	PAPPAS N, POPESCU-BELIS A. Multilingual hierarchical attention networks for document classification [EB/OL]. (2017-09-15)[2020-09-14]. https://arxiv.org/pdf/1707.00896v4.pdf.
13	ZHANG X, ZHAO J, LECUN Y. Character-level convolutional networks for text classification [EB/OL]. (2015-09-10)[2020-09-11]. https://arxiv.org/pdf/1509.01626v2.pdf.
14	TURIAN J P, RATINOV L A, BENGIO Y. Word representations: A simple and general method for semi-supervised learning [C]// Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. 2010: 384-394.
15	KINGMA D, BA J. Adam: A method for stochastic optimization [EB/OL]. (2015-07-30)[2020-10-16]. https://arxiv.org/pdf/1412.6980v8.pdf.
16	XIAO Y, CHO K. Efficient character-level document classification by combining convolution and recurrent layers [EB/OL]. (2016-02-01)[2020-10-16]. https://arxiv.org/pdf/1602.00367v1.pdf.
17	JOULIN A, GRAVE E, BOJANOWSKI P, et al. Bag of tricks for efficient text classification [EB/OL]. (2016-08-09)[2020-10-13]. https://arxiv.org/pdf/1607.01759v3.pdf.
18	QIAO C, HUANG B, NIU G, et al. A new method of region embedding for text classification [EB/OL]. (2018-01-30)[2020-10-16]. https://openreview.net/pdf?id=BkSDMA36Z.

数据集	任务	训练集(#s)/个	测试集(#s)/个	长度(#w)/个	类别
AG	news	120.0 k	7.6 k	45	4
DBP	Ontology	560.0 k	70.0 k	55	14
Yelp.P	SA	560.0 k	38.0 k	153	2
Yelp.F	SA	650.0 k	50.0 k	155	5
Yah.A	QA	1.4 M	60.0 k	112	10
Ama.F	SA	3.6 M	400.0 k	93	5
Ama.P	SA	3.0 M	650.0 k	91	2

数据集	ATTx/%	ATTy/%	ATTxy/%
AG	9.62	7.48	7.02
Yelp.P	6.13	5.78	5.69
Yah.A	39.60	27.00	26.32

数据集	FC/%	BFC/%
AG	7.35	7.16
Yelp.P	5.70	5.32
Yah.A	26.43	26.21

数据集	BFC/%	BLSTM-ATTy/%
AG	7.04	6.43
Yelp.P	4.71	2.87
Yah.A	25.42	24.74

数据集	Linear model	Word-level CNN	Char-level CNN	Char- CRNN	Fast- CRNN	VDCNN	DPCNN	Region.emb	LEAM	组合模型
AG	7.64	8.55	9.51	8.64	7.50	8.67	6.87	7.20	7.55	6.02
DBP	1.31	1.37	1.55	1.43	1.40	1.29	0.88	1.10	0.98	0.75
Yelp.P	4.36	4.60	4.88	5.51	4.30	4.28	2.64	4.70	4.69	2.54
Yelp.F	40.14	39.58	37.95	38.18	36.10	35.28	30.58	35.10	35.91	32.57
Yah.A	28.96	28.84	28.80	28.26	27.70	26.57	23.90	26.30	22.58	23.81
Ama.F	44.74	42.39	40.43	40.77	39.80	37.00	34.81	39.10		30.02
Ama.P	7.98	5.51	4.93	5.87	5.40	4.28	3.32	4.70		2.74

基于双向长短记忆网络和门控注意力的文本分类网络

Bi-directional long short-term memory and bi-directional gated attention networks for text classification

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 18

相关文章 5

编辑推荐

Metrics

本文评价

数据集	平均单词量/个
AG	128
DBP	110
Yelp.P	350
Yelp.F	350
Yah.A	300
Ama.F	300
Ama.P	300

[1]	梁艳春, 房爱莲. 基于多通道卷积神经网络的中文文本关系抽取[J]. 华东师范大学学报（自然科学版）, 2021, 2021(3): 96-104.
[2]	李小昌, 陈贝, 董启文, 陆雪松. 基于自编码器的旅行同伴挖掘[J]. 华东师范大学学报（自然科学版）, 2020, 2020(5): 179-188.
[3]	杨东明, 杨大为, 顾航, 洪道诚, 高明, 王晔. 面向初等数学的知识点关系提取研究[J]. 华东师范大学学报(自然科学版), 2019, 2019(5): 53-65.
[4]	傅裕, 李优, 林煜明, 周娅. 基于自注意力机制的冗长商品名称精简方法[J]. 华东师范大学学报(自然科学版), 2019, 2019(5): 113-122,167.
[5]	黄振龙，郑骏，胡文心. 基于类间可分性DAG-SVM的文本分类[J]. 华东师范大学学报(自然科学版), 2013, 2013(3): 209-218.