A review of machine reading comprehension for automatic QA

doi:10.3969/j.issn.1000-5641.2019.05.003

Abstract

Abstract: Artificial Intelligence (AI) is affecting every industry. Applying AI to education accelerates the structural reform of education and transforms traditional education into intelligent adaptive education. The automatic Question Answer system, based on deep learning, not only helps students to answer questions and acquire knowledge in real-time, but can also quickly gather student behavioral data and accelerate personalization of the educational process. Machine reading comprehension is the core module of an automatic Question Answer system, and it is an important technology to understand student problems, document content, and acquire knowledge quickly. With the revival of deep learning and the availability of large-scale reading comprehension datasets, a number of neural network-based machine reading models have been proposed over the past few years. The purpose of this review is three-fold:to introduce and review progress in machine reading comprehension; to compare and analyze the advantages and disadvantages between various neural machine reading models; and to summarize the relevant datasets and evaluation methods in the field of machine reading.

Key words: Artificial Intelligence, intellectual adaptation education, deep learning, machine reading comprehension

CLC Number:

TP391

YANG Kang, HANG Ding-jiang, GAO Ming. A review of machine reading comprehension for automatic QA[J]. Journal of East China Normal University(Natural Sc, 2019, 2019(5): 36-52.

References

[1] CHEN D Q. Neural reading comprehension and beyond[D]. CA:Standford University, 2018.
[2] LEHNERT W G. The process of question answering[R]. Yale Univ New Haven Conn, 1977.
[3] HIRSCHMAN L, LIGHT M, BRECK E, et al. Deep read:A reading comprehension system[C]//Proceedings of the 37th annual meeting of the Association for Computational Linguistics on Computational Linguistics. Association for Computational Linguistics, 1999:325-332.
[4] RILOFF E, THELEN M. A rule-based question answering system for reading comprehension tests[C]//Proceedings of the 2000 ANLP/NAACL Workshop on Reading comprehension tests as evaluation for computer-based language understanding sytems-Volume 6. Association for Computational Linguistics, 2000:13-19.
[5] RICHARDSON M, BURGES C J C, RENSHAW E. Mctest:A challenge dataset for the open-domain machine comprehension of text[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013:193-203.
[6] SACHAN M, DUBEY K, XING E, et al. Learning answer-entailing structures for machine comprehension[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1:Long Papers), 2015:239-249.
[7] NARASIMHAN K, BARZILAY R. Machine comprehension with discourse relations[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1:Long Papers), 2015:1253-1262.
[8] WANG H, BANSAL M, GIMPEL K, et al. Machine comprehension with syntax, frames, and semantics[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2:Short Papers), 2015:700-706.
[9] BENGIO Y, DUCHARME R, VINCENT P, et al. A neural probabilistic language model[J]. Journal of Machine Learning Research, 2003(3):1137-1155.
[10] MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality[C]//Advances in neural information processing systems, 2013:3111-3119.
[11] PENNINGTON J, SOCHER R, MANNING C. Glove:Global vectors for word representation[C]//Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 2014:1532-1543.
[12] HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural computation, 1997, 9(8):1735-1780.
[13] CHUNG J, GULCEHRE C, CHO K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXiv preprint, arXiv:1412.3555, 2014.
[14] RAJPURKAR P, ZHANG J, LOPYREV K, et al. Squad:100,000+questions for machine comprehension of text[J]. arXiv preprint, arXiv:1606.05250, 2016.
[15] NGUYEN T, ROSENBERG M, SONG X, et al. MS MARCO:A Human-Generated MAchine Reading COmprehension Dataset[J]. Neural Information Processing Systems, 2016.
[16] JOSHI M, CHOI E, WELD D S, et al. Triviaqa:A large scale distantly supervised challenge dataset for reading comprehension[J]. arXiv preprint, arXiv:1705.03551, 2017.
[17] TRISCHLER A, WANG T, YUAN X, et al. Newsqa:A machine comprehension dataset[J]. arXiv preprint, arXiv:1611.09830, 2016.
[18] KOČSKÝ T, SCHWARZ J, BLUNSOM P, et al. The narrativeqa reading comprehension challenge[J]. Transactions of the Association for Computational Linguistics, 2018(6):317-328.
[19] LAI G, XIE Q, LIU H, et al. Race:Large-scale reading comprehension dataset from examinations[J]. arXiv preprint, arXiv:1704.04683, 2017.
[20] DUNN M, SAGUN L, HIGGINS M, et al. Searchqa:A new q&a dataset augmented with context from a search engine[J]. arXiv preprint, arXiv:1704.05179, 2017.
[21] HE W, LIU K, LIU J, et al. Dureader:a chinese machine reading comprehension dataset from real-world applications[J]. arXiv preprint, arXiv:1711.05073, 2017.
[22] TAYLOR W L. "Cloze procedure":A new tool for measuring readability[J]. Journalism Bulletin, 1953, 30(4):415-433.
[23] BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[J]. arXiv preprint, arXiv:1409.0473, 2014.
[24] HERMANN K M, KOCISKÝ HERMANN K M, KOCISKY T, et al. Teaching machines to read and comprehend[C]//Advances in neural information processing systems. 2015:1693-1701.
[25] CHEN D, BOLTON J, MANNING C D. A thorough examination of the cnn/daily mail reading comprehension task[J]. arXiv preprint, arXiv:1606.02858, 2016.
[26] KADLEC R, SCHMID M, BAJGAR O, et al. Text understanding with the attention sum reader network[J]. arXiv preprint, arXiv:1603.01547, 2016.
[27] VINYALS O, FORTUNATO M, JAITLY N. Pointer networks[C]//Advances in Neural Information Processing Systems, 2015:2692-2700.
[28] CUI Y, LIU T, CHEN Z, et al. Consensus attention-based neural networks for chinese reading comprehension[J]. arXiv preprint, arXiv:1607.02250, 2016.
[29] CUI Y, CHEN Z, WEI S, et al. Attention-over-attention neural networks for reading comprehension[J]. arXiv preprint, arXiv:1607.04423, 2016.
[30] WANG S, JIANG J. Machine comprehension using match-lstm and answer pointer[J]. arXiv preprint, arXiv:1608.07905, 2016.
[31] SEO M, KEMBHAVI A, FARHADI A, et al. Bidirectional attention flow for machine comprehension[J]. arXiv preprint, arXiv:1611.01603, 2016.
[32] KIM Y. Convolutional neural networks for sentence classification[J]. arXiv preprint, arXiv:1408.5882, 2014.
[33] WANG W, YANG N, WEI F, et al. Gated self-matching networks for reading comprehension and question answering[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers), 2017:189-198.
[34] HUANG H Y, ZHU C, SHEN Y, et al. Fusionnet:Fusing via fully-aware attention with application to machine comprehension[J]. arXiv preprint, arXiv:1711.07341, 2017.
[35] LIU X, SHEN Y, DUH K, et al. Stochastic answer networks for machine reading comprehension[J]. arXiv preprint, arXiv:1712.03556, 2017.
[36] WESTON J, CHOPRA S, BORDES A. Memory networks[J]. arXiv preprint, arXiv:1410.3916, 2014.
[37] SUKHBAATAR S, WESTON J, FERGUS R. End-to-end memory networks[C]//Advances in neural information processing systems, 2015:2440-2448.
[38] DHINGRA B, LIU H, YANG Z, et al. Gated-attention readers for text comprehension[J]. arXiv preprint, arXiv:1606.01549, 2016.
[39] SORDONI A, BACHMAN P, TRISCHLER A, et al. Iterative alternating neural attention for machine reading[J]. arXiv preprint, arXiv:1606.02245, 2016.
[40] SHEN Y, HUANG P S, GAO J, et al. Reasonet:Learning to stop reading in machine comprehension[C]//Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2017:1047-1055.
[41] WILLIAMS R J. Simple statistical gradient-following algorithms for connectionist reinforcement learning[J]. Machine learning, 1992, 8(3/4):229-256.
[42] WU F, LAO N, BLITZER J, et al. Fast reading comprehension with convnets[J]. arXiv preprint, arXiv:1711.04352, 2017.
[43] YU A W, DOHAN D, LUONG M T, et al. Qanet:Combining local convolution with global self-attention for reading comprehension[J]. arXiv preprint, arXiv:1804.09541, 2018.
[44] DAUPHIN Y N, FAN A, AULI M, et al. Language modeling with gated convolutional networks[C]//Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017:933-941.
[45] GEHRING J, AULI M, GRANGIER D, et al. Convolutional sequence to sequence learning[C]//Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017:1243-1252.
[46] CHOLLET F. Xception:Deep learning with depthwise separable convolutions[C]//Proceedings of the IEEE conference on computer vision and pattern recognition, 2017:1251-1258.
[47] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in neural information processing systems, 2017:5998-6008.
[48] RADFORD A, NARASIMHAN K, SALIMANS T, et al. Improving language understanding with unsupervised learning[R/OL]. Technical report, OpenAI, 2018.[2019.08.01]. https://s3-us-west-2.amazonaws.com/openaiassets/research-covers/language-unsupervised/languageunderstandingpaper.pdf.
[49] DEVLIN J, CHANG M W, LEE K, et al. Bert:Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint, arXiv:1810.04805, 2018.
[50] HILL F, BORDES A, CHOPRA S, et al. The goldilocks principle:Reading children's books with explicit memory representations[J]. arXiv preprint, arXiv:1511.02301, 2015.