Journal of East China Normal University(Natural Science) ›› 2022, Vol. 2022 ›› Issue (2): 55-66.doi: 10.3969/j.issn.1000-5641.2022.02.007

• Computer Science • Previous Articles     Next Articles

Recognition of classroom learning behaviors based on the fusion of human pose estimation and object detection

Zejie WANG1,2, Chaomin SHEN1,2, Chun ZHAO3,4,*(), Xinmei LIU1,2, Jie CHEN1,2   

  1. 1. School of Computer Science and Technology, East China Normal University, Shanghai 200062, China
    2. Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai 200241, China
    3. Information Technology Service, East China Normal University, Shanghai 200062, China
    4. Department of Education Information Technology, East China Normal University, Shanghai 200062, China
  • Received:2020-11-04 Online:2022-03-25 Published:2022-03-28
  • Contact: Chun ZHAO


As a result of ongoing advances in artificial intelligence technology, the potential for learning analysis in teaching evaluation and educational data mining is gradually being recognized. In classrooms, artificial intelligence technology can help to enable automated student behavior analysis, so that teachers can effectively and intuitively grasp students’ learning behavior engagement; the technology, moreover, can provide data to support subsequent improvements in learning design and implementation of teaching interventions. The main scope of the research is as follows: Construct a classroom student behavior dataset that provides a basis for subsequent research; Propose a behavior detection method and a set of feasible, high-precision behavior recognition models. Based on the global features of the human posture extracted from the Openpose algorithm and the local features of the interactive objects extracted by the YOLO v3 algorithm, student behavior can be identified and analyzed to help improve recognition accuracy; Improve the model structure, compress and optimize the model, and reduce the consumption of computing power and time. Four behaviors closely related to the state of learning engagement: listening, turning sideways, bowing, and raising hands are recognized. The accuracy of the detection and recognition method on the verification set achieves 95.45%. The recognition speed and accuracy of common behaviors, such as playing with mobile phones and writing, are greatly improved compared to the original model.

Key words: learning behavior recognition, pose estimation, object detection, computer vision, deep learning

CLC Number: