华东师范大学学报（自然科学版） ›› 2022, Vol. 2022 ›› Issue (2): 55-66.doi: 10.3969/j.issn.1000-5641.2022.02.007

• 计算机科学 • 上一篇    下一篇

融合人体姿态估计和目标检测的学生课堂行为识别

王泽杰1,2, 沈超敏1,2, 赵春3,4,*(), 刘新妹1,2, 陈杰1,2   

  1. 1. 华东师范大学 计算机科学与技术学院, 上海　200062
    2. 华东师范大学 上海市多维度信息处理重点实验室, 上海　200241
    3. 华东师范大学 信息化治理办公室, 上海　200062
    4. 华东师范大学 教育信息技术学系, 上海　200062
  • 收稿日期:2020-11-04 出版日期:2022-03-25 发布日期:2022-03-28
  • 通讯作者: 赵春 E-mail:czhao@ecnu.edu.cn
  • 基金资助:
    国家自然科学基金(11771276, 61731009); 上海市“科技创新行动计划”人工智能科技支撑专项项目(20511100200); 上海市科学技术委员会资助项目(14DZ2260800)

Recognition of classroom learning behaviors based on the fusion of human pose estimation and object detection

Zejie WANG1,2, Chaomin SHEN1,2, Chun ZHAO3,4,*(), Xinmei LIU1,2, Jie CHEN1,2   

  1. 1. School of Computer Science and Technology, East China Normal University, Shanghai　200062, China
    2. Shanghai Key Laboratory of Multidimensional Information Processing, East China Normal University, Shanghai　200241, China
    3. Information Technology Service, East China Normal University, Shanghai　200062, China
    4. Department of Education Information Technology, East China Normal University, Shanghai　200062, China
  • Received:2020-11-04 Online:2022-03-25 Published:2022-03-28
  • Contact: Chun ZHAO E-mail:czhao@ecnu.edu.cn

RichHTML

163

PDF

2298

摘要：

在课堂教学中, 人工智能技术可以帮助实现学生行为分析自动化, 让教师能够高效且直观地掌握学生学习行为投入的情况, 为后续优化教学设计与实施教学干预提供数据支持. 构建了学生课堂行为数据集, 为后续研究工作提供了数据基础; 提出了一种行为检测方法及一套可行的高精度的行为识别模型, 利用OpenPose算法提取的人体姿态全局特征, 融合YOLO v3算法提取的交互物体局部特征, 对学生行为进行了识别分析, 提高了识别精度; 改进了模型结构, 压缩并优化了模型, 降低了空间与时间的消耗. 选取与学习投入状态紧密相关的4种行为: 正坐、侧身、低头和举手进行识别, 该检测与识别方法在验证集上的精度达到了95.45%, 在课堂上玩手机和书写等常见行为的识别精度较原模型有很大的提高.

关键词: 学习行为识别, 人体姿态估计, 目标检测, 计算机视觉, 深度学习

Abstract:

As a result of ongoing advances in artificial intelligence technology, the potential for learning analysis in teaching evaluation and educational data mining is gradually being recognized. In classrooms, artificial intelligence technology can help to enable automated student behavior analysis, so that teachers can effectively and intuitively grasp students’ learning behavior engagement; the technology, moreover, can provide data to support subsequent improvements in learning design and implementation of teaching interventions. The main scope of the research is as follows: Construct a classroom student behavior dataset that provides a basis for subsequent research; Propose a behavior detection method and a set of feasible, high-precision behavior recognition models. Based on the global features of the human posture extracted from the Openpose algorithm and the local features of the interactive objects extracted by the YOLO v3 algorithm, student behavior can be identified and analyzed to help improve recognition accuracy; Improve the model structure, compress and optimize the model, and reduce the consumption of computing power and time. Four behaviors closely related to the state of learning engagement: listening, turning sideways, bowing, and raising hands are recognized. The accuracy of the detection and recognition method on the verification set achieves 95.45%. The recognition speed and accuracy of common behaviors, such as playing with mobile phones and writing, are greatly improved compared to the original model.

Key words: learning behavior recognition, pose estimation, object detection, computer vision, deep learning

中图分类号: