{
  "paper_id": "cc49456d12867619add3d66e39c445ec",
  "paper_title": "Deep Convolutional Neural Networks for Human Action Recognition Using Depth Maps and Postures",
  "paper_authors": [
    "Aouaidjia Kamel",
    "Bin Sheng",
    "Po Yang",
    "Ping Li",
    "Ruimin Shen",
    "David Dagan Feng"
  ],
  "publication_date": "2018-07-11",
  "publication_venue": "Unknown",
  "abstract": "In this paper, we present a method (Action-Fusion) for human action recognition from depth maps and posture data using convolutional neural networks (CNNs). Two input descriptors are used for action representation. The first input is a depth motion image that accumulates consecutive depth maps of a human action, whilst the second input is a proposed moving joints descriptor which represents the motion of body joints over time. In order to maximize feature extraction for accurate action classification, three CNN channels are trained with different inputs. The first channel is trained with depth motion images (DMIs), the second channel is trained with both DMIs and moving joint descriptors together, and the third channel is trained with moving joint descriptors only. The action predictions generated from the three CNN channels are fused together for the final action classification. We propose several fusion score operations to maximize the score of the right action. The experiments show that the results of the fusing the output of three channels are better than using one channel or fusing two channels only. Our proposed method was evaluated on three public datasets: 1) Microsoft action 3-D dataset (MSRAction3D); 2) University of Texas at Dallas-multimodal human action dataset; and 3) multimodal action dataset (MAD) dataset. The testing results indicate that the proposed approach outperforms most of existing state-of-the-art methods, such as histogram of oriented 4-D normals and ActionNet on MSRAction3D. Although MAD dataset contains a high number of actions (35 actions) compared to existing action RGB-D datasets, this paper surpasses a state-of-the-art method on the dataset by 6.84%.",
  "keywords": [
    "Action recognition",
    "convolutional neural networks (CNNs)",
    "depth motion image (DMI)",
    "moving joints descriptor (MJD)"
  ],
  "paper_institutions": [
    "Shanghai Jiao Tong University",
    "Liverpool John Moores University",
    "Macau University of Science and Technology",
    "University of Sydney"
  ],
  "output_language": "zh",
  "provider": "doubao",
  "model": "ark-code-latest",
  "prompt_template": "simple",
  "extracted_at": "2026-08-16T19:46:50.010394Z",
  "template_tag": "simple",
  "summary": "### 摘要\n本文提出了一种基于深度图和姿态数据的人体动作识别方法（Action-Fusion），利用卷积神经网络（CNN）进行特征提取与分类。针对RGB-D动作识别中单一数据源（深度图或骨架姿态）的局限性，作者设计了两种输入描述符：深度运动图像（DMI）用于累积动作的深度变化，运动关节描述符（MJD）基于球坐标表示关节运动方向。模型包含三个CNN通道：通道1仅用DMI训练，通道2融合DMI和MJD（两个子通道拼接），通道3仅用MJD训练，并通过多种分数融合操作（如乘积、最大值）组合三个通道的输出，以最大化正确动作的得分。实验在MSRAction3D、UTD-MHAD和MAD三个公开数据集上评估，采用跨主体或5折交叉验证协议。结果显示，三通道融合方法（Fus5）在MSRAction3D上达到94.51%的准确率，优于多数现有方法，在包含35类动作的MAD数据集上达到91.86%，比现有方法高6.84%。值得注意的是，MJD描述符弥补了仅使用正面深度图的不足，而分数融合操作有效提升了分类性能。未来工作将探索在移动相机或机器人场景下采集自发动作数据，并验证方法的实际应用能力。\n### 关键要点\n- 提出Action-Fusion方法，结合深度图和姿态数据，通过三个CNN通道进行动作识别，有效弥补单一数据源的不足。\n- 设计新的运动关节描述符（MJD），使用球坐标（θ, φ, r）表示关节运动，相比笛卡尔坐标更能捕捉动作方向信息。\n- 深度运动图像（DMI）从正面视角累积深度变化，仅用正面视图即可达到与多视角方法相当的性能，计算复杂度更低。\n- CNN模型架构包含7×7和5×5卷积核，以及1×1卷积（Network in Network结构），并在池化层后直接使用全连接层，实验发现这比多层全连接更有效。\n- 在MSRAction3D上，三通道融合（Fus5）取得94.51%准确率，优于单个通道和双通道融合，证明多源特征互补的优势。\n- 在UTD-MHAD上，Fus4取得88.14%准确率，超过基线方法，验证了方法的泛化能力。\n- 在MAD数据集（35类动作）上，Fus5达到91.86%平均准确率，比现有方法高6.84%，表明方法可处理大规模类别。\n- 分数融合操作包括乘积（Prod）和最大值（Max）组合，Fus5结合两者并涉及三个通道，实验表明其性能最稳定且最优。\n- 预处理方面，DMI计算耗时远高于MJD（如65帧动作需0.835秒 vs 0.029秒），但总耗时仍低于多视角方法。\n- 训练时间与数据量相关，MSRAction3D约7分钟，UTD-MHAD约12分钟，MAD五折平均约196分钟，测试单输入仅0.07秒（MAD需五次预测平均）。\n- 方法在密集相似动作（如左右手臂动作）上仍能保持较高区分度，MJD提供了关键区分特征。\n- 未来工作包括收集移动相机或机器人场景下的自发动作数据，并研究在真实环境中的适用性。"
}