{
  "paper_id": "81b21b2cddef8d61943642b108a037f7",
  "paper_title": "Privacy-Preserving 3D Lidar-Based Multi-Modal Activity Recognition in Human-Robot Interaction",
  "paper_authors": [
    "Adel Baselizadeh",
    "Md Zia Uddin",
    "Weria Khaksar",
    "Diana Saplacan Lindblom",
    "Jim Torresen"
  ],
  "publication_date": "",
  "publication_venue": "2025 13th International Conference on Control, Mechatronics and Automation (ICCMA)",
  "abstract": "Human activity recognition (HAR) involves using sensors to collect human data, which is then analyzed to identify their activities. HAR has numerous applications across various fields. One significant area is human-robot interaction (HRI). Recognizing user activities provides robots with valuable insights into the user's status, enhancing the efficiency of HRI. However, developing HAR models presents privacy challenges due to the collection of user data, especially in robots, which are often equipped with diverse sensors. This paper proposes privacy-preserving HAR methods within the context of HRI. The paper investigates using privacy-preserving sensors, with a particular focus on 3D Lidar, to develop HAR models. It explores the integration of 3D Lidar with other sensors, including user-wearable sensors and robot-based sensors, including force/torque sensors, through multimodal deep learning (DL) approaches. Various DL-based sensor fusion methods, including data-level and feature-level fusion approaches, are thoroughly examined, and their accuracies for HAR are compared. A novel dataset was collected to train the multimodal DL models, capturing various user activities during HRI. This dataset leverages 10 different sensors, including 9 privacy-preserving sensors, along with an RGB camera for reference. The paper considers nine distinct user activities, including physical interactions with a robot and commanding a robot to perform specific tasks. The results indicate that integrating the sensors' data at the feature-level achieves an 80.73% accuracy in recognizing various user activities during HRI.",
  "keywords": [
    "Human activity recognition",
    "HAR",
    "human-robot interaction",
    "HRI",
    "3D Lidar",
    "multi-modal sensing",
    "privacy",
    "dataset"
  ],
  "paper_institutions": [],
  "output_language": "zh",
  "provider": "doubao",
  "model": "ark-code-latest",
  "prompt_template": "simple",
  "extracted_at": "2026-08-16T19:46:14.195143Z",
  "template_tag": "simple",
  "summary": "### 摘要\n本文针对人机交互（HRI）中的隐私保护问题，提出基于3D Lidar等隐私保护传感器的多模态深度学习方法用于人类活动识别（HAR）。研究背景是现有HAR模型多依赖可能侵犯隐私的传感器，而3D Lidar既能提供足够信息又保护隐私。论文核心解决方案是融合3D Lidar、用户腕部可穿戴传感器和机器人力/力矩传感器，采用数据级和特征级融合策略，并引入卷积LSTM（ConvLSTM）、PointNet++和Seq2Seq等架构。使用的数据集是作者收集的PriMA-Care，包含27种活动，本文选取其中9种与HRI相关的活动，涉及10种传感器（9种隐私保护+1个RGB用于参考）。实验设置中，将数据以20Hz同步，窗口为20个时间步（1秒）步长10，3D Lidar点云每帧128×1024点经裁剪至7300点，模型用约80%数据训练、20%测试。评估指标包括准确率、精确率、召回率和F1分数，结果显示仅用可穿戴传感器平均准确率0.5515，加入力/力矩数据级融合提升至0.5631，特征级融合0.5791，仅用3D Lidar为0.6508，全传感器特征级融合达到最优平均准确率0.6945（最大0.8073），证明特征级融合和多传感器集成的有效性。未来方向包括决策级融合、增加更多活动、集成热成像相机、使用更先进点云处理模型以及降维方法（ICA、PCA、LDA）自动裁剪点云。该方法的实际应用场景是护理机器人或个人助理机器人在家庭和护理设施中监测用户活动并保障隐私，同时受限于计算成本高、点云裁剪不自动化等约束。核心机制是：将3D Lidar点云逐帧经PointNet++提取128维特征，窗口内20个特征向量通过Seq2Seq编码器-解码器生成单一128维向量，与可穿戴和力/力矩传感器各自经ConvLSTM得到的128维特征拼接成384维向量，最后经多层感知机分类为9种活动。\n\n### 关键要点\n- 作者提出隐私保护的HAR方法，核心是利用3D Lidar传感器，它被视为隐私保护传感器，尤其适用于护理机器人场景。\n- 论文首次将3D Lidar与用户佩戴的腕部传感器和机器人力/力矩传感器集成用于HAR，这是该融合的首次尝试。\n- 采用两种融合方法：数据级融合（仅可穿戴和力/力矩，排除3D Lidar）和特征级融合（所有传感器）。\n- 使用了ConvLSTM架构处理时间序列传感器数据，包括CNN、LSTM和全连接层，处理窗口为20个时间步（1秒），步长10（0.5秒）。\n- 3D Lidar点云通过PointNet++处理，由于计算限制，将每个点云从131072点裁剪至7300点，训练每步约1小时。\n- 为解决不同传感器数据量不匹配问题，采用Seq2Seq机制将20个3D Lidar特征向量压缩为一个，实现特征级融合。\n- 实验使用PriMA-Care数据集，选取9种活动（含物理交互和机器人指令），以80%训练、20%测试进行公平比较。\n- 仅用可穿戴传感器平均准确率0.5515，最大0.5900，表明单一传感器无法充分捕获复杂活动。\n- 数据级融合（可穿戴+力/力矩）平均准确率0.5631，特征级融合提升至0.5791，说明特征级融合更有效。\n- 仅用3D Lidar平均准确率0.6508，最大0.7550，优于仅用可穿戴及其与力/力矩的融合。\n- 所有传感器特征级融合取得最佳性能：平均准确率0.6945，最大0.8073，同时精确率、召回率、F1也最高。\n- 混淆矩阵显示，仅可穿戴模型对涉及物理交互的活动A5（握住机器人手臂）准确率低，而力/力矩数据可改善A5分类，全传感器融合则综合了各方优势。\n- 论文指出主要限制包括最终准确率不算高（80.37%）、未采用决策级融合、仅用PointNet++处理点云、点云裁剪方法不精确，未来工作将针对这些方面改进。"
}