# Body Action Decoder 完整动作分类文档

> 创建日期: 2026-04-28
> 脚本路径: `scripts/utils/body_action_decoder.py`

---

## 概述

**Body Action Decoder** 支持以下肢体动作检测：

| 类别 | 动作数量 | 数据源 |
|------|----------|--------|
| **Face** | 12 | InsightFace (已有) |
| **Eyes** | 6 | MediaPipe Face Mesh (待安装) |
| **Mouth** | 6 | MediaPipe Face Mesh (待安装) |
| **Arms** | 9 | MediaPipe Pose (待安装) |
| **Hands** | 11 | MediaPipe Hand (待安装) |
| **Legs** | 9 | MediaPipe Pose (待安装) |
| **Feet** | 5 | MediaPipe Pose (待安装) |
| **Combined** | 9 | Multi-source 组合 |

---

## 一、Face Actions (已有 ✅)

### 1.1 Turn Actions (转身)

| Action | Description | Pattern |
|--------|-------------|---------|
| **turn_left** | 向左转 | frontal/three_quarter → profile_left |
| **turn_right** | 向右转 | frontal/three_quarter → profile_right |
| **turn_partial** | 部分转身 | frontal → three_quarter |
| **turn_full** | 完全转身 | profile_left → profile_right (or reverse) |
| **return_frontal** | 回正 | three_quarter/profile → frontal |
| **turn_to_three_quarter** | 转到侧面 | profile → three_quarter |

### 1.2 Pitch Actions (仰俯)

| Action | Description | Pattern |
|--------|-------------|---------|
| **look_up** | 向上看 | neutral → tilted_up |
| **look_down** | 向下看 | neutral → tilted_down |
| **return_neutral** | 回正 | tilted → neutral |

### 1.3 Complex Face Actions (复杂动作)

| Action | Description | Pattern |
|--------|-------------|---------|
| **shake_head** ⭐ | 摇头 | profile_left → profile_right → profile_left (5-30 frames) |
| **nod_head** ⭐ | 点头 | tilted_up → tilted_down → tilted_up (3-20 frames) |

---

## 二、Eye Actions (待安装 MediaPipe)

### 2.1 Basic Eye Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **blink** | 眨眼 | EAR < 0.2 for 1-3 frames |
| **close** | 闭眼 | EAR < 0.15 for > 10 frames |
| **wide_open** | 睁大眼 | EAR > 0.4 |
| **squint** | 眯眼 | EAR 0.15-0.25 |

**EAR (Eye Aspect Ratio)** 计算方式：
```
EAR = (|p2-p6| + |p3-p5|) / (2 × |p1-p4|)
```

### 2.2 Gaze Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **look_left** | 向左看 | iris_position_x < 0.3 |
| **look_right** | 向右看 | iris_position_x > 0.7 |
| **look_center** | 向前看 | iris_position_x 0.3-0.7 |

---

## 三、Mouth Actions (待安装 MediaPipe)

### 3.1 Basic Mouth Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **open** | 张嘴 | MAR > 0.5 |
| **close** | 闭嘴 | MAR < 0.2 |
| **smile** | 微笑 | mouth_corner_distance > threshold |
| **pout** | 嘟嘴 | lip_distance > threshold |

**MAR (Mouth Aspect Ratio)** 计算方式：
```
MAR = mouth_height / mouth_width
```

### 3.2 Dynamic Mouth Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **talk** ⭐ | 说话 | MAR oscillating 0.3-0.6 (min 10 frames) |
| **yawn** ⭐ | 打哈欠 | MAR > 0.7 (min 20 frames) |

---

## 四、Arm Actions (待安装 MediaPipe Pose)

### 4.1 Raise Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **raise_left** | 举起左手 | left_shoulder_y > elbow_y > wrist_y |
| **raise_right** | 举起右手 | right_shoulder_y > elbow_y > wrist_y |
| **raise_both** | 双手举起 | both arms raised |

### 4.2 Angle Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **extend_left** | 伸展左臂 | left_elbow_angle > 150° |
| **extend_right** | 伸展右臂 | right_elbow_angle > 150° |
| **fold_left** | 弯曲左臂 | left_elbow_angle < 90° |
| **fold_right** | 弯曲右臂 | right_elbow_angle < 90° |

### 4.3 Complex Arm Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **cross_arms** ⭐ | 双手交叉 | left_wrist_x > right_wrist_x AND overlapping |
| **wave** ⭐ | 挥手 | wrist_y oscillating ±20px (5-15 frames) |
| **point** | 指向 | index_finger extended, others folded |

---

## 五、Hand Actions (待安装 MediaPipe Hand)

### 5.1 Basic Hand Gestures

| Action | Description | Pattern |
|--------|-------------|---------|
| **open** | 张开手 | all 5 fingers extended |
| **fist** | 握拳 | all fingers folded into palm |
| **grab** | 抓取 | fingers folded, thumb opposing |

### 5.2 Specific Gestures

| Action | Description | Pattern |
|--------|-------------|---------|
| **thumbs_up** ⭐ | 点赞 | thumb extended upward, others folded |
| **peace** ⭐ | 剪刀手 | index + middle extended, others folded |
| **ok** ⭐ | OK 手势 | thumb + index touching |
| **point** | 指向 | index extended, others folded |

### 5.3 Contact Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **touch_face** | 摸脸 | hand near face region |
| **touch_hair** | 摸头发 | hand above head region |
| **pocket_left** | 左手插兜 | left_hand in hip region |
| **pocket_right** | 右手插兜 | right_hand in hip region |

### 5.4 Dynamic Hand Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **clap** ⭐ | 拍手 | hands together → apart (3-10 frames) |

---

## 六、Leg Actions (待安装 MediaPipe Pose)

### 6.1 Basic Leg Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **stand** | 站立 | hip_y < knee_y < ankle_y (vertical) |
| **sit** ⭐ | 姿 | hip_y ≈ knee_y (horizontal thigh) |
| **knee_bend** | 弯膝 | knee_angle < 120° |

### 6.2 Dynamic Leg Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **walk** ⭐ | 行走 | hip-knee-ankle oscillating (min 10 frames) |
| **run** ⭐ | 奔跑 | fast oscillating + knee_bend > 60° (min 10 frames) |
| **jump** ⭐ | 跳跃 | keypoints moving upward → landing (5-20 frames) |
| **kick** ⭐ | 踢腿 | one leg extended forward rapidly (3-15 frames) |

### 6.3 Cross Actions

| Action | Description | Pattern |
|--------|-------------|---------|
| **cross_left** | 左腿交叉 | left_ankle_x > right_ankle_x |
| **cross_right** | 右腿交叉 | right_ankle_x > left_ankle_x |

---

## 七、Feet Actions (待安装 MediaPipe Pose)

| Action | Description | Pattern |
|--------|-------------|---------|
| **tap** ⭐ | 轻踏 | ankle_y oscillating ±10px (3-15 frames) |
| **stomp** ⭐ | 重踏 | ankle_y large downward movement (min 3 frames) |
| **cross** | 交叉脚 | feet_x overlapping |
| **point_left** | 左脚前伸 | left_ankle_y < right_ankle_y |
| **point_right** | 右脚前伸 | right_ankle_y < left_ankle_y |

---

## 八、Combined Actions ⭐ (多源组合)

| Action | Description | Components |
|--------|-------------|------------|
| **thinking** | 思考姿势 | touch_face + look_down |
| **listening** | 倾听姿势 | turn_partial + mouth_open |
| **nodding_agreement** | 点头同意 | nod_head + smile |
| **shaking_disagreement** | 摇头不同意 | shake_head + frown |
| **waving_greeting** | 挥手打招呼 | wave + smile |
| **crossing_arms_defensive** | 双手交叉防御 | cross_arms + frontal_stable |
| **pointing_explaining** | 指向解释 | point + turn_partial |
| **stretching** | 伸展 | raise_both + look_up |
| **sitting_relaxed** | 放松坐姿 | sit + cross_arms |

---

## 九、MediaPipe Keypoint Indices

### 9.1 Pose Keypoints (33 points)

| Index | Keypoint | Description |
|-------|----------|-------------|
| **0** | nose | 鼻尖 |
| **11** | left_shoulder | 左肩 |
| **12** | right_shoulder | 右肩 |
| **13** | left_elbow | 左肘 |
| **14** | right_elbow | 右肘 |
| **15** | left_wrist | 左手腕 |
| **16** | right_wrist | 右手腕 |
| **23** | left_hip | 左髋 |
| **24** | right_hip | 右髋 |
| **25** | left_knee | 左膝 |
| **26** | right_knee | 右膝 |
| **27** | left_ankle | 左踝 |
| **28** | right_ankle | 右踝 |

### 9.2 Hand Keypoints (21 points per hand)

| Index | Keypoint | Description |
|-------|----------|-------------|
| **0** | wrist | 手腕 |
| **1-4** | thumb | 拇指 (CMC → TIP) |
| **5-8** | index | 食指 (MCP → TIP) |
| **9-12** | middle | 中指 (MCP → TIP) |
| **13-16** | ring | 无名指 (MCP → TIP) |
| **17-20** | pinky | 小指 (MCP → TIP) |

### 9.3 Face Mesh Keypoints (468 points)

| Region | Points | Description |
|--------|--------|-------------|
| **Eyes** | 33-133, 362-382 | 眼睛轮廓 + 瞳孔 |
| **Iris** | 468-477 | 虹膜位置 |
| **Mouth** | 61-308 | 嘴唇轮廓 |
| **Nose** | 1-98 | 鼻子 |

---

## 十、安装 MediaPipe

### 10.1 安装命令

```bash
# 安装 MediaPipe
pip install mediapipe==0.10.9

# 或使用 Homebrew Python
/opt/homebrew/bin/python3.11 -m pip install mediapipe==0.10.9
```

### 10.2 模型说明

| Model | Output | Description |
|-------|--------|-------------|
| **Holistic** | pose + face + hands | 全身关键点 (468 face + 33 pose + 42 hands) |
| **Pose** | 33 keypoints | 姿态估计 |
| **Face Mesh** | 468 keypoints | 面部网格 |
| **Hands** | 42 keypoints | 手部关键点 |

---

## 十一、使用方式

### 11.1 当前可用功能（Face）

```bash
# 仅使用 Face 数据（已有）
python3 scripts/utils/body_action_decoder.py \
  --face-json video.face_traced.json
```

### 11.2 完整功能（需安装 MediaPipe）

```bash
# 使用 Face + Pose + Hand 数据
python3 scripts/utils/body_action_decoder.py \
  --pose-json video.pose.json \
  --face-json video.face_traced.json \
  --hand-json video.hand.json \
  --output-json body_action_data.json
```

---

## 十二、输出结构

```json
{
  "face": [
    {"action": "turn_right", "description": "向右转"}
  ],
  "eyes": [
    {"action": "blink", "description": "眨眼", "ear": 0.18}
  ],
  "mouth": [
    {"action": "smile", "description": "微笑", "corner_distance": 12.5}
  ],
  "arms": [
    {"action": "raise_right", "description": "举起右手", "angle": 120.5}
  ],
  "hands": [
    {"action": "thumbs_up_right", "description": "右手点赞"}
  ],
  "legs": [
    {"action": "stand", "description": "站立"}
  ],
  "feet": [],
  "combined": [
    {"action": "waving_greeting", "description": "挥手打招呼", "components": ["wave", "smile"]}
  ]
}
```

---

## 十三、未来改进

| Phase | 功能 | 状态 |
|-------|------|------|
| **Phase 1** | Face Actions | ✅ 已完成 |
| **Phase 2** | Eye/Mouth Actions | ⏸ 待安装 MediaPipe Face Mesh |
| **Phase 3** | Arm/Hand Actions | ⏸ 待安装 MediaPipe Hand |
| **Phase 4** | Leg/Feet Actions | ⏸ 待安装 MediaPipe Pose |
| **Phase 5** | Combined Actions | ⏸ 待整合多源数据 |

---

## 版本信息

- 版本: 1.0
- 创建日期: 2026-04-28
- 状态: ✅ Face Actions 完成，其他待安装 MediaPipe
