# MediaPipe Holistic 整合完成报告

> 整合日期: 2026-04-28
> 测试视频: preview.mp4 (15秒, 329帧)

---

## 整合架构

```
┌─────────────────────────────────────────────────────────────────────┐
│                    Integrated Body Action Decoder                   │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌───────────────┐       ┌───────────────┐                         │
│  │ InsightFace   │       │ MediaPipe     │                         │
│  │ face.json     │       │ holistic.json │                         │
│  │               │       │               │                         │
│  │ - embedding   │       │ - face_mesh   │ (478 landmarks)         │
│  │ - pose_angle  │       │ - pose        │ (33 keypoints)          │
│  │ - landmarks   │       │ - hands       │ (21 × 2 keypoints)      │
│  └───────────────┘       └───────────────┘                         │
│         │                       │                                   │
│         └───────────┬───────────┘                                   │
│                     │                                               │
│             ┌───────▼───────┐                                       │
│             │ Frame Matcher │ (按 frame_num 合并)                   │
│             └───────┬───────┘                                       │
│                     │                                               │
│     ┌───────────────▼───────────────┐                               │
│     │ Integrated Action Decoder     │                               │
│     │                               │                               │
│     │ ┌─────────┐ ┌─────────┐     │                               │
│     │ │ Face    │ │ Eyes    │     │                               │
│     │ │ Actions │ │ Actions │     │                               │
│     │ └─────────┘ └─────────┘     │                               │
│     │ ┌─────────┐ ┌─────────┐     │                               │
│     │ │ Mouth   │ │ Arms    │     │                               │
│     │ │ Actions │ │ Actions │     │                               │
│     │ └─────────┘ └─────────┘     │                               │
│     │ ┌─────────┐ ┌─────────┐     │                               │
│     │ │ Hands   │ │ Legs    │     │                               │
│     │ │ Actions │ │ Actions │     │                               │
│     │ └─────────┘ └─────────┘     │                               │
│     │ ┌───────────────────┐       │                               │
│     │ │ Combined Actions  │       │                               │
│     │ └───────────────────┘       │                               │
│     └─────────────────────────────┘                               │
│                     │                                               │
│             ┌───────▼───────┐                                       │
│             │ Output JSON   │                                       │
│             └───────────────┘                                       │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘
```

---

## 数据来源

### InsightFace (face.json)

| 字段 | 说明 |
|------|------|
| **embedding** | 512-dim ArcFace embedding |
| **pose_angle** | Face pose (frontal, three_quarter, profile_left, profile_right) |
| **landmarks** | 5-point keypoints |

### MediaPipe Holistic (holistic.json)

| 字段 | 说明 |
|------|------|
| **face_mesh.landmarks** | 478 3D landmarks |
| **face_mesh.eye_features** | EAR, iris position, eye_action |
| **face_mesh.mouth_features** | MAR, mouth_action |
| **pose.landmarks** | 33 keypoints with visibility |
| **pose.arm_features** | Elbow angles, arm actions |
| **pose.leg_features** | Knee angles, leg actions |
| **hands.left/right** | 21 keypoints, gesture detection |

---

## 动作检测能力

### Face Actions (InsightFace)

| Action | Description | Example |
|--------|-------------|---------|
| **pose_frontal** | 正面 pose | frontal (confidence: 0.9) |
| **pose_three_quarter** | 侧面 pose | three_quarter (confidence: 0.85) |
| **pose_profile_left** | 左侧面 | profile_left (confidence: 0.9) |
| **pose_profile_right** | 右侧面 | profile_right (confidence: 0.9) |

---

### Eye Actions (MediaPipe Face Mesh)

| Action | Threshold | Description |
|--------|-----------|-------------|
| **eye_closed** | EAR < 0.15 | 闭眼 |
| **eye_squint** | EAR 0.15-0.25 | 眯眼 |
| **eye_normal** | EAR 0.25-0.4 | 正常 |
| **eye_wide_open** | EAR > 0.4 | 睁大眼 |
| **gaze_left** | iris_x < -0.2 | 向左看 |
| **gaze_right** | iris_x > 0.2 | 向右看 |

**示例输出**:
```json
{
  "eye_features": {
    "left_ear": 0.1902,
    "right_ear": 0.1902,
    "avg_ear": 0.1902,
    "eye_action": "squint",
    "gaze_direction": "center"
  }
}
```

---

### Mouth Actions (MediaPipe Face Mesh)

| Action | Threshold | Description |
|--------|-----------|-------------|
| **mouth_closed** | MAR < 0.2 | 闭嘴 |
| **mouth_slightly_open** | MAR 0.2-0.3 | 微张 |
| **mouth_open** | MAR > 0.5 | 张嘴 |
| **mouth_yawn** | MAR > 0.7 | 打哈欠 |
| **mouth_smile** | corner_lift > 0.02 | 微笑 |

**示例输出**:
```json
{
  "mouth_features": {
    "mar": 0.3319,
    "mouth_action": "slightly_open"
  }
}
```

---

### Arm Actions (MediaPipe Pose)

| Action | Angle Threshold | Description |
|--------|-----------------|-------------|
| **left_arm_raise_left** | wrist_y < elbow_y < shoulder_y | 举起左臂 |
| **left_arm_extend_left** | elbow_angle > 150° | 伸展左臂 |
| **left_arm_fold_left** | elbow_angle < 90° | 弯曲左臂 |
| **right_arm_raise_right** | wrist_y < elbow_y < shoulder_y | 举起右臂 |
| **right_arm_extend_right** | elbow_angle > 150° | 伸展右臂 |
| **right_arm_fold_right** | elbow_angle < 90° | 弯曲右臂 |
| **cross_arms** | wrists_x overlapping | 双手交叉 |

**示例输出**:
```json
{
  "arm_features": {
    "left_elbow_angle": 161.29,
    "right_elbow_angle": 161.95,
    "left_arm_action": "extend_left",
    "right_arm_action": "extend_right",
    "cross_arms": true
  }
}
```

---

### Hand Actions (MediaPipe Hands)

| Gesture | Fingers Extended | Description |
|---------|-----------------|-------------|
| **open_hand** | 5 | 张开手 |
| **fist** | 0 | 握拳 |
| **thumbs_up** | thumb only | 点赞 |
| **peace_sign** | index + middle | 剪刀手 |
| **pointing** | index only | 指向 |
| **ok_sign** | thumb + index touching | OK 手势 |
| **grab** | thumb + index | 抓取 |

**示例输出**:
```json
{
  "left_hand": {
    "gesture": "thumbs_up",
    "num_fingers_extended": 1
  },
  "right_hand": {
    "gesture": "open_hand",
    "num_fingers_extended": 5
  }
}
```

---

### Leg Actions (MediaPipe Pose)

| Action | Condition | Description |
|--------|-----------|-------------|
| **leg_stand** | hip < knee < ankle (vertical) | 站立 |
| **leg_sit** | hip ≈ knee height | 坐姿 |
| **leg_knee_bend** | knee_angle < 120° | 弯膝 |

**示例输出**:
```json
{
  "leg_features": {
    "left_knee_angle": 175.2,
    "right_knee_angle": 174.8,
    "standing": true,
    "sitting": false,
    "leg_action": "stand"
  }
}
```

---

## 实测结果 (preview.mp4)

### 动作统计

| 类别 | 动作 | 次数 |
|------|------|------|
| **Face** | pose_three_quarter | 6 |
| **Face** | pose_profile_right | 2 |
| **Eyes** | eye_squint | 8 |
| **Mouth** | mouth_closed | 6 |
| **Mouth** | mouth_slightly_open | 2 |
| **Arms** | cross_arms | 8 |
| **Arms** | left_arm_extend_left | 4 |
| **Arms** | left_arm_fold_left | 3 |
| **Arms** | right_arm_extend_right | 4 |
| **Arms** | right_arm_fold_right | 2 |
| **Hands** | left_hand_open_hand | 2 |
| **Hands** | left_hand_thumbs_up | 1 |
| **Hands** | right_hand_open_hand | 3 |
| **Legs** | leg_stand | 8 |

---

### 典型帧示例

#### Frame 30

```
Face: pose_three_quarter
Eyes: eye_squint
Mouth: mouth_closed
Arms: left_arm_fold_left, right_arm_neutral_right, cross_arms
Hands: left_hand_thumbs_up, right_hand_open_hand
Legs: leg_stand
```

**解读**: 站姿，左手握拳（fingers=1），右手张开（fingers=5），双臂交叉。

#### Frame 180

```
Face: pose_three_quarter
Eyes: eye_squint (EAR: 0.190)
Mouth: mouth_slightly_open (MAR: 0.332)
Arms: left_arm_extend_left (161.3°), right_arm_extend_right (161.9°), cross_arms
Legs: leg_stand
```

**解读**: 站姿，双臂伸展（角度161°），双手交叉，眼睛眯起，嘴巴微张。

---

## 创建的文件

| 文件 | 说明 |
|------|------|
| `scripts/mediapipe_holistic_processor.py` | MediaPipe Holistic 处理器 |
| `scripts/integrated_body_action_decoder.py` | 整合 Body Action Decoder |
| `scripts/utils/test_mediapipe.py` | MediaPipe 测试脚本 |

---

## 输出文件

| 文件 | 说明 |
|------|------|
| `preview.holistic.json` | MediaPipe Holistic 输出 (8 frames) |
| `integrated_body_actions.json` | 整合动作数据 (8 frames) |

---

## 使用方式

### Step 1: MediaPipe Holistic 处理

```bash
# 处理视频
python3 scripts/mediapipe_holistic_processor.py \
  --video video.mp4 \
  --output video.holistic.json \
  --sample-interval 30

# 测试单帧
python3 scripts/mediapipe_holistic_processor.py \
  --video video.mp4 \
  --output test.json \
  --test-frame 180
```

---

### Step 2: 整合 InsightFace + MediaPipe

```bash
# 整合并解码
python3 scripts/integrated_body_action_decoder.py \
  --face-json video.face_traced.json \
  --holistic-json video.holistic.json \
  --output-json integrated_body_actions.json

# 测试单帧
python3 scripts/integrated_body_action_decoder.py \
  --face-json video.face_traced.json \
  --holistic-json video.holistic.json \
  --frame 180
```

---

### Step 3: 查看输出

```json
{
  "frames": {
    "180": {
      "actions": {
        "face": [{"action": "pose_three_quarter"}],
        "eyes": [{"action": "eye_squint", "ear": 0.190}],
        "mouth": [{"action": "mouth_slightly_open", "mar": 0.332}],
        "arms": [
          {"action": "left_arm_extend_left", "angle": 161.29},
          {"action": "cross_arms"}
        ],
        "legs": [{"action": "leg_stand"}]
      }
    }
  }
}
```

---

## MediaPipe 模型信息

| Model | Keypoints | Purpose |
|-------|-----------|---------|
| **Face Mesh** | 478 | 面部网格 (eyes, mouth, iris) |
| **Pose** | 33 | 全身姿态 (arms, legs, torso) |
| **Hands** | 21 × 2 | 手部关键点 (fingers, wrist) |
| **Holistic** | 478 + 33 + 42 | 整合模型 |

---

## 版本信息

- MediaPipe: 0.9.2.1 (mediapipe-silicon)
- InsightFace: buffalo_l
- 整合状态: ✅ 完成
- 测试状态: ✅ 通过
