warren/momentry_core
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 2 Wiki Activity

cleanup: remove dead code and duplicate docs

Browse Source 
- Remove session-ses_2f27.md (161KB raw session log)
- Remove 49 ROOT_* duplicate files across REFERENCE/
- Remove 14 duplicate files between REFERENCE/ root and history/
- Remove asr_legacy.rs (dead code, replaced by asr.rs)
- Remove src/core/worker/ (duplicate JobWorker)
- Remove src/core/layers/ (empty directory)
- Remove 4 .bak files in src/
- Remove 7 dead private methods in worker/processor.rs
- Remove backup directory from git tracking
This commit is contained in:
Warren
2026-05-04 01:31:21 +08:00
parent ee81e343ce
commit e75c4d6f07
3270 changed files with 35190 additions and 53367 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs_v1.0/AI_AGENTS/CONTEXT/METADATA_PROCESSORS.md
-563
View File
@@ -1,563 +0,0 @@
# Momentry Core - Metadata 及 處理器總覽
本文檔說明 Momentry Core 中 chunks 資料表的 metadata 結構，以及各類處理器的輸出欄位。
## 1. Chunks 資料表結構
### 1.1 直接欄位 (Direct Columns)
這些欄位直接儲存於 chunks 資料表中：
| 欄位 | 類型 | 來源處理器 | 說明 |
|------|------|----------|------|
| `id` | serial | 系統 | 主鍵 |
| `uuid` | varchar(32) | 系統 | 影片 UUID |
| `chunk_id` | varchar(64) | 系統 | Chunk ID (如 sentence_0001) |
| `chunk_index` | integer | 系統 | 順序編號 |
| `chunk_type` | varchar(32) | 系統 | sentence/cut/time |
| `text_content` | text | ASR processor | 語音轉文字結果 |
| `content` | jsonb | - | 原始內容 (rule, data 等) |
| `metadata` | jsonb | 多個處理器 | 參閱下方 1.2 |
| `visual_stats` | jsonb | add_yolo_to_chunks.py | YOLO 識別結果 |
| `speaker_ids` | text[] | ASRX processor | 說話者 ID 陣列 |
| `face_ids` | integer[] | Face processor | 臉部 ID 陣列 |
| `summary_text` | text | generate_chunk_summaries.py | LLM 生成摘要 |
| `parent_chunk_id` | varchar(64) | 系統 | 父 chunk ID |
| `fps` | double | ffprobe | 幀率 |
| `start_frame` | bigint | ffprobe | 開始幀 |
| `end_frame` | bigint | ffprobe | 結束幀 |
| `metadata_version` | integer | 系統 | Metadata 版本 (5W1H, identity, visual) |
| `content_version` | integer | 系統 | Content 版本 (text_content, summary_text) |
| `created_at` | timestamp | 系統 | 建立時間 |
| `updated_at` | timestamp | 系統 | 最後更新時間 |
### 版本控制說明
| 欄位 | 說明 | 遞增時機 |
|------|------|----------|
| `metadata_version` | Metadata 版本 | 更新 5W1H, identity, visual 時 |
| `content_version` | Content 版本 | 更新 text_content, summary_text 時 |
| `updated_at` | 最後更新時間 | 任何更新時自動更新 |
**判別更新語法**:
```sql
-- 檢查哪些 chunk 需要重新生成 5W1H
SELECT chunk_id, metadata_version, content_version, updated_at
FROM dev.chunks
WHERE metadata_version < 1;
-- 檢查特定時間後的更新
SELECT chunk_id, updated_at
FROM dev.chunks
WHERE updated_at > '2024-01-01';
-- 檢查版本差異 (需要重新處理)
SELECT c.*
FROM dev.chunks c
WHERE c.metadata_version <
(SELECT MAX(metadata_version) FROM dev.chunks WHERE uuid = c.uuid);
```
## 11. 動態 Metadata 管理
### 11.1 欄位動態增減
Metadata JSONB 支援動態欄位，可根據處理器執行結果動態添加：
```python
# 動態添加欄位
metadata = existing_metadata or {}
metadata[field_name] = value
UPDATE chunks SET metadata = metadata || %s::jsonb
```
### 11.2 常見動態欄位
| 欄位 | 新增時機 | 來源處理器 |
|------|----------|------------|
| `chunk_5w1h` | 生成 summary | generate_chunk_summaries.py |
| `chunk_identity` | ASRX/Face 執行後 | 來源欄位聚合 |
| `chunk_visual` | YOLO 執行後 | add_yolo_to_chunks.py |
| `chunk_emotion` | 情緒分析 | future emotion_processor.py |
| `chunk_pose` | 姿勢辨識 | future pose_processor.py |
| `chunk_sentiment` | 情感分析 | future sentiment_processor.py |
### 11.3 版本升級策略
每次重大更新時遞增版本號：
```python
if新增重大欄位:
metadata_version += 1
# 記錄變更日誌
```
### 11.4 重跑機制
```bash
# 重跑特定版本後的 chunk
python scripts/generate_chunk_summaries.py --uuid <uuid> --min-version 1
# 查看版本分佈
SELECT metadata_version, COUNT(*)
FROM dev.chunks
GROUP BY metadata_version;
```
### 1.2 Metadata 結構 (JSONB)
`metadata` 欄位包含多個子欄位，由不同處理器產生：
```json
{
"chunk_5w1h": {
"who": "演員或角色",
"what": "主要動作或事件",
"when": "時間上下文",
"where": "地點",
"why": "目的或原因",
"how": "表達方式"
},
"chunk_identity": {
"speakers": ["speaker_001", "speaker_002"],
"faces": ["face_1", "face_3"]
},
"chunk_visual": {
"objects": ["person", "car", "tree"],
"places": ["street", "office"]
},
"structured_summary": {
"who": "Parent 級別角色",
"what": "Parent 級別動作",
...
}
}
```
| 子欄位 | 類型 | 來源處理器 | 說明 |
|--------|------|----------|------|
| `chunk_5w1h` | jsonb | generate_chunk_summaries.py | Chunk 級別的 5W1H + Emotion + Actions |
| `chunk_5w1h.who` | string | person | 人物名稱 (含來源標記) |
| `chunk_5w1h.what` | string | action | 具體動作 |
| `chunk_5w1h.when` | string | position | 場景中位置 (beginning/middle/end) |
| `chunk_5w1h.where` | string | location | 地點 |
| `chunk_5w1h.why` | string | purpose | 目的 |
| `chunk_5w1h.how` | string | manner | 表達方式 |
| `chunk_5w1h.emotion` | string | emotion | 情緒/語氣 |
| `chunk_5w1h.actions` | string[] | verbs | 動作動詞 |
| `chunk_identity` | jsonb | 來源欄位聚合 | speaker_ids + face_ids 資訊 |
| `chunk_visual` | jsonb | add_yolo_to_chunks.py | YOLO 物體識別結果 |
| `structured_summary` | jsonb | regenerate_parent_5w1h.py | Parent 級別 5W1H + tone + characters + key_events |
### chunk_5w1h 欄位說明 (Chunk 級)
| 欄位 | 類型 | 說明 | 範例 |
|------|------|------|------|
| `who` | string | 此 chunk 出現的角色 (含來源) | "John (SPEAKER_1), Mary (face_3)" |
| `what` | string | 此 chunk 的具體動作 | "Giving warning" |
| `when` | string | 相對時間位置 | "Mid-scene" |
| `where` | string | 地點 (如提及) | "Near taxi" |
| `why` | string | 此動作的目的 | "Warn about danger" |
| `how` | string | 表達/呈現方式 | "Urgent tone" |
| `emotion` | string | 情緒/語氣 | "Fearful, urgent" |
| `actions` | string[] | 動作動詞 | ["run", "shout", "warn"] |
**Prompt 增強內容**:
- 從 person_identities 取得驗證的人物名稱
- 包含 speaker_id 和 face_id 來源標記
- 視覺辨識: objects, places, actions
- Time range 傳入 chunk 時間範圍
- Emotion + Actions 額外欄位
### chunk_identity 欄位說明
| 欄位 | 類型 | 說明 | 範例 |
|------|------|------|------|
| `speakers` | string[] | 說話者 ID | ["speaker_001", "speaker_002"] |
| `faces` | string[] | 臉部 ID | ["face_1", "face_3"] |
| `global_identity` | string | 對應的全局人物 ID | "person_001" |
| `person_name` | string | 識別的人物名稱 | "John" |
> 說明：
> - `speakers`/`faces` 來自 ASRX/Face processor
> - `global_identity` 來自 `person_identities` 表，關聯 face_identity_id
> - `person_name` 來自 `person_identities.name`，經過確認的人物名稱
### 全域人物 Identity (person_identities 表)
每個影片會識別並記錄出現的人物，儲存於 `dev.person_identities` 表：
| 欄位 | 類型 | 說明 |
|------|------|------|
| `person_id` | varchar(255) | 人物唯一 ID (如 person_001) |
| `name` | varchar(255) | 人物名稱 (可確認) |
| `speaker_id` | varchar(255) | 對應的說話者 ID |
| `file_uuid` | varchar(255) | 影片 UUID |
| `face_identity_id` | integer | 對應的 global identity |
| `appearance_count` | integer | 出現次數 |
| `first_appearance_time` | double | 首次出現時間 |
| `last_appearance_time` | double | 最後出現時間 |
| `confidence` | double | 辨識信心度 |
| `is_confirmed` | boolean | 是否已確認 |
### 全域 Identity (face_identities 表)
跨影片的全局人物身份：
| 欄位 | 類型 | 說明 |
|------|------|------|
| `id` | serial | 主鍵 |
| `face_id` | integer | 臉部 ID |
| `name` | varchar(255) | 識別姓名 |
| `embedding` | blob | 人臉向量特徵 |
### 人物識別流程
Momentry 的人物識別分為三個層級：
```
層級 1: 原始識別 (chunks 表)
├── chunks.face_ids → 臉部 ID (local to chunk)
└── chunks.speaker_ids → 說話者 ID (local to chunk)
層級 2: 影片級識別 (person_identities 表)
├── person_id → 人物 ID (影片內唯一)
├── name → 識別出的人物名稱 (如 "John")
├── speaker_id → 對應的說話者
└── face_identity_id → 對應的全局 Identity
層級 3: 全局身份 (face_identities 表)
├── id → 全局唯一 ID
├── face_id → 臉部特徵 ID
├── name → 確認的姓名
└── embedding → 人臉向量 (用於比對)
```
**識別流程說明**:
```
Step 1: ASRX Processor
chunks.speaker_ids ← 說話者分離
Step 2: Face Processor
chunks.face_ids ← 臉部偵測
Step 3: Auto-identify
person_identities ← 合併 speaker + face (影片級)
Step 4: Global Matching
face_identities ← 人臉向量比對 (全局 Identity)
↑
合併相同人臉者為同一 Identity
```
**命名原則**:
- `person_id` = 角色名 (如 "John", "Adam")
- 而非 "Person_8"
- 透過 speaker 對應 + 手動確認
**範例**:
```sql
-- 取得影片中的人物列表
SELECT person_id, name, speaker_id, appearance_count
FROM dev.person_identities
WHERE file_uuid = '384b0ff44aaaa1f14cb2cd63b3fea966'
ORDER BY appearance_count DESC;
-- 取得 chunk 的人物
SELECT c.chunk_id, pi.name, pi.speaker_id
FROM dev.chunks c
JOIN dev.person_identities pi ON c.uuid = pi.file_uuid
WHERE c.chunk_id = 'sentence_0001';
```
### 取得 chunk 的人物資訊
```sql
-- 取得某 chunk 的人物
SELECT pi.name, pi.speaker_id, pi.appearance_count
FROM dev.person_identities pi
JOIN dev.chunks c ON c.uuid = pi.file_uuid
WHERE c.chunk_id = 'sentence_0001';
```
### chunk_visual 欄位說明
| 欄位 | 類型 | 說明 | 範例 |
|------|------|------|------|
| `objects` | string[] | YOLO 識別物體 | ["person", "car", "tree"] |
| `places` | string[] | Places365 識別地點 | ["street", "office"] |
## 2. 處理器對照表
### 2.1 ASR 處理器 (語音辨識)
**用途**：將影片音軌轉換為文字
| 處理器 | 輸出欄位 | 說明 |
|--------|---------|------|
| asr_processor_small_multilingual.py | text_content | Small 模型，多語言 |
| asr_processor_simplified.py | text_content | 簡化版 |
| asr_processor_contract_v1.py | text_content | 契約版本 v1 |
| asr_processor_contract_v2.py | text_content | 契約版本 v2 |
**輸出**：
- `text_content`: 語音轉文字結果
- 寫入 `chunks.content` 和 `chunks.text_content`
### 2.2 ASRX 處理器 (增強說話者辨識)
**用途**：說話者分離 (Diarization)
| 處理器 | 輸出欄位 | 說明 |
|--------|---------|------|
| asrx_processor.py | speaker_ids | 標準版 |
| asrx_processor_contract_v1.py | speaker_ids | 契約版 v1 |
**輸出**：
- `speaker_ids`: 說話者 ID 陣列，如 `["speaker_001", "speaker_002"]`
- 目前為空 `{}`，需執行後才會填充
### 2.3 Face 處理器 (臉部偵測)
**用途**：偵測並追蹤人臉
| 處理器 | 輸出欄位 | 說明 |
|--------|---------|------|
| analyze_video_faces.py | face_ids | 臉部偵測 |
**輸出**：
- `face_ids`: 臉部 ID 陣列，如 `[1, 3, 5]`
- 目前為空 `{}`，需執行後才會填充
### 2.4 YOLO 處理器 (物體識別)
**用途**：識別場景中的物體和地點
| 處理器 | 輸出欄位 | 說明 |
|--------|---------|------|
| add_yolo_to_chunks.py | visual_stats, chunk_visual | YOLO + Places365 |
**輸出**：
- `visual_stats`: 原始識別結果
- `metadata.chunk_visual`: 簡化格式 `{objects: [...], places: [...]}`
### 2.5 Summary 處理器 (生成摘要)
**用途**：生成 chunk 摘要和 5W1H 分析
| 處理器 | 輸出欄位 | 說明 |
|--------|---------|------|
| generate_chunk_summaries.py | summary_text, chunk_5w1h, chunk_identity, chunk_visual | LLM 生成 |
| regenerate_parent_5w1h.py | structured_summary | Parent 場景級 5W1H |
**輸入**：
- chunk.text_content
- parent_chunks.summary_text
- parent_chunks.metadata.structured_summary
- chunk.speaker_ids (用於 chunk_identity)
- chunk.face_ids (用於 chunk_identity)
- chunk.visual_stats (用於 chunk_visual)
**輸出**：
- `summary_text`: 2-3 句摘要
- `metadata.chunk_5w1h`: Who/What/When/Where/Why/How
- `metadata.chunk_identity`: speakers, faces
- `metadata.chunk_visual`: objects, places
## 3. Parent Chunks 結構
Parent chunks 代表場景 (scene) 層級：
| 欄位 | 類型 | 說明 |
|------|------|------|
| `id` | serial | 主鍵 |
| `uuid` | varchar(32) | 影片 UUID |
| `scene_order` | integer | 場景順序 |
| `summary_text` | text | 場景摘要 (LLM 生成) |
| `metadata` | jsonb | 包含 structured_summary |
### Parent Metadata 結構
```json
{
"structured_summary": {
"who": "主要角色",
"what": "主要事件",
"when": "時間線",
"where": "地點",
"why": "動機",
"how": "方式",
"tone": ["緊張", "懸疑", "溫馨"],
"characters": ["角色A", "角色B", "角色C"],
"key_events": ["事件1", "事件2", "事件3"],
"summary_5lines": "5行摘要..."
},
"auto_generated_by": "gemma4",
"chunk_count": 885
}
```
### structured_summary 欄位說明
| 欄位 | 類型 | 說明 | 範例 |
|------|------|------|------|
| `who` | string | 主要角色 | "Mr. Balletman, Adam" |
| `what` | string | 主要動作或事件 | "Escape attempt" |
| `when` | string | 時間上下文 | "During critical moment" |
| `where` | string | 地點 | "Near taxi" |
| `why` | string | 動機或原因 | "Evade capture" |
| `how` | string | 執行方式 | "Quickly moving to taxi" |
| `tone` | string[] | 語氣/情緒 | ["Urgent", "Tense", "Fearful"] |
| `characters` | string[] | 場景中的角色 | ["Mr. Balletman", "Adam", "Antagonist"] |
| `key_events` | string[] | 關鍵事件 | ["Decision to flee", "Warning given"] |
| `summary_5lines` | string | 5行摘要 | "Line 1\nLine 2..." |
## 4. Chunk 類型說明
| 類型 | 需要搜尋 | 說明 |
|------|----------|------|
| `sentence` | ✓ | 有 text_content，需向量化存入 Qdrant |
| `cut` | ✗ | 場景剪輯點，無文字內容 |
| `time` | ✗ | 時間區間標記，無文字 |
**搜尋適用性**：
- sentence: 有文字內容，可進行語意搜尋
- cut/time: 無文字，僅供時間定位使用
## 5. 處理流程 (Pipeline)
```
1. ffprobe → 取得影片資訊 (fps, frame count)
2. ASR processor → text_content
3. [ASRX processor] → speaker_ids (選用)
4. [Face processor] → face_ids (選用)
5. add_yolo_to_chunks.py → visual_stats
6. generate_chunk_summaries.py → summary_text + metadata
7. [vectorize_chunk_summaries.py] → Qdrant 向量
```
## 6. Qdrant Collections
| Collection | 向量類型 | 用途 |
|------------|----------|------|
| `momentry_dev_chunk_summaries` | nomic-embed-text | Chunk summary 語意搜尋 |
| `momentry_dev_vectors` | 原始向量 | 備用 |
## 7. API 回傳格式
Chunk Detail API 合併 chunk 和 parent 的 metadata：
```
metadata
├── chunk_5w1h (chunk 級)
├── chunk_identity (chunk 級)
├── chunk_visual (chunk 級)
├── structured_summary (parent 級) ← 只在有 parent 時
├── auto_generated_by
└── chunk_count
```
## 8. 執行狀態檢查
```bash
# 檢查 summary 生成進度
psql -h localhost -U accusys -d momentry -c "
SELECT COUNT(*) as total,
COUNT(CASE WHEN summary_text IS NOT NULL THEN 1 END) as generated
FROM dev.chunks WHERE chunk_type = 'sentence';"
# 檢查執行中的處理器
ps aux | grep -E "processor|generate" | grep -v grep
# 檢查 visual_stats
psql -h localhost -U accusys -d momentry -c "
SELECT COUNT(*) FROM dev.chunks WHERE visual_stats IS NOT NULL;"
```
## 9. 待執行處理器
### 人物識別處理器 (依序執行)
```bash
# Step 1: ASRX 執行說話者分離
python scripts/asrx_processor.py --uuid 384b0ff44aaaa1f14cb2cd63b3fea966
# Step 2: Face 執行臉部偵測
python scripts/analyze_video_faces.py --uuid 384b0ff44aaaa1f14cb2cd63b3fea966
# Step 3: Auto-identify 建立影片級人物
python scripts/auto_identify_persons.py --uuid 384b0ff44aaaa1f14cb2cd63b3fea966
# Step 4: 全局 Identity 比對 (需累積一定數量的 face_identities)
python scripts/match_faces_to_identities.py
# Step 5: 重新生成 chunk 5W1H (包含新的 identity 資訊)
python scripts/generate_chunk_summaries.py --uuid 384b0ff44aaaa1f14cb2cd63b3fea966
```
### 檢查待處理狀態
```bash
# 檢查 speaker_ids
psql -h localhost -U accusys -d momentry -c "
SELECT COUNT(*) FROM dev.chunks
WHERE speaker_ids IS NOT NULL AND array_length(speaker_ids, 1) > 0;"
# 檢查 face_ids
psql -h localhost -U accusys -d momentry -c "
SELECT COUNT(*) FROM dev.chunks
WHERE face_ids IS NOT NULL AND array_length(face_ids, 1) > 0;"
# 檢查 person_identities
psql -h localhost -U accusys -d momentry -c "
SELECT COUNT(*) FROM dev.person_identities
WHERE file_uuid = '384b0ff44aaaa1f14cb2cd63b3fea966';"
# 檢查 face_identities (全局)
psql -h localhost -U accusys -d momentry -c "
SELECT COUNT(*) FROM dev.face_identities;"
```
## 10. 自動化重新生成機制
### 觸發條件
當以下事件發生時，應自動重新生成 chunk 的 5W1H 和相關 metadata：
| 事件 | 觸發動作 |
|------|----------|
| 第一次執行 ASRX | 重新生成含 speaker_ids 的 5W1H |
| 第一次執行 Face | 重新生成含 face_ids 的 5W1H |
| 新增 chunk | 為新 chunk 生成 5W1H |
| 修改 chunk 內容 | 更新 5W1H 和 summary |
| 新增/修改 speaker | 重新生成含新 speaker 的 5W1H |
| 新增/修改 face | 重新生成含新 face 的 5W1H |
### 重新生成流程
```
事件觸發
↓
更新 speaker_ids / face_ids / person_identities
↓
呼叫 generate_chunk_summaries.py --uuid <uuid> --regenerate
↓
重新產生:
├── summary_text (2-3 句)
├── metadata.chunk_5w1h (Who/What/When/Where/Why/How)
├── metadata.chunk_identity (更新後的 speakers/faces)
└── metadata.chunk_visual (若 visual_stats 有更新)
```
### 重點
每次處理器執行後，Chunk metadata 會包含最新的：
1. **speaker_ids** → 進入 `chunk_identity.speakers`
2. **face_ids** → 進入 `chunk_identity.faces`
3. **person_identities** → 進入 `chunk_identity.person_name`
確保 LLM 產生的 5W1H 包含最新的角色資訊。
docs_v1.0/AI_AGENTS/CORE/AGENT_SPEC.md
-180
View File
@@ -1,180 +0,0 @@
---
document_type: "standard_doc"
service: "MOMENTRY_CORE"
title: "AI Agent 設計規範"
date: "2026-04-27"
version: "V1.1"
status: "active"
owner: "Warren"
created_by: "OpenCode"
tags:
- "AI Agent"
- "設計規範"
- "三層架構"
- "processing_status"
ai_query_hints:
- "查詢 AI Agent 設計規範的內容"
- "AI Agent 的三層架構定義"
- "Agent 類型列表"
- "Agent 進度追蹤方式"
- "processing_status JSONB agents 字段"
- "如何設計 AI Agent"
---
# AI Agent 設計規範 (Agent Design Specification)
| 項目 | 內容 |
|------|------|
| 建立者 | OpenCode |
| 建立時間 | 2026-04-25 |
| 文件版本 | V1.1 |
---
## 版本歷史
| 版本 | 日期 | 目的 | 操作人 | 工具/模型 |
|------|------|------|--------|-----------|
| V1.0 | 2026-04-25 | 定義 Momentry Core 中 AI Agent 的標準設計與職責 | OpenCode | OpenCode |
| V1.1 | 2026-04-27 | 添加 Agent 類型列表和進度追蹤（processing_status JSONB） | OpenCode | GLM-5 |
---
## 1. 核心概念
在 Momentry Core 系統中，處理邏輯分為三個層次，本規範專注於第三層：
| 層次 | 名稱 | 特性 | 範例 |
|------|------|------|------|
| **L1** | **Processor (處理器)** | **確定性 (Deterministic)**<br>輸入 A 必得輸出 B。通常為編譯型程式或腳本。 | FFmpeg, Whisper (ASR), YOLO |
| **L2** | **Rule (規則)** | **邏輯性 (Logic)**<br>基於明確的條件、正則表達式或時間軸聚合。 | 語句切分，時間重疊計算 |
| **L3** | **Agent (智能體)** | **推論性 (Probabilistic)**<br>依賴 LLM 進行語義理解、決策或生成。具備 Prompt 或 Workflow。 | 5W1H 推論，身份解析，摘要生成 |
---
## 2. Agent 職責 (Responsibilities)
AI Agent 負責處理那些傳統程式難以精確定義規則的任務。
**注意**: 在系統架構中，Agent 被視為一種 **資源 (Resource)**，與 Processor 和 Service 統一由 **資源註冊中心 (Resource Registry)** 管理。
1. **語義理解 (Semantic Understanding)**: 將非結構化數據（如 OCR 文字、雜訊 ASR 文本）轉化為結構化標籤 (5W1H)。
2. **跨模態匹配 (Cross-Modal Matching)**: 綜合視覺、聽覺和文本證據，判斷「畫面中的臉」是否為「資料庫中的人」。
3. **內容生成 (Content Generation)**: 為影片片段生成自然的摘要或標題。
4. **查詢解析 (Query Parsing)**: 將用戶的自然語言請求轉譯為系統可執行的 API 調用序列。
---
## 3. 標準設計結構 (Design Structure)
所有 AI Agent 的設計文件必須遵循以下結構：
### 3.1 檔案命名
* **格式**: `[AGENT_TYPE]_[PURPOSE].md`
* **範例**: `CONTEXT_5W1H_INFERENCE.md`
### 3.2 文件內容
#### 3.2.1 Agent 目標 (Goal)
簡短描述此 Agent 解決的業務問題。
> **範例**: 從雜亂的 YOLO 標籤和 OCR 文本中推論場景的「地點」和「天氣」資訊。
#### 3.2.2 輸入數據 (Input)
定義 Agent 接收的數據格式。通常來自 Processor 輸出或 Rule 產物。
* **來源**: `PROCESSORS/` 或 `CHUNKING/`
* **格式**: JSON, Text, List of Frames.
#### 3.2.3 核心邏輯 (Core Logic: Prompt / Workflow)
這是 Agent 的靈魂。
* **單一 Prompt Agent**: 提供完整的 System Prompt。
```markdown
## System Prompt
You are a scene analysis assistant...
```
* **多步 Workflow Agent**: 提供步驟圖或偽代碼。
```mermaid
graph TD
A[Start] --> B[Extract Entities]
B --> C[Verify with Knowledge Base]
C --> D[Output Result]
```
#### 3.2.4 輸出格式 (Output)
定義 Agent 產出的結構化數據 (通常為 JSON)。
```json
{
"who": ["Actor Name"],
"what": ["Action"],
"confidence": 0.95
}
```
#### 3.2.5 模型配置 (Model Config)
建議使用的模型類型及其原因。
* **推理模型 (Reasoning)**: `o1`, `R1` (用於複雜邏輯判斷)
* **生成模型 (Generation)**: `GPT-4o`, `Sonnet` (用於摘要)
* **本地模型 (Local)**: `Llama-3`, `Qwen` (用於隱私數據)
---
## 4. 開發工作流 (Development Workflow)
1. **定義需求**: 確定是否需要 AI 介入 (若規則可解，優先使用 Rule)。
2. **撰寫 Prompt**: 在文檔中迭代 Prompt，直到達到穩定輸出。
3. **工具串接**: 若需要外部數據 (如 TMDB)，定義 Tool 定義。
4. **實作封裝**: 將 Prompt/Workflow 封裝為 Rust/Python 模組，透過 API 調用。
---
## 5. 相關文件
* `UNIFIED_RESOURCE_REGISTRY.md` - 系統統一資源管理架構 (Agents 作為資源註冊)。
* `AI_DRIVEN_PROCESSOR_CONTRACT.md` - Processor 層級的整合合約。
* `CHUNKING_ARCHITECTURE.md` - Rule 層級的架構。
* `FILE_IDENTITY_API_DESIGN.md` - 全局架構。
---
## 6. Agent 類型列表
| Agent | 目的 | 觸發條件 | 文檔 |
|-------|------|----------|------|
| **Translation Agent** | 多語言翻譯 | 用戶手動觸發 | `AI_AGENTS/TRANSLATION/TEXT_TRANSLATION.md` |
| **5W1H Agent** | 場景分析（Who/What/When/Where/Why/How） | Rule 3 完成 | `AI_AGENTS/SUMMARIZATION/CHUNK_RULE_4_SUMMARY.md` |
| **Identity Agent** | 身份解析（Face/Speaker → Person） | Face/Speaker 完成 | `AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_WORKFLOW.md` |
---
## 7. Agent 進度追蹤
從 V1.2 起，所有 Agent 任務透過 `processing_status` JSONB 的 `agents` 字段追蹤。
### JSONB 範例
```json
{
"agents": {
"5w1h": {
"status": "running",
"scenes_processed": 5,
"scenes_total": 1332,
"progress_pct": 0.4
}
}
}
```
### 查詢 Agent 進度
```sql
SELECT processing_status->'agents'->'5w1h'->>'status' FROM videos WHERE uuid = 'xxx';
```
詳細規範請參考: `REFERENCE/PROCESSING_STATUS_JSONB_SPEC.md`
---
## 版本資訊
* 版本: V1.1
* 建立日期: 2026-04-25
* 文件更新: 2026-04-27
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_IDENTITY_TUTORIAL.md
-183
View File
@@ -1,183 +0,0 @@
# Face, Speaker, Person, Identity API 教學示範
本文件將以 1963 年電影《Charade》（謎中謎）為例，示範如何使用 API 管理 **Face** (臉孔)、**Person** (影片中的角色實體) 與 **Identity** (真實身份)。
## 核心概念定義
在開始之前，請區分以下名詞：
1. **Face (臉孔)**: 影像中偵測到的具體臉部特徵數據（向量）。
2. **Person (角色實體)**: 在特定影片中出現的角色。他是 Face + Speaker (說話者) 的集合體。
* *例如：影片 `384b0ff44aaaa1f14cb2cd63b3fea966` 中的 `Person_17`。*
3. **Identity (真實身份)**: 跨越所有影片的全域實體（如真實演員或新聞人物）。
* *例如：Cary Grant, Audrey Hepburn。*
---
## 前置準備
* **API URL**: `http://localhost:3003`
* **API Key**: `/`
* **目標影片 (Video UUID)**: `384b0ff44aaaa1f14cb2cd63b3fea966` (Charade)
---
## 情境設定
我們要在影片中識別兩位主角：
1. **Audrey Hepburn** (飾演 Reggie Lampert)
2. **Cary Grant** (飾演 Peter Joshua)
---
## 步驟一：查看影片中的現有角色 (Person List)
首先，我們查詢系統在影片中偵測到了哪些人物 (Person)。
```bash
curl -s "http://localhost:3003/api/v1/person/list?file_uuid=384b0ff44aaaa1f14cb2cd63b3fea966&limit=5" \
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
| python3 -m json.tool
```
**預期回應**：
你會看到類似如下的列表，其中包含系統自動分配的 `person_id` (例如 `Person_17`, `Person_4` 等)。
```json
{
"persons": [
{
"person_id": "Person_17",
"name": null,
"speaker_id": "SPEAKER_1",
"appearance_count": 1636
},
{
"person_id": "Person_4",
"name": null,
"speaker_id": "SPEAKER_0",
"appearance_count": 936
}
]
}
```
---
## 步驟二：建立身份並綁定角色 (Register Identity from Person)
假設經過人工確認，我們知道 `Person_17` 是 Audrey Hepburn。我們可以使用單一 API 同時完成 **「建立 Identity」** 與 **「綁定 Person」** 兩個動作。
### 範例 1: 註冊 Audrey Hepburn
我們指定 `Person_17` 為 "Audrey Hepburn"。系統會檢查此 Identity 是否存在；若不存在則建立，若已存在則直接綁定。
```bash
curl -s -X POST "http://localhost:3003/api/v1/identities/from-person" \
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
"person_id": "Person_17",
"identity_name": "Audrey Hepburn",
"metadata": { "role": "Reggie Lampert" }
}' | python3 -m json.tool
```
**預期回應**：
```json
{
"success": true,
"message": "Successfully registered identity 'Audrey Hepburn' and linked to person 'Person_17'",
"identity_id": 10,
"identity_name": "Audrey Hepburn",
"person_id": "Person_17"
}
```
*(註：此操作會自動將該影片中 `Person_17` 的名稱更新為 "Audrey Hepburn")*
### 範例 2: 註冊 Cary Grant
假設 `Person_4` 是 Cary Grant，我們進行同樣的操作。
```bash
curl -s -X POST "http://localhost:3003/api/v1/identities/from-person" \
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
"person_id": "Person_4",
"identity_name": "Cary Grant",
"metadata": { "role": "Peter Joshua" }
}' | python3 -m json.tool
```
**預期回應**：
```json
{
"success": true,
"message": "Successfully registered identity 'Cary Grant' and linked to person 'Person_4'",
"identity_id": 11,
"identity_name": "Cary Grant",
"person_id": "Person_4"
}
```
---
## 步驟三：查看全域身份庫 (List Identities)
現在我們可以查看所有已建立的「真實身份」，這些身份是跨影片通用的。
```bash
curl -s "http://localhost:3003/api/v1/identities?limit=10" \
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
| python3 -m json.tool
```
**預期回應**：
你應該能看到剛剛建立的 "Audrey Hepburn" 和 "Cary Grant"。
```json
[
{
"id": 11,
"name": "Cary Grant",
"metadata": { "role": "Peter Joshua" }
},
{
"id": 10,
"name": "Audrey Hepburn",
"metadata": { "role": "Reggie Lampert" }
}
]
```
---
## 步驟四：驗證綁定結果
再次查詢影片中的 `Person` 列表，確認名稱是否已自動更新。
```bash
curl -s "http://localhost:3003/api/v1/person/list?file_uuid=384b0ff44aaaa1f14cb2cd63b3fea966&limit=5" \
-H "X-API-Key: muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69" \
| python3 -m json.tool
```
**預期結果**：
原本的 `Person_17` 現在應該顯示為 `"name": "Audrey Hepburn"`。
---
## 常見問題 (FAQ)
**Q: 如果我想把「現有的 Person」綁定到「已經存在的 Identity」，要怎麼做？**
A: 使用相同的 `POST /api/v1/identities/from-person` API。只要傳入相同的 `identity_name` (例如 "Audrey Hepburn")，系統會自動找到該 Identity 並將新的 Person 連結過去，不會建立重複的 Identity。
**Q: Identity 和 Person 的差別是什麼？**
A: **Identity** 是真實世界的人（例如 "Tom Hanks"），這是全域共享的。
**Person** 是他在某部電影裡的具體出現（例如《阿甘正傳》裡的阿甘）。一個 Identity 可以對應多個影片中的多個 Person。
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_PROGRESS.md
-97
View File
@@ -1,97 +0,0 @@
# Face/Speaker/Person 分析完成度
**UUID**: `384b0ff44aaaa1f14cb2cd63b3fea966`
**视频**: Charade (1963) - ~115 min, 412,343 frames, 59.94 fps
**更新日期**: 2026-04-14
---
## 📊 数据统计
| 模块 | 状态 | 文件 | 数据量 |
|------|------|------|--------|
| **Face Detection** | ✅ 完成 | `384b0ff44aaaa1f14cb2cd63b3fea966.face.json` | 10,691 frames, 25,174 faces |
| **Face Clustering** | ✅ 完成 | `384b0ff44aaaa1f14cb2cd63b3fea966.face_clustered.json` | 302 unique Person IDs |
| **ASR (语音识别)** | ✅ 完成 | `384b0ff44aaaa1f14cb2cd63b3fea966.asr.json` | 1,011 segments |
| **ASRX (增强语音)** | ✅ 完成 | `384b0ff44aaaa1f14cb2cd63b3fea966.asrx.json` | - |
| **Pose (姿态)** | ✅ 完成 | `384b0ff44aaaa1f14cb2cd63b3fea966.pose.json` | - |
| **Speaker Diarization** | ⚠️ 未集成 | - | ASR segments 无 speaker 信息 |
---
## 🎯 Top 20 人物 (按帧数)
| Person ID | 帧数 | 说明 |
|-----------|------|------|
| Person_0 | 17,832 | 主角 (Cary Grant/Audrey Hepburn) |
| Person_17 | 1,636 | 主要配角 |
| Person_4 | 936 | 主要配角 |
| Person_25 | 217 | 次要角色 |
| Person_12 | 154 | 次要角色 |
| Person_46 | 122 | - |
| Person_70 | 119 | - |
| Person_8 | 109 | - |
| Person_3 | 109 | - |
| Person_124 | 97 | - |
| Person_37 | 95 | - |
| Person_176 | 90 | - |
| Person_34 | 85 | - |
| Person_80 | 78 | - |
| Person_50 | 73 | - |
| Person_94 | 73 | - |
| Person_33 | 63 | - |
| Person_21 | 58 | - |
| Person_14 | 57 | - |
| Person_7 | 57 | - |
**总计**: 302 个独立 Person ID，其中 282 个出现少于 57 帧。
---
## ⚠️ 未完成的整合
### 1. Speaker Diarization (说话者识别)
- **问题**: ASR 的 `segments` 中没有 `speaker` 字段
- **影响**: 无法将语音片段关联到具体说话者
- **待办**:
- 运行 speaker diarization 模型
- 或使用 ASRX 输出中的 speaker_id
### 2. Face ↔ Speaker 关联
- **脚本存在**: `scripts/sync_face_speaker_to_chunks.py`
- **状态**: 需要数据库支持 (chunks 表)
- **功能**: 将 face_ids 和 speaker_ids 写入 chunks 表
### 3. Face ↔ ASR 验证
- **文档存在**: `scripts/ASR_FACE_POSE_INTEGRATION.md`
- **状态**: 方案设计完成，但未执行
- **功能**: 使用 Face + Pose 验证 ASR 语句的置信度
### 4. 人物命名/识别
- **当前**: 只有机器生成的 Person_0, Person_1...
- **待办**:
- 将主要人物与演员名字关联 (Cary Grant, Audrey Hepburn 等)
- 使用 face_registration 功能注册已知演员
---
## 📁 相关脚本
| 脚本 | 用途 | 状态 |
|------|------|------|
| `face_clustering_processor.py` | 人脸聚类 | ✅ 已执行 |
| `fast_face_clustering_processor.py` | 快速人脸聚类 | 备选 |
| `sync_face_speaker_to_chunks.py` | 同步到数据库 | 待执行 |
| `match_speakers_to_chunks.py` | 匹配说话者 | 待执行 |
| `export_person_thumbnails.py` | 导出人物缩略图 | 可用 |
| `face_registration.py` | 人脸注册 | 可用 |
| `register_sample_faces.py` | 注册样本 | 可用 |
---
## 🔧 建议下一步
1. **检查 ASRX 输出** 是否有 speaker diarization 信息
2. **导出 Top 20 人物缩略图** 供人工识别
3. **关联主要演员名字** 到 Person_0, Person_17, Person_4 等
4. **执行 Face ↔ ASR 验证** 提升语音识别置信度
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_QUICK_START.md
-421
View File
@@ -1,421 +0,0 @@
# Face / Speaker / Person API 簡易指南
> **版本**: 1.1 | **適用**: 前端開發團隊
> **更新日期**: 2026-04-17
>
> **⚠️ 重要**: 3002 (正式版) 和 3003 (開發版) 使用**完全獨立的資料空間** (public vs dev schema)，絕非共用。開發版測試不會影響正式版資料。
---
## 快速開始
```bash
export BASE="http://localhost:3002"
export KEY="muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69"
export UUID="384b0ff44aaaa1f14cb2cd63b3fea966"
```
---
## 1. 用 uuid + chunk_id 查看 face / speaker / person
### 取得 chunk 內的人物
```bash
curl "$BASE/api/v1/chunks/sentence_0093/persons" \
-H "X-API-Key: $KEY"
```
```json
{
"success": true,
"chunk_id": "sentence_0093",
"persons": [
{
"person_id": "Person_0",
"name": "Person_0",
"confidence": 0.85,
"overlap_duration": 3.2
}
]
}
```
### 取得 chunk 的 speaker（從 content 欄位）
```bash
curl -X POST "$BASE/api/v1/search/universal" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"query": "", "uuid": "'$UUID'", "types": ["chunk"], "filters": {"speaker_id": "SPEAKER_0"}, "limit": 10}'
```
```json
{
"results": [
{
"type": "chunk",
"chunk_id": "sentence_0093",
"chunk_type": "sentence",
"start_frame": 29795,
"end_frame": 29963,
"fps": 59.94,
"start_time": 497.08,
"end_time": 499.88,
"text": "You could have the stamps.",
"speaker_id": "SPEAKER_0"
}
]
}
```
### 統一搜尋 chunk + face + person
```bash
curl -X POST "$BASE/api/v1/search/universal" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"query": "stamp", "uuid": "'$UUID'", "types": ["chunk", "person"], "limit": 10}'
```
```json
{
"query": "stamp",
"results": [
{
"type": "chunk",
"chunk_id": "sentence_1566",
"chunk_type": "sentence",
"start_frame": 329980,
"end_frame": 330040,
"fps": 59.94,
"start_time": 5506.84,
"end_time": 5507.84,
"text": "The envelope, but the stamps on it",
"speaker_id": "SPEAKER_0"
},
{
"type": "person",
"person_id": "Person_0",
"name": "Person_0",
"speaker_id": "SPEAKER_0",
"appearance_count": 17832
}
],
"total": 10,
"took_ms": 27
}
```
---
## 2. 選擇 face 並綁定 person
### 步驟 1: 列出所有人物
```bash
curl "$BASE/api/v1/person/list?min_appearances=100&has_speaker=true&limit=20" \
-H "X-API-Key: $KEY"
```
```json
{
"persons": [
{
"person_id": "Person_0",
"name": "Person_0",
"speaker_id": "SPEAKER_0",
"appearance_count": 17832
},
{
"person_id": "Person_17",
"name": "Person_17",
"speaker_id": "SPEAKER_1",
"appearance_count": 1636
}
],
"total": 9
}
```
### 步驟 2: 查看人物詳情 + 取得截圖
```bash
# 查看詳情
curl "$BASE/api/v1/person/Person_0" -H "X-API-Key: $KEY"
# 取得臉部截圖
curl "$BASE/api/v1/person/Person_0/thumbnail?file_uuid=$UUID" \
-H "X-API-Key: $KEY" -o person0_face.jpg
# 取得第 5 次出現的臉部截圖
curl "$BASE/api/v1/person/Person_0/thumbnail?file_uuid=$UUID&index=4" \
-H "X-API-Key: $KEY" -o person0_face_5.jpg
```
### 步驟 3: 綁定名稱（將 face 關聯到 person）
```bash
curl -X PATCH "$BASE/api/v1/person/Person_0" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"name": "Cary Grant", "is_confirmed": true}'
```
```json
{
"success": true,
"message": "Person 'Cary Grant' updated successfully",
"person_id": "Person_0"
}
```
### 步驟 4: 註冊新臉孔（建立參考樣本）
```bash
curl -X POST "$BASE/api/v1/face/register" \
-H "X-API-Key: $KEY" \
-F "image=@known_face.jpg" \
-F "name=Cary Grant" \
-F 'metadata={"imdb_id": "nm0000001"}'
```
---
## 3. 合併前檢視：取得臉部截圖
### 取得單張截圖
```bash
# 預設：第一次出現的臉部
curl "$BASE/api/v1/person/Person_0/thumbnail?file_uuid=$UUID" \
-H "X-API-Key: $KEY" -o face.jpg
# 指定第 N 次出現
curl "$BASE/api/v1/person/Person_0/thumbnail?file_uuid=$UUID&index=10" \
-H "X-API-Key: $KEY" -o face_10.jpg
```
### 找出相似人物（可能為同一人）
```bash
curl "$BASE/api/v1/person/Person_0/similar?threshold=0.5&limit=10" \
-H "X-API-Key: $KEY"
```
```json
{
"person_id": "Person_0",
"similar_persons": [
{
"person_id": "Person_4",
"name": "Person_4",
"speaker_id": "SPEAKER_0",
"similarity": 0.7
},
{
"person_id": "Person_25",
"name": "Person_25",
"speaker_id": "SPEAKER_0",
"similarity": 0.7
}
]
}
```
### 取得 AI 合併建議
```bash
curl -X POST "$BASE/api/v1/person/suggest" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"file_uuid": "'$UUID'"}'
```
```json
{
"merge_suggestions": [
{
"person_id": "Person_0",
"merge_with": ["Person_4", "Person_25"],
"confidence": 0.65,
"reasons": [
"All share speaker_id: SPEAKER_0",
"Primary Person_0 has 17832 appearances (89% of group)"
],
"action": "needs_review"
}
]
}
```
---
## 統一搜尋
### ⚠️ 重要：搜尋 chunks 時 uuid 為必填
**只有 `uuid + chunk_id` 組合才是唯一識別碼。** 單獨 `chunk_id` 在不同影片中可能重複。
```bash
# ✅ 正確：包含 uuid
curl -X POST "$BASE/api/v1/search/universal" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"query": "stamp", "uuid": "'$UUID'", "types": ["chunk"]}'
# ❌ 錯誤：缺少 uuid
curl -X POST "$BASE/api/v1/search/universal" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"query": "stamp", "types": ["chunk"]}'
# 回傳: {"error": "uuid is required for chunk search"}
```
---
## 4. 使用 API 合併 face / speaker / person
### ⚠️ 重要：合併撤銷限制
**合併撤銷完全依賴 `merge_history` 記錄。**
| 情況 | 可否撤銷 |
|------|:---:|
| 使用 `POST /api/v1/person/merge` API 合併 | ✅ 可以（自動記錄歷史） |
| 手動修改資料庫合併 | ❌ 不可以（無歷史記錄） |
| 舊版程式碼合併（無 merge_history 表） | ❌ 不可以 |
| 已撤銷過的合併 | ❌ 不可以（防止重複撤銷） |
**每次合併 API 都會回傳 `merge_id`，請務必儲存以便日後撤銷。**
### 執行合併
```bash
curl -X POST "$BASE/api/v1/person/merge" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"target_person_id": "Person_0",
"source_person_ids": ["Person_4", "Person_25"]
}'
```
```json
{
"success": true,
"message": "Merged 2 persons into Person_0",
"target_person_id": "Person_0",
"merge_id": "5b12e3ac-12fa-45c0-88e1-5cff67604a7d"
}
```
### 合併做了什麼？
```
合併前:
Person_0 (17832 幀, SPEAKER_0)
Person_4 (936 幀, SPEAKER_0)
Person_25 (217 幀, SPEAKER_0)
合併後:
Person_0 (17832+936+217=18985 幀, SPEAKER_0) ← 保留
Person_4 ← 刪除
Person_25 ← 刪除
```
### 撤銷合併
```bash
# 使用合併時回傳的 merge_id
curl -X POST "$BASE/api/v1/person/merge/undo" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"merge_id": "5b12e3ac-12fa-45c0-88e1-5cff67604a7d"}'
```
```json
{
"success": true,
"message": "Undo merge completed. Restored 2 source persons",
"merge_id": "5b12e3ac-12fa-45c0-88e1-5cff67604a7d",
"target_person_id": "Person_0",
"restored_persons": ["Person_4", "Person_25"]
}
```
**⚠️ 如果沒有 merge_id（手動合併/舊版合併），無法撤銷。**
### 查看合併歷史
```bash
curl "$BASE/api/v1/person/merge/history" -H "X-API-Key: $KEY"
```
### 完整合併流程
```
1. 取得建議 → POST /api/v1/person/suggest
2. 檢視截圖 → GET /api/v1/person/:id/thumbnail
3. 檢視相似 → GET /api/v1/person/:id/similar
4. 執行合併 → POST /api/v1/person/merge ← 儲存 merge_id!
5. 確認結果 → GET /api/v1/person/list
6. 如需撤銷 → POST /api/v1/person/merge/undo ← 需要 merge_id
```
---
## API 速查表
| 用途 | 方法 | 端點 |
|------|:---:|------|
| **查看 chunk 內人物** | GET | `/api/v1/chunks/:chunk_id/persons` |
| **搜尋人物** | GET | `/api/v1/search/persons?query=Person` |
| **列出人物** | GET | `/api/v1/person/list?limit=20` |
| **人物詳情** | GET | `/api/v1/person/:id` |
| **人物截圖** | GET | `/api/v1/person/:id/thumbnail?file_uuid=...` |
| **相似人物** | GET | `/api/v1/person/:id/similar` |
| **AI 建議** | POST | `/api/v1/person/suggest` |
| **綁定名稱** | PATCH | `/api/v1/person/:id` |
| **合併人物** | POST | `/api/v1/person/merge` |
| **撤銷合併** | POST | `/api/v1/person/merge/undo` |
| **合併歷史** | GET | `/api/v1/person/merge/history` |
| **統一搜尋** | POST | `/api/v1/search/universal` |
| **註冊臉孔** | POST | `/api/v1/face/register` |
---
## 錯誤處理
```bash
# 錯誤回應
curl -X POST "$BASE/api/v1/person/merge" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"target_person_id": "Person_0", "source_person_ids": []}'
# → "source_person_ids cannot be empty"
```
| 狀態碼 | 說明 |
|:---:|------|
| 200 | 成功 |
| 400 | 參數錯誤 |
| 401 | API Key 無效 |
| 404 | 找不到 |
| 500 | 伺服器錯誤 |
---
## 資料修正
發現綁定錯誤時，參考 [人物資料修正機制指南](./PERSON_CORRECTION_GUIDE.md)
| 錯誤類型 | 修正方式 |
|---------|---------|
| Speaker 綁錯 | `POST /person/:id/reassign-speaker` |
| 不該綁 Speaker | `POST /person/:id/unbind-speaker` |
| Appearance 分錯人 | `POST /person/:id/reassign-appearance` |
| 錯誤 Appearance | `POST /person/:id/remove-appearance` |
| 兩人被合併為一 | `POST /person/:id/split` |
| 錯誤合併 | `POST /person/merge/undo` |
| 錯誤命名 | `PATCH /person/:id` |
docs_v1.0/AI_AGENTS/IDENTITY/FACE_SPEAKER_PERSON_WORKFLOW.md
-372
View File
@@ -1,372 +0,0 @@
# Face to Identity Workflow Guide
> Version: V4.0 | Date: 2026-04-28
> Architecture: Two-layer (Face → Identity)
> Related: [FACE_TO_IDENTITY_FLOW.md](./FACE_TO_IDENTITY_FLOW.md)
---
## Overview
V4.0 架構實現 Face → Identity 直接綁定，移除 person_id 中間層，簡化工作流程。
### Key Changes (V3.x → V4.0)
| Change | V3.x | V4.0 |
|--------|------|------|
| **Architecture** | Three-layer (Face → Person → Identity) | Two-layer (Face → Identity) |
| **Person ID** | Video-local person_id | ❌ Removed |
| **Registration** | POST /identities/from-person | POST /identities/register |
| **Merge** | POST /person/merge | POST /agents/suggest/merge |
| **Candidates** | GET /person/list | GET /faces/candidates |
| **file_uuid** | Used everywhere | **file_uuid** |
---
## Workflow Visualization
```mermaid
graph TD
%% Nodes
Start((Start Analysis))
ListCandidates[List Face Candidates]
subgraph "Phase 1: Registration"
CheckIdentity{Identity Exists?}
Register[Register Identity]
Bind[Bind Faces]
end
subgraph "Phase 2: AI Analysis"
Suggest[Get AI Suggestions]
Review[Review Suggestions]
Merge[Execute Merge]
Confirm[Confirm Result]
end
End((Database Clean))
%% Flow
Start --> ListCandidates
ListCandidates --> CheckIdentity
CheckIdentity -- No --> Register
Register --> Bind
Bind --> Suggest
CheckIdentity -- Yes --> Bind
Bind --> Suggest
Suggest --> Review
Review -- Merge Recommended --> Merge
Review -- Bind Recommended --> Bind
Merge --> Confirm
Confirm --> End
style Start fill:#f9f,stroke:#333
style End fill:#bbf,stroke:#333
style Register fill:#dfd,stroke:#333
style Bind fill:#dfd,stroke:#333
```
---
## Phase 1: Registration
**Scenario**: You found unregistered faces and want to create a new identity.
### Step 1: List Face Candidates
```bash
curl -s "http://localhost:3003/api/v1/faces/candidates?min_confidence=0.8&pose_angle=frontal&limit=5" \
-H "X-API-Key: YOUR_KEY"
```
**Response**:
```json
{
"success": true,
"data": {
"candidates": [
{
"face_id": "face_100",
"file_uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
"frame": 100,
"timestamp": 5.2,
"pose_angle": "frontal",
"confidence": 0.92,
"trace_id": 2
}
],
"statistics": {
"total_candidates": 78,
"avg_confidence": 0.85
}
}
}
```
### Step 2: Register Identity
```bash
curl -X POST "http://localhost:3003/api/v1/identities/register" \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"face_ids": ["face_100", "face_150", "face_200"],
"name": "Audrey Hepburn",
"source": "manual",
"auto_bind_chunks": true
}'
```
**Response**:
```json
{
"success": true,
"data": {
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"name": "Audrey Hepburn",
"faces_bound": 3,
"chunks_bound": 10,
"speaker_ids": ["SPEAKER_0"],
"reference_vectors": {
"total": 3,
"angles": ["frontal"]
}
}
}
```
---
## Phase 2: AI Analysis
**Scenario**: You want AI to suggest potential merges or additional bindings.
### Step 1: Get AI Suggestions
```bash
curl -X POST "http://localhost:3003/api/v1/agents/suggest/clustering" \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"min_confidence": 0.8,
"pose_angles": ["frontal"],
"max_suggestions": 5
}'
```
**Response**:
```json
{
"success": true,
"data": {
"suggestions": [
{
"suggestion_id": "suggest_1",
"cluster_type": "high_confidence",
"confidence": 0.92,
"recommended_faces": [
{
"face_id": "face_100",
"pose_angle": "frontal",
"confidence": 0.95,
"is_primary": true
}
],
"cluster_stats": {
"total_faces": 50,
"avg_similarity": 0.89
},
"reason": "High confidence frontal faces from same trace",
"action": "register"
},
{
"suggestion_id": "suggest_2",
"cluster_type": "existing_identity",
"confidence": 0.88,
"identity_uuid": "a9a90105...",
"recommended_faces": [
{
"face_id": "face_300",
"confidence": 0.87
}
],
"reason": "Similar to Audrey Hepburn (0.88)",
"action": "bind"
}
]
}
}
```
### Step 2: Review & Execute
**Option A: Bind to Existing Identity**
```bash
curl -X POST "http://localhost:3003/api/v1/identities/a9a90105.../bind" \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"face_ids": ["face_300", "face_400"],
"auto_bind_chunks": true
}'
```
**Option B: Register New Identity**
```bash
curl -X POST "http://localhost:3003/api/v1/identities/register" \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"face_ids": ["face_500", "face_550"],
"name": "Cary Grant",
"source": "manual"
}'
```
### Step 3: Merge Identities
**Scenario**: Two identities are the same person.
```bash
curl -X POST "http://localhost:3003/api/v1/agents/suggest/merge" \
-H "X-API-Key: YOUR_KEY" \
-H "Content-Type: application/json" \
-d '{
"identity_uuids": ["a9a90105...", "b8b80206..."],
"threshold": 0.85
}'
```
**Response**:
```json
{
"success": true,
"data": {
"suggestions": [
{
"suggestion_type": "merge",
"confidence": 0.88,
"identities": [
{"identity_uuid": "a9a90105...", "name": "Person A", "face_count": 500},
{"identity_uuid": "b8b80206...", "name": "Person B", "face_count": 300}
],
"reason": "High embedding similarity (0.88)",
"recommended_action": {
"merge_target": "a9a90105...",
"merge_sources": ["b8b80206..."]
}
}
]
}
}
```
---
## Query Operations
### List Identities in a File
```bash
curl "http://localhost:3003/api/v1/files/384b0ff44aaaa1f14cb2cd63b3fea966/identities" \
-H "X-API-Key: YOUR_KEY"
```
### List Files for an Identity
```bash
curl "http://localhost:3003/api/v1/identities/a9a90105.../files" \
-H "X-API-Key: YOUR_KEY"
```
### List Faces for an Identity
```bash
curl "http://localhost:3003/api/v1/identities/a9a90105.../faces?limit=100" \
-H "X-API-Key: YOUR_KEY"
```
### List Chunks for an Identity
```bash
curl "http://localhost:3003/api/v1/identities/a9a90105.../chunks" \
-H "X-API-Key: YOUR_KEY"
```
---
## Demo Script
```bash
#!/bin/bash
# scripts/demo_identity_workflow_v4.sh
API_URL="http://localhost:3003"
API_KEY="YOUR_API_KEY"
echo "=== MOMENTRY IDENTITY WORKFLOW V4.0 ==="
# 1. List candidates
echo "STEP 1: Listing unregistered faces..."
curl -s "$API_URL/api/v1/faces/candidates?min_confidence=0.8&limit=5" \
-H "X-API-Key: $API_KEY" \
| python3 -m json.tool
# 2. Register identity
echo ""
echo "STEP 2: Registering Audrey Hepburn..."
curl -s -X POST "$API_URL/api/v1/identities/register" \
-H "X-API-Key: $API_KEY" \
-H "Content-Type: application/json" \
-d '{"face_ids": ["face_100"], "name": "Audrey Hepburn", "source": "manual"}' \
| python3 -m json.tool
# 3. Get AI suggestions
echo ""
echo "STEP 3: Getting AI suggestions..."
curl -s -X POST "$API_URL/api/v1/agents/suggest/clustering" \
-H "X-API-Key: $API_KEY" \
-H "Content-Type: application/json" \
-d '{"min_confidence": 0.8, "max_suggestions": 3}' \
| python3 -m json.tool
# 4. Bind faces to identity
echo ""
echo "STEP 4: Binding additional faces..."
curl -s -X POST "$API_URL/api/v1/identities/a9a90105.../bind" \
-H "X-API-Key: $API_KEY" \
-H "Content-Type: application/json" \
-d '{"face_ids": ["face_200"]}' \
| python3 -m json.tool
echo ""
echo "Demo Complete."
```
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| V4.0 | 2026-04-28 | Two-layer architecture, 15 endpoints |
| V3.x | 2026-04-10 | Three-layer architecture, 33 endpoints |
---
## Related Documents
- [IDENTITY_MANAGEMENT_API.md](./IDENTITY_MANAGEMENT_API.md): API design
- [FACE_TO_IDENTITY_FLOW.md](./FACE_TO_IDENTITY_FLOW.md): Binding flow
- [FILE_IDENTITIES_TABLE_SPEC.md](./FILE_IDENTITIES_TABLE_SPEC.md): Table schema
- [IDENTITY_API_SPEC.md](../IDENTITY_API_SPEC.md): Complete API spec
docs_v1.0/AI_AGENTS/IDENTITY/FACE_TO_IDENTITY_FLOW.md
-768
View File
@@ -1,768 +0,0 @@
# Face to Identity Binding Flow
> Version: V4.0 | Date: 2026-04-28
> Architecture: Two-layer (Face → Identity)
> Related: [FILE_IDENTITIES_TABLE_SPEC.md](./FILE_IDENTITIES_TABLE_SPEC.md)
---
## Overview
V4.0 架構實現 Face → Identity 直接綁定，移除 person_id 中間層。
### Key Principles
| Principle | Description |
|-----------|-------------|
| **Direct Binding** | Face 直接綁定到 Identity，無中間層 |
| **One-to-Many Reference** | Identity 擁有多個 Reference Vectors |
| **N:N File-Identity** | Identity 可跨多個 File |
| **Auto Chunk Binding** | Chunk 通過時間對齊自動綁定 |
---
## Data Model
```
┌─────────────────┐
│ face_detections│
├─────────────────┤
│ id │
│ file_uuid ─────┼───┐
│ frame │ │
│ timestamp │ │
│ trace_id │ │
│ pose_angle │ │
│ confidence │ │
│ embedding (512) │ │
│ identity_id ────┼───┼──┐
└─────────────────┘ │ │
│ │
┌─────────────────┐ │ │
│ files │ │ │
├─────────────────┤ │ │
│ uuid ◄──────────┼───┘ │
│ file_name │ │
│ duration │ │
└─────────────────┘ │
│
┌─────────────────┐ │
│ identities │ │
├─────────────────┤ │
│ id ◄────────────┼──────┘
│ uuid │
│ name │
│ source │
│ face_embedding │ (reference vector)
│ reference_data │ (JSONB, multiple vectors)
└─────────────────┘
│
│ N:N
▼
┌─────────────────┐
│ file_identities │
├─────────────────┤
│ file_uuid │
│ identity_id │
│ face_count │
│ speaker_count │
│ confidence │
└─────────────────┘
```
---
## Binding Workflows
### 1. Manual Registration (New Identity)
**Trigger**: User selects face(s) and assigns name
```
User Selection
│
▼
┌─────────────────────────┐
│ POST /identities/register │
├─────────────────────────┤
│ face_ids: ["face_100"] │
│ name: "Audrey Hepburn" │
│ source: "manual" │
│ auto_bind_chunks: true │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 1. Create Identity │
│ - identity_uuid │
│ - name, source │
│ - face_embedding │ (from first face)
│ - reference_data │ (selected vectors)
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 2. Bind Faces │
│ - Update face_detections │
│ - Set identity_id │
│ - Update file_identities │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 3. Auto Bind Chunks │
│ - Time alignment │
│ - Update chunk.metadata │
│ - Update file_identities.speaker_count │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 4. Select Reference Vectors │
│ - Trace-based selection │
│ - Pose diversity │
│ - Quality threshold │
└─────────────────────────┘
```
**Implementation**:
```rust
pub async fn register_identity(
db: &PgPool,
req: RegisterIdentityRequest,
) -> Result<Identity> {
let mut tx = db.begin().await?;
// 1. Get faces
let faces = sqlx::query_as!(
FaceDetection,
"SELECT * FROM face_detections WHERE id = ANY($1)",
&req.face_ids
)
.fetch_all(&mut *tx)
.await?;
// 2. Create identity
let identity = sqlx::query_as!(
Identity,
r#"
INSERT INTO identities (uuid, name, source, face_embedding, reference_data)
VALUES ($1, $2, $3, $4, $5)
RETURNING *
"#,
Uuid::new_v4().to_string(),
req.name,
req.source,
faces[0].embedding.clone(),
json!({
"vectors": vec![ReferenceVector {
embedding: faces[0].embedding.clone(),
pose_angle: faces[0].pose_angle.clone(),
quality: faces[0].confidence,
file_uuid: faces[0].file_uuid.clone(),
face_id: faces[0].id,
}],
"selection_strategy": "manual"
}),
)
.fetch_one(&mut *tx)
.await?;
// 3. Bind faces
for face in &faces {
sqlx::query!(
"UPDATE face_detections SET identity_id = $1 WHERE id = $2",
identity.id,
face.id
)
.execute(&mut *tx)
.await?;
// Update file_identities
update_file_identity_stats(
&mut tx,
&face.file_uuid,
identity.id,
1, // face_count +1
0, // speaker_count
Some(face.confidence),
Some(face.timestamp),
).await?;
}
// 4. Auto bind chunks
if req.auto_bind_chunks {
auto_bind_chunks_for_identity(&mut tx, &identity.id, &faces).await?;
}
tx.commit().await?;
Ok(identity)
}
```
---
### 2. Bind Faces to Existing Identity
**Trigger**: User selects face(s) and assigns to existing identity
```
User Selection
│
▼
┌────────────────────────────┐
│ POST /identities/:uuid/bind │
├────────────────────────────┤
│ face_ids: ["face_200"] │
│ auto_bind_chunks: true │
└────────────────────────────┘
│
▼
┌─────────────────────────┐
│ 1. Validate Identity │
│ - Check existence │
│ - Get reference_data │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 2. Bind Faces │
│ - Update face_detections │
│ - Set identity_id │
│ - Update file_identities │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 3. Update Reference Vectors │
│ - Add new vector if quality > threshold │
│ - Maintain diversity │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 4. Auto Bind Chunks │
│ - Time alignment │
└─────────────────────────┘
```
**Implementation**:
```rust
pub async fn bind_faces_to_identity(
db: &PgPool,
identity_uuid: &str,
req: BindFacesRequest,
) -> Result<()> {
let mut tx = db.begin().await?;
// 1. Get identity
let identity = sqlx::query_as!(
Identity,
"SELECT * FROM identities WHERE uuid = $1",
identity_uuid
)
.fetch_one(&mut *tx)
.await?;
// 2. Get faces
let faces = sqlx::query_as!(
FaceDetection,
"SELECT * FROM face_detections WHERE id = ANY($1)",
&req.face_ids
)
.fetch_all(&mut *tx)
.await?;
// 3. Bind faces
for face in &faces {
sqlx::query!(
"UPDATE face_detections SET identity_id = $1 WHERE id = $2",
identity.id,
face.id
)
.execute(&mut *tx)
.await?;
update_file_identity_stats(
&mut tx,
&face.file_uuid,
identity.id,
1,
0,
Some(face.confidence),
Some(face.timestamp),
).await?;
}
// 4. Update reference vectors
update_reference_vectors(&mut tx, &identity.id, &faces).await?;
// 5. Auto bind chunks
if req.auto_bind_chunks {
auto_bind_chunks_for_identity(&mut tx, &identity.id, &faces).await?;
}
tx.commit().await?;
Ok(())
}
```
---
### 3. Unbind Faces from Identity
**Trigger**: User removes face from identity
```
User Selection
│
▼
┌──────────────────────────────┐
│ POST /identities/:uuid/unbind │
├──────────────────────────────┤
│ face_ids: ["face_400"] │
└──────────────────────────────┘
│
▼
┌─────────────────────────┐
│ 1. Unbind Faces │
│ - Set identity_id = NULL │
│ - Update file_identities │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 2. Auto Unbind Chunks │
│ - Remove if no overlapping faces │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 3. Update Reference Vectors │
│ - Remove if vector source │
│ - Re-select if needed │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ 4. Check Identity Deletion │
│ - If face_count = 0, delete identity │
└─────────────────────────┘
```
---
### 4. Auto Chunk Binding
**Trigger**: Face binding/unbinding
**Principle**: Chunk 自動綁定，無需 Candidates/Suggest API
```
Face Timestamps
│
▼
┌─────────────────────────┐
│ Query Chunks by Time │
│ - chunk.start_time <= face.timestamp │
│ - chunk.end_time >= face.timestamp │
│ - Same file_uuid │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ Check Overlap │
│ - Count overlapping faces │
│ - Calculate confidence │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ Update Chunk Metadata │
│ - identity_id: ... │
│ - confidence: 0.85 │
│ - binding_source: "auto"│
│ - faces: ["face_100"] │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ Update file_identities │
│ - speaker_count += 1 │
└─────────────────────────┘
```
**Implementation**:
```rust
pub async fn auto_bind_chunks_for_identity(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
identity_id: &i64,
faces: &[FaceDetection],
) -> Result<()> {
for face in faces {
// Find overlapping chunks
let chunks = sqlx::query!(
r#"
SELECT id, metadata
FROM chunks
WHERE file_uuid = $1
AND start_time <= $2
AND end_time >= $2
"#,
face.file_uuid,
face.timestamp
)
.fetch_all(&mut **tx)
.await?;
for chunk in chunks {
let mut metadata: ChunkMetadata =
serde_json::from_value(chunk.metadata.clone()).unwrap_or_default();
// Update metadata
if !metadata.faces.contains(&face.id) {
metadata.faces.push(face.id);
}
metadata.identity_id = Some(*identity_id);
metadata.confidence = Some(face.confidence);
metadata.binding_source = "auto".to_string();
sqlx::query!(
r#"
UPDATE chunks
SET metadata = $1
WHERE id = $2
"#,
serde_json::to_value(metadata)?,
chunk.id
)
.execute(&mut **tx)
.await?;
// Update file_identities speaker_count
sqlx::query!(
r#"
UPDATE file_identities
SET speaker_count = speaker_count + 1
WHERE file_uuid = $1 AND identity_id = $2
"#,
face.file_uuid,
identity_id
)
.execute(&mut **tx)
.await?;
}
}
Ok(())
}
```
---
### 5. Reference Vector Selection
**Strategy**: Trace-based + Pose diversity
```
Face Detections (identity_id = X)
│
▼
┌─────────────────────────┐
│ Group by trace_id │
│ - Each trace = one person track │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ For each trace: │
│ - Find best frontal face │
│ - Find best profile faces │
│ - Quality > 0.85 │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ Select Top N Vectors │
│ - Max 5 per trace │
│ - Max 20 total │
│ - Prioritize quality │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ Store in reference_data │
│ {
│ "vectors": [...],
│ "selection_strategy": "trace_based",
│ "total_traces": 4,
│ "total_faces": 500
│ }
└─────────────────────────┘
```
**Implementation**:
```rust
pub async fn update_reference_vectors(
tx: &mut sqlx::Transaction<'_, sqlx::Postgres>,
identity_id: &i64,
new_faces: &[FaceDetection],
) -> Result<()> {
// Get all faces for this identity
let all_faces = sqlx::query_as!(
FaceDetection,
"SELECT * FROM face_detections WHERE identity_id = $1",
identity_id
)
.fetch_all(&mut **tx)
.await?;
// Group by trace_id
let mut trace_groups: HashMap<i32, Vec<&FaceDetection>> = HashMap::new();
for face in &all_faces {
trace_groups.entry(face.trace_id).or_default().push(face);
}
// Select vectors per trace
let mut selected_vectors = Vec::new();
for (_trace_id, faces) in trace_groups.iter() {
// Group by pose_angle
let mut pose_groups: HashMap<String, Vec<&FaceDetection>> = HashMap::new();
for face in faces {
pose_groups
.entry(face.pose_angle.clone())
.or_default()
.push(face);
}
// Select best from each pose (max 5 per trace)
for (_, pose_faces) in pose_groups.iter() {
let best = pose_faces
.iter()
.filter(|f| f.confidence > 0.85)
.max_by(|a, b| a.confidence.partial_cmp(&b.confidence).unwrap());
if let Some(face) = best {
selected_vectors.push(ReferenceVector {
embedding: face.embedding.clone(),
pose_angle: face.pose_angle.clone(),
quality: face.confidence,
file_uuid: face.file_uuid.clone(),
face_id: face.id,
});
}
}
}
// Sort by quality and take top 20
selected_vectors.sort_by(|a, b| b.quality.partial_cmp(&a.quality).unwrap());
selected_vectors.truncate(20);
// Update identity
sqlx::query!(
r#"
UPDATE identities
SET reference_data = $1
WHERE id = $2
"#,
json!({
"vectors": selected_vectors,
"selection_strategy": "trace_based",
"total_traces": trace_groups.len(),
"total_faces": all_faces.len(),
}),
identity_id
)
.execute(&mut **tx)
.await?;
Ok(())
}
```
---
## Query Workflows
### 1. List Identities in File
```bash
GET /api/v1/files/384b0ff44aaaa1f14cb2cd63b3fea966/identities
```
**SQL**:
```sql
SELECT
i.uuid AS identity_uuid,
i.name,
i.source,
fi.face_count,
fi.speaker_count,
fi.confidence
FROM file_identities fi
JOIN identities i ON i.id = fi.identity_id
WHERE fi.file_uuid = '384b0ff44aaaa1f14cb2cd63b3fea966'
ORDER BY fi.face_count DESC;
```
---
### 2. List Files for Identity
```bash
GET /api/v1/identities/a9a90105.../files
```
**SQL**:
```sql
SELECT
f.uuid AS file_uuid,
f.file_name,
f.duration,
fi.face_count,
fi.speaker_count,
fi.first_appearance,
fi.last_appearance,
fi.confidence
FROM file_identities fi
JOIN files f ON f.uuid = fi.file_uuid
WHERE fi.identity_id = 1
ORDER BY fi.face_count DESC;
```
---
### 3. List Faces for Identity
```bash
GET /api/v1/identities/a9a90105.../faces?limit=100
```
**SQL**:
```sql
SELECT
fd.id AS face_id,
fd.file_uuid,
fd.frame,
fd.timestamp,
fd.pose_angle,
fd.confidence,
fd.trace_id
FROM face_detections fd
WHERE fd.identity_id = 1
ORDER BY fd.timestamp
LIMIT 100;
```
---
### 4. List Unregistered Faces (Candidates)
```bash
GET /api/v1/faces/candidates?min_confidence=0.8&pose_angle=frontal
```
**SQL**:
```sql
SELECT
fd.id AS face_id,
fd.file_uuid,
fd.frame,
fd.timestamp,
fd.pose_angle,
fd.confidence,
fd.trace_id
FROM face_detections fd
WHERE fd.identity_id IS NULL
AND fd.confidence >= 0.8
AND fd.pose_angle = 'frontal'
ORDER BY fd.confidence DESC
LIMIT 100;
```
---
## Performance Considerations
### Indexing Strategy
```sql
-- Face queries
CREATE INDEX idx_face_detections_identity ON face_detections(identity_id)
WHERE identity_id IS NOT NULL;
CREATE INDEX idx_face_detections_candidates ON face_detections(confidence DESC)
WHERE identity_id IS NULL;
-- File identity queries
CREATE INDEX idx_file_identities_file_uuid ON file_identities(file_uuid);
CREATE INDEX idx_file_identities_identity_id ON file_identities(identity_id);
-- Chunk queries
CREATE INDEX idx_chunks_file_time ON chunks(file_uuid, start_time, end_time);
```
### Batch Operations
```rust
// Batch bind faces (recommended for >10 faces)
pub async fn batch_bind_faces(
db: &PgPool,
identity_id: i64,
face_ids: &[i64],
) -> Result<()> {
let mut tx = db.begin().await?;
// Single UPDATE statement
sqlx::query!(
"UPDATE face_detections SET identity_id = $1 WHERE id = ANY($2)",
identity_id,
face_ids
)
.execute(&mut *tx)
.await?;
// Batch update file_identities
// ... (use CTE or temp table)
tx.commit().await?;
Ok(())
}
```
---
## Error Handling
### Common Errors
| Error | Cause | Solution |
|-------|-------|----------|
| `Identity not found` | Invalid identity_uuid | Check UUID format |
| `Face already bound` | Face has identity_id | Unbind first |
| `Invalid face_ids` | Empty array or invalid IDs | Validate input |
| `Chunk overlap conflict` | Multiple identities in same chunk | Use latest binding |
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| V4.0 | 2026-04-28 | Two-layer architecture, direct binding |
---
## Related Documents
- [IDENTITY_MANAGEMENT_API.md](./IDENTITY_MANAGEMENT_API.md): API design
- [FILE_IDENTITIES_TABLE_SPEC.md](./FILE_IDENTITIES_TABLE_SPEC.md): Table schema
- [IDENTITY_AGENT_SPEC.md](./IDENTITY_AGENT_SPEC.md): Agent specification
docs_v1.0/AI_AGENTS/IDENTITY/FILE_IDENTITIES_TABLE_SPEC.md
-434
View File
@@ -1,434 +0,0 @@
# File Identities Table Specification
> Version: V4.0 | Date: 2026-04-28
> Architecture: Two-layer (Face → Identity)
> Relationship: N:N (Identity ↔ File)
---
## Overview
`file_identities` 表實現 Identity 與 File 的多對多關係，支援跨檔案身份追蹤。
### Key Features
| Feature | Description |
|---------|-------------|
| **N:N Relationship** | Identity 可跨多個 File，File 可包含多個 Identity |
| **Aggregate Stats** | 統計每個 File 中每個 Identity 的出現次數 |
| **Time Range** | 記錄首次/最後出現時間 |
| **Confidence** | 平均信心度 |
---
## Table Schema
```sql
CREATE TABLE file_identities (
id BIGSERIAL PRIMARY KEY,
file_uuid VARCHAR(64) NOT NULL,
identity_id BIGINT NOT NULL,
face_count INTEGER DEFAULT 0,
speaker_count INTEGER DEFAULT 0,
first_appearance DOUBLE PRECISION,
last_appearance DOUBLE PRECISION,
confidence DOUBLE PRECISION DEFAULT 0.0,
created_at TIMESTAMPTZ DEFAULT NOW(),
updated_at TIMESTAMPTZ DEFAULT NOW(),
CONSTRAINT fk_file_identities_file
FOREIGN KEY (file_uuid)
REFERENCES files(uuid)
ON DELETE CASCADE,
CONSTRAINT fk_file_identities_identity
FOREIGN KEY (identity_id)
REFERENCES identities(id)
ON DELETE CASCADE,
CONSTRAINT uq_file_identities
UNIQUE (file_uuid, identity_id)
);
CREATE INDEX idx_file_identities_file_uuid ON file_identities(file_uuid);
CREATE INDEX idx_file_identities_identity_id ON file_identities(identity_id);
CREATE INDEX idx_file_identities_confidence ON file_identities(confidence DESC);
```
---
## Column Descriptions
| Column | Type | Description | Example |
|--------|------|-------------|---------|
| `id` | BIGSERIAL | Primary key | `1` |
| `file_uuid` | VARCHAR(64) | File identifier (FK to files.uuid) | `384b0ff44aaaa1f14cb2cd63b3fea966` |
| `identity_id` | BIGINT | Identity ID (FK to identities.id) | `1` |
| `face_count` | INTEGER | Number of faces bound to identity in this file | `500` |
| `speaker_count` | INTEGER | Number of speaker segments bound | `10` |
| `first_appearance` | DOUBLE PRECISION | First appearance time in seconds | `5.2` |
| `last_appearance` | DOUBLE PRECISION | Last appearance time in seconds | `180.5` |
| `confidence` | DOUBLE PRECISION | Average confidence score | `0.86` |
| `created_at` | TIMESTAMPTZ | Record creation time | `2026-04-28T10:00:00Z` |
| `updated_at` | TIMESTAMPTZ | Record update time | `2026-04-28T12:00:00Z` |
---
## Relationships
### Identity → Files (One-to-Many)
```
identities (1) ──→ file_identities (N) ──→ files (N)
```
**Query**: List all files where an identity appears
```sql
SELECT
f.uuid AS file_uuid,
f.file_name,
fi.face_count,
fi.speaker_count,
fi.first_appearance,
fi.last_appearance,
fi.confidence
FROM file_identities fi
JOIN files f ON f.uuid = fi.file_uuid
WHERE fi.identity_id = ?
ORDER BY fi.face_count DESC;
```
### File → Identities (One-to-Many)
```
files (1) ──→ file_identities (N) ──→ identities (N)
```
**Query**: List all identities in a file
```sql
SELECT
i.uuid AS identity_uuid,
i.name,
i.source,
fi.face_count,
fi.speaker_count,
fi.confidence
FROM file_identities fi
JOIN identities i ON i.id = fi.identity_id
WHERE fi.file_uuid = ?
ORDER BY fi.face_count DESC;
```
---
## Data Flow
### 1. Face Binding
When a face is bound to an identity:
```sql
-- Step 1: Create file_identities record if not exists
INSERT INTO file_identities (file_uuid, identity_id, face_count, confidence)
VALUES (?, ?, 1, ?)
ON CONFLICT (file_uuid, identity_id)
DO UPDATE SET
face_count = file_identities.face_count + 1,
confidence = (file_identities.confidence * file_identities.face_count + EXCLUDED.confidence) / (file_identities.face_count + 1),
updated_at = NOW();
-- Step 2: Update first/last appearance
UPDATE file_identities
SET
first_appearance = LEAST(first_appearance, ?),
last_appearance = GREATEST(last_appearance, ?)
WHERE file_uuid = ? AND identity_id = ?;
```
### 2. Face Unbinding
When a face is unbound from an identity:
```sql
-- Step 1: Get face info before unbinding
SELECT file_uuid, confidence FROM face_detections WHERE id = ?;
-- Step 2: Update file_identities
UPDATE file_identities
SET
face_count = face_count - 1,
updated_at = NOW()
WHERE file_uuid = ? AND identity_id = ?;
-- Step 3: Delete if face_count = 0
DELETE FROM file_identities
WHERE file_uuid = ? AND identity_id = ? AND face_count = 0;
```
### 3. Chunk Binding (Auto)
When a chunk is auto-bound to an identity via time alignment:
```sql
-- Update speaker_count
UPDATE file_identities
SET
speaker_count = speaker_count + 1,
updated_at = NOW()
WHERE file_uuid = ? AND identity_id = ?;
```
---
## Indexes
| Index | Purpose |
|-------|---------|
| `idx_file_identities_file_uuid` | Query identities by file |
| `idx_file_identities_identity_id` | Query files by identity |
| `idx_file_identities_confidence` | Sort by confidence |
---
## Constraints
### Foreign Keys
| Constraint | On Delete | Description |
|------------|-----------|-------------|
| `fk_file_identities_file` | CASCADE | Delete file_identities when file is deleted |
| `fk_file_identities_identity` | CASCADE | Delete file_identities when identity is deleted |
### Unique Constraint
```sql
CONSTRAINT uq_file_identities UNIQUE (file_uuid, identity_id)
```
Ensures one record per file-identity pair.
---
## Query Patterns
### 1. Get Identity Files
```rust
pub async fn get_identity_files(
db: &PgPool,
identity_uuid: &str,
page: i64,
page_size: i64,
) -> Result<IdentityFilesResponse> {
let rows = sqlx::query_as!(
FileIdentityRow,
r#"
SELECT
f.uuid AS file_uuid,
f.file_name,
f.duration,
fi.face_count,
fi.speaker_count,
fi.first_appearance,
fi.last_appearance,
fi.confidence
FROM file_identities fi
JOIN files f ON f.uuid = fi.file_uuid
JOIN identities i ON i.id = fi.identity_id
WHERE i.uuid = $1
ORDER BY fi.face_count DESC
LIMIT $2 OFFSET $3
"#,
identity_uuid,
page_size,
(page - 1) * page_size
)
.fetch_all(db)
.await?;
Ok(IdentityFilesResponse { files: rows })
}
```
### 2. Get File Identities
```rust
pub async fn get_file_identities(
db: &PgPool,
file_uuid: &str,
page: i64,
page_size: i64,
) -> Result<FileIdentitiesResponse> {
let rows = sqlx::query_as!(
IdentityRow,
r#"
SELECT
i.uuid AS identity_uuid,
i.name,
i.source,
fi.face_count,
fi.speaker_count,
fi.confidence
FROM file_identities fi
JOIN identities i ON i.id = fi.identity_id
WHERE fi.file_uuid = $1
ORDER BY fi.face_count DESC
LIMIT $2 OFFSET $3
"#,
file_uuid,
page_size,
(page - 1) * page_size
)
.fetch_all(db)
.await?;
Ok(FileIdentitiesResponse { identities: rows })
}
```
### 3. Update Stats
```rust
pub async fn update_file_identity_stats(
db: &PgPool,
file_uuid: &str,
identity_id: i64,
face_count_delta: i32,
speaker_count_delta: i32,
confidence: Option<f64>,
timestamp: Option<f64>,
) -> Result<()> {
sqlx::query!(
r#"
INSERT INTO file_identities (file_uuid, identity_id, face_count, speaker_count, confidence, first_appearance, last_appearance)
VALUES ($1, $2, $3, $4, $5, $6, $6)
ON CONFLICT (file_uuid, identity_id)
DO UPDATE SET
face_count = file_identities.face_count + $3,
speaker_count = file_identities.speaker_count + $4,
confidence = CASE
WHEN $5 IS NOT NULL AND file_identities.face_count > 0
THEN (file_identities.confidence * file_identities.face_count + $5) / (file_identities.face_count + $3)
ELSE file_identities.confidence
END,
first_appearance = CASE
WHEN $6 IS NOT NULL
THEN LEAST(file_identities.first_appearance, $6)
ELSE file_identities.first_appearance
END,
last_appearance = CASE
WHEN $6 IS NOT NULL
THEN GREATEST(file_identities.last_appearance, $6)
ELSE file_identities.last_appearance
END,
updated_at = NOW()
"#,
file_uuid,
identity_id,
face_count_delta,
speaker_count_delta,
confidence,
timestamp
)
.execute(db)
.await?;
Ok(())
}
```
---
## Migration
### V3.x → V4.0
**Before (V3.x)**:
- `person_identities` table (303 records, 0 registered identities)
- One-to-many relationship (person → identities)
- Video-local person IDs
**After (V4.0)**:
- `file_identities` table (new)
- Many-to-many relationship (identity ↔ file)
- Global identity UUIDs
- Direct face → identity binding
### Migration Script
```sql
-- Step 1: Create file_identities table
CREATE TABLE file_identities ( ... );
-- Step 2: Populate from face_detections
INSERT INTO file_identities (file_uuid, identity_id, face_count, confidence, first_appearance, last_appearance)
SELECT
fd.file_uuid,
fd.identity_id,
COUNT(*) AS face_count,
AVG(fd.confidence) AS confidence,
MIN(fd.timestamp) AS first_appearance,
MAX(fd.timestamp) AS last_appearance
FROM face_detections fd
WHERE fd.identity_id IS NOT NULL
GROUP BY fd.file_uuid, fd.identity_id;
-- Step 3: Update speaker_count from chunks
UPDATE file_identities fi
SET speaker_count = (
SELECT COUNT(DISTINCT c.id)
FROM chunks c
WHERE c.file_uuid = fi.file_uuid
AND c.metadata->>'identity_id' = fi.identity_id::text
);
-- Step 4: Drop person_identities table
DROP TABLE IF EXISTS person_identities;
```
---
## Performance Considerations
### Index Strategy
| Query Pattern | Index |
|---------------|-------|
| Get identities by file | `idx_file_identities_file_uuid` |
| Get files by identity | `idx_file_identities_identity_id` |
| Sort by confidence | `idx_file_identities_confidence` |
### Query Optimization
1. **Use JOINs sparingly**: Fetch identity/file data separately when possible
2. **Pagination**: Always use `LIMIT` and `OFFSET`
3. **Batch updates**: Use transactions for bulk face binding
### Caching Strategy
```rust
// Redis cache key patterns
const CACHE_KEY_FILE_IDENTITIES: &str = "momentry:file_identities:{}";
const CACHE_KEY_IDENTITY_FILES: &str = "momentry:identity_files:{}";
// Cache TTL (5 minutes)
const CACHE_TTL: i64 = 300;
```
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| V4.0 | 2026-04-28 | Initial design (N:N relationship) |
---
## Related Documents
- [IDENTITY_MANAGEMENT_API.md](./IDENTITY_MANAGEMENT_API.md): Identity API design
- [IDENTITY_AGENT_SPEC.md](./IDENTITY_AGENT_SPEC.md): Identity Agent specification
- [FACE_TO_IDENTITY_FLOW.md](./FACE_TO_IDENTITY_FLOW.md): Face binding workflow
docs_v1.0/AI_AGENTS/IDENTITY/IDENTITY_AGENT_SPEC.md
-549
View File
@@ -1,549 +0,0 @@
---
document_type: "architecture_design"
service: "MOMENTRY_CORE"
title: "Identity Agent Design Specification"
date: "2026-04-28"
version: "V2.0"
status: "active"
owner: "Warren"
created_by: "OpenCode"
tags:
- "identity-agent"
- "agent"
- "face-clustering"
- "embedding-matching"
- "multi-file-aggregation"
ai_query_hints:
- "Identity Agent design specification"
- "Face to Identity inference flow"
- "Multi-file identity aggregation"
- "Embedding matching with pose adaptation"
related_documents:
- "AI_AGENTS/CORE/AGENT_SPEC.md"
- "AI_AGENTS/IDENTITY/IDENTITY_MANAGEMENT_API.md"
- "FILE_IDENTITIES_TABLE_SPEC.md"
---
# Identity Agent Design Specification
| Item | Content |
|------|---------|
| Creator | OpenCode |
| Date | 2026-04-28 |
| Version | V2.0 (Two-layer Architecture) |
---
## Version History
| Version | Date | Changes | Author |
|---------|------|---------|--------|
| V2.0 | 2026-04-28 | Two-layer architecture (Face → Identity) | OpenCode |
| V1.0 | 2026-04-27 | Initial design (three-layer) | OpenCode |
---
## Overview
Identity Agent is an L3 Agent in Momentry Core, responsible for inferring "Who is Who" from Face Processor outputs and aggregating identities across multiple files.
---
## Architecture Change (V1.0 → V2.0)
| Aspect | V1.0 (Deprecated) | V2.0 (Current) |
|--------|-------------------|----------------|
| **Layers** | Face → Person → Identity | Face → Identity (2 layers) |
| **person_identities** | Required table | Removed (deprecated) |
| **Binding** | Person → Identity | Face → Identity (direct) |
| **Chunks** | Person → Chunk | Face → Chunk (auto-bind by time) |
---
## Current Status
| Component | Status |
|-----------|--------|
| Face Processor | ✅ Implemented (InsightFace) |
| Face Tracker | ✅ Implemented (trace_id) |
| ASRX Processor | ✅ Implemented (WhisperX) |
| Identity Agent | 🔧 Pending implementation |
---
## 1. Agent Goals
### 1.1 Core Problem
**Question**: How to infer global Identity from Face embeddings across multiple files?
**Challenges**:
1. **Same person in different files**: Need cross-file matching
2. **Different poses**: frontal vs profile have different thresholds
3. **Temporal alignment**: Chunks need time-based binding
4. **Quality variance**: Low-quality faces need filtering
---
### 1.2 Agent Goals
Aggregate evidence across files to create/maintain global Identities:
| Evidence Source | Input | Output |
|-----------------|-------|--------|
| **Face Processor** | Face embedding + pose_angle | Face → identity_id |
| **Face Tracker** | trace_id (face tracking) | Trace statistics |
| **ASRX Processor** | Speaker segments | Chunk → identity_id (auto-bind) |
| **Identity Agent** | Face + trace + time | **Identity** (global) |
---
## 2. Data Flow (Two-layer)
```
File → InsightFace → face_full_traced.json
↓
face_id + embedding + pose_angle + trace_id
↓
Identity Agent
↓
┌─────────────────────────────────────┐
│ Step 1: Select unregistered face │
│ Step 2: Register identity │
│ Step 3: Embedding matching │
│ Step 4: Bind faces → identity_id │
│ Step 5: Auto-bind chunks │
└─────────────────────────────────────┘
↓
identities + file_identities tables
```
---
## 3. Input Data
### 3.1 Face Data Structure
```json
{
"file_uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
"fps": 59.94,
"metadata": {
"trace_stats": {
"total_traces": 4,
"long_traces": 3
}
},
"frames": {
"100": {
"faces": [
{
"face_id": "face_100",
"confidence": 0.92,
"embedding": [512-dim vector],
"pose_angle": {
"angle": "frontal",
"yaw": -5.2,
"pitch": 2.1,
"confidence": 0.95
},
"trace_id": 2,
"identity_id": null
}
]
}
},
"traces": {
"2": {
"trace_id": 2,
"total_appearances": 143,
"avg_confidence": 0.86,
"pose_distribution": {
"frontal": 20,
"profile_right": 125
}
}
}
}
```
---
### 3.2 Data Sources
| Data | Source File | Description |
|------|--------------|-------------|
| **Face frames** | `{uuid}.face_full_traced_v2.json` | Face detection + embedding + trace |
| **Speaker segments** | `{uuid}.asrx.json` | Speaker time segments |
| **Chunks** | `chunks` table | Sentence chunks (from pre_chunks) |
---
## 4. Core Logic
### 4.1 Inference Flow
```
┌─────────────────────────────────────────────────────────────────┐
│ Identity Agent Workflow │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Step 1: Candidates Query │
│ ───────────────────────────── │
│ Query: GET /api/v1/faces/candidates │
│ Filter: identity_id = NULL, confidence >= 0.8 │
│ Result: Unregistered faces list │
│ │
│ Step 2: AI Suggestion │
│ ───────────────── │
│ Query: POST /api/v1/agents/suggest/clustering │
│ Input: Unregistered faces │
│ Output: Cluster suggestions + recommended primary face │
│ │
│ Step 3: Identity Registration │
│ ───────────────────────────── │
│ Query: POST /api/v1/identities/register │
│ Input: face_ids + name │
│ Output: identity_uuid │
│ │
│ Step 4: Face Binding │
│ ───────────────── │
│ For each face in same trace: │
│ Calculate: embedding_similarity(face, identity.embedding) │
│ Apply: adaptive_threshold(pose_angle) │
│ If similarity > threshold: │
│ UPDATE face_detections SET identity_id = identity.id │
│ │
│ Step 5: Chunk Auto-Binding │
│ ───────────────────────────── │
│ For each face with identity_id: │
│ Query: chunks WHERE time overlaps face timestamp │
│ Update: chunk.metadata.identity_id = identity.uuid │
│ Update: chunk.metadata.chunk_identity.faces.push(face_id) │
│ │
│ Step 6: Statistics Aggregation │
│ ─────────────────────────────── │
│ Update: file_identities (face_count, speaker_count) │
│ Update: identities.metadata (global stats) │
│ │
└─────────────────────────────────────────────────────────────────┘
```
---
### 4.2 Adaptive Threshold
**Pose-based threshold strategy**:
```python
def get_adaptive_threshold(pose_angle: str) -> float:
"""Get matching threshold based on pose angle"""
thresholds = {
"frontal": 0.90, # Strict for frontal
"three_quarter": 0.85, # Moderate
"profile_left": 0.80, # Relaxed for profile
"profile_right": 0.80,
}
return thresholds.get(pose_angle, 0.75)
```
**Reasoning**:
- Frontal faces have best embedding quality → strict threshold
- Profile faces have distorted embedding → relaxed threshold
- Three_quarter is intermediate
---
### 4.3 Embedding Matching
```python
def match_face_to_identity(
face_embedding: List[float],
identity_embedding: List[float],
pose_angle: str
) -> Tuple[bool, float]:
"""Match face to identity with pose-adaptive threshold"""
similarity = cosine_similarity(face_embedding, identity_embedding)
threshold = get_adaptive_threshold(pose_angle)
is_match = similarity > threshold
return is_match, similarity
```
---
### 4.4 Chunk Auto-Binding
```python
def bind_chunks_to_identity(
identity_id: int,
file_uuid: str,
pool: PgPool
) -> int:
"""Auto-bind chunks by time alignment"""
# Get face time ranges
faces = sqlx::query(
"SELECT timestamp, pose_angle
FROM face_detections
WHERE identity_id = $1 AND file_uuid = $2"
).bind(identity_id).bind(file_uuid).fetch_all(pool)
# Find overlapping chunks
chunks_updated = 0
for face in faces:
chunks = sqlx::query(
"UPDATE chunks
SET metadata = jsonb_set(
metadata, '{chunk_identity}',
jsonb_build_object(
'identity_id', $1::text,
'binding_source', 'auto'
)
)
WHERE file_uuid = $2
AND ABS(start_time - $3) < 2.0"
).bind(identity_id).bind(file_uuid).bind(face.timestamp)
.execute(pool)
chunks_updated += chunks.rowcount()
return chunks_updated
```
---
## 5. Database Schema
### 5.1 identities Table
| Field | Type | Description |
|-------|------|-------------|
| `uuid` | UUID | identity_uuid (global) |
| `name` | VARCHAR | Identity name |
| `face_embedding` | VECTOR(512) | Reference embedding |
| `reference_data` | JSONB | Multi-angle reference vectors |
| `metadata` | JSONB | Global statistics |
---
### 5.2 file_identities Table (N:N)
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | UUID | File UUID |
| `identity_id` | BIGINT | Identity ID |
| `face_count` | INT | Faces in this file |
| `speaker_count` | INT | Speaker segments |
| `first_appearance` | FLOAT | First appearance time |
| `last_appearance` | FLOAT | Last appearance time |
| `confidence` | FLOAT | Avg confidence |
---
### 5.3 face_detections Table
| Field | Type | Description |
|-------|------|-------------|
| `identity_id` | BIGINT | Bound identity (direct) |
| `file_uuid` | UUID | File UUID |
| `pose_angle` | VARCHAR | Pose angle |
| `embedding` | VECTOR(512) | Face embedding |
| `trace_id` | INT | Trace ID (from Face Tracker) |
---
### 5.4 chunks.metadata Structure
```json
{
"chunk_identity": {
"faces": [100, 150],
"speakers": ["SPEAKER_0"],
"identity_id": "a9a90105-...",
"confidence": 0.88,
"binding_source": "auto"
}
}
```
---
## 6. API Design
### 6.1 Candidates API
```http
GET /api/v1/faces/candidates
?min_confidence=0.8
&pose_angle=frontal
&page=1
&page_size=15
&limit=100
```
**Response**:
```json
{
"candidates": [
{
"face_id": "face_100",
"pose_angle": "frontal",
"confidence": 0.92,
"trace_id": 2
}
]
}
```
---
### 6.2 Suggest API
```http
POST /api/v1/agents/suggest/clustering
{
"min_confidence": 0.8,
"max_suggestions": 5
}
```
**Response**:
```json
{
"suggestions": [
{
"cluster_type": "high_confidence",
"recommended_faces": ["face_100"],
"action": "register"
}
]
}
```
---
### 6.3 Register API
```http
POST /api/v1/identities/register
{
"face_ids": ["face_100"],
"name": "Person A",
"auto_bind_chunks": true
}
```
---
## 7. Multi-File Aggregation
### 7.1 Cross-File Matching
When a new file is processed:
1. **Query existing identities**: `SELECT * FROM identities`
2. **For each unregistered face**:
- Calculate similarity with all identity.face_embedding
- Apply adaptive threshold
- If match: bind to existing identity
3. **If no match**: create new identity
---
### 7.2 Statistics Update
```sql
-- Update file_identities after binding
INSERT INTO file_identities (
file_uuid, identity_id, face_count, confidence
)
SELECT
file_uuid,
identity_id,
COUNT(*),
AVG(confidence)
FROM face_detections
WHERE identity_id IS NOT NULL
GROUP BY file_uuid, identity_id
ON CONFLICT (file_uuid, identity_id)
DO UPDATE SET
face_count = EXCLUDED.face_count,
confidence = EXCLUDED.confidence;
```
---
## 8. Implementation Plan
### 8.1 Phase 1: Core Matching
| Task | Status |
|------|--------|
| Adaptive threshold function | Pending |
| Embedding matching logic | Pending |
| Face → Identity binding | Pending |
| Chunk auto-binding | Pending |
---
### 8.2 Phase 2: Candidates API
| Task | Status |
|------|--------|
| Candidates query endpoint | Pending |
| Pose distribution statistics | Pending |
| Trace-based filtering | Pending |
---
### 8.3 Phase 3: Suggest API
| Task | Status |
|------|--------|
| Clustering suggestion logic | Pending |
| Primary face recommendation | Pending |
| Merge suggestion | Pending |
---
### 8.4 Phase 4: Statistics
| Task | Status |
|------|--------|
| file_identities aggregation | Pending |
| identities.metadata update | Pending |
| Cross-file identity stats | Pending |
---
## 9. Key Decisions
| Decision | Reason |
|----------|--------|
| **Remove person_identities** | Middle layer adds complexity, unused (303 records, 0 registered) |
| **Face → Identity direct** | Simpler, embedding comparison is sufficient |
| **Adaptive threshold** | Pose affects embedding quality |
| **Chunk auto-bind** | Chunks follow faces by time alignment |
| **file_identities table** | Needed for N:N relationship tracking |
---
## 10. Metrics
| Metric | Target |
|--------|--------|
| **Matching accuracy** | > 90% for frontal |
| **False positive rate** | < 5% |
| **Processing speed** | 1000 faces/second |
| **Cross-file recall** | > 85% |
---
## Version Information
- Version: V2.0
- Architecture: Two-layer (Face → Identity)
- Date: 2026-04-28
- Status: Specification complete, implementation pending
docs_v1.0/AI_AGENTS/IDENTITY/IDENTITY_MANAGEMENT_API.md
-434
View File
@@ -1,434 +0,0 @@
# Momentry Identity Management API Guide
> Version: 4.0 | Updated: 2026-04-28
> Architecture: Two-layer (Face → Identity)
> Terminology: file_uuid, identity_uuid
---
## Overview
This guide demonstrates the complete workflow for:
- Choosing a video file
- Analyzing faces (unregistered candidates)
- Registering global identities
- Managing identity ↔ file relationships
---
## Terminology
| Term | Scope | Example |
|------|-------|---------|
| **file_uuid** | Video file identifier | `384b0ff44aaaa1f14cb2cd63b3fea966` |
| **identity_uuid** | Global identity identifier | `a9a90105-6d6b-...` |
| **face_id** | Single face detection | `face_100` |
| **trace_id** | Face tracking ID | `2` |
**Note**: `person_id` (video-local identifier) is deprecated. Use direct Face → Identity binding.
---
## 1. List Files
**Endpoint**: `GET /api/v1/files`
```bash
curl -s "http://127.0.0.1:3003/api/v1/files" \
-H "X-API-Key: YOUR_API_KEY" | jq .
```
**Response**:
```json
{
"success": true,
"data": {
"files": [
{
"file_uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
"file_name": "Charade_1963.mp4",
"duration": 6879.33,
"status": "completed"
}
]
}
}
```
---
## 2. List Unregistered Faces (Candidates)
**Endpoint**: `GET /api/v1/faces/candidates`
Query faces that have not been bound to any identity.
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `file_uuid` | UUID | No | - | Filter by file |
| `min_confidence` | float | No | 0.5 | Minimum confidence |
| `pose_angle` | string | No | - | Filter by pose (frontal/profile) |
| `page` | int | No | 1 | Page number |
| `page_size` | int | No | 15 | Items per page |
| `limit` | int | No | 100 | Total limit |
```bash
curl -s "http://127.0.0.1:3003/api/v1/faces/candidates?min_confidence=0.8" \
-H "X-API-Key: YOUR_API_KEY" | jq .
```
**Response**:
```json
{
"success": true,
"data": {
"candidates": [
{
"face_id": "face_100",
"file_uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
"frame": 100,
"timestamp": 5.2,
"pose_angle": "frontal",
"confidence": 0.92,
"trace_id": 2,
"embedding_quality": 0.88
}
],
"statistics": {
"total_candidates": 78,
"pose_distribution": {
"frontal": 20,
"profile_right": 30,
"three_quarter": 18
}
},
"pagination": {
"page": 1,
"page_size": 15,
"total": 78,
"total_pages": 6
}
}
}
```
---
## 3. AI Suggest Clustering
**Endpoint**: `POST /api/v1/agents/suggest/clustering`
AI Agent analyzes unregistered faces and suggests clustering.
```bash
curl -s -X POST "http://127.0.0.1:3003/api/v1/agents/suggest/clustering" \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_API_KEY" \
-d '{
"min_confidence": 0.8,
"pose_angles": ["frontal"],
"max_suggestions": 5
}' | jq .
```
**Response**:
```json
{
"success": true,
"data": {
"suggestions": [
{
"suggestion_id": "suggest_1",
"cluster_type": "high_confidence",
"confidence": 0.92,
"recommended_faces": [
{
"face_id": "face_100",
"pose_angle": "frontal",
"confidence": 0.95,
"is_primary": true
},
{
"face_id": "face_150",
"pose_angle": "frontal",
"confidence": 0.91
}
],
"cluster_stats": {
"total_faces": 50,
"avg_similarity": 0.89,
"trace_ids": [2, 3]
},
"reason": "High confidence frontal faces from same trace",
"action": "register"
}
]
}
}
```
---
## 4. Register Identity from Faces
**Endpoint**: `POST /api/v1/identities/register`
Register a new global identity from face candidates.
```bash
curl -s -X POST "http://127.0.0.1:3003/api/v1/identities/register" \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_API_KEY" \
-d '{
"face_ids": ["face_100", "face_150", "face_200"],
"name": "Audrey Hepburn",
"source": "manual",
"auto_bind_chunks": true
}' | jq .
```
**Response**:
```json
{
"success": true,
"data": {
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"name": "Audrey Hepburn",
"faces_bound": 3,
"chunks_bound": 10,
"speaker_ids": ["SPEAKER_0"],
"reference_vectors": {
"total": 3,
"angles": ["frontal", "three_quarter"]
}
}
}
```
---
## 5. Query Identity → Files
**Endpoint**: `GET /api/v1/identities/:identity_uuid/files`
List all files where this identity appears.
```bash
curl -s "http://127.0.0.1:3003/api/v1/identities/a9a90105.../files" \
-H "X-API-Key: YOUR_API_KEY" | jq .
```
**Response**:
```json
{
"success": true,
"data": {
"identity_uuid": "a9a90105...",
"name": "Audrey Hepburn",
"files": [
{
"file_uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
"file_name": "Charade_1963.mp4",
"face_count": 500,
"speaker_count": 10,
"first_appearance": 5.2,
"last_appearance": 180.5,
"confidence": 0.86
},
{
"file_uuid": "9760d0820f0cf9a7",
"file_name": "Breakfast_at_Tiffanys.mp4",
"face_count": 300,
"speaker_count": 5
}
],
"total_files": 2
}
}
```
---
## 6. Query File → Identities
**Endpoint**: `GET /api/v1/files/:file_uuid/identities`
List all identities appearing in a file.
```bash
curl -s "http://127.0.0.1:3003/api/v1/files/384b0ff44aaaa1f14cb2cd63b3fea966/identities" \
-H "X-API-Key: YOUR_API_KEY" | jq .
```
**Response**:
```json
{
"success": true,
"data": {
"file_uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
"file_name": "Charade_1963.mp4",
"identities": [
{
"identity_uuid": "a9a90105...",
"name": "Audrey Hepburn",
"face_count": 500,
"speaker_count": 10,
"confidence": 0.86
},
{
"identity_uuid": "b8b80206...",
"name": "Cary Grant",
"face_count": 450,
"speaker_count": 8
}
],
"total_identities": 2
}
}
```
---
## 7. Get Identity Detail
**Endpoint**: `GET /api/v1/identities/:identity_uuid`
```bash
curl -s "http://127.0.0.1:3003/api/v1/identities/a9a90105..." \
-H "X-API-Key: YOUR_API_KEY" | jq .
```
**Response**:
```json
{
"success": true,
"data": {
"identity_uuid": "a9a90105...",
"name": "Audrey Hepburn",
"source": "manual",
"identity_type": "person",
"global_stats": {
"total_files": 3,
"total_faces": 1500,
"total_speaker_segments": 30
},
"reference_vectors": {
"total": 4,
"angles": ["frontal", "profile_right", "three_quarter"],
"quality_avg": 0.875
}
}
}
```
---
## 8. Bind Additional Faces to Identity
**Endpoint**: `POST /api/v1/identities/:identity_uuid/bind`
Add more faces to an existing identity.
```bash
curl -s -X POST "http://127.0.0.1:3003/api/v1/identities/a9a90105.../bind" \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_API_KEY" \
-d '{
"face_ids": ["face_300", "face_400"],
"auto_bind_chunks": true
}' | jq .
```
**Response**:
```json
{
"success": true,
"data": {
"identity_uuid": "a9a90105...",
"faces_bound": 2,
"chunks_bound": 5,
"updated_stats": {
"total_faces": 1502,
"total_files": 3
}
}
}
```
---
## 9. Unbind Faces from Identity
**Endpoint**: `POST /api/v1/identities/:identity_uuid/unbind`
```bash
curl -s -X POST "http://127.0.0.1:3003/api/v1/identities/a9a90105.../unbind" \
-H "Content-Type: application/json" \
-H "X-API-Key: YOUR_API_KEY" \
-d '{
"face_ids": ["face_400"]
}' | jq .
```
---
## 10. Get Identity Thumbnail
**Endpoint**: `GET /api/v1/identities/:identity_uuid/thumbnail`
```bash
curl -s -o identity_thumbnail.jpg \
"http://127.0.0.1:3003/api/v1/identities/a9a90105.../thumbnail" \
-H "X-API-Key: YOUR_API_KEY"
```
---
## Complete Workflow Example
```
Step 1: List files → Choose Charade_1963.mp4
Step 2: List face candidates → Find high-confidence frontal faces
Step 3: AI suggest clustering → Get clustering recommendations
Step 4: Register identity → Create "Audrey Hepburn" with 3 faces
Step 5: Auto-bind chunks → 10 sentence chunks bound automatically
Step 6: Verify → Query identity → files (appears in 3 files)
```
---
## API Endpoints Summary
| Category | Endpoint | Description |
|----------|----------|-------------|
| **List** | `GET /api/v1/files` | List files |
| **List** | `GET /api/v1/identities` | List identities |
| **Candidates** | `GET /api/v1/faces/candidates` | Unregistered faces |
| **Suggest** | `POST /api/v1/agents/suggest/clustering` | AI clustering suggestions |
| **Register** | `POST /api/v1/identities/register` | Register new identity |
| **Bind** | `POST /api/v1/identities/:uuid/bind` | Bind faces to identity |
| **Detail** | `GET /api/v1/identities/:uuid` | Identity detail |
| **Relation** | `GET /api/v1/identities/:uuid/files` | Identity → Files (N:N) |
| **Relation** | `GET /api/v1/files/:uuid/identities` | File → Identities (N:N) |
---
## Changes from V3.x
| Change | V3.x | V4.0 |
|--------|------|------|
| **Architecture** | Face → Person → Identity | Face → Identity (2-layer) |
| **file_uuid** | file_uuid | file_uuid |
| **person_id** | 28 person API endpoints | Removed (deprecated) |
| **file_identities** | Not mentioned | Added (N:N relationship table) |
| **chunk candidates** | chunk candidates API | Removed (chunks auto-bind) |
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| V4.0 | 2026-04-28 | Two-layer architecture, file_uuid terminology |
| V3.5 | 2026-04-17 | Person-based workflow |
| V3.0 | 2026-04-10 | Initial identity management |
docs_v1.0/AI_AGENTS/IDENTITY/PHASE1_MIGRATION_PLAN.md
-282
View File
@@ -1,282 +0,0 @@
# Phase 1 Migration Plan: file_uuid → file_uuid
> Version: V4.0 | Date: 2026-04-28
> Status: Planning
---
## Overview
将所有 `file_uuid` 重命名为 `file_uuid`，统一术语定义。
### Impact Summary
| Category | Count | Priority |
|----------|-------|----------|
| **Migration SQL** | 6 files | High |
| **Rust API** | ~20 files | High |
| **Portal Vue** | 3 files | Medium |
| **Documents** | 121 refs | Low |
---
## Phase 1.1: Database Migration
### Tables Affected
| Table | Column | New Name |
|-------|--------|----------|
| `face_detections` | `file_uuid` | `file_uuid` |
| `face_clusters` | `file_uuid` | `file_uuid` |
| `person_identities` | `file_uuid` | `file_uuid` |
| `person_appearances` | `file_uuid` | `file_uuid` |
| `chunks` | `file_uuid` | `file_uuid` |
| `files` | - | (already has `uuid`) |
### Indexes Affected
| Old Index | New Index |
|-----------|-----------|
| `idx_face_detections_file_uuid` | `idx_face_detections_file_uuid` |
| `idx_face_clusters_file_uuid` | `idx_face_clusters_file_uuid` |
| `idx_person_identities_file_uuid` | `idx_person_identities_file_uuid` |
### Migration Script
```sql
-- Migration: 011_rename_file_uuid_to_file_uuid.sql
-- Date: 2026-04-28
BEGIN;
-- 1. face_detections
ALTER TABLE face_detections
RENAME COLUMN file_uuid TO file_uuid;
DROP INDEX IF EXISTS idx_face_detections_file_uuid;
CREATE INDEX idx_face_detections_file_uuid ON face_detections(file_uuid);
DROP INDEX IF EXISTS idx_face_detections_frame;
CREATE INDEX idx_face_detections_frame ON face_detections(file_uuid, frame_number);
-- 2. face_clusters
ALTER TABLE face_clusters
RENAME COLUMN file_uuid TO file_uuid;
DROP INDEX IF EXISTS idx_face_clusters_file_uuid;
CREATE INDEX idx_face_clusters_file_uuid ON face_clusters(file_uuid);
-- 3. person_identities (will be removed in Phase 2, but rename for consistency)
ALTER TABLE person_identities
RENAME COLUMN file_uuid TO file_uuid;
DROP INDEX IF EXISTS idx_person_identities_file_uuid;
CREATE INDEX idx_person_identities_file_uuid ON person_identities(file_uuid);
-- 4. person_appearances
ALTER TABLE person_appearances
RENAME COLUMN file_uuid TO file_uuid;
DROP INDEX IF EXISTS idx_person_appearances_file_uuid;
CREATE INDEX idx_person_appearances_file_uuid ON person_appearances(file_uuid);
DROP INDEX IF EXISTS idx_person_appearances_time;
CREATE INDEX idx_person_appearances_time ON person_appearances(file_uuid, start_time, end_time);
-- 5. chunks (if exists)
ALTER TABLE chunks
RENAME COLUMN file_uuid TO file_uuid;
-- 6. Update constraint names
ALTER TABLE face_detections
DROP CONSTRAINT IF EXISTS unique_detection_per_frame,
ADD CONSTRAINT unique_detection_per_frame UNIQUE (file_uuid, frame_number, x, y, width, height);
ALTER TABLE face_clusters
DROP CONSTRAINT IF EXISTS face_recognition_results_file_uuid_key,
ADD CONSTRAINT face_clusters_file_uuid_key UNIQUE (file_uuid);
ALTER TABLE person_identities
DROP CONSTRAINT IF EXISTS unique_person_identity,
ADD CONSTRAINT unique_person_identity UNIQUE (file_uuid, face_identity_id, speaker_id);
COMMIT;
```
---
## Phase 1.2: Rust API Migration
### Files Affected
| File | Changes |
|------|---------|
| `src/api/face_recognition.rs` | Rename struct fields |
| `src/api/videos.rs` | Rename endpoints |
| `src/api/identities.rs` | Update query params |
| `src/api/person_identity.rs` | (will be removed in Phase 2) |
| `src/core/db/*.rs` | Rename column bindings |
### Migration Steps
1. Rename struct fields:
```rust
// Before
pub struct FaceResult {
pub file_uuid: String,
}
// After
pub struct FaceResult {
pub file_uuid: String,
}
```
1. Rename route parameters:
```rust
// Before
"/api/v1/face/results/:file_uuid"
// After
"/api/v1/face/results/:file_uuid"
```
1. Update SQLx bindings:
```rust
// Before
sqlx::query!("WHERE file_uuid = $1", file_uuid)
// After
sqlx::query!("WHERE file_uuid = $1", file_uuid)
```
---
## Phase 1.3: Portal Migration
### Files Affected
| File | Changes |
|------|---------|
| `portal/src/views/IdentitiesView.vue` | Rename field references |
| `portal/src/views/PersonsView.vue` | Rename field references |
| `portal/src/views/IdentityDetailView.vue` | Rename field references |
| `portal/src-tauri/src/api/*.rs` | Rename struct fields |
### Migration Steps
1. Rename TypeScript interfaces:
```typescript
// Before
interface Identity {
file_uuid: string;
}
// After
interface Identity {
file_uuid: string;
}
```
1. Update Vue templates:
```vue
<!-- Before -->
<div>影片: {{ identity.file_uuid }}</div>
<!-- After -->
<div>影片: {{ identity.file_uuid }}</div>
```
---
## Phase 1.4: Document Migration
### Files Affected
- `docs_v1.0/**/*.md` (121 refs)
- `AGENTS.md` (already updated)
### Migration Steps
```bash
# Batch replacement (MacOS/Linux)
find docs_v1.0 -name "*.md" -type f \
-exec sed -i '' 's/file_uuid/file_uuid/g' {} \;
# Verify changes
grep -r "file_uuid" docs_v1.0/*.md | wc -l
```
---
## Execution Order
| Step | Description | Est. Time |
|------|-------------|-----------|
| 1 | Create DB migration script | 5 min |
| 2 | Run DB migration (dev schema) | 2 min |
| 3 | Update Rust API | 30 min |
| 4 | Update Portal | 20 min |
| 5 | Run tests | 10 min |
| 6 | Batch update docs | 5 min |
| **Total** | | **~1 hour** |
---
## Rollback Plan
```sql
-- Rollback migration
BEGIN;
ALTER TABLE face_detections RENAME COLUMN file_uuid TO file_uuid;
ALTER TABLE face_clusters RENAME COLUMN file_uuid TO file_uuid;
ALTER TABLE person_identities RENAME COLUMN file_uuid TO file_uuid;
ALTER TABLE person_appearances RENAME COLUMN file_uuid TO file_uuid;
ALTER TABLE chunks RENAME COLUMN file_uuid TO file_uuid;
-- Restore indexes
DROP INDEX idx_face_detections_file_uuid;
CREATE INDEX idx_face_detections_file_uuid ON face_detections(file_uuid);
-- ... (repeat for other tables)
COMMIT;
```
---
## Test Commands
```bash
# After migration, verify API still works
cargo run --bin momentry_playground -- server
# Test endpoints
curl "http://localhost:3003/api/v1/files/384b0ff44aaaa1f14cb2cd63b3fea966"
curl "http://localhost:3003/api/v1/files/384b0ff44aaaa1f14cb2cd63b3fea966/identities"
# Run tests
cargo test --lib
cargo clippy --lib
```
---
## Status Checklist
- [ ] Create migration script (011_rename_file_uuid.sql)
- [ ] Test migration on dev schema
- [ ] Update Rust API
- [ ] Update Portal
- [ ] Run cargo test
- [ ] Run cargo clippy
- [ ] Batch update docs
- [ ] Verify all endpoints work
---
## Next Phase
After Phase 1 completion:
- **Phase 2**: Architecture simplification (remove person_identities table)
- **Phase 3**: Implement new binding logic
- **Phase 4**: Portal UI update
docs_v1.0/AI_AGENTS/IDENTITY/PHASE2_MIGRATION_SUMMARY.md
-113
View File
@@ -1,113 +0,0 @@
# Phase 2 Migration Summary
> Version: V4.0 | Date: 2026-04-28
> Status: Completed (Code Ready, Migration Pending)
---
## Completed Tasks
| Task | Status | Details |
|------|--------|---------|
| **DB Migration Scripts** | ✅ | 026, 027, 028 created |
| **New Binding API** | ✅ | identity_binding_v4.rs (473 lines) |
| **Routes Registration** | ✅ | 5 new endpoints |
| **Module Export** | ✅ | mod.rs updated |
---
## New API Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| `/api/v1/identities/register` | POST | Register identity from face_ids |
| `/api/v1/identities/:uuid/bind` | POST | Bind faces to identity |
| `/api/v1/identities/:uuid/unbind` | POST | Unbind faces from identity |
| `/api/v1/faces/candidates` | GET | List unregistered faces |
| `/api/v1/files/:uuid/identity-stats` | GET | Get file identity stats |
---
## Migration Files Created
| File | Purpose |
|------|---------|
| `migrations/025_rename_video_uuid_to_file_uuid.sql` | Rename columns |
| `migrations/026_create_file_identities_table.sql` | N:N relationship table |
| `migrations/027_add_identity_id_to_face_detections.sql` | Add foreign key |
| `migrations/028_drop_person_identities_table.sql` | Remove old architecture |
---
## Files Modified
| File | Changes |
|------|--------|
| `src/api/mod.rs` | Add identity_binding_v4 module |
| `src/api/server.rs` | Register new routes |
| `src/api/identity_binding_v4.rs` | New binding logic |
---
## Next Steps
### 1. Run DB Migrations
```bash
# Connect to dev schema
psql -U accusys -d momentry -c "SET search_path TO dev;"
# Run migrations
psql -U accusys -d momentry -f migrations/025_rename_video_uuid_to_file_uuid.sql
psql -U accusys -d momentry -f migrations/026_create_file_identities_table.sql
psql -U accusys -d momentry -f migrations/027_add_identity_id_to_face_detections.sql
psql -U accusys -d momentry -f migrations/028_drop_person_identities_table.sql
```
### 2. Update SQLx Cache
```bash
cargo sqlx prepare
```
### 3. Test New Endpoints
```bash
cargo run --bin momentry_playground -- server
# Test candidates API
curl "http://localhost:3003/api/v1/faces/candidates?min_confidence=0.8"
# Test register API
curl -X POST "http://localhost:3003/api/v1/identities/register" \
-H "Content-Type: application/json" \
-d '{"face_ids": [100], "name": "Test Person"}'
```
---
## Compilation Status
- **Code Structure**: ✅ Correct
- **Type Safety**: ⏸ Pending DB migration
- **SQLx Cache**: ⏸ Need `cargo sqlx prepare` after migration
---
## Architecture Comparison
| Aspect | V3.x | V4.0 |
|--------|------|------|
| **Binding Layer** | 3 (Face → Person → Identity) | 2 (Face → Identity) |
| **Tables** | person_identities + person_appearances | file_identities |
| **API Endpoints** | 33 | 15 |
| **Person ID** | Video-local | ❌ Removed |
| **Chunk Binding** | Manual | Auto (time alignment) |
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| V4.0 | 2026-04-28 | Two-layer architecture complete |
docs_v1.0/AI_AGENTS/IDENTITY/V4_MIGRATION_COMPLETE.md
-119
View File
@@ -1,119 +0,0 @@
# V4.0 Migration Complete
> Date: 2026-04-28 19:50
> Status: ✅ Successfully Completed
---
## Summary
### Phase 1: Terminology Migration (video_uuid → file_uuid)
| Task | Status | Files Modified |
|------|--------|----------------|
| **DB Migration 025** | ✅ | 4 tables renamed |
| **Rust API** | ✅ | 11 files |
| **Portal Vue/Tauri** | ✅ | 6 files |
| **Documents** | ✅ | 117 MD files |
### Phase 2: Architecture Simplification
| Task | Status | Details |
|------|--------|---------|
| **DB Migration 026** | ✅ | file_identities table created |
| **DB Migration 027** | ✅ | identity_id FK added |
| **DB Migration 028** | ✅ | person_identities dropped |
| **SQLx Fix** | ✅ | 5 JSONB bindings fixed |
| **Compilation** | ✅ | cargo check --lib passed |
| **Tests** | ✅ | 178 tests passed |
| **Clippy** | ✅ | 119 warnings (minor) |
---
## Files Fixed (JSONB Issues)
| File | Line | Fix |
|------|------|-----|
| src/api/identities.rs | 274 | .bind(serde_json::to_string(...)) |
| src/api/face_recognition.rs | 337 | .bind(serde_json::to_string(...)) |
| src/api/person_identity.rs | 1508 | .bind(serde_json::to_string(...)) |
| src/api/person_identity.rs | 2287 | .bind(serde_json::to_string(...)) |
| src/core/worker/job_runner.rs | 105 | serde_json::json!({"status": "COMPLETED"}) |
---
## Database State (dev schema)
```sql
-- Tables Created
file_identities ✅
- file_uuid, identity_id, face_count, confidence
-- Tables Renamed
face_detections.video_uuid → file_uuid ✅
face_clusters.video_uuid → file_uuid ✅
-- Tables Deleted
person_identities ✅
person_appearances ✅
```
---
## Build Status
```bash
# Compilation
cargo check --lib ✅
cargo build --lib ✅
# Tests
cargo test --lib ✅ (178 passed)
# Linting
cargo clippy --lib ✅ (119 warnings, minor)
# SQLx Cache
cargo sqlx prepare ✅ (.sqlx updated)
```
---
## Remaining Tasks (Optional)
| Task | Priority | Status |
|------|----------|--------|
| Create identity_binding_v4.rs | Medium | Pending |
| Remove person_identity.rs | Low | Pending |
| Update Portal UI for new endpoints | Low | Pending |
---
## Migration Summary
| Aspect | V3.x | V4.0 |
|--------|------|------|
| **video_uuid** | Used everywhere | **file_uuid** |
| **person_identities** | 303 records | **Removed** |
| **file_identities** | N/A | **Created** |
| **Architecture** | 3-layer | **2-layer** |
| **Compilation** | Broken | **Fixed** |
| **Tests** | - | **178 passed** |
---
## Next Steps
1. Test API endpoints manually
2. Create identity_binding_v4.rs with proper JSONB handling
3. Update Portal UI to use new endpoints
4. Document API changes in AGENTS.md
---
## Key Lessons
1. **SQLx JSONB**: Must use `serde_json::json!()` for compile-time checks
2. **Batch replacements**: Use sed -i for large-scale renaming
3. **DB Migration**: Test on dev schema first, fix errors incrementally
4. **Compilation**: Fix one error at a time, run cargo check frequently
docs_v1.0/AI_AGENTS/IDENTITY/V4_MIGRATION_STATUS.md
-121
View File
@@ -1,121 +0,0 @@
# V4.0 Migration Status
> Date: 2026-04-28
---
## Completed Tasks
### Phase 1: Terminology Migration (video_uuid → file_uuid)
| Task | Status | Details |
|------|--------|---------|
| **DB Migration 025** | ✅ | face_detections, face_clusters, person_identities renamed |
| **Rust API** | ✅ | 11 files batch replaced |
| **Portal** | ✅ | 6 Vue/Tauri files |
| **Documents** | ✅ | 117 MD files |
### Phase 2: Architecture Simplification
| Task | Status | Details |
|------|--------|---------|
| **DB Migration 026** | ✅ | file_identities table created |
| **DB Migration 027** | ✅ | identity_id FK added to face_detections |
| **DB Migration 028** | ✅ | person_identities + person_appearances dropped |
| **New Binding API** | ⏸ | identity_binding_v4.rs (SQLx compile error) |
---
## Current Issue
**SQLx Compile Error**: "invalid input syntax for type json"
Cause: identities.metadata column is JSONB, but SQLx requires exact type matching during compile-time checks.
---
## Database State
```sql
-- Tables Created
file_identities (N:N relationship)
- file_uuid, identity_id, face_count, confidence
-- Tables Renamed
face_detections.video_uuid → file_uuid
face_clusters.video_uuid → file_uuid
-- Tables Deleted
person_identities ✅
person_appearances ✅
```
---
## Next Steps
### Option A: Fix SQLx (Recommended)
1. Remove identity_binding_v4.rs temporarily
2. Run `cargo sqlx prepare` to update cache
3. Fix SQL queries with proper JSONB binding
4. Re-add identity_binding_v4.rs
### Option B: Use SQLX_OFFLINE
```bash
SQLX_OFFLINE=true cargo build --lib
cargo sqlx prepare
```
### Option C: Skip for Now
Keep existing person_identity.rs API, migrate later when database is stable.
---
## Test Commands
```bash
# Verify tables
psql -U accusys -d momentry -c "\dt dev.*"
# Check columns
psql -U accusys -d momentry -c "
SELECT table_name, column_name
FROM information_schema.columns
WHERE table_schema = 'dev'
AND column_name = 'file_uuid'
ORDER BY table_name;
"
# Build (if SQLx fixed)
cargo build --lib
cargo test --lib
```
---
## Files Modified
| File | Lines |
|------|-------|
| migrations/025_rename_video_uuid_to_file_uuid.sql | 42 |
| migrations/026_create_file_identities_table.sql | 39 |
| migrations/027_add_identity_id_to_face_detections.sql | 30 |
| migrations/028_drop_person_identities_table.sql | 29 |
| src/api/identity_binding_v4.rs | 310 |
| src/api/mod.rs | +1 line |
| src/api/server.rs | +1 line |
---
## Migration Summary
| Aspect | V3.x | V4.0 |
|--------|------|------|
| **video_uuid** | Used everywhere | **file_uuid** |
| **person_identities** | 303 records | **Removed** |
| **file_identities** | N/A | **Created** |
| **API Endpoints** | 33 | 15 (pending) |
| **Binding Logic** | 3-layer | 2-layer (pending) |
docs_v1.0/AI_AGENTS/SEARCH/SEARCH_PROMPTS.md
-139
View File
@@ -1,139 +0,0 @@
---
document_type: "reference_doc"
service: "MOMENTRY_CORE"
title: "搜尋範例 Prompt"
date: "2026-04-25"
version: "V1.0"
status: "active"
owner: "Warren"
created_by: "OpenCode"
tags:
- "prompt"
- "搜尋範例"
ai_query_hints:
- "查詢 搜尋範例 Prompt 的內容"
- "搜尋範例 Prompt 的主要目的是什麼？"
- "如何操作或實施 搜尋範例 Prompt？"
---
# 搜尋範例 Prompt
## 基本搜尋測試
### 1. 簡單關鍵字搜尋
```bash
curl -X POST http://localhost:3002/api/v1/search \
-H "Content-Type: application/json" \
-d '{"query": "charade", "limit": 5}'
```
### 2. 電影相關詞
```
charade
woody allen
audrey hepburn
classic movie
old time movie
romantic comedy
```
### 3. 場景描述
```
widowed woman
secret agent
chase scene
paris
```
---
## 進階搜尋測試
### 4. 短語搜尋
```bash
curl -X POST http://localhost:3002/api/v1/search \
-H "Content-Type: application/json" \
-d '{"query": "fun plot twists", "limit": 3}'
```
### 5. 情感/描述詞
```
charming performances
hilarious
suspenseful
dramatic
```
### 6. 動作場景
```
running
chase
fighting
dancing
```
---
## 整合範例
### n8n Workflow
```
搜尋詞: "charade"
→ 取得 chunk 的 start_time, end_time
→ 組裝成影片 URL
→ 回傳給用戶
```
### PHP 範例
```php
$searchTerms = ['charade', 'woody', 'audrey', 'classic'];
// 搜尋每個詞
foreach ($searchTerms as $term) {
$ch = curl_init('http://localhost:3002/api/v1/search');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
'query' => $term,
'limit' => 5
]));
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
$response = curl_exec($ch);
$data = json_decode($response, true);
// 處理結果
foreach ($data['results'] as $result) {
echo "{$result['text']} (score: {$result['score']})\n";
}
}
```
---
## 預期回傳格式
```json
{
"results": [
{
"uuid": "a1b10138a6bbb0cd",
"chunk_id": "sentence_0006",
"chunk_type": "sentence",
"start_time": 48.8,
"end_time": 55.44,
"text": "fun plot twists, Woody Dialog and charming performances...",
"score": 0.526
}
],
"query": "charade"
}
```
---
## 測試檢查清單
- [ ] 基本關鍵字搜尋
- [ ] n8n 整合格式
- [ ] 影片時戳取得
- [ ] 多筆結果排序
- [ ] 不同 chunk_type 搜尋
docs_v1.0/AI_AGENTS/SUMMARIZATION/CHUNK_RULE_4_SUMMARY.md
-231
View File
@@ -1,231 +0,0 @@
---
document_type: "architecture_design"
service: "MOMENTRY_CORE"
title: "Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0)"
date: "2026-04-21"
version: "V1.0"
status: "active"
owner: "Warren"
created_by: "OpenCode"
tags:
- "momentry"
- "core"
- "摘要分析級檢索"
- "rule"
ai_query_hints:
- "查詢 Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0) 的內容"
- "Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0) 的主要目的是什麼？"
- "如何操作或實施 Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0)？"
---
# Momentry Core Chunk Rule 4: 摘要分析級檢索 (Summary 5W1H Chunk) (v1.0)
| 項目 | 內容 |
|------|------|
| 建立者 | OpenCode |
| 建立時間 | 2026-04-21 |
| 文件版本 | V1.0 |
---
## 版本歷史
| 版本 | 日期 | 目的 | 操作人 | 工具/模型 |
|------|------|------|--------|-----------|
| V1.0 | 2026-04-21 | 定義 Rule 4: 基於 LLM 5W1H 分析的最高層級摘要結構 | OpenCode | OpenCode / Qwen3.6-Plus |
---
## 0. 設計目標
**Rule 4** 的核心概念是**「情節理解」(Storyline Understanding)**。透過將多個場景 (Rule 3) 聚合，並利用大型語言模型 (Gemma4) 進行深度分析，提取 5W1H 結構化資訊，使系統能夠回答複雜的「情節相關問題」。
- **核心原則**: 5-10 個場景 (Rule 3) = 1 個摘要區塊 (Summary Chunk)。
- **結構**: 頂層 Parent Chunk。
- **特徵**: 包含 LLM 生成的完整摘要與 **5W1H** (Who, What, When, Where, Why, How) 分析結果。
- **優勢**: 支援宏觀劇情檢索、人物動線追蹤與複雜問答 (RAG)。
---
## 1. 數據源與聚合邏輯
Rule 4 是處理管線的終點，依賴 **Rule 3** 的產出以及 **LLM 服務**。
1. **Rule 3 Chunks (Primary)**: 提供場景級的文本摘要與元數據。
- *聚合策略*: 將連續的 5-10 個 Rule 3 Chunks 視為一個「敘事區塊」。
2. **LLM Processor (Gemma4)**:
- *任務*: 讀取該區塊內所有 Rule 3 的摘要與 ASR 文本。
- *輸出*:
- **Summary**: 流暢的劇情描述。
- **5W1H**: 結構化的關鍵要素提取。
3. **Visual/Audio Retention**:
- 保留區塊內所有出現過的 `face_ids` (Who) 和 `objects` (What/Where)。
---
## 2. Chunk 結構定義
### 2.1 資料庫結構 (PostgreSQL)
```sql
CREATE TABLE chunks_rule4 (
id UUID PRIMARY KEY,
asset_uuid UUID NOT NULL,
chunk_type VARCHAR(20) DEFAULT 'summary',
-- 時間軸 (繼承自第一個與最後一個 Rule 3 子區塊)
start_frame INT NOT NULL,
end_frame INT NOT NULL,
start_time_sec DOUBLE PRECISION,
end_time_sec DOUBLE PRECISION,
-- LLM 生成內容
summary TEXT NOT NULL, -- 劇情摘要
analysis_5w1h JSONB, -- 結構化分析結果
-- 聚合元數據
faces JSONB, -- 區塊內所有人物
objects JSONB, -- 區塊內重要物件
-- 向量索引
embedding vector(768), -- 摘要與 5W1H 的混合向量
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- 關聯子區塊
ALTER TABLE parent_chunks ADD COLUMN rule4_parent_id UUID REFERENCES chunks_rule4(id);
```
### 2.2 5W1H 結構 (JSONB)
```json
{
"who": ["Cary Grant", "Audrey Hepburn"], // 主要人物 (對應 Face ID)
"what": ["Searching for the stamps", "Car chase"], // 核心事件
"where": ["Paris", "Bank", "Car"], // 地點/場景 (對應 Visual Objects)
"when": "Night", // 時間背景 (對應 Time of day)
"why": "To pay off a debt", // 動機
"how": "By sneaking into the vault" // 手段/過程
}
```
### 2.3 JSON 產出範例
```json
{
"chunk_id": "550e...0004",
"type": "summary",
"summary": "Peter 和 Regina 計劃潛入銀行金庫尋找郵票。他們在夜間開車前往，途中遭遇巡邏隊盤查，但最終利用機智脫身。",
"start_frame": 5000,
"end_frame": 8000,
"analysis_5w1h": {
"who": ["peter_joshua", "regina_lampert"],
"what": ["heist_planning", "evasion"],
"where": ["car", "street", "bank_exterior"],
"when": "night",
"why": "retrieve_stamps",
"how": "stealth_deception"
},
"metadata": {
"rule3_count": 7
}
}
```
---
## 3. 搜尋能力定義
Rule 4 是 **RAG (Retrieval-Augmented Generation)** 的核心數據源。
### 3.1 劇情摘要搜尋 (Plot Search)
- **場景**: "這部片在講什麼？"、"他們找到郵票了嗎？"
- **邏輯**:
1. 搜尋 `summary` 向量。
2. 返回包含該情節的完整摘要區塊。
### 3.2 5W1H 結構化查詢 (Structured Query)
- **場景**: "找出所有 **Cary Grant (Who)** 在 **車上 (Where)** 的片段"。
- **邏輯**:
1. 過濾 `analysis_5w1h` JSONB 欄位。
2. `who` 包含 "Cary Grant" **AND** `where` 包含 "car"。
3. 這種查詢比傳統關鍵字搜索更精準，因為它是經過 LLM 理解後的結構化數據。
### 3.3 動機與原因搜尋 (Why/How)
- **場景**: "他為什麼要偷東西？"
- **邏輯**:
1. 針對 `analysis_5w1h.why` 進行語意比對。
---
## 4. 處理流程 (LLM Pipeline)
Rule 4 的生成需要呼叫 `llm_engine` (Gemma4) 服務。
### 4.1 演算法邏輯 (Pseudocode)
```python
# 輸入: rule3_chunks (List of Scene Chunks)
# 1. 分組 (每 5-10 個場景一組)
for group in chunks(rule3_chunks, size=7):
# 2. 準備 LLM 上下文
context_text = "\n".join([chunk.summary for chunk in group])
context_objects = aggregate_objects(group)
prompt = f"""
Analyze the following video scenes and extract the 5W1H information.
Scenes:
{context_text}
Return JSON format:
{{
"summary": "A brief summary of these scenes.",
"5w1h": {{
"who": ["List of characters"],
"what": ["Main events"],
...
}}
}}
"""
# 3. 呼叫 LLM (Gemma4 via Service Registry)
response = llm_service.chat(prompt)
result = parse_json(response)
# 4. 建立 Rule 4 Chunk
rule4_chunk = {
"summary": result["summary"],
"analysis_5w1h": result["5w1h"],
"start_frame": group[0].start_frame,
"end_frame": group[-1].end_frame,
"faces": aggregate_faces(group),
"objects": aggregate_objects(group)
}
# 5. 儲存並關聯
rule4_id = store_rule4_chunk(rule4_chunk)
for chunk in group:
link_rule3_to_rule4(chunk.id, rule4_id)
```
---
## 5. 總結
Rule 4 將 Momentry 從「影片搜尋引擎」提升為**「影片知識圖譜」**。
| 特性 | 實作方式 |
|------|----------|
| **粒度** | 情節/敘事區塊 (5-10 場景) |
| **核心技術** | LLM 5W1H 提取 (Gemma4) |
| **數據結構** | 摘要文本 + JSONB 5W1H 結構 |
| **向量內容** | 混合向量 (Summary + 5W1H) |
| **適用場景** | 問答系統 (RAG)、劇情回顧、複雜條件過濾 |
**四層架構總覽：**
1. **Rule 1 (Sentence)**: 精確台詞檢索。
2. **Rule 2 (Visual)**: 畫面物件檢索。
3. **Rule 3 (Scene)**: 場景上下文檢索。
4. **Rule 4 (Summary)**: 劇情理解與知識問答。
docs_v1.0/AI_AGENTS/TRANSLATION/TEXT_TRANSLATION.md
-166
View File
@@ -1,166 +0,0 @@
# 翻譯 Agent (Translation Agent) 設計文件
| 項目 | 內容 |
|------|------|
| 建立者 | OpenCode |
| 建立時間 | 2026-04-25 |
| 文件版本 | V1.0 |
| 用途 | 提供多語言文本翻譯服務 (應用於 Portal Chunk Detail) |
---
## 1. Agent 概覽
Translation Agent 負責將系統中的非結構化文本（如 Chunk 內容、摘要、5W1H 推論結果）翻譯為使用者指定的語言。
在 Portal 的 **Chunk Search Detail** 頁面，當使用者瀏覽不同語言的影片內容時，此 Agent 提供即時翻譯支援。
### 1.1 資源註冊資訊 (Resource Registry)
當 Agent 啟動時，將向 **Resource Registry** 註冊以下資訊：
```json
{
"resource_id": "agent_text_translation_v1",
"resource_type": "agent",
"capabilities": ["translate_text", "detect_language", "batch_translate"],
"category": "text_processing",
"config": {
"default_model": "gpt-4o-mini",
"fallback_model": "local-llama-3-8b",
"max_tokens": 4096,
"supported_languages": ["zh-TW", "en-US", "ja-JP", "ko-KR"]
}
}
```
---
## 2. 核心設計
### 2.1 輸入格式 (Input)
Agent 接收來自 Portal 或內部 API 的 JSON 請求：
```json
{
"text": "He walked into the room and saw a large red car.",
"target_language": "zh-TW",
"source_language": "auto",
"context": {
"domain": "movie_subtitle",
"glossary": {
"red car": "紅色跑車"
}
}
}
```
- `text`: 待翻譯文本。
- `target_language`: 目標語言 (BCP 47 格式)。
- `context` (可選): 提供領域資訊或專有名詞對照表 (Glossary) 以提高準確度。
### 2.2 輸出格式 (Output)
Agent 回傳標準化 JSON：
```json
{
"translated_text": "他走進房間，看到一輛紅色跑車。",
"source_language_detected": "en-US",
"confidence": 0.98,
"usage": {
"input_tokens": 12,
"output_tokens": 15
}
}
```
---
## 3. Prompt 設計 (System Prompt)
為了確保翻譯風格符合 Momentry Core 的專業性（如準確的影視術語），我們使用以下 System Prompt：
```text
You are a professional translator for Momentry Core, a digital asset management system specializing in video analysis.
## Guidelines:
1. **Accuracy**: Translate the meaning accurately, maintaining the original tone.
2. **Context Awareness**: If a glossary is provided in the context, strictly follow it.
3. **Style**:
- For subtitles: Keep it concise and natural for reading.
- For technical terms (e.g., 5W1H, metadata): Use standard industry translations.
4. **Format**: Preserve any JSON structure, markdown, or timestamps present in the input text. Do not translate code blocks.
5. **Output**: Return ONLY the translated text in the requested format unless asked otherwise.
```
---
## 4. API 端點設計
### 4.1 單一翻譯
```http
POST /api/v1/agents/translate
Content-Type: application/json
X-Resource-Id: agent_text_translation_v1
{
"text": "...",
"target_language": "zh-TW"
}
```
### 4.2 批次翻譯 (Batch Translation)
針對 Chunk Detail 頁面可能一次顯示多個段落，支援批次翻譯：
```http
POST /api/v1/agents/translate/batch
Content-Type: application/json
{
"items": [
{ "id": "chunk_001", "text": "..." },
{ "id": "chunk_002", "text": "..." }
],
"target_language": "zh-TW"
}
```
---
## 5. 錯誤處理與容錯
- **模型降級 (Fallback)**: 若 `gpt-4o-mini` 超時或不可用，自動切換至本地模型 `local-llama-3-8b`。
- **Token 超長**: 若文本超過 `max_tokens`，自動進行分段翻譯 (Split & Translate)。
- **無效語言**: 若 `target_language` 不在支援列表中，回傳 `400 Bad Request`。
---
## 6. Portal 整合範例 (Chunk Detail)
在 Portal 的 `ChunkDetailView.vue` 中，翻譯功能的調用流程如下：
1. 使用者點擊「翻譯為 繁體中文」按鈕。
2. Portal 發送 POST 請求至 `/api/v1/agents/translate`。
3. 取得結果後，在不重新整理頁面的情況下更新 UI (顯示 `translated_text`)。
```typescript
// Portal 前端調用範例
async function translateChunkText(text: string, targetLang: string) {
const response = await fetch('/api/v1/agents/translate', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text, target_language: targetLang })
});
return response.json();
}
```
---
## 版本資訊
- 版本: V1.0
- 建立日期: 2026-04-25