# Momentry Core 多模態語義搜尋設計文檔 V5.0 (全集)

**更新日期**: 2026-04-10
**版本**: V5.0 (Final Integration)
**狀態**: 設計完成，準備實作

---

## 1. 系統架構總覽

```mermaid
graph TD
    %% 樣式定義
    classDef storage fill:#e1f5fe,stroke:#01579b,stroke-width:2px;
    classDef processor fill:#fff3e0,stroke:#e65100,stroke-width:2px;
    classDef identity fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px;
    classDef search fill:#f3e5f5,stroke:#4a148c,stroke-width:2px;

    subgraph "1. Input Sources (輸入源)"
        Video[Video/Audio File]
    end

    subgraph "2. Analysis Processors (分析模組 - 模組化)"
        ASR[ASR Processor\n(Whisper)]:::processor
        ASRX[ASRX Processor\n(SpeechBrain)]:::processor
        YOLO[YOLO Processor\n(Object Detection)]:::processor
        OCR[OCR Processor\n(Text Recognition)]:::processor
        FACE[Face Processor\n(Face ID/Cluster)]:::processor
        POSE[Pose Processor\n(KeyPoints 33)]:::processor
        SCENE[Scene Processor\n(Places365)]:::processor
        AUDIO[AUDIO EVENT Processor\n(PANNs/YAMNet)]:::processor
        POSE_ANALYZER[Pose Analyzer Processor\n(Action/Gesture/Sports)]:::processor
        CONTEXT[Context Inference Processor\n(Rule Engine + LLM)]:::processor
        SPORTS[Sports Classifier Processor\n(Rule Engine)]:::processor
    end

    subgraph "3. Identity & Binding (身份與綁定)"
        VoiceBind[Voice Binding\n(Speaker -> Talent)]:::identity
        FaceBind[Face Binding\n(Face -> Talent)]:::identity
        RoleBind[Role Casting\n(Talent -> Character)]:::identity
        TalentDB[(Talents DB\nVoice/Face Embeddings)]:::storage
        CharDB[(Characters DB\nRoles/Multi-lang)]:::storage
    end

    subgraph "4. Data Storage (數據存儲)"
        PG[(PostgreSQL\nChunks/Relations/Metadata)]:::storage
        Qdrant[(Qdrant\nVector Search Engine)]:::storage
    end

    subgraph "5. Search & Query (搜尋與查詢)"
        SearchProc[Search Processor\n(LLM Parser -> Hybrid Query)]:::search
        User[User / API]
    end

    %% 數據流向
    Video --> ASR
    Video --> ASRX
    Video --> YOLO
    Video --> OCR
    Video --> FACE
    Video --> POSE
    Video --> SCENE
    Video --> AUDIO

    %% 處理結果 -> 特徵提取/分析
    POSE --> POSE_ANALYZER
    POSE --> SPORTS
    AUDIO --> AUDIO
    SCENE --> CONTEXT
    YOLO --> CONTEXT
    YOLO --> SPORTS
    ASRX --> CONTEXT
    ASR --> CONTEXT
    POSE_ANALYZER --> SPORTS
    
    %% 結果寫入
    ASR --> PG
    ASRX --> PG
    YOLO --> PG
    OCR --> PG
    FACE --> PG
    SCENE --> PG
    AUDIO --> PG
    POSE_ANALYZER --> PG
    CONTEXT --> PG
    SPORTS --> PG

    %% 綁定邏輯
    ASRX -.->|Speaker ID| VoiceBind
    FACE -.->|Face ID| FaceBind
    VoiceBind --> TalentDB
    FaceBind --> TalentDB
    TalentDB --> RoleBind
    CharDB --> RoleBind
    RoleBind -.->|Resolve Names| PG

    %% 向量同步
    PG -.->|Sync Payload & Vectors| Qdrant

    %% 搜尋邏輯
    User --> SearchProc
    SearchProc --> PG
    SearchProc --> Qdrant
```

---

## 2. 核心處理模組清單 (Processor Modules)

每個模組遵循單一職責原則 (Single Responsibility Principle)。

| 模組名稱 | 職責 (Responsibility) | 核心技術/模型 | 輸出維度 (Dimension) |
|:---|:---|:---|:---|
| **ASR** | 語音轉文字 | Whisper (small/int8 + VAD) | **Text Content** (語音內容) |
| **ASRX** | 說話人分離/聲紋提取 | SpeechBrain (ECAPA-TDNN) | **Voice ID**, **Speaker Embedding** (192-dim) |
| **YOLO** | 物體檢測 | YOLOv8 (COCO 80 classes) | **Object** (車輛、武器、物品、運動裝備) |
| **OCR** | 畫面文字識別 | EasyOCR / PaddleOCR | **Text** (字幕、招牌、文件) |
| **FACE** | 人臉檢測與聚類 | RetinaFace / ArcFace | **Face ID**, **Face Embedding** |
| **POSE** | 骨架關鍵點提取 | MediaPipe / YOLO-Pose | **Keypoints** (33 點坐標) |
| **POSE ANALYZER** | 動作/手勢解碼 | Heuristics (規則引擎) | **Action** (站/坐/臥/揮手/打鬥/泳姿/旋轉) |
| **SCENE** | 場景分類 | Places365 (ResNet18) | **Location** (Macro/Semantic/Raw 三層級) |
| **AUDIO EVENT** | 環境/特效音識別 | PANNs / YAMNet | **Audio Event** (槍聲/雨聲/狗叫/樂器/哨音) |
| **CONTEXT INFERENCE** | 環境/氛圍推論 | Rule Engine + LLM | **Context** (季節/溫度/節慶/天氣) |
| **SPORTS CLASSIFIER** | 運動項目識別 | Multi-Modal Rule Engine | **Sport Type** (棒球/足球/游泳/跳水/滑冰...) |

---

## 3. 數據架構設計 (Data Architecture)

### 3.1 Chunk 定義 (Video Chunk)
**定義**: 特定視頻文件 (`uuid`) 內，從 `start_frame` 到 `end_frame` 之間的**連續畫面**。
**存儲**:
* **PostgreSQL**: 權威主數據 (Metadata, Relations, Complex Queries).
* **Qdrant**: 向量檢索與 Payload 過濾 (Fast Retrieval).

### 3.2 數據庫 Schema (PostgreSQL)

```sql
-- ==========================================
-- 1. 核心 Chunk 表
-- ==========================================
CREATE TABLE chunks (
    id              BIGSERIAL PRIMARY KEY,
    uuid            VARCHAR(32) NOT NULL,         -- 視頻 ID
    chunk_id        VARCHAR(64) NOT NULL,
    
    -- 物理邊界定義 (核心)
    start_frame     BIGINT NOT NULL,              
    end_frame       BIGINT NOT NULL,              
    fps             FLOAT8 NOT NULL,              
    duration_sec    FLOAT8 GENERATED ALWAYS AS ((end_frame - start_frame) / fps) STORED,
    
    -- 2. 人 (Who)
    speaker_ids     TEXT[] DEFAULT '{}',          -- 觀察到的說話人 ID (Speaker X)
    face_ids        TEXT[] DEFAULT '{}',          -- 觀察到的人臉 ID (Face Y)
    
    -- 3. 事 (What) - 語音與行為
    text_content    TEXT,                         -- ASR 文本
    action_tags     TEXT[] DEFAULT '{}',          -- Pose 動作 (e.g. ['running', 'fighting', 'swimming'])
    audio_events    TEXT[] DEFAULT '{}',          -- 音頻事件 (e.g. ['gunshot', 'scream', 'whistle'])
    event_tags      JSONB DEFAULT '[]',           -- 融合事件 (e.g. [{"tag":"gunfight", "score":0.8}])
    sport_type      VARCHAR(32),                  -- 運動項目 (e.g. 'baseball', 'diving')
    sport_actions   TEXT[] DEFAULT '{}',          -- 運動細分動作 (e.g. ['pitching', 'smash'])
    sport_sequence  JSONB DEFAULT '[]',           -- 動作序列 (e.g. ["takeoff", "twist", "entry"])
    
    -- 4. 地 (Where) & 物 (Object)
    scene_raw       TEXT[] DEFAULT '{}',          -- Places365 原始標籤
    scene_semantic  TEXT[] DEFAULT '{}',          -- 高層語義 (e.g. ['office', 'indoor'])
    object_tags     TEXT[] DEFAULT '{}',          -- YOLO 物件 (e.g. ['car', 'gun', 'baseball_bat'])
    
    -- 5. 上下文 (Context)
    context_season  VARCHAR(16),                  -- 'winter', 'summer'
    context_temp    VARCHAR(16),                  -- 'hot', 'cold'
    context_weather VARCHAR(16),                  -- 'rainy', 'snowy'
    context_festivals TEXT[] DEFAULT '{}',        -- ['christmas', 'halloween']
    
    -- 向量與索引
    vector_ids      JSONB,                        -- 指向 Qdrant Point ID
    created_at      TIMESTAMPTZ DEFAULT NOW(),
    
    UNIQUE(chunk_id),
    UNIQUE(uuid, chunk_id)
);

CREATE INDEX idx_chunks_uuid ON chunks(uuid);
CREATE INDEX idx_chunks_frame_range ON chunks(uuid, start_frame, end_frame);
CREATE INDEX idx_chunks_attrs ON chunks USING GIN (scene_semantic, object_tags, audio_events);

-- ==========================================
-- 6. 身份綁定表 (Identity Binding)
-- ==========================================

-- 真實人才庫 (Talent)
CREATE TABLE talents (
    id              BIGSERIAL PRIMARY KEY,
    real_name       TEXT,
    voice_embedding VECTOR(192),                  -- 聲紋參考向量 (ECAPA-TDNN)
    face_embedding  VECTOR(512)                   -- 人臉參考向量 (ArcFace)
);

-- 劇中角色庫 (Character)
CREATE TABLE characters (
    id              BIGSERIAL PRIMARY KEY,
    file_uuid      TEXT NOT NULL,
    name            TEXT NOT NULL,                -- 角色名
    language_track  TEXT DEFAULT 'original',      -- 語言軌道 (dub_zh_tw, dub_en)
    is_voice_only   BOOLEAN DEFAULT FALSE,        -- 無臉角色 (動畫/旁白/AI)
    metadata        JSONB DEFAULT '{}'
);

-- 飾演關係 (Casting)
CREATE TABLE castings (
    talent_id       BIGINT REFERENCES talents(id),
    character_id    BIGINT REFERENCES characters(id),
    track_type      TEXT DEFAULT 'original',
    PRIMARY KEY (talent_id, character_id, track_type)
);

-- 綁定映射 (Signal -> Talent)
CREATE TABLE identity_bindings (
    binding_type    VARCHAR(32),                  -- 'face', 'speaker'
    binding_value   VARCHAR(64),                  -- 機器 ID (e.g. 'face_1', 'speaker_3')
    talent_id       BIGINT REFERENCES talents(id),
    UNIQUE(binding_type, binding_value)
);
```

### 3.3 Qdrant Payload 結構 (扁平化過濾)

```json
{
  "uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
  "chunk_id": "chunk_001",
  "start_frame": 100,
  "end_frame": 200,
  
  "who_is_present": ["Alice", "Bob"],
  "who_is_speaking": ["Alice"],
  
  "what_happening": ["arguing", "shouting"],
  "what_objects": ["person", "table"],
  "what_audio": ["raised_voice"],
  "sport_type": null,
  
  "where_semantic": ["office", "indoor"],
  "where_weather": null,
  
  "context_season": null,
  "context_time": "day"
}
```

---

## 4. 搜尋維度 (5W1H + Context + Sports)

### 4.1 人 (Person / Who)
* **身份解析**: `speaker_X` / `face_Y` -> `talent` -> `character`.
* **屬性過濾**: 性別、年齡、體型、五官、服裝 (VLM/Heuristics).
* **聲紋檢索**: 上傳音頻片段 -> Cosine Similarity (ECAPA-TDNN 192-dim).

### 4.2 事 (Event / What)
* **語音語義**: ASR 文本向量檢索.
* **視覺行為**: Pose Analyzer 標籤 (打架、擁抱、揮手).
* **融合事件**: `gunfight`, `romantic_scene`, `interview` (多信號規則融合).

### 4.3 時 (Time / When)
* **精確幀**: `start_frame`, `end_frame`.
* **相對時間**: "最後 5 分鐘".

### 4.4 地 (Location / Where)
* **場景語義**: Places365 -> 宏觀/語義/原始三層映射 (e.g., `beach` -> `outdoor`).
* **天氣/環境**: `rainy`, `sunny`, `night` (Context Inference).

### 4.5 物 (Object / Which)
* **YOLO 物件**: `car`, `gun`, `dog`.
* **音頻物件**: `siren`, `barking`.

### 4.6 上下文 (Context)
* **季節**: `winter` (雪/圍巾), `summer` (泳衣/太陽眼鏡).
* **節慶**: `christmas` (聖誕樹/鈴鐺聲), `cny` (鞭炮/紅燈籠).

### 4.7 運動 (Sports)
* **球類**: 棒球 (球棒/打擊聲/揮棒), 籃球 (運球聲/投籃), 足球 (哨音/踢球).
* **水上/冰上運動 (詳細特徵)**:
  * **🏊 游泳 (Swimming)**:
    * *場景*: `swimming_pool`, `water`.
    * *物件*: `goggles`, `swim_cap`, `lane_rope`.
    * *動作*: `freestyle_stroke` (自由式), `breaststroke` (蛙式), `butterfly` (蝶式), `backstroke` (仰式).
    * *音頻*: `water_splash` (水花聲), `rhythmic_breathing` (規律換氣聲).
  * **🤿 跳水 (Diving)**:
    * *場景*: `diving_board`, `platform_10m`.
    * *動作序列*: `takeoff` (起跳) → `aerial_twist` (空中翻轉) → `entry` (入水).
    * *音頻*: `high_pitch_whistle` (哨音) → `massive_splash` (巨大入水聲).
  * **⛸️ 滑冰 (Ice Skating)**:
    * *場景*: `ice_rink`, `winter`.
    * *物件*: `ice_skates`, `barrier`.
    * *動作*: `gliding` (滑行), `spinning` (旋轉), `jumping` (跳躍).
    * *音頻*: `blade_on_ice` (冰刀摩擦聲), `classical_music` (花滑配樂).

---

## 5. 搜尋執行流程 (Search Workflow)

### 5.1 用戶輸入
> *"找一下昨天在辦公室，那個穿西裝的男人在生氣地罵人，旁邊還有狗叫的片段。"*

### 5.2 LLM 解析 (`Search Processor`)
```json
{
  "who": {
    "clothing": ["suit"],
    "expression": ["angry"],
    "gender": "male"
  },
  "where": {
    "semantic": ["office"]
  },
  "what": {
    "action": ["arguing", "shouting"],
    "audio_event": ["dog_bark"]
  },
  "when": {
    "relative": "yesterday"
  }
}
```

### 5.3 混合查詢 (Hybrid Query)

1. **解析身份 (Who)**:
    * 查詢 `identity_bindings`，找到符合 "穿西裝男人" 的機器 ID (`face_5`).
2. **構建 SQL (PostgreSQL)**:
    ```sql
    SELECT chunk_id, start_frame, end_frame FROM chunks
    WHERE uuid = '384b0ff44aaaa1f14cb2cd63b3fea966'
    AND 'face_5' = ANY(face_ids)
    AND scene_semantic @> ARRAY['office']
    AND action_tags @> ARRAY['arguing', 'shouting']
    AND audio_events @> ARRAY['dog_bark'];
    ```
3. **構建 Vector Search (Qdrant)**:
    * 如果 SQL 結果為空或用戶語意模糊，切換至 Qdrant Payload Filter + Vector Similarity.
4. **返回結果**:
    * Chunk 列表，包含精確的 `start_frame`, `end_frame`.

---

## 6. 實施路線圖 (Implementation Roadmap)

### Phase 1: 基礎設施與 Schema (第 1 週)
* [ ] 執行 PostgreSQL Schema V5 更新 (Chunks, Talents, Castings, Bindings, Sports).
* [ ] 建立 Qdrant Collection (`momentry_chunks`)，配置 Multi-Vector 和 Payload 索引.
* [ ] 編寫 `scene_hierarchy_processor.py` (場景映射層).
* [ ] 編寫 `scene_mapping.json`.

### Phase 2: 信號提取模組 (第 2-3 週)
* [ ] 部署 `audio_event_processor.py` (PANNs/YAMNet).
* [ ] 部署 `pose_analyzer_processor.py` (基礎規則：站/坐/揮手/打鬥/泳姿).
* [ ] 部署 `context_inference_processor.py` (季節/節慶/天氣推斷).
* [ ] 部署 `sports_classifier_processor.py` (運動分類規則引擎).
* [ ] 確保所有處理器的輸出能正確映射並寫入 `chunks` 表.

### Phase 3: 身份綁定系統 (第 4 週)
* [ ] 部署 `voice_embedding_extractor.py` (聲紋提取與比對).
* [ ] 實現 `identity_resolver.py`：將機器 ID 綁定到 `talents` 和 `characters`.
* [ ] 提供 API: `POST /api/v1/person/bind`.

### Phase 4: 搜尋引擎整合 (第 5 週)
* [ ] 開發 `search_processor.py` (LLM Parser + SQL Builder).
* [ ] 實現 `POST /api/v1/search/smart` 端點.
* [ ] 測試複雜查詢 (人+事+時+地+物+上下文+運動).

### Phase 5: 優化與前端對接 (第 6 週)
* [ ] 性能優化 (索引調整、查詢緩存).
* [ ] 前端搜尋介面展示多維度過濾條件.
* [ ] 前端視頻播放器跳轉至精確 `start_frame`.

---

此設計文檔已涵蓋所有需求，確立了 Momentry Core 作為一個**高度模組化、多模態、支持深度語義搜尋**的系統架構。所有討論過的維度 (包括運動、配音、動畫、聲紋) 均已整合。
