feat: Rule2 TKG relationship chunks + Phase0-1 Qdrant integration
Phase 0: TKG builder populate face_detections from face.json - Fix face.json parser for pose_angle format - Call store_traced_faces.py to set trace_id - Skip if trace_id already populated Phase 1: Qdrant face embeddings integration - Add FaceEmbeddingDb module (src/core/db/face_embedding_db.rs) - Create dev_face_embeddings collection (dim=512) - Store 1122 face embeddings with pose metadata - API: init_collection, batch_upsert, search_similar Rule2: TKG edges → relationship chunks - Design: RULE2_TKG_RELATIONSHIP_V1.0.md - Implementation: rule2_ingest.rs - ChunkType::Relationship added - Edge types: SPEAKS_AS, MUTUAL_GAZE, CO_OCCURS_WITH, HAS_APPEARANCE, WEARS - Auto-trigger on TKG rebuild API: - POST /api/v1/file/:file_uuid/rule2 (vectorization) - POST /api/v1/file/:file_uuid/tkg/rebuild (auto Rule2) Test: 75 relationship chunks created + vectorized
This commit is contained in:
@@ -0,0 +1,378 @@
|
|||||||
|
<!-- module: tkg -->
|
||||||
|
<!-- description: Temporal Knowledge Graph — rebuild, nodes, edges, processor counts -->
|
||||||
|
<!-- depends: 05_process, 07_identity -->
|
||||||
|
|
||||||
|
## Temporal Knowledge Graph (TKG)
|
||||||
|
|
||||||
|
TKG is a time-aligned knowledge graph built from multi-processor outputs (face, yolo, ocr, pose, asrx, gaze, lip, appearance). It produces 9 node types and 14 edge types stored in `dev.tkg_nodes` and `dev.tkg_edges`.
|
||||||
|
|
||||||
|
### Node Types
|
||||||
|
|
||||||
|
| Node Type | Description | Key Properties |
|
||||||
|
|-----------|-------------|----------------|
|
||||||
|
| `face_trace` | A tracked face identity over time | `trace_id`, `face_count`, `avg_confidence` |
|
||||||
|
| `gaze_trace` | Gaze direction over time | `direction` (frontal/left/right/up/down + diagonals) |
|
||||||
|
| `lip_trace` | Lip movement synced with speech | `speaker_id`, `lip_area_range` |
|
||||||
|
| `text_trace` | Spoken text aligned to time | `speaker_id`, `text`, `start_time`, `end_time` |
|
||||||
|
| `appearance_trace` | Human appearance (clothing) over time | `clothing_color`, `upper_cloth`, `lower_cloth` |
|
||||||
|
| `skin_tone_trace` | Fitzpatrick skin tone classification | `fitzpatrick_type` (I–VI) |
|
||||||
|
| `accessory` | Detected accessories | `type` (glasses/hat/etc.), `confidence` |
|
||||||
|
| `object` | YOLO-detected object | `class`, `confidence`, `frame_count` |
|
||||||
|
| `speaker` | ASRX speaker segment | `speaker_id`, `segment_count`, `total_duration` |
|
||||||
|
|
||||||
|
### Edge Types
|
||||||
|
|
||||||
|
| Edge Type | Source → Target | Description |
|
||||||
|
|-----------|-----------------|-------------|
|
||||||
|
| `co_occurs` | object ↔ object | Two objects appear together in same frame |
|
||||||
|
| `speaker_face` | speaker ↔ face_trace | Speaker matched to face trace via lip sync |
|
||||||
|
| `face_face` | face_trace ↔ face_trace | Two face traces interact (mutual gaze) |
|
||||||
|
| `mutual_gaze` | gaze_trace ↔ gaze_trace | Two people looking at each other |
|
||||||
|
| `lip_sync` | lip_trace ↔ text_trace | Lip movement aligned with spoken text |
|
||||||
|
| `has_appearance` | face_trace ↔ appearance_trace | Face has specific appearance |
|
||||||
|
| `wears` | face_trace ↔ accessory | Face wears an accessory |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `POST /api/v1/file/:file_uuid/tkg/rebuild`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Rebuild the Temporal Knowledge Graph for a file. Reads processor JSON outputs (face, yolo, ocr, pose, asrx, gaze, lip, appearance) and generates TKG nodes and edges. Clears existing nodes/edges for the file first, then rebuilds from scratch.
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/rebuild" \
|
||||||
|
-H "X-API-Key: $KEY"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
|
||||||
|
"result": {
|
||||||
|
"face_trace_nodes": 16,
|
||||||
|
"gaze_trace_nodes": 16,
|
||||||
|
"lip_trace_nodes": 12,
|
||||||
|
"text_trace_nodes": 24,
|
||||||
|
"appearance_trace_nodes": 8,
|
||||||
|
"skin_tone_trace_nodes": 5,
|
||||||
|
"accessory_nodes": 3,
|
||||||
|
"object_nodes": 26,
|
||||||
|
"speaker_nodes": 4,
|
||||||
|
"co_occurrence_edges": 94,
|
||||||
|
"speaker_face_edges": 12,
|
||||||
|
"face_face_edges": 8,
|
||||||
|
"mutual_gaze_edges": 2,
|
||||||
|
"lip_sync_edges": 10,
|
||||||
|
"has_appearance_edges": 16,
|
||||||
|
"wears_edges": 3
|
||||||
|
},
|
||||||
|
"error": null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `success` | boolean | True if rebuild completed |
|
||||||
|
| `file_uuid` | string | 32-char hex UUID |
|
||||||
|
| `result` | object | Node and edge counts by type |
|
||||||
|
| `error` | string/null | Error message if failed |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `POST /api/v1/file/:file_uuid/tkg/nodes`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Query TKG nodes with pagination and optional type filter.
|
||||||
|
|
||||||
|
#### Request Parameters
|
||||||
|
|
||||||
|
| Field | Type | Required | Default | Description |
|
||||||
|
|-------|------|----------|---------|-------------|
|
||||||
|
| `node_type` | string | No | all | Filter by node type: `face_trace`, `gaze_trace`, `lip_trace`, `text_trace`, `appearance_trace`, `skin_tone_trace`, `accessory`, `object`, `speaker` |
|
||||||
|
| `page` | integer | No | 1 | Page number |
|
||||||
|
| `page_size` | integer | No | 100 | Items per page (max 500) |
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Get all face_trace nodes
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/nodes" \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"node_type": "face_trace", "page": 1, "page_size": 50}'
|
||||||
|
|
||||||
|
# Get all nodes
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/nodes" \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{}'
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
|
||||||
|
"total": 16,
|
||||||
|
"page": 1,
|
||||||
|
"page_size": 50,
|
||||||
|
"nodes": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"node_type": "face_trace",
|
||||||
|
"external_id": "trace_0",
|
||||||
|
"label": "Face Trace 0",
|
||||||
|
"properties": {
|
||||||
|
"trace_id": 0,
|
||||||
|
"face_count": 142,
|
||||||
|
"avg_confidence": 0.87
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `success` | boolean | Always true on 200 |
|
||||||
|
| `file_uuid` | string | 32-char hex UUID |
|
||||||
|
| `total` | integer | Total matching node count |
|
||||||
|
| `page` | integer | Current page |
|
||||||
|
| `page_size` | integer | Items per page |
|
||||||
|
| `nodes` | array | Array of node objects |
|
||||||
|
| `nodes[].id` | integer | Database primary key |
|
||||||
|
| `nodes[].node_type` | string | Node type (see table above) |
|
||||||
|
| `nodes[].external_id` | string | External identifier (e.g., `trace_0`, `gaze_1`) |
|
||||||
|
| `nodes[].label` | string | Human-readable label |
|
||||||
|
| `nodes[].properties` | object | Type-specific properties as JSON |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `POST /api/v1/file/:file_uuid/tkg/edges`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Query TKG edges with pagination and optional filters.
|
||||||
|
|
||||||
|
#### Request Parameters
|
||||||
|
|
||||||
|
| Field | Type | Required | Default | Description |
|
||||||
|
|-------|------|----------|---------|-------------|
|
||||||
|
| `edge_type` | string | No | all | Filter by edge type: `co_occurs`, `speaker_face`, `face_face`, `mutual_gaze`, `lip_sync`, `has_appearance`, `wears` |
|
||||||
|
| `source_type` | string | No | — | Filter by source node type |
|
||||||
|
| `target_type` | string | No | — | Filter by target node type |
|
||||||
|
| `page` | integer | No | 1 | Page number |
|
||||||
|
| `page_size` | integer | No | 100 | Items per page (max 500) |
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Get all co_occurrence edges
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/edges" \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"edge_type": "co_occurs"}'
|
||||||
|
|
||||||
|
# Get edges between face_trace and speaker nodes
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/edges" \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"source_type": "speaker", "target_type": "face_trace"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
|
||||||
|
"total": 94,
|
||||||
|
"page": 1,
|
||||||
|
"page_size": 100,
|
||||||
|
"edges": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"edge_type": "co_occurs",
|
||||||
|
"source_node_id": 10,
|
||||||
|
"target_node_id": 15,
|
||||||
|
"properties": {
|
||||||
|
"frame_count": 45,
|
||||||
|
"confidence": 0.92
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `success` | boolean | Always true on 200 |
|
||||||
|
| `file_uuid` | string | 32-char hex UUID |
|
||||||
|
| `total` | integer | Total matching edge count |
|
||||||
|
| `page` | integer | Current page |
|
||||||
|
| `page_size` | integer | Items per page |
|
||||||
|
| `edges` | array | Array of edge objects |
|
||||||
|
| `edges[].id` | integer | Database primary key |
|
||||||
|
| `edges[].edge_type` | string | Edge type |
|
||||||
|
| `edges[].source_node_id` | integer | Source node ID (FK to tkg_nodes) |
|
||||||
|
| `edges[].target_node_id` | integer | Target node ID (FK to tkg_nodes) |
|
||||||
|
| `edges[].properties` | object | Edge-specific properties as JSON |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `GET /api/v1/file/:file_uuid/tkg/node/:node_id`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Get detail for a specific TKG node including its connected edges.
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "$API/api/v1/file/$FILE_UUID/tkg/node/1" \
|
||||||
|
-H "X-API-Key: $KEY"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"node": {
|
||||||
|
"id": 1,
|
||||||
|
"node_type": "face_trace",
|
||||||
|
"external_id": "trace_0",
|
||||||
|
"label": "Face Trace 0",
|
||||||
|
"properties": {
|
||||||
|
"trace_id": 0,
|
||||||
|
"face_count": 142,
|
||||||
|
"avg_confidence": 0.87
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"connected_edges": [
|
||||||
|
{
|
||||||
|
"id": 5,
|
||||||
|
"edge_type": "co_occurs",
|
||||||
|
"source_node_id": 1,
|
||||||
|
"target_node_id": 10,
|
||||||
|
"properties": {"frame_count": 45}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"edge_count": 3
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `success` | boolean | Always true on 200 |
|
||||||
|
| `node` | object | Node detail (same format as nodes query) |
|
||||||
|
| `connected_edges` | array | Edges connected to this node |
|
||||||
|
| `edge_count` | integer | Total connected edge count |
|
||||||
|
|
||||||
|
#### Error Codes
|
||||||
|
|
||||||
|
| HTTP | When |
|
||||||
|
|------|------|
|
||||||
|
| `404` | Node not found |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `GET /api/v1/file/:file_uuid/processor-counts`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Get counts of processor JSON output files for a file. Scans the output directory for `{file_uuid}.{processor}.json` files and extracts frame counts, segment counts, and chunk counts from each file.
|
||||||
|
|
||||||
|
Supports short UUID prefix matching (e.g., `d3f9ae8e` → resolves to full `d3f9ae8e471a1fc4d47022c66091b920`).
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "$API/api/v1/file/$FILE_UUID/processor-counts" \
|
||||||
|
-H "X-API-Key: $KEY"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
|
||||||
|
"output_dir": "/Users/accusys/momentry/output_dev",
|
||||||
|
"processors": [
|
||||||
|
{
|
||||||
|
"processor": "cut",
|
||||||
|
"has_json": true,
|
||||||
|
"frame_count": 5391,
|
||||||
|
"segment_count": null,
|
||||||
|
"chunk_count": null,
|
||||||
|
"last_modified": "2026-06-16T18:48:01.987241061+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"processor": "face",
|
||||||
|
"has_json": true,
|
||||||
|
"frame_count": 1112,
|
||||||
|
"segment_count": null,
|
||||||
|
"chunk_count": null,
|
||||||
|
"last_modified": "2026-06-18T17:21:37.408383765+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"processor": "asrx",
|
||||||
|
"has_json": true,
|
||||||
|
"frame_count": null,
|
||||||
|
"segment_count": 6,
|
||||||
|
"chunk_count": null,
|
||||||
|
"last_modified": "2026-06-18T17:21:40.872063642+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"processor": "story",
|
||||||
|
"has_json": true,
|
||||||
|
"frame_count": null,
|
||||||
|
"segment_count": null,
|
||||||
|
"chunk_count": 12,
|
||||||
|
"last_modified": "2026-06-18T17:22:00.000000000+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"processor": "mediapipe",
|
||||||
|
"has_json": false,
|
||||||
|
"frame_count": null,
|
||||||
|
"segment_count": null,
|
||||||
|
"chunk_count": null,
|
||||||
|
"last_modified": null
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `file_uuid` | string | Full 32-char hex UUID (resolved from prefix) |
|
||||||
|
| `output_dir` | string | Output directory scanned |
|
||||||
|
| `processors` | array | Per-processor output info |
|
||||||
|
| `processors[].processor` | string | Processor name |
|
||||||
|
| `processors[].has_json` | boolean | Whether JSON file exists |
|
||||||
|
| `processors[].frame_count` | integer/null | Total frames processed (frame-based processors) |
|
||||||
|
| `processors[].segment_count` | integer/null | Segment count (ASRX segments, etc.) |
|
||||||
|
| `processors[].chunk_count` | integer/null | Chunk count (Story chunks, etc.) |
|
||||||
|
| `processors[].last_modified` | string/null | ISO 8601 timestamp of last modification |
|
||||||
|
|
||||||
|
#### Error Codes
|
||||||
|
|
||||||
|
| HTTP | When |
|
||||||
|
|------|------|
|
||||||
|
| `404` | File UUID not found in database |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Updated: 2026-06-20 12:00:00*
|
||||||
@@ -0,0 +1,235 @@
|
|||||||
|
---
|
||||||
|
title: Rule 2 TKG Relationship Chunks V1.0
|
||||||
|
version: 1.0
|
||||||
|
date: 2026-06-20
|
||||||
|
author: OpenCode
|
||||||
|
status: approved
|
||||||
|
---
|
||||||
|
|
||||||
|
# Rule 2 TKG Relationship Chunks V1.0
|
||||||
|
|
||||||
|
| Scope | Status | Applicable to | Binary |
|
||||||
|
|-------|--------|---------------|--------|
|
||||||
|
| TKG relationship vectorization | Approved | `momentry_playground`, `momentry` | Both |
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Rule 2 creates **relationship chunks** by converting TKG edges into searchable, vectorized units. Each TKG edge becomes a chunk with LLM-generated natural language description, enabling semantic search for relationship queries.
|
||||||
|
|
||||||
|
**Key Change:** Original Rule 2 (YOLO frame objects) is deprecated due to COCO classes being too generic. New Rule 2 focuses on TKG relationships.
|
||||||
|
|
||||||
|
## Data Flow
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────┐
|
||||||
|
│ UPSTREAM: TKG Builder │
|
||||||
|
│ │
|
||||||
|
│ tkg_nodes: face_trace, speaker, object, etc. │
|
||||||
|
│ tkg_edges: speaker_face, mutual_gaze, co_occurs, etc. │
|
||||||
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼ after TKG complete
|
||||||
|
│
|
||||||
|
┌─────────────────────────────────────────────────────────┐
|
||||||
|
│ RULE 2 PROCESSING │
|
||||||
|
│ │
|
||||||
|
│ Triggered by: │
|
||||||
|
│ 1. Worker auto: job_worker.rs after TKG completes │
|
||||||
|
│ 2. HTTP API: POST /api/v1/file/:file_uuid/rule2 │
|
||||||
|
│ │
|
||||||
|
│ ingest_rule2(file_uuid): │
|
||||||
|
│ ├─ Query tkg_edges by type (priority order) │
|
||||||
|
│ ├─ For each edge: │
|
||||||
|
│ │ ├─ Resolve source_node / target_node │
|
||||||
|
│ │ ├─ Resolve identity names (if face_trace) │
|
||||||
|
│ │ ├─ Build context JSON │
|
||||||
|
│ │ ├─ call_llm(context) → text_content │
|
||||||
|
│ │ └─ INSERT INTO chunk (chunk_type='relationship') │
|
||||||
|
│ │ │
|
||||||
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────┐
|
||||||
|
│ DOWNSTREAM: vectorize_chunks() │
|
||||||
|
│ │
|
||||||
|
│ SELECT ... WHERE chunk_type='relationship' │
|
||||||
|
│ AND embedding IS NULL │
|
||||||
|
│ │
|
||||||
|
│ 1. embedder.embed_document(text_content) → vector │
|
||||||
|
│ 2. db.store_vector() → PG chunk.embedding │
|
||||||
|
│ 3. qdrant.upsert_vector() → momentry_rule2 collection │
|
||||||
|
│ │
|
||||||
|
└─────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Edge Type Priority
|
||||||
|
|
||||||
|
| Priority | Edge Type | Description | Example Output |
|
||||||
|
|----------|-----------|-------------|----------------|
|
||||||
|
| P0 | `speaker_face` | Speaker ↔ Face trace | "SPEAKER_01 以 Cary Grant 的身份說話,從 frame 100 到 350" |
|
||||||
|
| P0 | `mutual_gaze` | Two face traces looking at each other | "Cary Grant 和 Grace Kelly 互相看對方 24 幀,起始於 frame 450" |
|
||||||
|
| P1 | `face_face` | Two face traces co-occurring | "Cary Grant 和 Grace Kelly 同框 180 幀" |
|
||||||
|
| P1 | `co_occurs` | Object ↔ Object co-occurrence | "物件 'car' 和 'person' 在同一畫面出現 60 幀" |
|
||||||
|
| P2 | `has_appearance` | Face trace ↔ Appearance trace | "Cary Grant 穿著藍色上衣,戴眼鏡" |
|
||||||
|
| P2 | `wears` | Face trace ↔ Accessory | "Cary Grant 戴帽子,信心值 0.82" |
|
||||||
|
|
||||||
|
## Chunk Data Structure
|
||||||
|
|
||||||
|
### Content JSON (`content` column)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"edge_type": "speaker_face",
|
||||||
|
"edge_id": 123,
|
||||||
|
"source_node": {
|
||||||
|
"id": 45,
|
||||||
|
"node_type": "speaker",
|
||||||
|
"external_id": "SPEAKER_01",
|
||||||
|
"label": "SPEAKER_01"
|
||||||
|
},
|
||||||
|
"target_node": {
|
||||||
|
"id": 67,
|
||||||
|
"node_type": "face_trace",
|
||||||
|
"external_id": "trace_5",
|
||||||
|
"label": "Face Trace 5",
|
||||||
|
"identity_name": "Cary Grant"
|
||||||
|
},
|
||||||
|
"properties": {
|
||||||
|
"first_frame": 100,
|
||||||
|
"last_frame": 350,
|
||||||
|
"frame_count": 250,
|
||||||
|
"lip_sync_confidence": 0.85
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Text Content (`text_content` column)
|
||||||
|
|
||||||
|
LLM-generated natural language description in Traditional Chinese:
|
||||||
|
|
||||||
|
```
|
||||||
|
"SPEAKER_01 以 Cary Grant 的身份說話,從 frame 100 到 frame 350,唇語同步信心值 0.85"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Metadata JSON (`metadata` column)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"source_type": "speaker",
|
||||||
|
"target_type": "face_trace",
|
||||||
|
"has_identity": true,
|
||||||
|
"identity_source": "tmdb"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## LLM Prompt Template
|
||||||
|
|
||||||
|
```text
|
||||||
|
你是影片關係描述專家。請用繁體中文描述以下人物/物件關係:
|
||||||
|
|
||||||
|
關係類型: {edge_type}
|
||||||
|
來源節點: {source_node.node_type} - {source_node.external_id}
|
||||||
|
身份名稱: {identity_name} (如果有)
|
||||||
|
目標節點: {target_node.node_type} - {target_node.external_id}
|
||||||
|
身份名稱: {identity_name} (如果有)
|
||||||
|
關係屬性:
|
||||||
|
- 起始幀: {first_frame}
|
||||||
|
- 結束幀: {last_frame}
|
||||||
|
- 幀數: {frame_count}
|
||||||
|
- 信心值: {confidence}
|
||||||
|
|
||||||
|
要求:
|
||||||
|
1. 使用自然語言,不要輸出 JSON
|
||||||
|
2. 包含時間範圍(幀號)
|
||||||
|
3. 包含人物名字(如有 identity)
|
||||||
|
4. 簡潔,20-50 字
|
||||||
|
5. 用繁體中文
|
||||||
|
|
||||||
|
範例輸出:
|
||||||
|
"SPEAKER_01 以 Cary Grant 的身份說話,從 frame 100 到 frame 350"
|
||||||
|
"Cary Grant 和 Grace Kelly 互相看對方 24 幀,起始於 frame 450"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Edge → Chunk Conversion Rules
|
||||||
|
|
||||||
|
### speaker_face Edge
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// Source: speaker node
|
||||||
|
// Target: face_trace node
|
||||||
|
// Properties: first_frame, last_frame, lip_sync_confidence
|
||||||
|
|
||||||
|
let text_content = call_llm(format!(
|
||||||
|
"SPEAKER {} 對應 face trace {},身份 {},frame {}-{}",
|
||||||
|
speaker_id, trace_id, identity_name, first_frame, last_frame
|
||||||
|
));
|
||||||
|
```
|
||||||
|
|
||||||
|
### mutual_gaze Edge
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// Source: face_trace node A
|
||||||
|
// Target: face_trace node B
|
||||||
|
// Properties: first_frame, gaze_frame_count, yaw_a_avg, yaw_b_avg
|
||||||
|
|
||||||
|
let text_content = call_llm(format!(
|
||||||
|
"人物 {} 和 {} 互相看對方 {} 幀,起始於 frame {}",
|
||||||
|
identity_a, identity_b, gaze_frame_count, first_frame
|
||||||
|
));
|
||||||
|
```
|
||||||
|
|
||||||
|
### has_appearance Edge
|
||||||
|
|
||||||
|
```rust
|
||||||
|
// Source: face_trace node
|
||||||
|
// Target: appearance_trace node
|
||||||
|
// Properties: clothing colors, accessories
|
||||||
|
|
||||||
|
let text_content = call_llm(format!(
|
||||||
|
"人物 {} 穿著 {} 上衣,{} 下衣",
|
||||||
|
identity_name, upper_color, lower_color
|
||||||
|
));
|
||||||
|
```
|
||||||
|
|
||||||
|
## Search Contribution
|
||||||
|
|
||||||
|
| Search Path | Mechanism | Rule 2 Contribution |
|
||||||
|
|-------------|-----------|-------------------|
|
||||||
|
| **Semantic search** (Qdrant) | `chunk_type='relationship'` → embedding query | LLM descriptions enable natural language queries |
|
||||||
|
| **Keyword search** (BM25 ILIKE) | `text_content ILIKE '%互相看%'` | Relationship keywords searchable |
|
||||||
|
| **Agent tkg_query** | Direct edge queries | Rule 2 complements with vectorized search |
|
||||||
|
| **identity_text** | Reverse lookup | "誰戴眼鏡" → has_appearance chunks |
|
||||||
|
|
||||||
|
## Trigger Points
|
||||||
|
|
||||||
|
| Trigger | Location | Condition |
|
||||||
|
|---------|----------|-----------|
|
||||||
|
| Worker auto | `job_worker.rs` | After TKG builder completes |
|
||||||
|
| HTTP API | `POST /api/v1/file/:file_uuid/rule2` | Manual trigger |
|
||||||
|
| Pipeline | `pipeline_core::execute_rule2` | Called by other modules |
|
||||||
|
|
||||||
|
## Edge Cases
|
||||||
|
|
||||||
|
| Scenario | Behavior |
|
||||||
|
|----------|----------|
|
||||||
|
| No tkg_edges | Returns 0 immediately with info log |
|
||||||
|
| Edge without identity | Use node external_id (e.g., "trace_5") in description |
|
||||||
|
| LLM call fails | Fallback to template-based description |
|
||||||
|
| Multiple edges same type | Each edge becomes separate chunk |
|
||||||
|
|
||||||
|
## Qdrant Collection
|
||||||
|
|
||||||
|
| Property | Value |
|
||||||
|
|----------|-------|
|
||||||
|
| Collection name | `momentry_rule2` |
|
||||||
|
| Vector size | 768 (nomic-embed-text-v2-moe) |
|
||||||
|
| Distance | Cosine |
|
||||||
|
| Payload | `{chunk_id, file_uuid, edge_type, source_type, target_type}` |
|
||||||
|
|
||||||
|
## Version History
|
||||||
|
|
||||||
|
| Version | Date | Author | Change |
|
||||||
|
|---------|------|--------|--------|
|
||||||
|
| 1.0 | 2026-06-20 | OpenCode | Initial design: TKG edges → relationship chunks |
|
||||||
@@ -0,0 +1,378 @@
|
|||||||
|
<!-- module: tkg -->
|
||||||
|
<!-- description: Temporal Knowledge Graph — rebuild, nodes, edges, processor counts -->
|
||||||
|
<!-- depends: 05_process, 07_identity -->
|
||||||
|
|
||||||
|
## Temporal Knowledge Graph (TKG)
|
||||||
|
|
||||||
|
TKG is a time-aligned knowledge graph built from multi-processor outputs (face, yolo, ocr, pose, asrx, gaze, lip, appearance). It produces 9 node types and 14 edge types stored in `dev.tkg_nodes` and `dev.tkg_edges`.
|
||||||
|
|
||||||
|
### Node Types
|
||||||
|
|
||||||
|
| Node Type | Description | Key Properties |
|
||||||
|
|-----------|-------------|----------------|
|
||||||
|
| `face_trace` | A tracked face identity over time | `trace_id`, `face_count`, `avg_confidence` |
|
||||||
|
| `gaze_trace` | Gaze direction over time | `direction` (frontal/left/right/up/down + diagonals) |
|
||||||
|
| `lip_trace` | Lip movement synced with speech | `speaker_id`, `lip_area_range` |
|
||||||
|
| `text_trace` | Spoken text aligned to time | `speaker_id`, `text`, `start_time`, `end_time` |
|
||||||
|
| `appearance_trace` | Human appearance (clothing) over time | `clothing_color`, `upper_cloth`, `lower_cloth` |
|
||||||
|
| `skin_tone_trace` | Fitzpatrick skin tone classification | `fitzpatrick_type` (I–VI) |
|
||||||
|
| `accessory` | Detected accessories | `type` (glasses/hat/etc.), `confidence` |
|
||||||
|
| `object` | YOLO-detected object | `class`, `confidence`, `frame_count` |
|
||||||
|
| `speaker` | ASRX speaker segment | `speaker_id`, `segment_count`, `total_duration` |
|
||||||
|
|
||||||
|
### Edge Types
|
||||||
|
|
||||||
|
| Edge Type | Source → Target | Description |
|
||||||
|
|-----------|-----------------|-------------|
|
||||||
|
| `co_occurs` | object ↔ object | Two objects appear together in same frame |
|
||||||
|
| `speaker_face` | speaker ↔ face_trace | Speaker matched to face trace via lip sync |
|
||||||
|
| `face_face` | face_trace ↔ face_trace | Two face traces interact (mutual gaze) |
|
||||||
|
| `mutual_gaze` | gaze_trace ↔ gaze_trace | Two people looking at each other |
|
||||||
|
| `lip_sync` | lip_trace ↔ text_trace | Lip movement aligned with spoken text |
|
||||||
|
| `has_appearance` | face_trace ↔ appearance_trace | Face has specific appearance |
|
||||||
|
| `wears` | face_trace ↔ accessory | Face wears an accessory |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `POST /api/v1/file/:file_uuid/tkg/rebuild`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Rebuild the Temporal Knowledge Graph for a file. Reads processor JSON outputs (face, yolo, ocr, pose, asrx, gaze, lip, appearance) and generates TKG nodes and edges. Clears existing nodes/edges for the file first, then rebuilds from scratch.
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/rebuild" \
|
||||||
|
-H "X-API-Key: $KEY"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
|
||||||
|
"result": {
|
||||||
|
"face_trace_nodes": 16,
|
||||||
|
"gaze_trace_nodes": 16,
|
||||||
|
"lip_trace_nodes": 12,
|
||||||
|
"text_trace_nodes": 24,
|
||||||
|
"appearance_trace_nodes": 8,
|
||||||
|
"skin_tone_trace_nodes": 5,
|
||||||
|
"accessory_nodes": 3,
|
||||||
|
"object_nodes": 26,
|
||||||
|
"speaker_nodes": 4,
|
||||||
|
"co_occurrence_edges": 94,
|
||||||
|
"speaker_face_edges": 12,
|
||||||
|
"face_face_edges": 8,
|
||||||
|
"mutual_gaze_edges": 2,
|
||||||
|
"lip_sync_edges": 10,
|
||||||
|
"has_appearance_edges": 16,
|
||||||
|
"wears_edges": 3
|
||||||
|
},
|
||||||
|
"error": null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `success` | boolean | True if rebuild completed |
|
||||||
|
| `file_uuid` | string | 32-char hex UUID |
|
||||||
|
| `result` | object | Node and edge counts by type |
|
||||||
|
| `error` | string/null | Error message if failed |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `POST /api/v1/file/:file_uuid/tkg/nodes`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Query TKG nodes with pagination and optional type filter.
|
||||||
|
|
||||||
|
#### Request Parameters
|
||||||
|
|
||||||
|
| Field | Type | Required | Default | Description |
|
||||||
|
|-------|------|----------|---------|-------------|
|
||||||
|
| `node_type` | string | No | all | Filter by node type: `face_trace`, `gaze_trace`, `lip_trace`, `text_trace`, `appearance_trace`, `skin_tone_trace`, `accessory`, `object`, `speaker` |
|
||||||
|
| `page` | integer | No | 1 | Page number |
|
||||||
|
| `page_size` | integer | No | 100 | Items per page (max 500) |
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Get all face_trace nodes
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/nodes" \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"node_type": "face_trace", "page": 1, "page_size": 50}'
|
||||||
|
|
||||||
|
# Get all nodes
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/nodes" \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{}'
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
|
||||||
|
"total": 16,
|
||||||
|
"page": 1,
|
||||||
|
"page_size": 50,
|
||||||
|
"nodes": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"node_type": "face_trace",
|
||||||
|
"external_id": "trace_0",
|
||||||
|
"label": "Face Trace 0",
|
||||||
|
"properties": {
|
||||||
|
"trace_id": 0,
|
||||||
|
"face_count": 142,
|
||||||
|
"avg_confidence": 0.87
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `success` | boolean | Always true on 200 |
|
||||||
|
| `file_uuid` | string | 32-char hex UUID |
|
||||||
|
| `total` | integer | Total matching node count |
|
||||||
|
| `page` | integer | Current page |
|
||||||
|
| `page_size` | integer | Items per page |
|
||||||
|
| `nodes` | array | Array of node objects |
|
||||||
|
| `nodes[].id` | integer | Database primary key |
|
||||||
|
| `nodes[].node_type` | string | Node type (see table above) |
|
||||||
|
| `nodes[].external_id` | string | External identifier (e.g., `trace_0`, `gaze_1`) |
|
||||||
|
| `nodes[].label` | string | Human-readable label |
|
||||||
|
| `nodes[].properties` | object | Type-specific properties as JSON |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `POST /api/v1/file/:file_uuid/tkg/edges`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Query TKG edges with pagination and optional filters.
|
||||||
|
|
||||||
|
#### Request Parameters
|
||||||
|
|
||||||
|
| Field | Type | Required | Default | Description |
|
||||||
|
|-------|------|----------|---------|-------------|
|
||||||
|
| `edge_type` | string | No | all | Filter by edge type: `co_occurs`, `speaker_face`, `face_face`, `mutual_gaze`, `lip_sync`, `has_appearance`, `wears` |
|
||||||
|
| `source_type` | string | No | — | Filter by source node type |
|
||||||
|
| `target_type` | string | No | — | Filter by target node type |
|
||||||
|
| `page` | integer | No | 1 | Page number |
|
||||||
|
| `page_size` | integer | No | 100 | Items per page (max 500) |
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Get all co_occurrence edges
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/edges" \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"edge_type": "co_occurs"}'
|
||||||
|
|
||||||
|
# Get edges between face_trace and speaker nodes
|
||||||
|
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/edges" \
|
||||||
|
-H "X-API-Key: $KEY" \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"source_type": "speaker", "target_type": "face_trace"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
|
||||||
|
"total": 94,
|
||||||
|
"page": 1,
|
||||||
|
"page_size": 100,
|
||||||
|
"edges": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"edge_type": "co_occurs",
|
||||||
|
"source_node_id": 10,
|
||||||
|
"target_node_id": 15,
|
||||||
|
"properties": {
|
||||||
|
"frame_count": 45,
|
||||||
|
"confidence": 0.92
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `success` | boolean | Always true on 200 |
|
||||||
|
| `file_uuid` | string | 32-char hex UUID |
|
||||||
|
| `total` | integer | Total matching edge count |
|
||||||
|
| `page` | integer | Current page |
|
||||||
|
| `page_size` | integer | Items per page |
|
||||||
|
| `edges` | array | Array of edge objects |
|
||||||
|
| `edges[].id` | integer | Database primary key |
|
||||||
|
| `edges[].edge_type` | string | Edge type |
|
||||||
|
| `edges[].source_node_id` | integer | Source node ID (FK to tkg_nodes) |
|
||||||
|
| `edges[].target_node_id` | integer | Target node ID (FK to tkg_nodes) |
|
||||||
|
| `edges[].properties` | object | Edge-specific properties as JSON |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `GET /api/v1/file/:file_uuid/tkg/node/:node_id`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Get detail for a specific TKG node including its connected edges.
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "$API/api/v1/file/$FILE_UUID/tkg/node/1" \
|
||||||
|
-H "X-API-Key: $KEY"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"node": {
|
||||||
|
"id": 1,
|
||||||
|
"node_type": "face_trace",
|
||||||
|
"external_id": "trace_0",
|
||||||
|
"label": "Face Trace 0",
|
||||||
|
"properties": {
|
||||||
|
"trace_id": 0,
|
||||||
|
"face_count": 142,
|
||||||
|
"avg_confidence": 0.87
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"connected_edges": [
|
||||||
|
{
|
||||||
|
"id": 5,
|
||||||
|
"edge_type": "co_occurs",
|
||||||
|
"source_node_id": 1,
|
||||||
|
"target_node_id": 10,
|
||||||
|
"properties": {"frame_count": 45}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"edge_count": 3
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `success` | boolean | Always true on 200 |
|
||||||
|
| `node` | object | Node detail (same format as nodes query) |
|
||||||
|
| `connected_edges` | array | Edges connected to this node |
|
||||||
|
| `edge_count` | integer | Total connected edge count |
|
||||||
|
|
||||||
|
#### Error Codes
|
||||||
|
|
||||||
|
| HTTP | When |
|
||||||
|
|------|------|
|
||||||
|
| `404` | Node not found |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `GET /api/v1/file/:file_uuid/processor-counts`
|
||||||
|
|
||||||
|
**Auth**: Required
|
||||||
|
**Scope**: file-level
|
||||||
|
|
||||||
|
Get counts of processor JSON output files for a file. Scans the output directory for `{file_uuid}.{processor}.json` files and extracts frame counts, segment counts, and chunk counts from each file.
|
||||||
|
|
||||||
|
Supports short UUID prefix matching (e.g., `d3f9ae8e` → resolves to full `d3f9ae8e471a1fc4d47022c66091b920`).
|
||||||
|
|
||||||
|
#### Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -s "$API/api/v1/file/$FILE_UUID/processor-counts" \
|
||||||
|
-H "X-API-Key: $KEY"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Response (200)
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
|
||||||
|
"output_dir": "/Users/accusys/momentry/output_dev",
|
||||||
|
"processors": [
|
||||||
|
{
|
||||||
|
"processor": "cut",
|
||||||
|
"has_json": true,
|
||||||
|
"frame_count": 5391,
|
||||||
|
"segment_count": null,
|
||||||
|
"chunk_count": null,
|
||||||
|
"last_modified": "2026-06-16T18:48:01.987241061+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"processor": "face",
|
||||||
|
"has_json": true,
|
||||||
|
"frame_count": 1112,
|
||||||
|
"segment_count": null,
|
||||||
|
"chunk_count": null,
|
||||||
|
"last_modified": "2026-06-18T17:21:37.408383765+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"processor": "asrx",
|
||||||
|
"has_json": true,
|
||||||
|
"frame_count": null,
|
||||||
|
"segment_count": 6,
|
||||||
|
"chunk_count": null,
|
||||||
|
"last_modified": "2026-06-18T17:21:40.872063642+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"processor": "story",
|
||||||
|
"has_json": true,
|
||||||
|
"frame_count": null,
|
||||||
|
"segment_count": null,
|
||||||
|
"chunk_count": 12,
|
||||||
|
"last_modified": "2026-06-18T17:22:00.000000000+00:00"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"processor": "mediapipe",
|
||||||
|
"has_json": false,
|
||||||
|
"frame_count": null,
|
||||||
|
"segment_count": null,
|
||||||
|
"chunk_count": null,
|
||||||
|
"last_modified": null
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `file_uuid` | string | Full 32-char hex UUID (resolved from prefix) |
|
||||||
|
| `output_dir` | string | Output directory scanned |
|
||||||
|
| `processors` | array | Per-processor output info |
|
||||||
|
| `processors[].processor` | string | Processor name |
|
||||||
|
| `processors[].has_json` | boolean | Whether JSON file exists |
|
||||||
|
| `processors[].frame_count` | integer/null | Total frames processed (frame-based processors) |
|
||||||
|
| `processors[].segment_count` | integer/null | Segment count (ASRX segments, etc.) |
|
||||||
|
| `processors[].chunk_count` | integer/null | Chunk count (Story chunks, etc.) |
|
||||||
|
| `processors[].last_modified` | string/null | ISO 8601 timestamp of last modification |
|
||||||
|
|
||||||
|
#### Error Codes
|
||||||
|
|
||||||
|
| HTTP | When |
|
||||||
|
|------|------|
|
||||||
|
| `404` | File UUID not found in database |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
*Updated: 2026-06-20 12:00:00*
|
||||||
+510
-13
@@ -38,10 +38,17 @@ pub fn trace_agent_routes() -> Router<crate::api::types::AppState> {
|
|||||||
get(get_cooccurrence),
|
get(get_cooccurrence),
|
||||||
)
|
)
|
||||||
.route("/api/v1/file/:file_uuid/tkg/rebuild", post(rebuild_tkg))
|
.route("/api/v1/file/:file_uuid/tkg/rebuild", post(rebuild_tkg))
|
||||||
|
.route("/api/v1/file/:file_uuid/rule2", post(ingest_rule2))
|
||||||
.route(
|
.route(
|
||||||
"/api/v1/file/:file_uuid/representative-frame",
|
"/api/v1/file/:file_uuid/representative-frame",
|
||||||
get(get_representative_frame),
|
get(get_representative_frame),
|
||||||
)
|
)
|
||||||
|
.route("/api/v1/file/:file_uuid/tkg/nodes", post(query_tkg_nodes))
|
||||||
|
.route("/api/v1/file/:file_uuid/tkg/edges", post(query_tkg_edges))
|
||||||
|
.route(
|
||||||
|
"/api/v1/file/:file_uuid/tkg/node/:node_id",
|
||||||
|
get(get_tkg_node_detail),
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Debug, Deserialize)]
|
#[derive(Debug, Deserialize)]
|
||||||
@@ -961,22 +968,52 @@ async fn rebuild_tkg(
|
|||||||
State(state): State<crate::api::types::AppState>,
|
State(state): State<crate::api::types::AppState>,
|
||||||
Path(file_uuid): Path<String>,
|
Path(file_uuid): Path<String>,
|
||||||
) -> Json<TkgRebuildResponse> {
|
) -> Json<TkgRebuildResponse> {
|
||||||
|
use crate::core::chunk::rule2_ingest::ingest_rule2;
|
||||||
|
use tracing::info;
|
||||||
|
|
||||||
let result = crate::core::processor::tkg::build_tkg(&state.db, &file_uuid, &OUTPUT_DIR).await;
|
let result = crate::core::processor::tkg::build_tkg(&state.db, &file_uuid, &OUTPUT_DIR).await;
|
||||||
|
|
||||||
match result {
|
match result {
|
||||||
Ok(r) => Json(TkgRebuildResponse {
|
Ok(r) => {
|
||||||
success: true,
|
let total_edges = r.speaker_face_edges
|
||||||
file_uuid,
|
+ r.mutual_gaze_edges
|
||||||
result: Some(serde_json::json!({
|
+ r.face_face_edges
|
||||||
"face_trace_nodes": r.face_trace_nodes,
|
+ r.co_occurrence_edges
|
||||||
"object_nodes": r.object_nodes,
|
+ r.has_appearance_edges
|
||||||
"speaker_nodes": r.speaker_nodes,
|
+ r.wears_edges;
|
||||||
"co_occurrence_edges": r.co_occurrence_edges,
|
|
||||||
"speaker_face_edges": r.speaker_face_edges,
|
if total_edges > 0 {
|
||||||
"face_face_edges": r.face_face_edges,
|
info!("[TKG] {} relationship edges found, triggering Rule 2 ingestion...", total_edges);
|
||||||
})),
|
match ingest_rule2(state.db.pool(), &file_uuid).await {
|
||||||
error: None,
|
Ok(count) => info!("[TKG] Rule 2 created {} relationship chunks", count),
|
||||||
}),
|
Err(e) => info!("[TKG] Rule 2 ingestion failed: {}", e),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Json(TkgRebuildResponse {
|
||||||
|
success: true,
|
||||||
|
file_uuid,
|
||||||
|
result: Some(serde_json::json!({
|
||||||
|
"face_trace_nodes": r.face_trace_nodes,
|
||||||
|
"gaze_trace_nodes": r.gaze_trace_nodes,
|
||||||
|
"lip_trace_nodes": r.lip_trace_nodes,
|
||||||
|
"text_trace_nodes": r.text_trace_nodes,
|
||||||
|
"appearance_trace_nodes": r.appearance_trace_nodes,
|
||||||
|
"skin_tone_trace_nodes": r.skin_tone_trace_nodes,
|
||||||
|
"accessory_nodes": r.accessory_nodes,
|
||||||
|
"object_nodes": r.object_nodes,
|
||||||
|
"speaker_nodes": r.speaker_nodes,
|
||||||
|
"co_occurrence_edges": r.co_occurrence_edges,
|
||||||
|
"speaker_face_edges": r.speaker_face_edges,
|
||||||
|
"face_face_edges": r.face_face_edges,
|
||||||
|
"mutual_gaze_edges": r.mutual_gaze_edges,
|
||||||
|
"lip_sync_edges": r.lip_sync_edges,
|
||||||
|
"has_appearance_edges": r.has_appearance_edges,
|
||||||
|
"wears_edges": r.wears_edges,
|
||||||
|
})),
|
||||||
|
error: None,
|
||||||
|
})
|
||||||
|
}
|
||||||
Err(e) => Json(TkgRebuildResponse {
|
Err(e) => Json(TkgRebuildResponse {
|
||||||
success: false,
|
success: false,
|
||||||
file_uuid,
|
file_uuid,
|
||||||
@@ -1097,3 +1134,463 @@ async fn get_stranger_thumbnail(
|
|||||||
|
|
||||||
get_trace_thumbnail_inner(&state, &file_uuid, trace_id).await
|
get_trace_thumbnail_inner(&state, &file_uuid, trace_id).await
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── TKG Node/Edge Query APIs ─────────────────────────────────────
|
||||||
|
|
||||||
|
fn t(name: &str) -> String {
|
||||||
|
let schema = std::env::var("DATABASE_SCHEMA").unwrap_or_else(|_| "dev".to_string());
|
||||||
|
if schema == "public" {
|
||||||
|
name.to_string()
|
||||||
|
} else {
|
||||||
|
format!("{}.{}", schema, name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Deserialize)]
|
||||||
|
struct TkgNodesRequest {
|
||||||
|
node_type: Option<String>,
|
||||||
|
page: Option<i64>,
|
||||||
|
page_size: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize)]
|
||||||
|
struct TkgNodeInfo {
|
||||||
|
id: i64,
|
||||||
|
node_type: String,
|
||||||
|
external_id: String,
|
||||||
|
label: String,
|
||||||
|
properties: serde_json::Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize)]
|
||||||
|
struct TkgNodesResponse {
|
||||||
|
success: bool,
|
||||||
|
file_uuid: String,
|
||||||
|
total: i64,
|
||||||
|
page: i64,
|
||||||
|
page_size: i64,
|
||||||
|
nodes: Vec<TkgNodeInfo>,
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn query_tkg_nodes(
|
||||||
|
State(state): State<crate::api::types::AppState>,
|
||||||
|
Path(file_uuid): Path<String>,
|
||||||
|
Json(req): Json<TkgNodesRequest>,
|
||||||
|
) -> Result<Json<TkgNodesResponse>, (StatusCode, Json<serde_json::Value>)> {
|
||||||
|
let nodes_table = t("tkg_nodes");
|
||||||
|
let page = req.page.unwrap_or(1).max(1);
|
||||||
|
let page_size = req.page_size.unwrap_or(100).max(1).min(500);
|
||||||
|
let offset = (page - 1) * page_size;
|
||||||
|
|
||||||
|
let (where_clause, count_args, query_args) = if let Some(ref node_type) = req.node_type {
|
||||||
|
(
|
||||||
|
"WHERE file_uuid = $1 AND node_type = $2".to_string(),
|
||||||
|
vec![serde_json::json!([&file_uuid, node_type])],
|
||||||
|
vec![serde_json::json!([
|
||||||
|
&file_uuid, node_type, page_size, offset
|
||||||
|
])],
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
(
|
||||||
|
"WHERE file_uuid = $1".to_string(),
|
||||||
|
vec![serde_json::json!([&file_uuid])],
|
||||||
|
vec![serde_json::json!([&file_uuid, page_size, offset])],
|
||||||
|
)
|
||||||
|
};
|
||||||
|
|
||||||
|
let total: i64 = if let Some(ref node_type) = req.node_type {
|
||||||
|
sqlx::query_scalar(&format!(
|
||||||
|
"SELECT COUNT(*) FROM {} {}",
|
||||||
|
nodes_table, where_clause
|
||||||
|
))
|
||||||
|
.bind(&file_uuid)
|
||||||
|
.bind(node_type)
|
||||||
|
.fetch_one(state.db.pool())
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
(
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
Json(serde_json::json!({"error": e.to_string()})),
|
||||||
|
)
|
||||||
|
})?
|
||||||
|
} else {
|
||||||
|
sqlx::query_scalar(&format!(
|
||||||
|
"SELECT COUNT(*) FROM {} {}",
|
||||||
|
nodes_table, where_clause
|
||||||
|
))
|
||||||
|
.bind(&file_uuid)
|
||||||
|
.fetch_one(state.db.pool())
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
(
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
Json(serde_json::json!({"error": e.to_string()})),
|
||||||
|
)
|
||||||
|
})?
|
||||||
|
};
|
||||||
|
|
||||||
|
let query = format!(
|
||||||
|
"SELECT id, node_type, external_id, label, properties FROM {} {} ORDER BY id LIMIT ${} OFFSET ${}",
|
||||||
|
nodes_table, where_clause,
|
||||||
|
if req.node_type.is_some() { 3 } else { 2 },
|
||||||
|
if req.node_type.is_some() { 4 } else { 3 }
|
||||||
|
);
|
||||||
|
|
||||||
|
let rows: Vec<(i64, String, String, String, serde_json::Value)> =
|
||||||
|
if let Some(ref node_type) = req.node_type {
|
||||||
|
sqlx::query_as(&query)
|
||||||
|
.bind(&file_uuid)
|
||||||
|
.bind(node_type)
|
||||||
|
.bind(page_size)
|
||||||
|
.bind(offset)
|
||||||
|
.fetch_all(state.db.pool())
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
(
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
Json(serde_json::json!({"error": e.to_string()})),
|
||||||
|
)
|
||||||
|
})?
|
||||||
|
} else {
|
||||||
|
sqlx::query_as(&query)
|
||||||
|
.bind(&file_uuid)
|
||||||
|
.bind(page_size)
|
||||||
|
.bind(offset)
|
||||||
|
.fetch_all(state.db.pool())
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
(
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
Json(serde_json::json!({"error": e.to_string()})),
|
||||||
|
)
|
||||||
|
})?
|
||||||
|
};
|
||||||
|
|
||||||
|
let nodes = rows
|
||||||
|
.into_iter()
|
||||||
|
.map(
|
||||||
|
|(id, node_type, external_id, label, properties)| TkgNodeInfo {
|
||||||
|
id,
|
||||||
|
node_type,
|
||||||
|
external_id,
|
||||||
|
label,
|
||||||
|
properties,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
Ok(Json(TkgNodesResponse {
|
||||||
|
success: true,
|
||||||
|
file_uuid,
|
||||||
|
total,
|
||||||
|
page,
|
||||||
|
page_size,
|
||||||
|
nodes,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Deserialize)]
|
||||||
|
struct TkgEdgesRequest {
|
||||||
|
edge_type: Option<String>,
|
||||||
|
source_type: Option<String>,
|
||||||
|
target_type: Option<String>,
|
||||||
|
page: Option<i64>,
|
||||||
|
page_size: Option<i64>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize)]
|
||||||
|
struct TkgEdgeInfo {
|
||||||
|
id: i64,
|
||||||
|
edge_type: String,
|
||||||
|
source_node_id: i64,
|
||||||
|
target_node_id: i64,
|
||||||
|
properties: serde_json::Value,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize)]
|
||||||
|
struct TkgEdgesResponse {
|
||||||
|
success: bool,
|
||||||
|
file_uuid: String,
|
||||||
|
total: i64,
|
||||||
|
page: i64,
|
||||||
|
page_size: i64,
|
||||||
|
edges: Vec<TkgEdgeInfo>,
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn query_tkg_edges(
|
||||||
|
State(state): State<crate::api::types::AppState>,
|
||||||
|
Path(file_uuid): Path<String>,
|
||||||
|
Json(req): Json<TkgEdgesRequest>,
|
||||||
|
) -> Result<Json<TkgEdgesResponse>, (StatusCode, Json<serde_json::Value>)> {
|
||||||
|
let edges_table = t("tkg_edges");
|
||||||
|
let nodes_table = t("tkg_nodes");
|
||||||
|
let page = req.page.unwrap_or(1).max(1);
|
||||||
|
let page_size = req.page_size.unwrap_or(100).max(1).min(500);
|
||||||
|
let offset = (page - 1) * page_size;
|
||||||
|
|
||||||
|
let mut conditions = vec!["e.file_uuid = $1".to_string()];
|
||||||
|
let mut param_idx = 2i32;
|
||||||
|
let mut joins = String::new();
|
||||||
|
|
||||||
|
if let Some(ref edge_type) = req.edge_type {
|
||||||
|
conditions.push(format!("e.edge_type = ${}", param_idx));
|
||||||
|
param_idx += 1;
|
||||||
|
}
|
||||||
|
if req.source_type.is_some() || req.target_type.is_some() {
|
||||||
|
joins = format!(
|
||||||
|
" JOIN {} sn ON e.source_node_id = sn.id JOIN {} tn ON e.target_node_id = tn.id",
|
||||||
|
nodes_table, nodes_table
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if let Some(ref source_type) = req.source_type {
|
||||||
|
conditions.push(format!("sn.node_type = ${}", param_idx));
|
||||||
|
param_idx += 1;
|
||||||
|
}
|
||||||
|
if let Some(ref target_type) = req.target_type {
|
||||||
|
conditions.push(format!("tn.node_type = ${}", param_idx));
|
||||||
|
param_idx += 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
let where_clause = conditions.join(" AND ");
|
||||||
|
let count_query = format!(
|
||||||
|
"SELECT COUNT(*) FROM {} e {} WHERE {}",
|
||||||
|
edges_table, joins, where_clause
|
||||||
|
);
|
||||||
|
|
||||||
|
let total: i64 = {
|
||||||
|
let mut q = sqlx::query_scalar::<_, i64>(&count_query).bind(&file_uuid);
|
||||||
|
if let Some(ref edge_type) = req.edge_type {
|
||||||
|
q = q.bind(edge_type);
|
||||||
|
}
|
||||||
|
if let Some(ref source_type) = req.source_type {
|
||||||
|
q = q.bind(source_type);
|
||||||
|
}
|
||||||
|
if let Some(ref target_type) = req.target_type {
|
||||||
|
q = q.bind(target_type);
|
||||||
|
}
|
||||||
|
q.fetch_one(state.db.pool()).await.map_err(|e| {
|
||||||
|
(
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
Json(serde_json::json!({"error": e.to_string()})),
|
||||||
|
)
|
||||||
|
})?
|
||||||
|
};
|
||||||
|
|
||||||
|
let query = format!(
|
||||||
|
"SELECT e.id, e.edge_type, e.source_node_id, e.target_node_id, e.properties FROM {} e {} WHERE {} ORDER BY e.id LIMIT ${} OFFSET ${}",
|
||||||
|
edges_table, joins, where_clause, param_idx, param_idx + 1
|
||||||
|
);
|
||||||
|
|
||||||
|
let rows: Vec<(i64, String, i64, i64, serde_json::Value)> = {
|
||||||
|
let mut q = sqlx::query_as(&query).bind(&file_uuid);
|
||||||
|
if let Some(ref edge_type) = req.edge_type {
|
||||||
|
q = q.bind(edge_type);
|
||||||
|
}
|
||||||
|
if let Some(ref source_type) = req.source_type {
|
||||||
|
q = q.bind(source_type);
|
||||||
|
}
|
||||||
|
if let Some(ref target_type) = req.target_type {
|
||||||
|
q = q.bind(target_type);
|
||||||
|
}
|
||||||
|
q.bind(page_size)
|
||||||
|
.bind(offset)
|
||||||
|
.fetch_all(state.db.pool())
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
(
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
Json(serde_json::json!({"error": e.to_string()})),
|
||||||
|
)
|
||||||
|
})?
|
||||||
|
};
|
||||||
|
|
||||||
|
let edges = rows
|
||||||
|
.into_iter()
|
||||||
|
.map(
|
||||||
|
|(id, edge_type, source_node_id, target_node_id, properties)| TkgEdgeInfo {
|
||||||
|
id,
|
||||||
|
edge_type,
|
||||||
|
source_node_id,
|
||||||
|
target_node_id,
|
||||||
|
properties,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
Ok(Json(TkgEdgesResponse {
|
||||||
|
success: true,
|
||||||
|
file_uuid,
|
||||||
|
total,
|
||||||
|
page,
|
||||||
|
page_size,
|
||||||
|
edges,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize)]
|
||||||
|
struct TkgNodeWithEdges {
|
||||||
|
node: TkgNodeInfo,
|
||||||
|
incoming_edges: Vec<TkgEdgeInfo>,
|
||||||
|
outgoing_edges: Vec<TkgEdgeInfo>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Serialize)]
|
||||||
|
struct TkgNodeDetailResponse {
|
||||||
|
success: bool,
|
||||||
|
file_uuid: String,
|
||||||
|
node: Option<TkgNodeWithEdges>,
|
||||||
|
error: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn get_tkg_node_detail(
|
||||||
|
State(state): State<crate::api::types::AppState>,
|
||||||
|
Path((file_uuid, node_id)): Path<(String, i64)>,
|
||||||
|
) -> Json<TkgNodeDetailResponse> {
|
||||||
|
let nodes_table = t("tkg_nodes");
|
||||||
|
let edges_table = t("tkg_edges");
|
||||||
|
|
||||||
|
let node: Option<(i64, String, String, String, serde_json::Value)> = sqlx::query_as(
|
||||||
|
&format!("SELECT id, node_type, external_id, label, properties FROM {} WHERE file_uuid = $1 AND id = $2", nodes_table)
|
||||||
|
)
|
||||||
|
.bind(&file_uuid).bind(node_id)
|
||||||
|
.fetch_optional(state.db.pool()).await.ok().flatten();
|
||||||
|
|
||||||
|
match node {
|
||||||
|
Some((id, node_type, external_id, label, properties)) => {
|
||||||
|
let incoming: Vec<TkgEdgeInfo> = sqlx::query_as(
|
||||||
|
&format!("SELECT id, edge_type, source_node_id, target_node_id, properties FROM {} WHERE file_uuid = $1 AND target_node_id = $2", edges_table)
|
||||||
|
)
|
||||||
|
.bind(&file_uuid).bind(node_id)
|
||||||
|
.fetch_all(state.db.pool()).await.unwrap_or_default()
|
||||||
|
.into_iter().map(|(id, edge_type, source_node_id, target_node_id, properties)| {
|
||||||
|
TkgEdgeInfo { id, edge_type, source_node_id, target_node_id, properties }
|
||||||
|
}).collect();
|
||||||
|
|
||||||
|
let outgoing: Vec<TkgEdgeInfo> = sqlx::query_as(
|
||||||
|
&format!("SELECT id, edge_type, source_node_id, target_node_id, properties FROM {} WHERE file_uuid = $1 AND source_node_id = $2", edges_table)
|
||||||
|
)
|
||||||
|
.bind(&file_uuid).bind(node_id)
|
||||||
|
.fetch_all(state.db.pool()).await.unwrap_or_default()
|
||||||
|
.into_iter().map(|(id, edge_type, source_node_id, target_node_id, properties)| {
|
||||||
|
TkgEdgeInfo { id, edge_type, source_node_id, target_node_id, properties }
|
||||||
|
}).collect();
|
||||||
|
|
||||||
|
Json(TkgNodeDetailResponse {
|
||||||
|
success: true,
|
||||||
|
file_uuid,
|
||||||
|
node: Some(TkgNodeWithEdges {
|
||||||
|
node: TkgNodeInfo {
|
||||||
|
id,
|
||||||
|
node_type,
|
||||||
|
external_id,
|
||||||
|
label,
|
||||||
|
properties,
|
||||||
|
},
|
||||||
|
incoming_edges: incoming,
|
||||||
|
outgoing_edges: outgoing,
|
||||||
|
}),
|
||||||
|
error: None,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
None => Json(TkgNodeDetailResponse {
|
||||||
|
success: false,
|
||||||
|
file_uuid,
|
||||||
|
node: None,
|
||||||
|
error: Some("Node not found".to_string()),
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ── Rule 2 Ingest ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct IngestRule2Response {
|
||||||
|
success: bool,
|
||||||
|
file_uuid: String,
|
||||||
|
rule2_chunks: i64,
|
||||||
|
vectorized_chunks: Option<i64>,
|
||||||
|
error: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn ingest_rule2(
|
||||||
|
State(state): State<crate::api::types::AppState>,
|
||||||
|
Path(file_uuid): Path<String>,
|
||||||
|
) -> Result<Json<IngestRule2Response>, (StatusCode, Json<serde_json::Value>)> {
|
||||||
|
use crate::core::chunk::rule2_ingest::ingest_rule2;
|
||||||
|
use crate::core::embedding::Embedder;
|
||||||
|
use crate::core::db::schema;
|
||||||
|
use crate::core::db::qdrant_db::{QdrantDb, VectorPayload};
|
||||||
|
use tracing::info;
|
||||||
|
|
||||||
|
let result = ingest_rule2(state.db.pool(), &file_uuid).await;
|
||||||
|
|
||||||
|
match result {
|
||||||
|
Ok(rule2_chunks) => {
|
||||||
|
info!(
|
||||||
|
"[Rule2API] {} relationship chunks created for {}",
|
||||||
|
rule2_chunks, file_uuid
|
||||||
|
);
|
||||||
|
|
||||||
|
// Auto-vectorize relationship chunks
|
||||||
|
let embedder = Embedder::new("embeddinggemma-300m".to_string());
|
||||||
|
let qdrant = QdrantDb::new();
|
||||||
|
let pool = state.db.pool();
|
||||||
|
let chunk_table = schema::table_name("chunk");
|
||||||
|
|
||||||
|
let rows: Vec<(String, String, i64, i64, f64, f64)> = sqlx::query_as(&format!(
|
||||||
|
"SELECT chunk_id, text_content, start_frame, end_frame, start_time, end_time \
|
||||||
|
FROM {} WHERE file_uuid = $1 AND chunk_type = 'relationship' \
|
||||||
|
AND embedding IS NULL AND (text_content IS NOT NULL AND text_content != '')",
|
||||||
|
chunk_table
|
||||||
|
))
|
||||||
|
.bind(&file_uuid)
|
||||||
|
.fetch_all(pool)
|
||||||
|
.await
|
||||||
|
.map_err(|e| {
|
||||||
|
(
|
||||||
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
|
Json(serde_json::json!({"error": e.to_string()})),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
let mut vectorized = 0i64;
|
||||||
|
for (chunk_id, text, start_frame, end_frame, start_time, end_time) in &rows {
|
||||||
|
if text.is_empty() {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if let Ok(vector) = embedder.embed_document(&text).await {
|
||||||
|
if state.db.store_vector(&chunk_id, &vector, &file_uuid).await.is_ok() {
|
||||||
|
let payload = VectorPayload {
|
||||||
|
file_uuid: file_uuid.clone(),
|
||||||
|
chunk_id: chunk_id.clone(),
|
||||||
|
chunk_type: "relationship".to_string(),
|
||||||
|
start_frame: *start_frame,
|
||||||
|
end_frame: *end_frame,
|
||||||
|
start_time: *start_time,
|
||||||
|
end_time: *end_time,
|
||||||
|
text: Some(text.clone()),
|
||||||
|
};
|
||||||
|
if qdrant.upsert_vector(&chunk_id, &vector, payload).await.is_ok() {
|
||||||
|
vectorized += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(Json(IngestRule2Response {
|
||||||
|
success: true,
|
||||||
|
file_uuid,
|
||||||
|
rule2_chunks: rule2_chunks as i64,
|
||||||
|
vectorized_chunks: Some(vectorized),
|
||||||
|
error: None,
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
Err(e) => Ok(Json(IngestRule2Response {
|
||||||
|
success: false,
|
||||||
|
file_uuid,
|
||||||
|
rule2_chunks: 0,
|
||||||
|
vectorized_chunks: None,
|
||||||
|
error: Some(e.to_string()),
|
||||||
|
})),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,10 +1,12 @@
|
|||||||
pub mod rule1_ingest;
|
pub mod rule1_ingest;
|
||||||
|
pub mod rule2_ingest;
|
||||||
pub mod rule3_ingest;
|
pub mod rule3_ingest;
|
||||||
pub mod splitter;
|
pub mod splitter;
|
||||||
pub mod trace_ingest;
|
pub mod trace_ingest;
|
||||||
pub mod types;
|
pub mod types;
|
||||||
|
|
||||||
pub use rule1_ingest::execute_rule1;
|
pub use rule1_ingest::execute_rule1;
|
||||||
|
pub use rule2_ingest::ingest_rule2;
|
||||||
pub use rule3_ingest::ingest_rule3;
|
pub use rule3_ingest::ingest_rule3;
|
||||||
pub use splitter::{AsrSegment, ChunkSplitter};
|
pub use splitter::{AsrSegment, ChunkSplitter};
|
||||||
pub use trace_ingest::ingest_traces;
|
pub use trace_ingest::ingest_traces;
|
||||||
|
|||||||
@@ -0,0 +1,335 @@
|
|||||||
|
use crate::core::db::schema;
|
||||||
|
use anyhow::{Context, Result};
|
||||||
|
use serde_json::Value;
|
||||||
|
use sqlx::PgPool;
|
||||||
|
use tracing::{info, warn};
|
||||||
|
|
||||||
|
fn t(name: &str) -> String {
|
||||||
|
let schema = std::env::var("DATABASE_SCHEMA").unwrap_or_else(|_| "dev".to_string());
|
||||||
|
if schema == "public" {
|
||||||
|
name.to_string()
|
||||||
|
} else {
|
||||||
|
format!("{}.{}", schema, name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Executes Rule 2 Ingestion: TKG edges → relationship chunks.
|
||||||
|
///
|
||||||
|
/// 1. Query tkg_edges by priority order.
|
||||||
|
/// 2. Resolve source/target nodes and identities.
|
||||||
|
/// 3. Generate natural language description (template-based).
|
||||||
|
/// 4. Insert chunks with chunk_type='relationship'.
|
||||||
|
pub async fn ingest_rule2(pool: &PgPool, file_uuid: &str) -> Result<usize> {
|
||||||
|
let edges_table = t("tkg_edges");
|
||||||
|
let nodes_table = t("tkg_nodes");
|
||||||
|
let chunk_table = t("chunk");
|
||||||
|
let fd_table = t("face_detections");
|
||||||
|
let id_table = t("identities");
|
||||||
|
let videos_table = t("videos");
|
||||||
|
|
||||||
|
// Get video fps
|
||||||
|
let fps: f64 = sqlx::query_scalar(&format!(
|
||||||
|
"SELECT COALESCE(fps, 25.0) FROM {} WHERE file_uuid = $1",
|
||||||
|
videos_table
|
||||||
|
))
|
||||||
|
.bind(file_uuid)
|
||||||
|
.fetch_optional(pool)
|
||||||
|
.await?
|
||||||
|
.unwrap_or(25.0);
|
||||||
|
|
||||||
|
// Priority order for edge types (matching TKG edge_type values)
|
||||||
|
let edge_types = vec![
|
||||||
|
"SPEAKS_AS",
|
||||||
|
"MUTUAL_GAZE",
|
||||||
|
"CO_OCCURS_WITH",
|
||||||
|
"HAS_APPEARANCE",
|
||||||
|
"WEARS",
|
||||||
|
];
|
||||||
|
|
||||||
|
let mut count = 0;
|
||||||
|
let mut tx = pool.begin().await?;
|
||||||
|
|
||||||
|
for edge_type in &edge_types {
|
||||||
|
// Query edges of this type
|
||||||
|
let edges: Vec<(i64, String, String, Value)> = sqlx::query_as(&format!(
|
||||||
|
"SELECT id, source_node_id::text, target_node_id::text, properties \
|
||||||
|
FROM {} WHERE file_uuid = $1 AND edge_type = $2",
|
||||||
|
edges_table
|
||||||
|
))
|
||||||
|
.bind(file_uuid)
|
||||||
|
.bind(edge_type)
|
||||||
|
.fetch_all(&mut *tx)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
info!(
|
||||||
|
"Rule2: {} {} edges for file_uuid={}",
|
||||||
|
edges.len(),
|
||||||
|
edge_type,
|
||||||
|
file_uuid
|
||||||
|
);
|
||||||
|
|
||||||
|
for (edge_id, source_id_str, target_id_str, properties) in &edges {
|
||||||
|
// Parse source/target node IDs
|
||||||
|
let source_id: i64 = source_id_str.parse().context("Invalid source_node_id")?;
|
||||||
|
let target_id: i64 = target_id_str.parse().context("Invalid target_node_id")?;
|
||||||
|
|
||||||
|
// Query source node
|
||||||
|
let source_node: Option<(String, String, String, Value)> = sqlx::query_as(&format!(
|
||||||
|
"SELECT node_type, external_id, label, properties FROM {} WHERE id = $1",
|
||||||
|
nodes_table
|
||||||
|
))
|
||||||
|
.bind(source_id)
|
||||||
|
.fetch_optional(&mut *tx)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
// Query target node
|
||||||
|
let target_node: Option<(String, String, String, Value)> = sqlx::query_as(&format!(
|
||||||
|
"SELECT node_type, external_id, label, properties FROM {} WHERE id = $1",
|
||||||
|
nodes_table
|
||||||
|
))
|
||||||
|
.bind(target_id)
|
||||||
|
.fetch_optional(&mut *tx)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if source_node.is_none() || target_node.is_none() {
|
||||||
|
warn!("Rule2: Missing node for edge {}", edge_id);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
let (src_type, src_ext_id, src_label, _src_props) = source_node.unwrap();
|
||||||
|
let (tgt_type, tgt_ext_id, tgt_label, tgt_props) = target_node.unwrap();
|
||||||
|
|
||||||
|
// Resolve identity names for face_trace nodes (inline)
|
||||||
|
let src_identity: Option<String> = if src_type == "face_trace" {
|
||||||
|
let trace_id: i32 = src_ext_id
|
||||||
|
.replace("trace_", "")
|
||||||
|
.parse()
|
||||||
|
.context("Invalid trace_id")?;
|
||||||
|
sqlx::query_scalar(&format!(
|
||||||
|
"SELECT i.name FROM {} fd \
|
||||||
|
JOIN {} i ON i.id = fd.identity_id \
|
||||||
|
WHERE fd.file_uuid = $1 AND fd.trace_id = $2 AND fd.identity_id IS NOT NULL \
|
||||||
|
LIMIT 1",
|
||||||
|
fd_table, id_table
|
||||||
|
))
|
||||||
|
.bind(file_uuid)
|
||||||
|
.bind(trace_id)
|
||||||
|
.fetch_optional(&mut *tx)
|
||||||
|
.await?
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
};
|
||||||
|
|
||||||
|
let tgt_identity: Option<String> = if tgt_type == "face_trace" {
|
||||||
|
let trace_id: i32 = tgt_ext_id
|
||||||
|
.replace("trace_", "")
|
||||||
|
.parse()
|
||||||
|
.context("Invalid trace_id")?;
|
||||||
|
sqlx::query_scalar(&format!(
|
||||||
|
"SELECT i.name FROM {} fd \
|
||||||
|
JOIN {} i ON i.id = fd.identity_id \
|
||||||
|
WHERE fd.file_uuid = $1 AND fd.trace_id = $2 AND fd.identity_id IS NOT NULL \
|
||||||
|
LIMIT 1",
|
||||||
|
fd_table, id_table
|
||||||
|
))
|
||||||
|
.bind(file_uuid)
|
||||||
|
.bind(trace_id)
|
||||||
|
.fetch_optional(&mut *tx)
|
||||||
|
.await?
|
||||||
|
} else {
|
||||||
|
None
|
||||||
|
};
|
||||||
|
|
||||||
|
// Extract time range from properties
|
||||||
|
let first_frame = properties
|
||||||
|
.get("first_frame")
|
||||||
|
.and_then(|v| v.as_i64())
|
||||||
|
.unwrap_or(0);
|
||||||
|
let last_frame = properties
|
||||||
|
.get("last_frame")
|
||||||
|
.or_else(|| properties.get("end_frame"))
|
||||||
|
.and_then(|v| v.as_i64())
|
||||||
|
.unwrap_or(first_frame);
|
||||||
|
let frame_count = properties
|
||||||
|
.get("frame_count")
|
||||||
|
.or_else(|| properties.get("gaze_frame_count"))
|
||||||
|
.and_then(|v| v.as_i64())
|
||||||
|
.unwrap_or(last_frame - first_frame);
|
||||||
|
|
||||||
|
// Build context for description
|
||||||
|
let context = serde_json::json!({
|
||||||
|
"edge_type": edge_type,
|
||||||
|
"source_node": {
|
||||||
|
"node_type": src_type,
|
||||||
|
"external_id": src_ext_id,
|
||||||
|
"label": src_label,
|
||||||
|
"identity_name": src_identity,
|
||||||
|
},
|
||||||
|
"target_node": {
|
||||||
|
"node_type": tgt_type,
|
||||||
|
"external_id": tgt_ext_id,
|
||||||
|
"label": tgt_label,
|
||||||
|
"identity_name": tgt_identity,
|
||||||
|
"properties": tgt_props,
|
||||||
|
},
|
||||||
|
"properties": {
|
||||||
|
"first_frame": first_frame,
|
||||||
|
"last_frame": last_frame,
|
||||||
|
"frame_count": frame_count,
|
||||||
|
"confidence": properties.get("confidence").and_then(|v| v.as_f64()),
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// Generate text_content (template-based)
|
||||||
|
let text_content = generate_description(&context);
|
||||||
|
|
||||||
|
// Build chunk_id
|
||||||
|
let chunk_id = format!("rel_{}", edge_id);
|
||||||
|
|
||||||
|
// Build content JSON
|
||||||
|
let content = serde_json::json!({
|
||||||
|
"edge_type": edge_type,
|
||||||
|
"edge_id": edge_id,
|
||||||
|
"source_node": {
|
||||||
|
"id": source_id,
|
||||||
|
"node_type": src_type,
|
||||||
|
"external_id": src_ext_id,
|
||||||
|
"label": src_label,
|
||||||
|
"identity_name": src_identity,
|
||||||
|
},
|
||||||
|
"target_node": {
|
||||||
|
"id": target_id,
|
||||||
|
"node_type": tgt_type,
|
||||||
|
"external_id": tgt_ext_id,
|
||||||
|
"label": tgt_label,
|
||||||
|
"identity_name": tgt_identity,
|
||||||
|
},
|
||||||
|
"properties": properties,
|
||||||
|
});
|
||||||
|
|
||||||
|
// Build metadata
|
||||||
|
let metadata = serde_json::json!({
|
||||||
|
"source_type": src_type,
|
||||||
|
"target_type": tgt_type,
|
||||||
|
"has_identity": src_identity.is_some() || tgt_identity.is_some(),
|
||||||
|
});
|
||||||
|
|
||||||
|
// Insert chunk
|
||||||
|
let start_time = first_frame as f64 / fps;
|
||||||
|
let end_time = last_frame as f64 / fps;
|
||||||
|
|
||||||
|
sqlx::query(&format!(
|
||||||
|
"INSERT INTO {} (file_uuid, chunk_id, chunk_type, \
|
||||||
|
start_frame, end_frame, fps, start_time, end_time, \
|
||||||
|
text_content, content, metadata) \
|
||||||
|
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) \
|
||||||
|
ON CONFLICT (file_uuid, chunk_id) DO NOTHING",
|
||||||
|
chunk_table
|
||||||
|
))
|
||||||
|
.bind(file_uuid)
|
||||||
|
.bind(&chunk_id)
|
||||||
|
.bind("relationship")
|
||||||
|
.bind(first_frame)
|
||||||
|
.bind(last_frame)
|
||||||
|
.bind(fps)
|
||||||
|
.bind(start_time)
|
||||||
|
.bind(end_time)
|
||||||
|
.bind(&text_content)
|
||||||
|
.bind(&content)
|
||||||
|
.bind(&metadata)
|
||||||
|
.execute(&mut *tx)
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
count += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
tx.commit().await?;
|
||||||
|
info!(
|
||||||
|
"Rule2: {} relationship chunks created for file_uuid={}",
|
||||||
|
count, file_uuid
|
||||||
|
);
|
||||||
|
Ok(count)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Generate natural language description for a relationship (template-based).
|
||||||
|
fn generate_description(context: &Value) -> String {
|
||||||
|
let edge_type = context.get("edge_type").and_then(|v| v.as_str()).unwrap_or("");
|
||||||
|
let src = context.get("source_node").unwrap();
|
||||||
|
let tgt = context.get("target_node").unwrap();
|
||||||
|
let props = context.get("properties").unwrap();
|
||||||
|
|
||||||
|
let src_identity = src.get("identity_name").and_then(|v| v.as_str());
|
||||||
|
let tgt_identity = tgt.get("identity_name").and_then(|v| v.as_str());
|
||||||
|
let src_ext_id = src.get("external_id").and_then(|v| v.as_str()).unwrap_or("");
|
||||||
|
let tgt_ext_id = tgt.get("external_id").and_then(|v| v.as_str()).unwrap_or("");
|
||||||
|
|
||||||
|
let first_frame = props.get("first_frame").and_then(|v| v.as_i64()).unwrap_or(0);
|
||||||
|
let last_frame = props.get("last_frame").and_then(|v| v.as_i64()).unwrap_or(first_frame);
|
||||||
|
let frame_count = props.get("frame_count").and_then(|v| v.as_i64()).unwrap_or(0);
|
||||||
|
|
||||||
|
let src_display = src_identity.unwrap_or(src_ext_id);
|
||||||
|
let tgt_display = tgt_identity.unwrap_or(tgt_ext_id);
|
||||||
|
|
||||||
|
match edge_type {
|
||||||
|
"SPEAKS_AS" => {
|
||||||
|
format!(
|
||||||
|
"SPEAKER {} 以 {} 的身份說話,從 frame {} 到 frame {}",
|
||||||
|
src_ext_id, tgt_display, first_frame, last_frame
|
||||||
|
)
|
||||||
|
}
|
||||||
|
"MUTUAL_GAZE" => {
|
||||||
|
format!(
|
||||||
|
"{} 和 {} 互相看對方 {} 幀,起始於 frame {}",
|
||||||
|
src_display, tgt_display, frame_count, first_frame
|
||||||
|
)
|
||||||
|
}
|
||||||
|
"CO_OCCURS_WITH" => {
|
||||||
|
// Check if both nodes are face_trace (face-face co-occurrence)
|
||||||
|
let src_type = src.get("node_type").and_then(|v| v.as_str()).unwrap_or("");
|
||||||
|
let tgt_type = tgt.get("node_type").and_then(|v| v.as_str()).unwrap_or("");
|
||||||
|
if src_type == "face_trace" && tgt_type == "face_trace" {
|
||||||
|
format!(
|
||||||
|
"{} 和 {} 同框 {} 幀,從 frame {} 到 frame {}",
|
||||||
|
src_display, tgt_display, frame_count, first_frame, last_frame
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
format!(
|
||||||
|
"{} 和 {} 在同一畫面出現",
|
||||||
|
src_display, tgt_display
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
"HAS_APPEARANCE" => {
|
||||||
|
let tgt_props = tgt.get("properties").unwrap();
|
||||||
|
let upper_color = tgt_props
|
||||||
|
.get("color_features")
|
||||||
|
.and_then(|c| c.get("dominant_colors"))
|
||||||
|
.and_then(|d| d.as_array())
|
||||||
|
.and_then(|arr| arr.first())
|
||||||
|
.and_then(|c| c.as_str());
|
||||||
|
format!(
|
||||||
|
"{} 穿著 {} 上衣",
|
||||||
|
src_display,
|
||||||
|
upper_color.unwrap_or("未知顏色")
|
||||||
|
)
|
||||||
|
}
|
||||||
|
"WEARS" => {
|
||||||
|
let tgt_props = tgt.get("properties").unwrap();
|
||||||
|
let accessory_type = tgt_props.get("type").and_then(|t| t.as_str());
|
||||||
|
let confidence = tgt_props.get("confidence").and_then(|c| c.as_f64());
|
||||||
|
format!(
|
||||||
|
"{} 戴 {},信心值 {:.2}",
|
||||||
|
src_display,
|
||||||
|
accessory_type.unwrap_or("配件"),
|
||||||
|
confidence.unwrap_or(0.0)
|
||||||
|
)
|
||||||
|
}
|
||||||
|
_ => {
|
||||||
|
format!(
|
||||||
|
"{} 與 {} 有 {} 關係,frame {}-{}",
|
||||||
|
src_display, tgt_display, edge_type, first_frame, last_frame
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -9,6 +9,7 @@ pub enum ChunkType {
|
|||||||
Cut,
|
Cut,
|
||||||
Trace,
|
Trace,
|
||||||
Story,
|
Story,
|
||||||
|
Relationship,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl ChunkType {
|
impl ChunkType {
|
||||||
@@ -19,6 +20,7 @@ impl ChunkType {
|
|||||||
ChunkType::Cut => "cut",
|
ChunkType::Cut => "cut",
|
||||||
ChunkType::Trace => "trace",
|
ChunkType::Trace => "trace",
|
||||||
ChunkType::Story => "story",
|
ChunkType::Story => "story",
|
||||||
|
ChunkType::Relationship => "relationship",
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,488 @@
|
|||||||
|
use anyhow::{Context, Result};
|
||||||
|
use reqwest::Client;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use std::collections::HashMap;
|
||||||
|
|
||||||
|
pub struct FaceEmbeddingDb {
|
||||||
|
client: Client,
|
||||||
|
base_url: String,
|
||||||
|
api_key: String,
|
||||||
|
collection_name: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
||||||
|
pub struct FaceEmbeddingPayload {
|
||||||
|
pub file_uuid: String,
|
||||||
|
pub trace_id: i32,
|
||||||
|
pub frame: i64,
|
||||||
|
pub bbox_x: f64,
|
||||||
|
pub bbox_y: f64,
|
||||||
|
pub bbox_w: f64,
|
||||||
|
pub bbox_h: f64,
|
||||||
|
pub confidence: f64,
|
||||||
|
pub yaw: f64,
|
||||||
|
pub pitch: f64,
|
||||||
|
pub roll: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Debug, Clone, Deserialize)]
|
||||||
|
pub struct FaceEmbeddingPoint {
|
||||||
|
pub id: String,
|
||||||
|
pub vector: Vec<f32>,
|
||||||
|
pub payload: FaceEmbeddingPayload,
|
||||||
|
pub score: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FaceEmbeddingDb {
|
||||||
|
pub fn new() -> Self {
|
||||||
|
let schema = std::env::var("DATABASE_SCHEMA").unwrap_or_else(|_| "dev".to_string());
|
||||||
|
let collection_name = format!("{}_face_embeddings", schema);
|
||||||
|
|
||||||
|
let base_url =
|
||||||
|
std::env::var("QDRANT_URL").unwrap_or_else(|_| "http://localhost:6333".to_string());
|
||||||
|
let api_key = std::env::var("QDRANT_API_KEY")
|
||||||
|
.unwrap_or_else(|_| "Test3200Test3200Test3200".to_string());
|
||||||
|
|
||||||
|
Self {
|
||||||
|
client: Client::new(),
|
||||||
|
base_url,
|
||||||
|
api_key,
|
||||||
|
collection_name,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn init_collection(&self) -> Result<()> {
|
||||||
|
let url = format!("{}/collections/{}", self.base_url, self.collection_name);
|
||||||
|
|
||||||
|
let response = self
|
||||||
|
.client
|
||||||
|
.get(&url)
|
||||||
|
.header("api-key", &self.api_key)
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if response.status().is_success() {
|
||||||
|
tracing::info!("[FaceEmbedding] Collection {} already exists", self.collection_name);
|
||||||
|
return Ok(());
|
||||||
|
}
|
||||||
|
|
||||||
|
let create_url = format!("{}/collections/{}", self.base_url, self.collection_name);
|
||||||
|
let body = serde_json::json!({
|
||||||
|
"vectors": {
|
||||||
|
"size": 512,
|
||||||
|
"distance": "Cosine"
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
self.client
|
||||||
|
.put(&create_url)
|
||||||
|
.header("api-key", &self.api_key)
|
||||||
|
.header("Content-Type", "application/json")
|
||||||
|
.json(&body)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.context("Failed to create face embeddings collection")?;
|
||||||
|
|
||||||
|
tracing::info!("[FaceEmbedding] Created collection {} (dim=512)", self.collection_name);
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn upsert_embedding(
|
||||||
|
&self,
|
||||||
|
point_id: &str,
|
||||||
|
embedding: &[f32],
|
||||||
|
payload: &FaceEmbeddingPayload,
|
||||||
|
) -> Result<()> {
|
||||||
|
let url = format!(
|
||||||
|
"{}/collections/{}/points?wait=true",
|
||||||
|
self.base_url, self.collection_name
|
||||||
|
);
|
||||||
|
|
||||||
|
let body = serde_json::json!({
|
||||||
|
"points": [{
|
||||||
|
"id": point_id,
|
||||||
|
"vector": embedding,
|
||||||
|
"payload": payload
|
||||||
|
}]
|
||||||
|
});
|
||||||
|
|
||||||
|
let response = self
|
||||||
|
.client
|
||||||
|
.put(&url)
|
||||||
|
.header("api-key", &self.api_key)
|
||||||
|
.header("Content-Type", "application/json")
|
||||||
|
.json(&body)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.context("Failed to upsert face embedding")?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let text = response.text().await.unwrap_or_default();
|
||||||
|
anyhow::bail!("Qdrant upsert failed: {}", text);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn batch_upsert(
|
||||||
|
&self,
|
||||||
|
points: Vec<(String, Vec<f32>, FaceEmbeddingPayload)>,
|
||||||
|
) -> Result<usize> {
|
||||||
|
if points.is_empty() {
|
||||||
|
return Ok(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
let url = format!(
|
||||||
|
"{}/collections/{}/points?wait=true",
|
||||||
|
self.base_url, self.collection_name
|
||||||
|
);
|
||||||
|
|
||||||
|
let body = serde_json::json!({
|
||||||
|
"points": points.iter().map(|(id, vec, payload)| {
|
||||||
|
// Parse id as u64 for Qdrant (requires integer or UUID)
|
||||||
|
let id_num: u64 = id.parse().unwrap_or(0);
|
||||||
|
serde_json::json!({
|
||||||
|
"id": id_num,
|
||||||
|
"vector": vec,
|
||||||
|
"payload": payload
|
||||||
|
})
|
||||||
|
}).collect::<Vec<_>>()
|
||||||
|
});
|
||||||
|
|
||||||
|
let response = self
|
||||||
|
.client
|
||||||
|
.put(&url)
|
||||||
|
.header("api-key", &self.api_key)
|
||||||
|
.header("Content-Type", "application/json")
|
||||||
|
.json(&body)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.context("Failed to batch upsert face embeddings")?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let text = response.text().await.unwrap_or_default();
|
||||||
|
anyhow::bail!("Qdrant batch upsert failed: {}", text);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(points.len())
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn search_similar(
|
||||||
|
&self,
|
||||||
|
query_embedding: &[f32],
|
||||||
|
file_uuid: Option<&str>,
|
||||||
|
limit: usize,
|
||||||
|
threshold: f64,
|
||||||
|
) -> Result<Vec<FaceEmbeddingPoint>> {
|
||||||
|
let url = format!(
|
||||||
|
"{}/collections/{}/points/search",
|
||||||
|
self.base_url, self.collection_name
|
||||||
|
);
|
||||||
|
|
||||||
|
let mut filter = serde_json::json!({});
|
||||||
|
if let Some(fu) = file_uuid {
|
||||||
|
filter = serde_json::json!({
|
||||||
|
"must": [{
|
||||||
|
"key": "file_uuid",
|
||||||
|
"match": { "value": fu }
|
||||||
|
}]
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
let body = serde_json::json!({
|
||||||
|
"vector": query_embedding,
|
||||||
|
"limit": limit,
|
||||||
|
"with_payload": true,
|
||||||
|
"with_vector": false,
|
||||||
|
"filter": filter
|
||||||
|
});
|
||||||
|
|
||||||
|
let response = self
|
||||||
|
.client
|
||||||
|
.post(&url)
|
||||||
|
.header("api-key", &self.api_key)
|
||||||
|
.header("Content-Type", "application/json")
|
||||||
|
.json(&body)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.context("Failed to search face embeddings")?;
|
||||||
|
|
||||||
|
let status = response.status();
|
||||||
|
let text = response.text().await.unwrap_or_default();
|
||||||
|
|
||||||
|
if !status.is_success() {
|
||||||
|
anyhow::bail!("Qdrant search failed: {} - {}", status, text);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct SearchResult {
|
||||||
|
result: Vec<PointResult>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct PointResult {
|
||||||
|
id: serde_json::Value,
|
||||||
|
score: f64,
|
||||||
|
payload: HashMap<String, serde_json::Value>,
|
||||||
|
}
|
||||||
|
|
||||||
|
let parsed: SearchResult = serde_json::from_str(&text)
|
||||||
|
.context("Failed to parse Qdrant search response")?;
|
||||||
|
|
||||||
|
let results: Vec<FaceEmbeddingPoint> = parsed
|
||||||
|
.result
|
||||||
|
.into_iter()
|
||||||
|
.filter(|r| r.score >= threshold)
|
||||||
|
.map(|r| {
|
||||||
|
let id = match r.id {
|
||||||
|
serde_json::Value::String(s) => s,
|
||||||
|
serde_json::Value::Number(n) => n.to_string(),
|
||||||
|
_ => "unknown".to_string(),
|
||||||
|
};
|
||||||
|
let payload = FaceEmbeddingPayload {
|
||||||
|
file_uuid: r.payload.get("file_uuid")
|
||||||
|
.and_then(|v| v.as_str()).unwrap_or("").to_string(),
|
||||||
|
trace_id: r.payload.get("trace_id")
|
||||||
|
.and_then(|v| v.as_i64()).unwrap_or(0) as i32,
|
||||||
|
frame: r.payload.get("frame")
|
||||||
|
.and_then(|v| v.as_i64()).unwrap_or(0),
|
||||||
|
bbox_x: r.payload.get("bbox_x")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
bbox_y: r.payload.get("bbox_y")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
bbox_w: r.payload.get("bbox_w")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
bbox_h: r.payload.get("bbox_h")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
confidence: r.payload.get("confidence")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
yaw: r.payload.get("yaw")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
pitch: r.payload.get("pitch")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
roll: r.payload.get("roll")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
};
|
||||||
|
FaceEmbeddingPoint {
|
||||||
|
id,
|
||||||
|
vector: vec![], // Not returned with_vector=false
|
||||||
|
payload,
|
||||||
|
score: r.score,
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
Ok(results)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn get_embeddings_by_trace(
|
||||||
|
&self,
|
||||||
|
file_uuid: &str,
|
||||||
|
trace_id: i32,
|
||||||
|
) -> Result<Vec<(String, Vec<f32>)>> {
|
||||||
|
let url = format!(
|
||||||
|
"{}/collections/{}/points/scroll",
|
||||||
|
self.base_url, self.collection_name
|
||||||
|
);
|
||||||
|
|
||||||
|
let body = serde_json::json!({
|
||||||
|
"limit": 1000,
|
||||||
|
"with_payload": true,
|
||||||
|
"with_vector": true,
|
||||||
|
"filter": {
|
||||||
|
"must": [
|
||||||
|
{"key": "file_uuid", "match": { "value": file_uuid }},
|
||||||
|
{"key": "trace_id", "match": { "value": trace_id }}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let response = self
|
||||||
|
.client
|
||||||
|
.post(&url)
|
||||||
|
.header("api-key", &self.api_key)
|
||||||
|
.header("Content-Type", "application/json")
|
||||||
|
.json(&body)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.context("Failed to scroll face embeddings")?;
|
||||||
|
|
||||||
|
let status = response.status();
|
||||||
|
let text = response.text().await.unwrap_or_default();
|
||||||
|
|
||||||
|
if !status.is_success() {
|
||||||
|
anyhow::bail!("Qdrant scroll failed: {} - {}", status, text);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct ScrollResult {
|
||||||
|
result: ScrollPoints,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct ScrollPoints {
|
||||||
|
points: Vec<PointResult>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct PointResult {
|
||||||
|
id: serde_json::Value,
|
||||||
|
vector: Vec<f32>,
|
||||||
|
}
|
||||||
|
|
||||||
|
let parsed: ScrollResult = serde_json::from_str(&text)
|
||||||
|
.context("Failed to parse Qdrant scroll response")?;
|
||||||
|
|
||||||
|
let results: Vec<(String, Vec<f32>)> = parsed
|
||||||
|
.result
|
||||||
|
.points
|
||||||
|
.into_iter()
|
||||||
|
.map(|r| {
|
||||||
|
let id = match r.id {
|
||||||
|
serde_json::Value::String(s) => s,
|
||||||
|
serde_json::Value::Number(n) => n.to_string(),
|
||||||
|
_ => "unknown".to_string(),
|
||||||
|
};
|
||||||
|
(id, r.vector)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
Ok(results)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn get_all_embeddings_for_file(
|
||||||
|
&self,
|
||||||
|
file_uuid: &str,
|
||||||
|
) -> Result<Vec<(String, Vec<f32>, FaceEmbeddingPayload)>> {
|
||||||
|
let url = format!(
|
||||||
|
"{}/collections/{}/points/scroll",
|
||||||
|
self.base_url, self.collection_name
|
||||||
|
);
|
||||||
|
|
||||||
|
let body = serde_json::json!({
|
||||||
|
"limit": 10000,
|
||||||
|
"with_payload": true,
|
||||||
|
"with_vector": true,
|
||||||
|
"filter": {
|
||||||
|
"must": [
|
||||||
|
{"key": "file_uuid", "match": { "value": file_uuid }}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let response = self
|
||||||
|
.client
|
||||||
|
.post(&url)
|
||||||
|
.header("api-key", &self.api_key)
|
||||||
|
.header("Content-Type", "application/json")
|
||||||
|
.json(&body)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.context("Failed to scroll face embeddings")?;
|
||||||
|
|
||||||
|
let status = response.status();
|
||||||
|
let text = response.text().await.unwrap_or_default();
|
||||||
|
|
||||||
|
if !status.is_success() {
|
||||||
|
anyhow::bail!("Qdrant scroll failed: {} - {}", status, text);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct ScrollResult {
|
||||||
|
result: ScrollPoints,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct ScrollPoints {
|
||||||
|
points: Vec<PointResult>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct PointResult {
|
||||||
|
id: serde_json::Value,
|
||||||
|
vector: Vec<f32>,
|
||||||
|
payload: HashMap<String, serde_json::Value>,
|
||||||
|
}
|
||||||
|
|
||||||
|
let parsed: ScrollResult = serde_json::from_str(&text)
|
||||||
|
.context("Failed to parse Qdrant scroll response")?;
|
||||||
|
|
||||||
|
let results: Vec<(String, Vec<f32>, FaceEmbeddingPayload)> = parsed
|
||||||
|
.result
|
||||||
|
.points
|
||||||
|
.into_iter()
|
||||||
|
.map(|r| {
|
||||||
|
let id = match r.id {
|
||||||
|
serde_json::Value::String(s) => s,
|
||||||
|
serde_json::Value::Number(n) => n.to_string(),
|
||||||
|
_ => "unknown".to_string(),
|
||||||
|
};
|
||||||
|
let payload = FaceEmbeddingPayload {
|
||||||
|
file_uuid: r.payload.get("file_uuid")
|
||||||
|
.and_then(|v| v.as_str()).unwrap_or("").to_string(),
|
||||||
|
trace_id: r.payload.get("trace_id")
|
||||||
|
.and_then(|v| v.as_i64()).unwrap_or(0) as i32,
|
||||||
|
frame: r.payload.get("frame")
|
||||||
|
.and_then(|v| v.as_i64()).unwrap_or(0),
|
||||||
|
bbox_x: r.payload.get("bbox_x")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
bbox_y: r.payload.get("bbox_y")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
bbox_w: r.payload.get("bbox_w")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
bbox_h: r.payload.get("bbox_h")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
confidence: r.payload.get("confidence")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
yaw: r.payload.get("yaw")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
pitch: r.payload.get("pitch")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
roll: r.payload.get("roll")
|
||||||
|
.and_then(|v| v.as_f64()).unwrap_or(0.0),
|
||||||
|
};
|
||||||
|
(id, r.vector, payload)
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
|
||||||
|
Ok(results)
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn delete_file_embeddings(&self, file_uuid: &str) -> Result<usize> {
|
||||||
|
let url = format!(
|
||||||
|
"{}/collections/{}/points/delete?wait=true",
|
||||||
|
self.base_url, self.collection_name
|
||||||
|
);
|
||||||
|
|
||||||
|
let body = serde_json::json!({
|
||||||
|
"filter": {
|
||||||
|
"must": [
|
||||||
|
{"key": "file_uuid", "match": { "value": file_uuid }}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let response = self
|
||||||
|
.client
|
||||||
|
.post(&url)
|
||||||
|
.header("api-key", &self.api_key)
|
||||||
|
.header("Content-Type", "application/json")
|
||||||
|
.json(&body)
|
||||||
|
.send()
|
||||||
|
.await
|
||||||
|
.context("Failed to delete face embeddings")?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let text = response.text().await.unwrap_or_default();
|
||||||
|
anyhow::bail!("Qdrant delete failed: {}", text);
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(0)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Default for FaceEmbeddingDb {
|
||||||
|
fn default() -> Self {
|
||||||
|
Self::new()
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -32,12 +32,14 @@ pub trait VectorStore: Send + Sync {
|
|||||||
async fn search(&self, query_vector: &[f32], limit: usize) -> Result<Vec<SearchResult>>;
|
async fn search(&self, query_vector: &[f32], limit: usize) -> Result<Vec<SearchResult>>;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub mod face_embedding_db;
|
||||||
pub mod identity_merge_history;
|
pub mod identity_merge_history;
|
||||||
pub mod mongodb_db;
|
pub mod mongodb_db;
|
||||||
pub mod postgres_db;
|
pub mod postgres_db;
|
||||||
pub mod qdrant_db;
|
pub mod qdrant_db;
|
||||||
pub mod redis_client;
|
pub mod redis_client;
|
||||||
pub mod redis_db;
|
pub mod redis_db;
|
||||||
|
pub use face_embedding_db::{FaceEmbeddingDb, FaceEmbeddingPayload, FaceEmbeddingPoint};
|
||||||
pub use identity_merge_history::{
|
pub use identity_merge_history::{
|
||||||
AliasEntry, FacesTransferred, IdentityMergeHistory, IdentityMergeHistoryStore,
|
AliasEntry, FacesTransferred, IdentityMergeHistory, IdentityMergeHistoryStore,
|
||||||
IdentitySnapshot, MergeHistoryEntry, MergeHistoryQuery, MergeParams, TargetIdentitySnapshot,
|
IdentitySnapshot, MergeHistoryEntry, MergeHistoryQuery, MergeParams, TargetIdentitySnapshot,
|
||||||
@@ -56,3 +58,10 @@ pub use redis_client::{
|
|||||||
ProgressMessage, RedisClient,
|
ProgressMessage, RedisClient,
|
||||||
};
|
};
|
||||||
pub use redis_db::RedisDb;
|
pub use redis_db::RedisDb;
|
||||||
|
pub mod qdrant_workspace;
|
||||||
|
pub mod workspace_sqlite;
|
||||||
|
pub use qdrant_workspace::{QdrantWorkspace, ScrolledPoint, WorkspaceScrollResult};
|
||||||
|
pub use workspace_sqlite::{
|
||||||
|
workspace_path, ChunkRow, FaceDetectionBatchItem, FaceDetectionRow, PreChunkRow,
|
||||||
|
ProcessorResultRow, SpeakerDetectionBatchItem, SpeakerDetectionRow, WorkspaceDb,
|
||||||
|
};
|
||||||
|
|||||||
+1474
-17
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user