warren/momentry_core
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Compare commits

base: warren/momentry_core:e3066c3f49e0e08e91e4b924b4164bcf6214a9c3
Branches Tags
warren/momentry_core:main
warren/momentry_core:v1.1.0 warren/momentry_core:v0.1.1 warren/momentry_core:v0.9.20260325_151200 warren/momentry_core:v0.9.20260325_144654
...
compare: warren/momentry_core:main
Branches Tags
warren/momentry_core:main
warren/momentry_core:v1.1.0 warren/momentry_core:v0.1.1 warren/momentry_core:v0.9.20260325_151200 warren/momentry_core:v0.9.20260325_144654

122 Commits
e3066c3f49 ... main

Author SHA1 Message Date
Accusys 5d0c771a2b fix: use environment variables for VLM/LLM endpoints and models 
- trace_vlm_caption.py: use MOMENTRY_LLM_VISION_URL and MOMENTRY_LLM_VISION_MODEL
- scene_vlm_caption.py: use MOMENTRY_LLM_VISION_URL and MOMENTRY_LLM_VISION_MODEL
- Changed from Ollama /api/generate to OpenAI-compatible /v1/chat/completions format
- Added embedding server environment variables
 2026-07-27 13:49:08 +08:00
Accusys 39a2cbc65b fix: face group name read consistency, sync_file_status fix, cleanup ghost records, identity_agent replaced with face_dedup 
- get_face_groups_handler: COALESCE(tp.name, tn.label) for name consistency
- sync_file_status: compare JSON vs pre_chunks (not chunk table)
- face consistency: compare frames.len() not total_faces
- cleanup 2 ghost records with NULL file_name/file_path
- replace identity_agent with face_dedup in pipeline stages
- remove identity_agent_api.rs and all references
- update required_processors to match actual processors
- update AGENTS.md with team responsibilities
- add Studio pipeline changes documentation
 2026-07-27 02:15:51 +08:00
Accusys fcdeab82e6 feat: save key_frame.jpg and key_face.jpg for each face trace during TKG build 
- Extract full frame at best confidence frame using ffmpeg
- Crop face region using average bbox coordinates
- Save to .faces/{file_uuid}/{trace_id}/key_frame.jpg and key_face.jpg
- TKG node properties store full file paths for UI display
- Supports future extension for cut representative frames
 2026-07-21 04:08:16 +08:00
Accusys 6f5d71763f feat: auto-populate key_frame and key_face in TKG face_track nodes 
- FacePoint now includes confidence and point_id from Qdrant
- TKG build tracks best face (highest confidence) per trace
- key_frame = frame number of best face, key_face = Qdrant point ID
- Every face trace now has representative frame and face for UI display
 2026-07-21 03:56:25 +08:00
Accusys 644516769c docs: update TKG, profile, identity module docs with trace management and external_id fix 
- 15_tkg.md: external_id format changed to trace_N, added trace management section (delete/restore/merge)
- 18_profile.md: note external_id matching fix
- 07_identity.md: note bind/unbind external_id fix
- regenerated user docs (doc/)
 2026-07-21 03:22:09 +08:00
Accusys 52bec30c90 feat: unify TKG face_track external_id to trace_N, add trace restore endpoint 
- tkg.rs: use trace_N as primary external_id, delete duplicate face_track_N nodes
- trace_agent_api.rs: add POST /trace/:id/restore to undo soft delete
- ensure 1:1 mapping: each trace has exactly one TKG node with full profile
 2026-07-21 03:19:17 +08:00
Accusys 0ad6927d25 feat: fix TKG external_id mismatch, add trace delete/merge endpoints, fix no-faces progress update 
- identity_binding.rs: match both face_track_N and trace_N external_id formats in bind/unbind/create_pending_person
- profile.rs: fix external_id matching and parameter ordering in trace profile updates
- trace_agent_api.rs: add DELETE /trace/:id (soft/hard delete) and POST /trace/:src/merge/:target endpoints
- qdrant_db.rs: add delete_points_by_filter method
- tkg.rs: add debug logging for scroll and parse operations
- job_worker.rs: update progress for no-faces TKG path
 2026-07-21 02:45:34 +08:00
Accusys d25b5921f7 fix: delete all progress keys on unregister 
- Delete progress:{uuid}:pipeline, :agent, :tkg
- Prevents stale Redis data from blocking new registrations
 2026-07-20 23:45:48 +08:00
Accusys 29d2cc7315 fix: wait for Qdrant face points before TKG build 
- Add Qdrant count verification before TKG build
- Wait up to 30 seconds for Qdrant to have all face embeddings
- Prevents race condition where TKG reads incomplete data
- Add count_points method to QdrantDb
 2026-07-20 23:26:42 +08:00
Accusys bc914ce3df fix: handle no_faces case - TKG proceeds without waiting for face_traced.json 
- Check face.json status before waiting for face_traced.json
- If status='no_faces', spawn TKG immediately (no face_track nodes)
- If status='has_faces', wait for face_traced.json
- Prevents infinite wait when video has no faces
 2026-07-20 22:34:09 +08:00
Accusys 5ec18ed520 fix: TKG build waits for face_traced.json to ensure face_track nodes are built 
- Add check for face_traced.json existence before spawning TKG build
- Prevents race condition where TKG reads Qdrant before trace_id is written
- Logs clearly when waiting and when proceeding
 2026-07-20 22:20:33 +08:00
Accusys 70e6915246 fix: remove error status handling that caused TKG to build without face_track nodes 2026-07-20 21:59:53 +08:00
Accusys 7dc910e2de feat: add cluster-agent endpoint, fix OCR labeling, fix ingestion blocking 
- Add POST /api/v1/file/:file_uuid/cluster-agent endpoint for on-demand face clustering
- Fix OCR chunks being labeled as ASRX: use ChunkType.as_str() instead of {:?}
- Rule 1 now deletes old chunks before re-inserting to avoid stale data
- Add fallback face_traced.json when store_traced_faces.py fails
- ingestion_complete now handles status='error' to unblock jobs
- VLM describe tool now uses trace_id with pre-extracted face crops
 2026-07-20 21:48:27 +08:00
Accusys 244af51edf docs: add file status sync API delivery document for Studio team 2026-07-19 22:00:34 +08:00
Accusys 2ed56b7973 feat: add sync-status endpoint to update file status based on processor files 2026-07-19 21:58:03 +08:00
Accusys 84dda3b5bc docs: add full file paths section to pose/appearance POC document 2026-07-19 20:45:42 +08:00
Accusys d0b7bfa56b feat: add pose and appearance endpoints for face detail display 
- Add GET /api/v1/file/:file_uuid/pose endpoint
- Add GET /api/v1/file/:file_uuid/appearance endpoint
- Keypoint deduplication (highest confidence)
- Add face-groups endpoint for Studio integration
- Add profile routes (trace-profile, file-profile)
- Update API documentation
 2026-07-19 20:43:35 +08:00
Accusys 455c6b81e6 fix: improve search ranking for semantic results 
- Lower title match score: 0.9 → 0.75
- Boost keyword score: capped 0.1 → boosted 0.1-0.85
- Skip text_match filter for semantic-only results
- Semantic results now rank higher than title/keyword

Verification:
- 'unfamiliar technology': 5 semantic results (score 0.68-0.74)
- 'storage': 60 semantic + 45 keyword + 60 title merged
- 'AUDIO MONITORING': 2 keyword (0.94) + 3 semantic (0.66-0.69)
 2026-07-19 14:40:58 +08:00
Accusys 3c924e1f83 fix: agent search uses smart_search for content queries 
- Update system prompt to prioritize smart_search over find_file
- Add source prefix ([OCR], [ASRX], [ASRX+OCR]) to exec_smart_search results
- Users can now see chunk content with source indicators
 2026-07-19 14:21:12 +08:00
Accusys 87aa7e0c40 feat: add source prefix to search results 
- Add [OCR], [ASRX], [ASRX+OCR] prefix to text_content
- Add content field to SemanticSearchResult struct
- Update SQL queries to include content field
- Helps users distinguish the source of search results
 2026-07-19 14:06:06 +08:00
Accusys 5e83ee7dac fix: keyword search for OCR text 
- Fix OCR confidence threshold: 0.5 → 0.2
- Fix fetch_ocr_texts frame calculation (use start_frame from DB)
- Fix text_match filter to use text_content instead of summary
- Process all merged results (not just top 30) to include keyword results
- Add logging for keyword search debugging

Fixes issue where OCR text like 'AUDIO MONITORING' and 'thunderbolt'
could not be found via keyword search.
 2026-07-19 13:33:31 +08:00
Accusys 0d9aa7b470 refactor: agent search defaults to all files, no longer asks which file 
- Removed '先確認用戶在問哪部影片' — no longer asks users to identify file
- Removed all 反问/ask-back rules (specify-type, ask-type, suggestions)
- Rule 1 now: 所有查詢都預設搜尋全部影片
- Smart/universal search file_uuid parameter noted as optional
- Simplified system prompt by ~40% (19 lines vs 47 lines)
 2026-07-11 09:26:55 +08:00
Accusys e4b8e9924c fix: agent_search tool descriptions + HAND_OBJECT casing + rule2 edge types 
- agent_search.rs: text_region -> text_trace, removed skin_tone_trace
  (never built, misleading LLM agents)
- tkg.rs + scan.rs: hand_object -> HAND_OBJECT for consistency with
  all other edge types (CO_OCCURS_WITH, SPEAKS_AS, etc.)
- rule2_ingest.rs: added LIP_SYNC and HAND_OBJECT to edge_type
  priority list so these edges generate relationship chunks
 2026-07-11 08:50:38 +08:00
Accusys 7635ad8d4d fix: correct TKG node_type/edge_type mismatches in scan.rs and tkg.rs 
Mismatches fixed (builder vs query layer):
- 'text_region' -> 'text_trace' (3 locations: scan.rs count + tkg.rs LIP_SYNC
  query + tkg.rs LIP_SYNC node ID lookup)
- 'yolo_object' -> 'object' (2 queries in scan.rs)
- 'FACE_TO_FACE' -> CO_OCCURS_WITH + JOIN with tkg_nodes (2 queries in
  scan.rs, so face_face_edges correctly counts only face-track pairs)
- 'HAND_OBJECT' -> 'hand_object' (2 queries in scan.rs)

Impact: ingestion_status and file stats endpoints now return accurate
counts; LIP_SYNC edges now resolve their target text nodes.
 2026-07-11 08:40:24 +08:00
Accusys b98a362de5 refactor: cargo fmt across 27 files + behavioral fixes 
Behavioral changes:
- postgres_db: reorder default processors (cut first), prevent overwriting completed status
- qdrant_db: fix scroll pagination exit condition (next.is_none())
- job_worker: add idempotency guards for face trace / TKG build; better error logging
- main.rs: add LineWriter for stdout buffering

Remaining diff is cargo fmt reformatting (line wrapping, import ordering).
 2026-07-11 02:03:28 +08:00
Accusys 701727fd08 test(tkg): add integration tests for TKG optimizations 
- test_build_frame_index: verifies frame-to-point index creation
- test_build_frame_index_empty: edge case for empty input
- test_find_trace_in_frame_exact_match: exact bbox match
- test_find_trace_in_frame_closest_match: closest bbox match
- test_find_trace_in_frame_no_matching_frame: missing frame returns None
- test_find_trace_in_frame_empty_index: empty index returns None
- test_node_id_map_lookup: verifies HashMap-based node ID lookup

Note: Pre-existing test compilation errors (E0063 in asr/asrx/face) block test execution.
 2026-07-10 07:44:37 +08:00
Accusys 01f8b89636 perf(tkg): batch INSERT for co_occurrence edges using QueryBuilder 
- Collect edges in Vec first, then batch insert in chunks of 100
- Uses sqlx::query_builder::QueryBuilder for efficient batch INSERT
- Reduces SQL round-trips from O(edges) to O(edges/100)
- Maintains ON CONFLICT DO UPDATE semantics
 2026-07-10 01:10:14 +08:00
Accusys 2da0ada34a feat(tkg): add Qdrant retry logic with exponential backoff 
- scroll_face_points now retries up to 3 times on failure
- Exponential backoff: 1s, 2s, 4s between attempts
- Logs retry attempts for debugging
- Prevents silent failures when Qdrant is temporarily unavailable
 2026-07-10 00:58:30 +08:00
Accusys 13abb15595 refactor(tkg): extract shared t() function to schema module 
- Create src/core/db/schema.rs with t() and table_name() functions
- Remove duplicate t() from tkg.rs and trace_agent_api.rs
- postgres_db.rs uses schema::table_name() from shared module
 2026-07-10 00:08:12 +08:00
Accusys 5a941f857c refactor(tkg): remove dead code - FaceTraceRow, FaceDetectionRow, build_face_track_nodes_from_face_json 2026-07-09 23:45:59 +08:00
Accusys 1ac0a144d5 perf(tkg): pre-build node ID map to eliminate N+1 SQL in edge builders 
- Add build_node_id_map() to fetch all node IDs once into HashMap
- Pass node_id_map to all 6 edge builders
- build_co_occurrence_edges: uses map lookup instead of per-face SELECT
- build_speaker_face_edges: uses map lookup instead of per-trace SELECT
- Remove redundant Qdrant scroll in build_co_occurrence_edges
- Expected: 90%+ reduction in SQL queries during edge building
 2026-07-09 23:44:06 +08:00
Accusys 356edb16be perf(tkg): eliminate N+1 Qdrant HTTP requests in TKG builders 
- Add build_frame_index() for O(1) frame-to-face lookup
- Add find_trace_in_frame() for in-memory bbox matching
- Share face_points across all 6 TKG builders (was 4x redundant fetches)
- Remove get_trace_for_face() (31,666 HTTP requests → 0)
- Change Qdrant scroll error from warn to error
- build_lip_track_nodes_from_pg: uses frame_index instead of per-face HTTP
- build_appearance_trace_nodes: uses frame_index instead of per-face HTTP
- Expected speedup: ~200-600x for lip_track build on large files
 2026-07-09 16:05:50 +08:00
Accusys 149226ff17 fix: production service auto-start via launchd + fix worker retry loop 
- Replaced old wrapper script with two launchd plists (server + worker)
- Both services auto-start at login via RunAtLoad + KeepAlive
- Removed MOMENTRY_FORCE_RETRY=true from .env to prevent infinite retries
- Added REDIS_URL and all required env vars to launchd plists
- Truncated 3.6GB of oversized logs (2.3G worker + 1.3G launchd stdout)
- Deleted 1214 orphaned processor_results records
- Updated run-server-3002.sh to manage via launchctl
- Removed obsolete wrapper scripts
 2026-07-07 05:41:09 +08:00
Accusys bb606f52f5 feat: independent TKG processing mechanism with operation logging 
- Created src/core/tkg/ module with service, log, and models
- Added tkg_operation_log table for tracking all TKG operations
- TkgService: build, rebuild (with force option), delete, get_operations
- Updated rebuild endpoint with force parameter
- Added GET /api/v1/file/:file_uuid/tkg for operation history
- Added DELETE /api/v1/file/:file_uuid/tkg for TKG deletion
- TKG rebuild now always triggers Rule 2 (even with 0 edges)
- Full audit trail for all TKG operations (create/update/delete/rebuild)
 2026-07-07 03:15:59 +08:00
Accusys f56bdb7fbb fix: ingestion-status shows all steps done when job is completed 
When a job status is 'completed', all ingestion steps are marked as 'done'
regardless of node/edge counts. This fixes the UI showing pending steps
for completed jobs with 0 nodes/edges (e.g., videos with single speaker).
 2026-07-06 22:08:11 +08:00
Accusys 3067896b0f fix: comprehensive fixes for ingestion and TKG 
1. Add Redis PipelineProgress cleanup in unregister_internal
   - Deletes progress key when file is unregistered

2. Fix ingestion_complete asr_status query
   - Query ASRX first, then ASR
   - Filter out NULL values to avoid false negatives

3. Fix TKG 0 nodes/edges support
   - TKG is considered done if face traces are complete
   - TKG may create 0 nodes/edges for videos with minimal content

4. Add ASRX fallback to ASR segments
   - When ASRX has 0 segments but ASR has segments, use ASR segments
   - Ensures at least one ASRX output when ASR has content
 2026-07-06 21:14:08 +08:00
Accusys 146d3cedb2 fix: update PipelineProgress when job completes via essential_completed path 
When a job completes via the essential_completed branch (all essential
processors done but some non-essential failed), the PipelineProgress
was not being updated to 100%, causing a discrepancy where the status
showed 'completed' but progress showed <100%.

Now PipelineProgress.mark_completed() is called in both completion paths.
 2026-07-06 19:08:56 +08:00
Accusys 765db8ae9f fix: TKG FPS calculation for ASRX fallback segments 
When ASRX falls back to ASR segments, end_frame may be 0.
The FPS calculation now handles this case correctly by checking
both end_frame > 0 and end_time > 0 before dividing.

This prevents division by zero and incorrect FPS values when
processing videos with ASRX fallback segments.
 2026-07-06 16:08:32 +08:00
Accusys dd63dbff9b fix: support videos with no audio or no faces 
1. trace_done now checks for 'no_faces' status in face_traced.json
   - Videos with no detected faces now complete correctly
   - Previously stuck because trace_count=0 returned false

2. ASRX fallback to ASR segments includes start_frame/end_frame
   - Added _convert_asr_segments_to_asrx helper function
   - TKG can now process fallback segments correctly

This allows processing of:
- Videos with no audio track (ASR: no_audio_track)
- Videos with no faces (face_traced.json: no_faces)
 2026-07-06 15:54:14 +08:00
Accusys 27660f48e4 fix: ASRX fallback to ASR segments on error instead of silent_audio 
When ASRX processing fails (error or exception), it now falls back to
ASR segments instead of returning silent_audio with 0 segments.

This ensures asrx_segments >= asr_segments always, fixing the bug where
ASRX would report 0 segments even when ASR detected speech.
 2026-07-06 14:23:53 +08:00
Accusys e4fdbbc18a fix: clear stale PipelineProgress when new job starts 
When a file is re-registered, the old PipelineProgress in Redis
was not cleared, causing the Portal to show 'completed' status
even though the new job was still running.

Now PipelineProgress is deleted when a new job starts processing.
 2026-07-06 13:07:53 +08:00
Accusys 004ff9ad48 fix: ingestion_complete now checks TKG nodes before completing job 
Previously, ingestion_complete only checked ASR and face traces,
causing jobs to complete before TKG was triggered. This resulted in
TKG nodes being 0 even after job completion.

Now ingestion_complete also checks if TKG nodes exist for the file.
The job stays in 'running' status until TKG completes.
 2026-07-06 12:24:25 +08:00
Accusys 552f539bdf fix: remove TKG guard to prevent deadlock 
The TKG guard was checking if nodes exist before spawning TKG build.
This caused a deadlock when:
1. TKG spawns (async)
2. ingestion_complete checks → false (nodes not yet created)
3. Returns Ok(false), job stays running
4. Next poll: guard checks nodes → no nodes yet (TKG still running)
5. Skips TKG → deadlock!

Since build_tkg uses ON CONFLICT (idempotent), it's safe to call
multiple times. Removed the guard to fix the deadlock.
 2026-07-06 11:28:40 +08:00
Accusys 221aa4c4cc fix: worker detects deleted jobs and skips them 
When a job is deleted from the database (e.g., by unregister),
the worker now checks if the job still exists before processing.
If the job no longer exists, it skips it instead of getting stuck.

This fixes the issue where unregistering a file would leave the
worker stuck trying to process a non-existent job.
 2026-07-06 10:28:11 +08:00
Accusys 799ede5a0e feat: OCR independent chunks + TMDb seed with file_uuid 
- Rule 1 now creates OCR-only chunks instead of merging into ASRX
- generate_seed_embeddings.py supports --file-uuid parameter
- get_seeds() filters by file_uuid
- identity_matcher.py uses file_uuid for seed matching
- Push QDRANT_API_KEY to Python subprocesses
- Face clustering uses frame+bbox matching instead of face_id
- Portal uses JWT authentication
- FilesView filter logic fixed
 2026-07-06 08:56:56 +08:00
Accusys cb604b74ec docs: update Rule 1 OCR independent chunks documentation 
Updated Searchable_Chunk_Rules.md and pipeline.md to reflect:
- Phase 1: ASRX segments (pure speech, NO OCR merge)
- Phase 2: OCR-only chunks (all OCR frames grouped by proximity)
- New stats API steps: rule1_ocr, rule1_ocr_chunks
 2026-07-05 23:36:56 +08:00
Accusys e91d51cc5e feat: OCR independent chunks (方案 A) + stats API 
Rule 1 now creates OCR chunks separately from ASRX segments:
- Phase 1: ASRX segments (pure speech, NO OCR merge)
- Phase 2: OCR-only chunks (all OCR frames grouped by proximity)

Added OCR statistics to ingestion status API:
- rule1_ocr: shows OCR pre_chunks count
- rule1_ocr_chunks: shows OCR-only chunks count

Example: FilmRiot_test now has 32 ASRX + 3 OCR-only = 35 chunks
Stats: rule1_sentence: 35, rule1_ocr: 30, rule1_ocr_chunks: 3
 2026-07-05 23:31:06 +08:00
Accusys 5a3f791ecd feat: only run identity agent if file has seed identities 
Identity agent now checks Qdrant _seeds collection for the file's
seed identity photos before running. If no seeds exist, the agent
is skipped to avoid unnecessary processing.

Flow:
1. Job completes (Face + ASRX done)
2. Check Qdrant _seeds for file_uuid
3. If seeds exist → run identity agent
4. If no seeds → skip identity agent
 2026-07-05 22:22:22 +08:00
Accusys 0b82aa875c feat: Rule 1 now creates chunks for OCR-only text 
Previously Rule 1 only created chunks from ASRX segments, merging OCR
text where frame ranges overlapped. OCR text that didn't overlap with
any ASRX segment was ignored.

Now Rule 1 has two phases:
1. Process ASRX segments (merge OCR where overlapping) - existing behavior
2. Create chunks for OCR-only text (frames not covered by ASRX)

OCR-only chunks are grouped by consecutive frames (within 5 frames)
to avoid creating too many single-frame chunks.

Example: ASRX 819 + OCR-only 4 = 823 sentence chunks
 2026-07-05 22:06:35 +08:00
Accusys 465552f8b2 fix: remove duplicate 'asr' pre_chunks storage in ASRX handler 
Bug: ASRX handler stored pre_chunks as BOTH 'asrx' and 'asr' types.
This caused confusion because Rule 1 queries 'asrx' type, but only
'asr' type existed in the database (asrx type was deleted or never stored).

Fix: Remove the duplicate 'asr' storage (lines 530-542).
ASRX handler now only stores 'asrx' type pre_chunks to workspace SQLite.
PostgreSQL pre_chunks are stored by processor.rs with correct 'asrx' type.

This ensures Rule 1 can find ASRX pre_chunks correctly.
 2026-07-05 19:49:03 +08:00
Accusys 5fcd5212d5 fix: FilesView merge logic and computed property 
Bug: When regFiles API fails, all files get 'unregistered' status even if
they are actually registered. Also, computed property was using reference
to files.value instead of a copy, which could cause mutation issues.

Fix:
- Fetch scan results FIRST (source of truth for files on disk)
- Use scan API's is_registered field as fallback status
- Only override with regFiles data if file exists in scan results
- Computed property now uses [...files.value] to create a copy
- Skip files from regFiles that don't exist on disk (deleted)
 2026-07-04 23:42:42 +08:00
Accusys 53f28ac458 fix: FilesView filter logic - remove 'registered_scan' status 
Bug: Scan files were getting status 'registered_scan' which doesn't match
any filter value (unregistered/pending/processing/completed/indexed/unindexed).
When toggling filters on/off, files would disappear because their status
didn't match any valid filter.

Fix:
- Removed 'registered_scan' status entirely
- Fetch regFiles FIRST to get real statuses
- Scan files default to 'unregistered' status
- regFiles overlay with actual status (pending/processing/completed)
- Increased regFiles page_size to 200 for larger libraries
 2026-07-04 22:48:23 +08:00
Accusys 7fc4dcbddb feat: add media type and indexed status filters to FilesView 
Frontend:
- Add media type filter (全部/影片/照片)
- Add indexed status filter (未入庫/已入庫)
- Show media type column with icons
- Fix status filter to handle indexed/unindexed correctly
- Determine media type from file extension

Backend:
- Add total_chunks field to FileItem API response
- Query chunk counts efficiently in batch with IN clause
- Frontend uses total_chunks to determine is_indexed status
 2026-07-04 22:41:51 +08:00
Accusys 96e13e40cb fix: all_completed now checks ALL expected processors have results 
Bug: all_completed only checked existing results, not missing processors.
If a processor (like pose) never created a result row, all_completed would
still return true and mark the job as completed.

Fix: all_completed now checks that every processor in job_processors has
a corresponding completed result. Added logging for missing processors.

Also fixed:
- any_pending now checks all expected processors, not just existing results
- Added missing_processors detection and logging
 2026-07-04 22:09:38 +08:00
Accusys 4e8c0ea5b9 fix: skip empty ASRX segments in Rule 1, fix chunk_id numbering 
- Skip chunks where both ASRX text and OCR text are empty
- Use count-based chunk_id instead of index to avoid gaps
- This ensures PostgreSQL and Qdrant chunk counts match
 2026-07-04 12:41:40 +08:00
Accusys e4d6fbac50 feat: add search_by_appearance agent tool for clothing color search 
- New Python script: clothing_color_search.py
- New agent tool: search_by_appearance (red, blue, green, etc.)
- Uses appearance.json person bboxes + HSV color analysis
- Returns matched frames with confidence scores
 2026-07-02 22:22:07 +08:00
Accusys 78364afc51 fix: keyword search - add text_content field and CJK support 
- Added text_content field to SearchResult and SemanticSearchResult
- Added get_chunk_by_id_no_embedding for keyword results without embedding requirement
- Fixed search_bm25 to use position-based ranking for CJK/Korean content
- Fixed sqlx column mapping with explicit alias
- Skip text_match filter for keyword-only results
- Use text_content as fallback when summary is empty
 2026-07-02 21:16:38 +08:00
Accusys 5a9d4325d8 fix: face thumbnail crop using bbox parameters 
- Added bbox_x, bbox_y, bbox_w, bbox_h fields to ThumbQuery
- face_thumbnail now uses bbox params for ffmpeg crop filter
- Frontend passes bboxX/Y/Width/Height which maps to bbox_x/y/w/h
 2026-07-02 18:31:54 +08:00
Accusys 3035c6db5d fix: add /api/v1/face-thumbnail route for People view thumbnails 
- Frontend calls /api/v1/face-thumbnail?uuid=...&frame=...
- Backend only had /api/v1/file/:file_uuid/thumbnail
- Added compat route and uuid field to ThumbQuery
 2026-07-02 18:13:38 +08:00
Accusys 28a4e9b1b8 fix: worker/processor.rs ASRX 使用正確的 start_frame 
- 使用 segment.start_frame 取代 i (sequential index)
- data JSON 加入 start_frame, end_frame
 2026-07-02 17:10:24 +08:00
Accusys 3943075a9b fix: ASRX pre_chunks 使用正確的 start_frame 
- pipeline/mod.rs: 使用 segment.start_frame 取代 i (sequential index)
- data JSON 加入 end_time, start_frame, end_frame 供 rule1_ingest 使用
- 確保 ASRX pre_chunks 有正確的 frame 資訊
 2026-07-02 17:08:50 +08:00
Accusys e2b3858b67 fix: job never completes - processor_results.file_uuid is NULL 
- ingestion_complete query used file_uuid column which is always NULL
- Changed to JOIN processor_results with monitor_jobs on job_id
- All stuck jobs now complete successfully
 2026-07-02 16:42:24 +08:00
Accusys bd6d108ade fix: remove trace_chunks from API + fix OCR frame calculation 
- Removed trace_chunks field from PostgresStats struct
- Removed trace_chunks query from get_file_stats and get_ingestion_status
- Fixed OCR fetch_ocr_texts to compute frames from start_time*FPS
- Updated scan.rs to use separate count_nodes/count_edges functions
 2026-07-02 16:23:25 +08:00
Accusys d4c26deae2 fix: pipeline progress computed from DB state instead of Redis 
- get_pipeline_progress_handler now queries actual DB counts
- Fixed processor_results query (requires JOIN with monitor_jobs)
- Card progress bar and right-click content now consistent
 2026-07-02 15:11:25 +08:00
Accusys 619b056ada fix: TKG stats API returning 0 - count_by_type used wrong column 
- tkg_nodes has no edge_type column, query was failing silently
- Split into count_nodes(node_type) and count_edges(edge_type)
- Fixed text_region → text_trace node type name
- Also: OCR frame fix in rule1 (end_frame computed from end_time+FPS)
 2026-07-02 14:53:47 +08:00
Accusys 6507766ea2 fix: Qdrant collection name + PipelineProgress accumulation 
- scan.rs: rule1 collection 'momentry_public_rule1_v2' → 'momentry_rule1'
- progress.rs: publish_pipeline_progress now reads existing progress and merges stages
 2026-07-02 13:44:45 +08:00
Accusys 64f29d614b fix: Rule 1/TKG trigger conditions + essential_failed guard 
- Rule 1 trigger: has_asr_or_asrx → has_asrx (wait for ASRX pre_chunks)
- P3/P4 triggers: has_asr_or_asrx → has_asrx (need ASRX data)
- Add essential_failed check: job only fails if essential processor fails
- P2/P3/P4 triggers: all_completed → has_face && has_asrx
- Add publish_pipeline_progress calls at each pipeline stage
 2026-07-02 13:31:38 +08:00
Accusys 3eabd45882 fix: ASRX duplication, TKG edges, trace ingest, and add pipeline progress publishing 
- ASRX handler no longer stores duplicate 'asr' pre_chunks
- Pre_chunks storage made idempotent (delete-before-insert)
- Rule 1 + trace_ingest changed to query 'asrx' not 'asr'
- Trace chunks removed (dynamic from TKG/Qdrant)
- TKG scroll_face_points fixed: trace_id >= 1 (not == 1)
- TKG AsrxSegmentEntry: start/end -> start_time/end_time (match ASRX JSON)
- Unregister error handling: log instead of silent discard
- Add publish_pipeline_progress calls at each pipeline stage
  (processors, rule1, face_trace, identity_agent, TKG, rule2, completion)
 2026-07-02 10:43:46 +08:00
Accusys d791d138f2 fix: API endpoints for file_uuid filtering of pending identities 
- get_file_identities: UNION face_detections + file_identities
- list_identities: add file_bindings from file_identities table
- Add back /api/v1/traces/unassigned route
- Total count query now includes file_identities

Frontend can now:
- Filter pending identities by file_uuid
- Filter pending faces (unassigned traces) by file_uuid
 2026-06-26 14:26:36 +08:00
Accusys bd7d8c77bf feat: add migrate_manual_file_identities.py 
Migrate identities.file_uuid to file_identities table for consistent structure
 2026-06-26 13:55:10 +08:00
Accusys 6f1a560d06 fix: add script_dir() method to PythonExecutor 2026-06-26 13:46:23 +08:00
Accusys 67caf09732 feat: tmdb_agent now inserts identities and file_identities to DB 
- tmdb_agent.py: INSERT identities with status='pending'
- tmdb_agent.py: INSERT file_identities (file_uuid → identity_id)
- identity.json: file_bindings includes file_uuid, movie_id, character
- backfill_file_identities.py: migrate existing TMDb identities
- Tested: 27 Charade cast identities linked to file
 2026-06-26 13:39:08 +08:00
Accusys 6cbc11efda feat: add confirm_identity API endpoint 
- Add POST /api/v1/agents/identity/confirm endpoint
- Calls confirm_identity.py to bind trace to identity
- Updates TKG, Qdrant _faces, PG face_detections, _seeds
- Optional Round 2 propagation after confirmation
- Fix trace_id=0 check in confirm_identity.py (use 'is not None')
- Document API endpoint in 08_identity_agent.md
 2026-06-26 08:30:03 +08:00
Accusys 615f9da2df fix: identity status - TMDb and user_defined identities start as 'pending' (確認制) 2026-06-26 02:16:46 +08:00
Accusys a2f2b7918a fix: add trace_id and status to face_track nodes, force update properties on rebuild 2026-06-26 00:19:00 +08:00
Accusys 0c3f385b1f remove: skin_tone_trace node type 
- skin_tone is a person attribute (like height), not trace attribute
- Remove build_skin_tone_trace_nodes function
- Remove skin_tone_trace_nodes from TkgResult and API response
- Remove skin_tone_trace from documentation tables
 2026-06-25 19:21:05 +08:00
Accusys fd2edd5736 fix: TKG rebuild type mismatch and face_track nodes 
- Fix trace_id type mismatch (INT4 vs i64) with explicit ::bigint cast
- Change build_face_track_nodes to use from_pg version
- Add skin_tone_trace_nodes to API response
- Add #[derive(Serialize)] to TkgResult
- Fix Unicode panic in text label truncation
- Add push_existing_embeddings.py script
 2026-06-25 11:23:53 +08:00
Accusys ecb0e9c7d0 feat: add /api/v1/health public endpoint 
- Add public health routes at /api/v1/health, /api/v1/health/detailed, /api/v1/health/consistency
- Make health functions and response types public
- Public routes bypass auth middleware (unlike protected /api/v1/* routes)
 2026-06-25 10:05:33 +08:00
Accusys 4273576612 feat: implement skin_tone_trace node builder and standardize TKG node naming 
- Add build_skin_tone_trace_nodes() to tkg.rs (Fitzpatrick I-VI classification)
- Add skin_tone_trace_nodes field to TkgResult
- Standardize node naming: _trace -> _track (text uses _region)
- Add external_id format column to Node Types table
- Add storage names to Edge Types table
- Create TKG_FORMATION_V1.0.md with Phase 0-4 definition, flow diagram, queries
- Add cross-reference from identity_agent_v4.0.md to TKG Formation
- Update Python scripts to executable mode
 2026-06-25 03:09:16 +08:00
Accusys 406b2d5524 docs: update TKG documentation for Identity Agent V4.0 
- Add new file: 2026-06-25_identity_agent_v4.0.md (M4 workspace)
  - Complete architecture overview
  - All phases completed
  - Thresholds, components, test results

- Update: API_WORKSPACE/modules/15_tkg.md
  - Correct node type: face_trace → face_track
  - Add text_region (replaces text_trace)
  - Add Identity Agent integration section
  - face_track status values (pending/suggested/confirmed/stranger)
  - Example face_track node with identity properties
 2026-06-25 02:27:34 +08:00
Accusys 4b4d37b332 fix: qdrant_request empty body handling (use 'is not None' check) 
Fix qdrant_request() to properly handle empty dict {} as body.
Python's 'if body' evaluates to False for empty dict, causing EOF error.

Changed:
- data = json.dumps(body).encode() if body is not None else None

Also cleaned up count_seeds() to use consistent body passing.
 2026-06-25 02:19:07 +08:00
Accusys b19b1a8c46 fix: count_seeds empty body handling 
Fix count_seeds() to always pass valid JSON body to Qdrant count API.
Empty dict {} was causing EOF error when no source filter provided.
 2026-06-25 02:02:17 +08:00
Accusys d20819b03b feat: add manual_seed.py for user-selected face trace seed creation 
Implements:
- create_identity(): Create PG identity (source='manual')
- create_manual_seed(): Full flow from trace → seed → confirm
  - Get trace centroid embedding from Qdrant _faces
  - Create identity in PG
  - Push to Qdrant _seeds
  - Confirm trace binding (TKG + Qdrant + PG)
  - Auto-trigger Round 2 propagation
- list_pending_traces(): List traces for user selection
- run_propagation(): Auto propagation trigger

Usage:
  # List pending traces
  python manual_seed.py --file-uuid <uuid> --list

  # Create seed from trace
  python manual_seed.py --file-uuid <uuid> --trace-id 1 --name 'John Doe'

  # Custom UUID
  python manual_seed.py --file-uuid <uuid> --trace-id 1 --name 'John Doe' --identity-uuid xxx

  # No propagation
  python manual_seed.py --file-uuid <uuid> --trace-id 1 --name 'John Doe' --no-propagate

Flow: select trace → label → create identity → push seed → auto-bind → propagate
 2026-06-25 01:49:53 +08:00
Accusys b5e3adf5de feat: add generate_seed_embeddings.py for TMDb profile extraction 
Implements:
- get_tmdb_identities(): Query PG for TMDb identities with profile photos
- download_tmdb_image(): Download profile image from TMDb (handles full URL or path)
- extract_face_embedding(): CoreML FaceNet 512D embedding extraction
- generate_seed_embeddings(): Full flow: download → extract → push to _seeds

TMDb image handling:
- Supports both full URL (https://...) and path (/xxx.jpg)
- Uses 'original' size for better quality (replaces /w185)

Usage:
  python generate_seed_embeddings.py                # All TMDb identities
  python generate_seed_embeddings.py --limit 10    # Limit to 10
  python generate_seed_embeddings.py --dry-run     # Don't push to Qdrant

Tested: 3 seeds successfully pushed (Cary Grant, Audrey Hepburn, Walter Matthau)
 2026-06-25 01:45:48 +08:00
Accusys 4198a74002 feat: add confirm_identity.py for identity binding confirmation 
Implements:
- confirm_single_trace(): Confirm identity binding for one trace
  - Update TKG face_track node: status='confirmed'
  - Update Qdrant _faces: identity_uuid for all points
  - Update PG face_detections: identity_id
  - Add trace centroid to _seeds (source='propagation')
  - Auto-trigger Round 2 matching

- batch_confirm_from_json(): Batch confirm from suggestions file
  - Confirm multiple suggestions from identity_matcher output
  - Final propagation after all confirmations

- run_round_2_propagation(): Auto propagation trigger
  - Get confirmed traces from TKG nodes
  - Build identity_map for propagation
  - Run identity_matcher.py Round 2

Usage:
  python confirm_identity.py --file-uuid <uuid> --trace-id 1 --identity-id 1 --identity-uuid xxx --name 'Tom Hanks'
  python confirm_identity.py --file-uuid <uuid> --json suggestions.json
  python confirm_identity.py --file-uuid <uuid> --json suggestions.json --no-propagate
 2026-06-25 01:38:00 +08:00
Accusys 21b9f500d9 feat: add TKG node marking for Identity Agent suggestions 
TKG Helper (scripts/utils/tkg_helper.py):
- mark_face_track_suggested(): Mark node as 'suggested' with pending identity info
- mark_face_track_confirmed(): Mark node as 'confirmed' with identity_ref
- mark_face_track_stranger(): Mark node as 'stranger' with stranger_ref
- batch_mark_suggestions(): Batch mark multiple traces
- batch_mark_strangers(): Batch mark stranger clusters
- get_face_track_nodes(): Get all face_track nodes for a file
- get_pending_face_tracks(): Get nodes with status='pending'
- get_suggested_face_tracks(): Get nodes with status='suggested'

Identity Matcher updates:
- Add --mark-tkg flag to update TKG nodes after matching
- Integrates with tkg_helper for batch operations

Node properties schema:
- status: pending | suggested | confirmed | stranger
- pending_identity_name/uuid/id: suggested identity info
- suggested_by: tmdb | propagation | manual
- confidence: matching score
- identity_ref: confirmed identity reference
 2026-06-25 01:11:05 +08:00
Accusys 6851cb4734 feat: add identity_matcher.py for multi-angle face matching 
Implements:
- match_faces_round_1: TMDb seeds → traces (TH=0.55)
- match_faces_round_2: Confirmed traces → pending (TH=0.55)
- match_faces_round_3_plus: Propagation (TH=0.50)
- cluster_strangers: Greedy merge unmatched traces (TH=0.40)
- multi_angle_match: max(cosine(seed, rep)) across 3 representatives
- cosine_similarity: Vector similarity calculation

Usage:
  python identity_matcher.py --file-uuid <uuid> --round 1
  python identity_matcher.py --file-uuid <uuid> --round 2 --confirmed-traces 1,2,3
  python identity_matcher.py --file-uuid <uuid> --round 1 --stranger

Output: JSON with suggestions {trace_id: {identity_id, uuid, name, score, suggested_by}}
 2026-06-25 00:57:22 +08:00
Accusys 580c4b4017 feat: add _seeds collection helper functions for Identity Agent 
- Add ensure_seeds_collection(): create _seeds collection (512D, Cosine)
- Add push_seed_embedding(): push identity seed with payload {identity_id, uuid, name, source, file_uuid, trace_id, tmdb_id}
- Add get_seeds(): get all seeds (optional source filter)
- Add search_seeds(): cosine search against seeds
- Add delete_seed(): delete seed by identity_id
- Add count_seeds(): count seeds (optional source filter)
- Add get_trace_representatives(): get 3 representatives per trace for multi-angle matching
- Add get_trace_centroid(): get centroid embedding for a trace
- Add update_identity_in_faces(): update identity_id/uuid for all face points with trace_id

Point ID strategy: identity_id directly as point_id for _seeds collection
All functions tested successfully
 2026-06-25 00:47:25 +08:00
Accusys 9fbb4f9b48 feat: add Qdrant _faces collection embedding push 
- Add qdrant_faces.py utility module for _faces collection operations
- Modify face_processor.py to push embeddings to Qdrant (CoreML extraction re-enabled)
- Modify store_traced_faces.py to update trace_id in Qdrant after face tracking
- Collection schema: 512D vectors, Cosine distance, fixed name '_faces'
- Payload: file_uuid, frame, trace_id, bbox, confidence, identity_id/uuid, stranger_id
- Batch size: 100 (default), configurable via QDRANT_BATCH_SIZE env var
- Error handling: face_processor.py exits with error if Qdrant push fails
 2026-06-25 00:23:20 +08:00
Accusys 074cdcdbed refactor: remove face embedding architecture - single Qdrant _faces collection 
- Delete FaceEmbeddingDb module (face_embedding_db.rs)
- Stub match_faces_iterative, generate_seed_embeddings, tmdb_match_handler
- Remove sync_trace_embeddings, populate_face_embeddings_to_qdrant
- Remove embedding from face.json output (face_processor.py)
- Remove embedding from PG UPDATE (store_traced_faces.py)
- Remove workspace traces staging (checkin.rs, qdrant_workspace.rs)
- Fix tests: add pose_angle to Face, hand_nodes to TkgResult

Disabled functions (need reimplement with _faces):
- match_faces_iterative (identity agent)
- generate_seed_embeddings (TMDb seeds)
- tmdb_match_handler (TMDb matching)
- cluster_face_embeddings, search_similar_faces
- merge_traces_within_cuts
 2026-06-24 22:27:09 +08:00
Accusys 360cb991e1 feat: add queued status + FIFO queue ordering 
- Add Queued variant to VideoStatus enum
- Trigger sets videos.status='queued' instead of staying 'pending'
- Worker sets videos.status='processing' on pickup
- list_monitor_jobs_by_status ORDER BY created_at ASC (FIFO)
- queue_position counts both 'pending' and 'queued' jobs
 2026-06-24 05:18:40 +08:00
Accusys 14e886cc08 feat: progressive multi-round face matching + pending person API 
- Identity agent: per-face max matching, multi-round with derived
  seeds from high-confidence faces, angle diversity filter (cosine sim < 0.90)
- Pending person API: POST /file/:file_uuid/pending-person
  + GET /file/:file_uuid/pending-persons with status=pending, source=manual
- Update API docs (07_identity.md)
 2026-06-24 03:42:04 +08:00
Accusys 766a1d9a6d feat: Swift Face Pose integration + TKG 方案 B 
Major Changes:
- swift_face_pose: output pose angles (yaw/pitch/roll) in face.json
- face_processor.py: call swift_face_pose (dual output: face.json + pose.json)
- Face struct: add pose_angle field
- TKG 方案 B: gaze/lip_track nodes from face.json (no face_detections dependency)
- Chunk cleanup: delete old data before rebuild (avoid duplicate key)
- Hand nodes: classify by hand_type + gesture (15 combinations)
- HAND_OBJECT edges: bbox spatial matching (174 matches)

Test Results:
- Blake Jones: 8 faces, pose_angle ✓, 66 nodes, 174 edges
- FilmRiot: 394 faces, pose_angle ✓, 35 nodes, 39 edges
- Left hands: 132, Right hands: 2

Architecture:
- All TKG nodes built from JSON files (face.json, hand.json, yolo.json)
- Swift processors: sample_interval=3 (Face/Pose/Hand sync)
- Cleanup functions: delete_tkg_nodes_by_uuid, delete_tkg_edges_by_uuid
 2026-06-23 05:47:24 +08:00
Accusys e1e2da2140 fix: processor-counts API + ASRX field name conversion 
- Fix processor-counts API to correctly read JSON counts:
  - YOLO: use frames.length (was returning null)
  - CUT: prioritize scenes.length over frame_count
  - Result: YOLO 1963 frames, CUT 25 scenes (correct)

- Fix ASRX field name conversion:
  - Convert start_time/end_time → start/end for ASRX compatibility
  - Prefer frame-based positioning over time-based

- Document issues in issues_2026-06-21.md:
  - Issue 6: ASRX field name mismatch
  - Issue 7: processor-counts API null values
 2026-06-22 23:33:39 +08:00
Accusys db8bb8fa95 fix(tkg): handle null identity_id + remove skin_tone nodes 
- Fix Phase 2.5 null handling in build_gaze/lip_track_nodes
  - Use query_scalar::<_, Option<i64>> + flatten() for nullable fields
  - Prevents 'unexpected null' decoding errors

- Remove skin_tone_trace_nodes from TKG build
  - Delete build_skin_tone_trace_nodes function (110 lines)
  - Remove from TkgResult struct and API response
  - Skin tone should be independent function, not in TKG

Result: TKG rebuild now completes successfully
- Nodes: 40 (face_track, gaze_track, text_region, appearance)
- Edges: 2967 (co_occurrence edges increased from 21 → 2964)
 2026-06-22 16:39:47 +08:00
Accusys 70e849d3ae refactor: remove Rule 3, Story, and Caption processors 
- Remove Rule 3 (Scene Chunking) from worker auto-trigger
- Remove rule3_ingest.rs and related imports
- Remove Story/Caption from playground module parsing
- Clean up scan.rs Rule 3 display
- Fix ASRX field name conversion (start_time -> start)

Reason: Story/5W1H/Scene accuracy too poor - will redesign later
 2026-06-22 15:34:02 +08:00
Accusys 22f13eca4b fix(cut): change ffprobe output format to default=nk=0 
- Problem: compact=p=0:nk=1 outputs pipe-delimited format without pts_time=
- Fix: default=nk=0 outputs pts_time=XXX format that parser can match
- Result: Charade scene detection from 1 scene -> 833 scenes (correct)
 2026-06-22 13:25:16 +08:00
Accusys 30b252ac95 fix: pre_chunks schema + TMDb movie name extraction 
- pre_chunks: add chunk_type, text_content columns; drop NOT NULL on
  coordinate_type/coordinate_index (INSERT statements reference these
  columns but CREATE TABLE was missing them)
- run_migrations: add ALTER TABLE for existing databases
- extract_movie_name: filter noise words (youtube, fps, 24fps, 1080p,
  pure digits) so 'Charade_YouTube_24fps' → 'Charade'
- run-server-3002.sh: add companion worker startup (matching 3003 script)
 2026-06-22 11:55:12 +08:00
Accusys f4de741d5b fix: add appearance back to processor list, keep mediapipe/story filtered out 2026-06-22 09:20:16 +08:00
Accusys c93b54efeb fix: filter deprecated processors from trigger API requests 2026-06-22 09:15:02 +08:00
Accusys 4ba248513e fix: correct processor list - remove deprecated mediapipe/appearance/story, fix auto-pipeline order 
- ProcessorType::all(): remove MediaPipe, Appearance, Story (mediapipe replaced by Swift)
- files.rs auto-pipeline: fix order to cut,asr,asrx,yolo,ocr,face,pose (was missing asr)
- postgres_db.rs run_migrations(): rewrite to auto-create all 38 tables idempotently
 2026-06-22 08:49:41 +08:00
Accusys 7e548f8b08 release: v1.3.0 - TKG node type renaming 
Changes:
- Rust: face_trace → face_track (45 occurrences in 8 files)
- Rust: gaze_trace → gaze_track, lip_trace → lip_track
- Python: tkg_builder.py unified + pipeline_checklist.py fixed
- Swift: swift_hand.swift hand state detection (empty vs holding)

Node type changes:
  face_trace    → face_track
  person_trace  → body_track
  gaze_trace    → gaze_track
  lip_trace     → lip_track
  hand_trace    → hand_track
  speaker       → speaker_segment
  object        → detected_object
  text_trace    → text_region

Migration:
  PUBLIC schema: 12970 + 892 + 305 rows updated
 2026-06-22 07:18:21 +08:00
Accusys bce9435823 feat: add Level 2/3 dynamic feature extraction CLI 
- test_level2_level3.py: on-demand extraction script
- Level 2: face, torso, leg, arm regions (medium)
- Level 3: glasses, earrings, watch (fine details)
- Demonstrates dynamic calculation from keypoints
 2026-06-22 03:26:12 +08:00
Accusys d0858f288a docs: add CLI usage for TKG Level 1 builder 
- Add Usage section with CLI commands
- TKG Level 1 builder: python scripts/tkg_level1_builder.py
- Query example for person_trace nodes
 2026-06-22 03:24:04 +08:00
Accusys 9e0a0227ea docs: update Appearance_Feature_System with shot type detection 
- Add reference units table (eye/head/shoulder width)
- Add BODY_PROPORTIONS constants for validation
- Add shot type detection section (full_body/medium_shot/close_up)
- Add height estimation strategies per shot type
- Update code examples with head_width and proportion_ratios
 2026-06-22 02:50:45 +08:00
Accusys d94b96d884 feat: add shot type detection and proportion-based height estimation 
- detect_shot_type(): classify full_body/medium_shot/close_up
- estimate height using shoulder_width × 3.8 (~171cm) for close-up
- add BODY_PROPORTIONS constants for validation
- head position ratio + bbox aspect ratio → shot type
- enables filtering full-body shots in video search
 2026-06-22 02:47:01 +08:00
Accusys 606f31f13c feat: add appearance feature system with coordinate/scale fixes 
- Add Appearance_Feature_System_V1.0.md design doc
- Add proportion_calculator.py for body proportions (height, body shape)
- Add feature_extractor.py for hierarchical feature extraction
- Add tkg_level1_builder.py for TKG person_trace nodes
- Fix mediapipe_holistic_processor.py to output Top-Left pixels
- Add MediaPipe format conversion in proportion_calculator

Coordinate system alignment:
- Swift Pose: Top-Left pixels (Y-flip done in swift_pose.swift)
- MediaPipe: Top-Left pixels (norm→pixel conversion added)
 2026-06-22 02:27:03 +08:00
Accusys 97180aa7cd fix: add environment variable exports to startup scripts 
- Added MOMENTRY_OUTPUT_DIR, DATABASE_SCHEMA, MOMENTRY_REDIS_PREFIX exports
- Created run-worker-3002.sh for standalone worker
- Created config/ directory with environment-specific files
- Updated AGENTS.md with critical variables section and release checklist

This fixes Python subprocess environment variable inheritance issue
where store_traced_faces.py was using wrong output directory.
 2026-06-21 21:21:32 +08:00
Accusys e949ac793d docs: face_detections deprecation plan - analysis and future migration 
Analysis Results:
- 12 PostgreSQL fallback functions (TKG builders)
- 11 API modules with direct queries
- Identity binding: critical dependency

Current Status:
- Cannot deprecate now (Production stability)
- PostgreSQL fallback necessary
- Qdrant collection empty (0 points)

Recommendations:
- Keep PostgreSQL fallback for safety
- Document migration path
- New features use Qdrant/TKG
- Gradual migration in future (6+ months)

Migration Priority:
- P1: identity_binding.rs (TKG-based)
- P2: identity_agent_api.rs
- P3: identity_api.rs
- P4: Other APIs

Conclusion: face_detections cannot be deprecated yet due to:
- Production Qdrant empty
- API dependencies (identity binding)
- Stability requirements

Status: Draft (no immediate deprecation)
 2026-06-21 05:24:12 +08:00
Accusys 01dae66285 test: Production (3002) Phase 2.6-2.7 release test 
Test Results:
- Health check: 20 identities ✅
- File info: Success ✅
- Rule2 chunks: 75 ✅
- TKG rebuild: Failed (face.json missing)

Status:
- Phase 2.6-2.7 code: Implemented ✅
- PostgreSQL fallback: Active (Qdrant empty)
- Rule2 identity resolution: Working ✅
- Qdrant collection: Green, 0 points

Recommendations:
- Keep Production running with PostgreSQL fallback
- New videos will auto-fill Qdrant collection
- Production performance: ~1.85s (PG fallback)
 2026-06-21 05:20:39 +08:00
Accusys 6ede2a443c release: Phase 2.6-2.7 to production (3002) - edges migration and identity resolution 
Release: 2026-06-21 05:15
Binary: Jun 21 05:14 (34MB)
PID: 95567

Features:
- Phase 2.6: All edges from Qdrant (co_occurrence, face_face, speaker_face)
- Phase 2.7: Identity resolution for gaze_trace/lip_trace nodes
- Rule2: Extended for face_trace/gaze_trace/lip_trace node types

Architecture:
- Complete TKG-only identity resolution
- PostgreSQL fallback for empty Qdrant
- Estimated 3.6x edges performance improvement

Backup: momentry_backup_20260621_phase25

Commits:
- e214106d: Phase 2.7 identity resolution
- Phase 2.6 commits: edges migration to Qdrant

Status: ✅ Release successful
 2026-06-21 05:17:34 +08:00
Accusys e214106d48 feat: Phase 2.7 identity resolution for gaze/lip trace nodes 
Implementation:
- gaze_trace nodes: Query face_trace identity_id, add to properties
- lip_trace nodes: Query face_trace identity_id, add to properties
- Rule2: Extend identity resolution to support gaze_trace/lip_trace node types

Architecture:
- All face-related nodes now have identity_id in TKG properties
- Rule2 unified identity resolution for face_trace/gaze_trace/lip_trace
- TKG-only approach (no face_detections dependency for identity)

Code Changes:
- src/core/processor/tkg.rs: Add identity_id query in gaze/lip builders
- src/core/chunk/rule2_ingest.rs: Extend node_type condition

Docs:
- docs_v1.0/DESIGN/TKG_PHASE2_7_IDENTITY_RESOLUTION.md

Status: Implementation complete, pending test with valid file
 2026-06-21 05:12:13 +08:00
Accusys 2cfcfdd1af feat: Phase 2.6 edges migration to Qdrant (TKG-only architecture) 
Phase 2.6.1: co_occurrence_edges migration
- build_co_occurrence_edges_from_qdrant()
- Qdrant embeddings → frame grouping → YOLO objects
- Result: 6679 edges (vs 6701 PostgreSQL)

Phase 2.6.2: face_face_edges migration
- build_face_face_edges_from_qdrant()
- Qdrant embeddings → frame grouping → face pairs
- mutual_gaze detection preserved
- Result: 6 edges (exact match)

Phase 2.6.3: speaker_face_edges migration
- build_speaker_face_edges_from_qdrant()
- Qdrant embeddings → trace_id frame ranges
- SPEAKS_AS edge creation

Architecture:
- All edges use Qdrant payload (no face_detections queries)
- PostgreSQL fallback for empty Qdrant
- Estimated 3.6x performance improvement

Testing:
- Playground (3003): ✓ All Phase 2.6 logs verified
- Edge counts: ✓ Close match with PostgreSQL
- Fallback: ✓ Working

Docs:
- docs_v1.0/DESIGN/TKG_PHASE2_6_EDGES_MIGRATION.md
- docs_v1.0/M4_workspace/2026-06-21_phase2_6_test.md
 2026-06-21 04:47:49 +08:00
Accusys 0afc70fc5b test: Production (3002) Phase 2.5 release verification 
Test results:
- TKG rebuild: 1.75s (2.4x faster than Playground)
- gaze_trace_nodes: 21 (PostgreSQL fallback)
- lip_trace_nodes: 21 (PostgreSQL fallback)
- Rule2 chunks: 75 ✓

Findings:
- Production faster than Playground (1.75s vs 4.2s)
- Qdrant collection empty (0 points)
- Using PostgreSQL fallback for Phase 2.5
- New videos will auto-populate Qdrant

Status: ✅ Release successful
 2026-06-21 04:31:52 +08:00
Accusys 721c343486 release: Phase 2.5 to production (3002) - gaze_trace and lip_trace Qdrant migration 
Release: 2026-06-21 02:35
Binary: Jun 21 02:33
PID: 16386

Features:
- Phase 2.5.1: gaze_trace_nodes from Qdrant
- Phase 2.5.2: lip_trace_nodes from Qdrant + face.json
- Qdrant collection: momentry_face_embeddings (dim=512)

Verification:
- gaze_trace_nodes: 21 ✓
- lip_trace_nodes: 21 ✓
- Rule2 chunks: 75 ✓
- Performance: TKG rebuild 1.85s ✓

Backup: momentry_backup_20260619
 2026-06-21 03:12:38 +08:00
Accusys c39805bb8e feat: Phase 2.5 gaze_trace and lip_trace Qdrant migration + Charade Q&A test 
Phase 2.5.1: gaze_trace_nodes from Qdrant
- build_gaze_trace_nodes_from_qdrant()
- Read trace_id, frame, bbox from Qdrant payload
- Compute gaze stats (yaw, pitch, roll, gaze direction, blink)
- No PostgreSQL face_detections dependency

Phase 2.5.2: lip_trace_nodes from Qdrant + face.json
- build_lip_trace_nodes_from_qdrant()
- Match trace_id using Qdrant embeddings + face.json bbox
- Compute lip stats (openness, variance, speaking frames)
- Fixed face.json bbox structure (x,y,width,height not bbox object)

Test results:
- 23 gaze_trace nodes from Qdrant
- 23 lip_trace nodes from Qdrant + face.json
- 51 lip_sync edges created
- Charade Q&A: 20 identities, 75 relationship chunks

Docs:
- TKG_PHASE2_NONFACE_MIGRATION_V1.0.md (migration plan)
- 2026-06-21_charade_qa_test.md (Q&A test report)
 2026-06-21 02:17:08 +08:00
Accusys 23c440104b feat: Phase 2-3 TKG-only architecture 
Phase 2.1: build_face_trace_nodes_from_qdrant()
- Read trace_id, frame, bbox directly from Qdrant payload
- No dependency on face_detections table

Phase 2.3: Rule2 queries TKG nodes
- identity resolution from tkg_nodes.properties.identity_id
- TKG-only architecture (Phase 2.3)

Phase 3: Identity Agent updates TKG nodes
- match_faces_iterative() updates tkg_nodes.properties
- bind_identity_trace() syncs identity_id to TKG
- unbind_identity() removes identity_id from TKG

Test results:
- 23 face_trace nodes from Qdrant (Phase 2.1)
- 75 relationship chunks (Rule2)
- TKG rebuild: Phase0 → Phase1 → Phase2
 2026-06-21 01:30:04 +08:00
Accusys 2f2ccc94f7 feat: Identity Agent query Qdrant for face embeddings 
Phase 1.4: Modify match_faces_iterative to use Qdrant

Changes:
- match_faces_iterative() now queries FaceEmbeddingDb
- Fallback to PostgreSQL if Qdrant is empty
- Group embeddings by trace_id from Qdrant payload
- Sample 3-angle embeddings (front, mid, back)
- Match against TMDb seeds (threshold=0.50)
- Propagate to unmatched traces
- Update face_detections.identity_id in PostgreSQL

New functions:
- match_faces_iterative() - Qdrant-based matching
- match_faces_iterative_pg() - PostgreSQL fallback

Flow:
1. Load TMDb identities with face_embedding
2. Query Qdrant for file embeddings
3. Sample 3 embeddings per trace
4. Match against TMDb seeds
5. Propagate matches iteratively
6. Update identity_id in PostgreSQL
 2026-06-21 00:31:25 +08:00
Accusys 3ad6f8740a feat: Rule2 TKG relationship chunks + Phase0-1 Qdrant integration 
Phase 0: TKG builder populate face_detections from face.json
- Fix face.json parser for pose_angle format
- Call store_traced_faces.py to set trace_id
- Skip if trace_id already populated

Phase 1: Qdrant face embeddings integration
- Add FaceEmbeddingDb module (src/core/db/face_embedding_db.rs)
- Create dev_face_embeddings collection (dim=512)
- Store 1122 face embeddings with pose metadata
- API: init_collection, batch_upsert, search_similar

Rule2: TKG edges → relationship chunks
- Design: RULE2_TKG_RELATIONSHIP_V1.0.md
- Implementation: rule2_ingest.rs
- ChunkType::Relationship added
- Edge types: SPEAKS_AS, MUTUAL_GAZE, CO_OCCURS_WITH, HAS_APPEARANCE, WEARS
- Auto-trigger on TKG rebuild

API:
- POST /api/v1/file/:file_uuid/rule2 (vectorization)
- POST /api/v1/file/:file_uuid/tkg/rebuild (auto Rule2)

Test: 75 relationship chunks created + vectorized
 2026-06-21 00:22:41 +08:00
Accusys 17e4e15860 feat: add Vision LLM integration (CLIP + Qwen3-VL cascade) 
- Add Qwen3-VL dynamic management (start/stop/status CLI)
- Add CLIP + Qwen3-VL cascade detection strategy
- Add Vision CLI commands (vision start/stop/status, detect)
- Add cascade_vision processor module
- Add clip processor module
- Add qwen_vl_manager module

Changes:
- scripts/start_qwen3vl.sh, stop_qwen3vl.sh: Qwen3-VL management scripts
- src/core/vision/: Qwen3-VL manager module
- src/core/processor/cascade_vision.rs: CLIP + Qwen3-VL cascade logic
- src/core/processor/clip.rs: CLIP classification and detection
- src/api/clip_api.rs: CLIP API endpoints
- src/cli/vision.rs: Vision CLI implementation
- src/cli/args.rs: Add Vision and Detect commands
- src/main.rs: Integrate Vision CLI
- src/core/mod.rs: Add vision module
- src/core/processor/mod.rs: Add cascade_vision module
 2026-06-13 16:25:52 +08:00
Accusys 834b0d4865 feat: score-based search, LLM re-ranking endpoint, video title search, pipeline module 
Core search changes:
- Replace RRF with score-based merge (max of semantic/keyword/identity)
- Add video title ILIKE search for brand/name queries (score 0.9)
- Add /api/v1/search/llm-smart endpoint with Gemma 4 re-ranking
- Fix LLM JSON parsing (markdown fences, empty responses)

Infrastructure:
- Rebuild Qdrant collection (clear 347K contaminated points)
- Add dotenv loading to main.rs for config parity
- Implement store_pre_chunk in postgres_db.rs

Pipeline module (WordPress):
- store-asrx, rule1, vectorize, phase1, complete endpoints
- CLI commands for pipeline operations

Docs:
- SEARCH_SCORE_IMPROVEMENT.md (score-based merge proposal)
 2026-06-04 07:40:41 +08:00
Accusys e1572907ae feat: ASRX hybrid pipeline, identity history, worker fixes, checkpoint system 2026-06-02 07:13:23 +08:00
3282 changed files with 8406854 additions and 15462 deletions
Expand all files Collapse all files
.env.development
+28 -5
View File
@@ -41,8 +41,8 @@ MOMENTRY_PYTHON_PATH=/Users/accusys/momentry_core/venv/bin/python
MOMENTRY_SCRIPTS_DIR=/Users/accusys/momentry_core/scripts
# Logging
RUST_LOG=debug
MOMENTRY_LOG_LEVEL=debug
RUST_LOG=info
MOMENTRY_LOG_LEVEL=info
# Media
MOMENTRY_MEDIA_BASE_URL=https://wp.momentry.ddns.net
@@ -72,10 +72,33 @@ REDIS_CACHE_TTL_VIDEO_META=3600
# TMDb Integration (probe phase - auto-create identities from movie metadata)
TMDB_API_KEY=e9cde52197f6f8df4d9db99da93db1fb
MOMENTRY_TMDB_PROBE_ENABLED=true
# LLM for 5W1H summary (points to M5 Gemma4)
MOMENTRY_LLM_SUMMARY_URL=http://127.0.0.1:8082/v1/chat/completions
MOMENTRY_LLM_SUMMARY_MODEL=google_gemma-4-26B-A4B-it-Q5_K_M.gguf
# LLM Configuration
# Agent Search uses Ollama (llama3.1:8b) - OpenAI-compatible endpoint
MOMENTRY_LLM_CHAT_URL=http://localhost:11434/v1/chat/completions
MOMENTRY_LLM_CHAT_MODEL=llama3.1:8b
# VLM uses llama.cpp (llava-v1.6-vicuna-13b)
MOMENTRY_LLM_VISION_URL=http://localhost:8091/v1/chat/completions
MOMENTRY_LLM_VISION_MODEL=llava-v1.6-vicuna-13b
# Summary LLM uses Ollama
MOMENTRY_LLM_SUMMARY_URL=http://localhost:11434/v1/chat/completions
MOMENTRY_LLM_SUMMARY_MODEL=llama3.1:8b
MOMENTRY_LLM_SUMMARY_ENABLED=true
# Embedding (ANE CoreML server)
MOMENTRY_EMBED_URL=http://localhost:11436
# === Binary & Data Paths (for start_momentry.sh) ===
MOMENTRY_LOG_DIR=/Users/accusys/momentry/logs
MOMENTRY_PG_BIN_DIR=/Users/accusys/pgsql/18.3/bin
MOMENTRY_PG_DATA_DIR=/Users/accusys/pgsql/data
MOMENTRY_QDRANT_BIN=/Users/accusys/.cargo/bin/qdrant
MOMENTRY_QDRANT_STORAGE_DIR=/Users/accusys/momentry/qdrant_storage
MOMENTRY_LLAMACPP_BIN=/Users/accusys/llama/bin/llama-server
MOMENTRY_LLM_A4B_MODEL_PATH=/Users/accusys/models/google_gemma-4-26B-A4B-it-Q5_K_M.gguf
MOMENTRY_LLM_A4B_MMPROJ_PATH=/Users/accusys/models/gemma-4-26B-A4B-it.mmproj-f16.gguf
MOMENTRY_LLM_E4B_MODEL_PATH=/Users/accusys/models/gemma-4-E4B-it-Q4_K_M.gguf
MOMENTRY_LLM_E4B_MMPROJ_PATH=/Users/accusys/models/mmproj-gemma-4-E4B-it-BF16.gguf
MOMENTRY_OLLAMA_BIN=/Users/accusys/bin/ollama
MOMENTRY_PLAYGROUND_BIN=target/debug/momentry_playground
.env.example
+10
View File
@@ -32,6 +32,16 @@ MOMENTRY_LLM_SUMMARY_URL=http://127.0.0.1:8082/v1/chat/completions
MOMENTRY_LLM_SUMMARY_MODEL=google_gemma-4-26B-A4B-it-Q5_K_M.gguf
MOMENTRY_LLM_SUMMARY_TIMEOUT=120
# LLM Chat (A4B)
MOMENTRY_LLM_CHAT_URL=http://127.0.0.1:8082/v1/chat/completions
MOMENTRY_LLM_CHAT_MODEL=google_gemma-4-26B-A4B-it-Q5_K_M.gguf
MOMENTRY_LLM_CHAT_TIMEOUT=120
# LLM Vision (E4B)
MOMENTRY_LLM_VISION_URL=http://127.0.0.1:8083/v1/chat/completions
MOMENTRY_LLM_VISION_MODEL=gemma-4-E4B-it-Q4_K_M.gguf
MOMENTRY_LLM_VISION_TIMEOUT=120
# === Paths ===
MOMENTRY_OUTPUT_DIR=/Users/accusys/momentry/output_dev
MOMENTRY_BACKUP_DIR=/Users/accusys/momentry/backup
.gitignore
+30 -1
View File
@@ -15,6 +15,35 @@ __pycache__/
node_modules/
*.log
/tmp/
*.log
*.diff
*.bundle
*.probe.json
*.cut.json
.qdrant-initialized
dump.rdb
fix55.js
checksums.sha256
scripts/swift_processors/.build/
.opencode/
.vscode/
backups/
logs/
output/
models/
data/
storage/
thumbnails/
services/
model_checkpoints/
release/delivery/
release/system/
release/phase*/
release/dev_*.sql
release/migrate_*.sql
release/files/
package-lock.json
package.json
portal/dist/
portal/src-tauri/icons/
momentry_runtime/logs/
AGENTS.md
+176 -9
View File
@@ -14,23 +14,34 @@ Rust-based digital asset management system with video analysis and RAG capabilit
- **🔴 DELETE / REMOVE / DROP / CLEAR 任何資料前必須先問使用者「要刪嗎？」獲得明確同意後才能執行**
- **🔴 Qdrant collection 刪除、DB truncate、檔案刪除、資料清空 — 一律要先問**
- **🔴 不確定是否該刪 → 先問，不要自己決定**
- **🔴 改變議題前必須先存檔紀錄**：使用 `todowrite` 工具或建立紀錄文件（如 `docs_v1.0/M4_workspace/YYYY-MM-DD_topic_handoff.md`），確保上下文不丟失
### 開發範圍界定
| 範圍 | 狀態 | 說明 |
|------|------|------|
| `momentry_core_0.1/` | ✅ **可開發** | Momentry Core 主要開發目錄 |
| `momentry_core_0.1/portal/` | ✅ **可開發** | Tauri Portal 前端 |
| `momentry_core_0.1/src/` | ✅ **可開發** | Rust 後端程式碼 |
| `/Users/accusys/wordpress/` | ❌ **禁止修改** | WordPress/Marcom 團隊負責 |
| `~/momentry_core/` | ✅ **可開發** | Momentry Core 後端（Rust），Core team 負責 |
| `~/momentry_core/portal/` | ⚠️ **僅供測試** | Tauri Portal 僅用於測試，非正式前端 |
| `~/momentry_studio/` | ❌ **禁止修改** | Momentry Studio 前端，**Studio team 負責** |
| `/Users/accusys/wordpress/` | ❌ **禁止修改** | WordPress 網站僅供參考，已被 Studio 取代 |
| n8n 工作流 | ❌ **禁止修改** | 自動化流程，與 dev 無關 |
| WordPress/n8n 資料庫 table | ❌ **禁止修改** | Marcom 團隊管理，與 dev 無關 |
### 團隊職責劃分
| 團隊 | 負責專案 | 目錄 |
|------|---------|------|
| **Core team** | Momentry Core 後端 | `~/momentry_core/` |
| **Studio team** | Momentry Studio 前端 | `~/momentry_studio/` |
| **Marcom team** | WordPress 網站（已停用） | `/Users/accusys/wordpress/` |
### 開發環境
| 服務 | Port | 用途 | 命令 |
|------|------|------|------|
| Playground | 3003 | **唯一開發環境** | `cargo run --bin momentry_playground -- server` |
| Production | 3002 | ❌ 禁止修改 | `cargo run -- server` (僅 release 時) |
| Portal (Tauri) | 1420 | 前端開發 | `npm run tauri dev` |
| 服務 | Port | 用途 | 命令 | 狀態 |
|------|------|------|------|------|
| Playground | 3003 | ~~開發環境~~ (暫停) | `cargo run --bin momentry_playground -- server` | 🔴 已暫停 (節省 memory) |
| Production | 3002 | **開發 + 生產環境** | `cargo run -- server` | 🟢 運行中 (debug binary) |
| Portal (Tauri) | 1420 | 前端開發 | `npm run tauri dev` | - |
> **注意 (2026-07-25)**: Playground (3003) 已暫停服務。Production (3002) 改為直接用於開發測試。新功能可直接部署至 3002。
> **注意 (2026-07-23)**: 為節省記憶體，Playground (3003) 已關閉。Production (3002) 目前使用 debug binary 運行（已套用 smart_search 修復）。正式 release 時需重新 build release binary。
### 日誌與啟動
| 服務 | 日誌路徑 | 啟動方式 |
@@ -233,6 +244,77 @@ grep -i "error\|panic\|FAIL" logs/momentry_*.log | tail -20
| `momentry_playground` | Development | 3003 | `momentry_dev:` | `.env.development` |
| `momentry_player` | Video player | - | - | - |
## LLM Services
### 環境一致性原則
**生產環境 (port 3002) 與 Playground (port 3003) 使用相同的 LLM/VLM/Embedding 服務。**
### LLM Configuration
| 用途 | Model | 服務 | Port | API 格式 |
|------|-------|------|------|----------|
| **Agent Search** | `llama3.1:8b` | Ollama | 11434 | `/v1/chat/completions` |
| **VLM（視覺）** | `llava-v1.6-vicuna-13b` | llama.cpp | 8091 | `/v1/chat/completions` |
| **Embedding** | `embeddinggemma-300m` | Python | 11436 | Custom |
### Model 檔案位置
```
/Users/accusys/models/
├── llava-v1.6-vicuna-13b.Q4_K_M.gguf (VLM model)
├── mmproj-model-f16.gguf (VLM mmproj)
├── embeddinggemma-300M-Q8_0.gguf (Embedding model)
├── gemma-4-E4B-it-Q4_K_M.gguf (Text LLM)
└── google_gemma-4-26B-A4B-it-Q5_K_M.gguf (Text LLM)
```
### 啟動命令
**Ollama (llama3.1:8b)**
```bash
ollama serve # Service
ollama run llama3.1:8b # Interactive
curl http://localhost:11434/api/chat # API endpoint
```
**llama.cpp (llava-v1.6-vicuna-13b)**
```bash
/Users/accusys/llama/bin/llama-server \
-m /Users/accusys/models/llava-v1.6-vicuna-13b.Q4_K_M.gguf \
--mmproj /Users/accusys/models/mmproj-model-f16.gguf \
--host 0.0.0.0 \
--port 8091 \
-ngl 99 \
-c 4096
```
**Embedding (embeddinggemma-300m)**
```bash
python3 scripts/embeddinggemma_server.py --port 11436
```
### VLM 用途
- **Face trace VLM**: 描述人物外貌（衣著、顏色、配件）
- **Scene VLM**: 場景分析
- **Agent `analyze_frame`**: 畫面分析工具
### Agent Search 語言
- **預設使用英文回答**（除非用戶明確要求其他語言）
- System prompt 已明確規範 LLM 必須使用英文回應
### 環境變數
```bash
# .env.development
MOMENTRY_LLM_CHAT_URL=http://localhost:11434/api/chat
MOMENTRY_LLM_CHAT_MODEL=llama3.1:8b
MOMENTRY_LLM_VISION_URL=http://localhost:8091/v1/chat/completions
MOMENTRY_LLM_VISION_MODEL=llava-v1.6-vicuna-13b
```
## Testing
```bash
@@ -271,6 +353,8 @@ cargo check --all-features
- Use Rust 2021 edition
- Use tracing for logging (not println!)
- Keep lines under 100 characters
- **Always provide absolute paths when referencing files** — use full paths like `/Users/accusys/momentry_core/src/main.rs` instead of relative paths like `src/main.rs`
- **All document references MUST include full paths** — when listing files to modify, API endpoints, or cross-references in docs, always use absolute paths (e.g., `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/19_people_api.md`)
### Imports (order: std → external → local)
```rust
@@ -406,6 +490,40 @@ cargo run --features player --bin momentry_player -- -o
- `MOMENTRY_PYTHON_PATH` - Python path (default: `/opt/homebrew/bin/python3.11`)
- `MOMENTRY_SCRIPTS_DIR` - Scripts directory
### Critical Variables for Startup Scripts
**IMPORTANT**: Startup scripts must explicitly `export` these variables for Python subprocess inheritance.
#### Production (3002)
Required exports in `run-server-3002.sh` and `run-worker-3002.sh`:
```bash
export MOMENTRY_OUTPUT_DIR=/Users/accusys/momentry/output
export DATABASE_SCHEMA=public
export MOMENTRY_REDIS_PREFIX=momentry:
export MOMENTRY_SERVER_PORT=3002
```
#### Playground (3003)
Required exports in `run-server-3003.sh`:
```bash
export DATABASE_SCHEMA=dev
export MOMENTRY_SERVER_PORT=3003
export MOMENTRY_REDIS_PREFIX=momentry_dev:
export MOMENTRY_OUTPUT_DIR=/Users/accusys/momentry/output_dev
```
#### Why This Matters
- Rust process loads `.env` via `dotenv`
- Python subprocess inherits environment from Rust process
- Without explicit `export`, dotenv variables are only available inside Rust
- Python scripts like `store_traced_faces.py` will use hardcoded defaults if not exported
#### Config Directory
Environment-specific configuration files:
- `config/production.env` - Production-specific variables
- `config/development.env` - Development-specific variables
- `config/test.env` - Test environment (if needed)
### Processor Timeouts
- `MOMENTRY_ASR_TIMEOUT` - ASR timeout in seconds (default: 3600)
- `MOMENTRY_CUT_TIMEOUT` - CUT timeout in seconds (default: 3600)
@@ -624,6 +742,16 @@ git push origin main
pg_dump -U accusys -d momentry --schema-only > "$RELEASE_DIR/schema_v0.X.X.sql"
```
5. **驗證環境變數配置**
- ✅ Startup scripts export all required environment variables
- ✅ Python scripts don't use hardcoded paths
- ✅ Environment variables consistent across:
- `.env` / `.env.development`
- Startup script `export`
- Python script `os.environ.get()`
- ✅ Config directory has environment-specific files
- ✅ AGENTS.md documents all required exports
### 重要性
- 避免 release binary 與 current source code 不一致
- 方便追蹤特定 release 的程式碼狀態
@@ -818,3 +946,42 @@ Before creating any file in `docs_v1.0/` (API_WORKSPACE, GUIDES, REFERENCE, DESI
完整交付程序（M4_workspace → M5 → Release → Deploy → Public）見：
`docs_v1.0/OPERATIONS/DELIVERY_PROCEDURE.md`
## Session Summary (2026-07-01: Search Mode Fixes)
### Goal
Fix search modes: Keyword BM25 ranking + People search migration to Qdrant + Qdrant scroll pagination
### Done
- **Keyword/BM25 search (`search_bm25`)**: Replaced hardcoded 1.0 score with PostgreSQL FTS (`ts_rank` + `plainto_tsquery`). Now ranks results by relevance instead of flat 1.0.
- **Smart search merge**: Passes real FTS score through instead of fixed 0.5, so keyword-only results are properly differentiated.
- **Qdrant scroll_points**: Added `offset` parameter for pagination support; new `scroll_all_points()` method handles multi-page scroll automatically.
- **get_identity_traces**: Fixed broken pagination loop (always fetched same first 1000 points) by switching to `scroll_all_points`.
- **People search (`search_persons_internal`)**: Replaced `face_detections` JOIN in universal search with Qdrant `_faces` scroll + Rust aggregation (count per identity per file, frame→second via FPS).
- **People search (`search_persons_by_query`)**: Same migration for the REST API person search endpoint.
- **Payload field fix**: `_faces` uses `frame` (integer) not `timestamp_secs` (float). Fixed both `search_persons_internal` and `search_persons_by_query` to read `frame` and convert via `frame / fps`.
### Key Files Changed
- `src/core/db/qdrant_db.rs`: `scroll_points` → offset pagination, new `scroll_all_points`
- `src/api/identity_binding.rs`: Use `scroll_all_points` instead of broken loop
- `src/api/universal_search.rs`: Rewrote `search_persons_internal` and `search_persons_by_query` to use Qdrant
- `src/core/db/postgres_db.rs`: `search_bm25` → PostgreSQL FTS ranking
- `src/api/search.rs`: Pass real FTS scores in merge, removed unused `KEYWORD_FIXED_SCORE`
### Done This Session
- **Qdrant scroll pagination**: `scroll_points` now accepts `offset` param + returns `next_page_offset`; new `scroll_all_points()` handles multi-page scroll automatically
- **get_identity_traces pagination fix**: No longer fetches same 1000 points in infinite loop
- **Keyword BM25**: `search_bm25` replaced hardcoded 1.0 score with PostgreSQL `ts_rank` + `plainto_tsquery`; `smart_search` passes real FTS scores instead of fixed 0.5
- **People search → Qdrant**: Both `search_persons_internal` and `search_persons_by_query` replaced `face_detections` JOIN with Qdrant `_faces` scroll + Rust aggregation (count/group/sort). Fixed `timestamp_secs` → `frame` + `frame/fps` conversion
- **list_face_candidates → Qdrant**: `identities.rs` unbound faces query now scrolls `_faces` with `is_null: identity_id` filter, sorts by confidence DESC in Rust
- **list_unassigned_traces → Qdrant**: `identities.rs` unbound traces query now scrolls `_faces` with `is_null: identity_id` + `trace_id > 0` filter, groups by (file_uuid, trace_id) in Rust, picks best face per trace
- **get_identity_chunks → identity_bindings**: Replaced `face_detections` frame-range JOIN with `identity_bindings` + `chunk.metadata->>'trace_id'`
- **postgres_db.rs 5 remaining READs → Qdrant**: `get_trace_count_by_file`, `get_trace_frame_count_distribution`, `get_identity_files`, `get_identity_faces`, `get_file_faces` all migrated to `_faces` scroll + Rust aggregation
- **agent/tools.rs fully migrated**: `exec_find_file`, `exec_list_files`, `exec_tkg_query` (8 sub-queries), `exec_identity_text`, `exec_identities_search` — all face_detections JOINs replaced with Qdrant scroll or identity_bindings
- **job_worker.rs + storage.rs**: Remaining face_detections READs migrated to Qdrant scroll
### Remaining face_detections references (all inactive/safe)
- Schema definition (CREATE TABLE/INDEX in `postgres_db.rs`)
- `store_face_detections_batch` — already skipped (Phase 1)
- `workspace_sqlite.rs` — local processing DB, separate from PG
- `bin/release.rs` — standalone release utility
Cargo.lock
Generated
+2
View File
@@ -636,6 +636,8 @@ dependencies = [
"compression-core",
"flate2",
"memchr",
"zstd",
"zstd-safe",
]
[[package]]
Cargo.toml
+9 -1
View File
@@ -55,7 +55,7 @@ sqlx = { version = "0.8", features = ["runtime-tokio", "postgres", "sqlite", "js
mongodb = { version = "2", features = ["tokio-runtime"] }
bson = { version = "2", features = ["chrono-0_4"] }
qdrant-client = "1.7"
reqwest = { version = "0.12", features = ["json", "gzip"] }
reqwest = { version = "0.12", features = ["json", "gzip", "zstd"] }
pgvector = { version = "0.3", features = ["sqlx"] }
# HTTP Server
@@ -134,6 +134,14 @@ path = "src/bin/integrated_player.rs"
name = "release"
path = "src/bin/release.rs"
[[bin]]
name = "vectorize_missing"
path = "src/bin/vectorize_missing.rs"
[[bin]]
name = "sync_qdrant_from_pg"
path = "src/bin/sync_qdrant_from_pg.rs"
[[bin]]
name = "service"
path = "src/bin/service.rs"
check_jobs.rs
+26
View File
@@ -0,0 +1,26 @@
use sqlx::postgres::PgPoolOptions;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let pool = PgPoolOptions::new()
.max_connections(1)
.connect("postgres://accusys@localhost:5432/momentry")
.await?;
let row: Option<(i32, String, String, Option<String>)> = sqlx::query_as(
"SELECT id, uuid, status, processors FROM monitor_jobs WHERE uuid = 'd8acb03870f0cc9b14e01f14a7bf24d6' ORDER BY id DESC LIMIT 1"
)
.fetch_optional(&pool)
.await?;
if let Some((id, uuid, status, processors)) = row {
println!("Job ID: {}", id);
println!("UUID: {}", uuid);
println!("Status: {}", status);
println!("Processors: {:?}", processors);
} else {
println!("No job found for this UUID");
}
Ok(())
}
check_jobs_status.sh
Executable
+13
View File
@@ -0,0 +1,13 @@
#!/bin/bash
# Query PostgreSQL monitor_jobs status
# Using Rust code to execute SQL
echo "Jobs in PostgreSQL:"
cat << 'SQL' > query_jobs.sql
SELECT uuid, status, processors, created_at::date
FROM monitor_jobs
ORDER BY created_at DESC
LIMIT 10;
SQL
echo "SQL query created. Need to execute via API or Rust..."
clear_failed_processor.sql
+10
View File
@@ -0,0 +1,10 @@
-- Delete failed face processor result to allow retry
DELETE FROM processor_results
WHERE job_id = 62
AND processor = 'face'
AND status = 'failed';
-- Check remaining processor_results for this job
SELECT id, processor, status, retry_count
FROM processor_results
WHERE job_id = 62;
config/README.md
+150 -77
View File
@@ -1,105 +1,178 @@
# Momentry Core 配置管理
# Momentry Core Config Management
## 目錄結構
## Directory Structure
```
momentry_core_0.1/
├── .env.example # 配置模板（已納入版本控制）
├── .env # 本地配置（已從版本控制排除）
├── .env.local # 本地覆蓋配置（已從版本控制排除）
├── .env.example # Template (version controlled)
├── .env # Local config (gitignored)
├── .env.development # Playground dev overrides (gitignored)
├── .env.local # Local overrides (gitignored)
├── config/
│ └── README.md # 本文件
└── src/core/config.rs # 配置代碼
│ ├── README.md # This file
│ └── port_registry.tsv # Central port registry
└── src/core/config.rs # Config code with lazy_static env reading
```
## 配置加載順序
## Load Order
1. `.env` - 默認本地配置
2. `.env.local` - 本地覆蓋（最高優先級）
For `momentry_playground` (development):
1. `.env` — shared defaults
2. `.env.development` — dev-specific overrides (loaded by playground binary)
## 環境變數列表
For `momentry` (production):
1. `.env` — production config
### 數據庫配置
In Rust: `config.rs` reads env vars with lazy_static, falling back to hardcoded defaults.
| 變數 | 說明 | 默認值 |
|------|------|--------|
| `DATABASE_URL` | PostgreSQL 連接字串 | `postgres://accusys@localhost:5432/momentry` |
## Environment Variables
### Redis 配置
### Server
| 變數 | 說明 | 默認值 |
|------|------|--------|
| `REDIS_URL` | Redis 連接字串 | `redis://:accusys@localhost:6379` |
| `REDIS_PASSWORD` | Redis 密碼 | `accusys` |
| Variable | Description | Default |
|----------|-------------|---------|
| `MOMENTRY_SERVER_PORT` | Server port (3002=prod, 3003=dev) | `3002` |
| `MOMENTRY_REDIS_PREFIX` | Redis key prefix | `momentry:` (prod), `momentry_dev:` (dev) |
### 存儲路徑
### Database
| 變數 | 說明 | 默認值 |
|------|------|--------|
| `MOMENTRY_OUTPUT_DIR` | 輸出目錄 | `/Users/accusys/momentry/output` |
| `MOMENTRY_BACKUP_DIR` | 備份目錄 | `/Users/accusys/momentry/backup/momentry` |
| `MOMENTRY_SCRIPTS_DIR` | 腳本目錄 | `/Users/accusys/momentry_core_0.1/scripts` |
| `MOMENTRY_PYTHON_PATH` | Python 路徑 | `/opt/homebrew/bin/python3.11` |
| Variable | Description | Default |
|----------|-------------|---------|
| `DATABASE_URL` | PostgreSQL connection string | `postgres://accusys@localhost:5432/momentry` |
| `DATABASE_SCHEMA` | Schema for dev isolation | `dev` |
| `MONGODB_URL` | MongoDB connection string | `mongodb://localhost:27017` |
| `MONGODB_DATABASE` | MongoDB database name | `momentry` (prod), `momentry_dev` (dev) |
| `MONGODB_CACHE_ENABLED` | MongoDB cache toggle | `true` |
| `MONGODB_CACHE_TTL_VIDEOS` | Cache TTL for videos | `300` |
| `MONGODB_CACHE_TTL_SEARCH` | Cache TTL for search | `300` |
| `MONGODB_CACHE_TTL_HYBRID_SEARCH` | Cache TTL for hybrid search | `600` |
| `MONGODB_CACHE_TTL_VIDEO_META` | Cache TTL for video metadata | `3600` |
### 處理器超時（秒）
### Redis
| 變數 | 說明 | 默認值 |
|------|------|--------|
| `MOMENTRY_ASR_TIMEOUT` | ASR 處理超時 | `3600` |
| `MOMENTRY_CUT_TIMEOUT` | CUT 處理超時 | `3600` |
| `MOMENTRY_DEFAULT_TIMEOUT` | 默認超時 | `7200` |
| Variable | Description | Default |
|----------|-------------|---------|
| `REDIS_URL` | Redis connection string | `redis://:accusys@localhost:6379` |
| `REDIS_PASSWORD` | Redis password | `accusys` |
| `REDIS_CACHE_TTL_HEALTH` | Health check cache TTL | `30` |
| `REDIS_CACHE_TTL_VIDEO_META` | Video metadata cache TTL | `3600` |
### 日誌
### Qdrant
| 變數 | 說明 | 默認值 |
|------|------|--------|
| `RUST_LOG` | 日誌級別 | `info` |
| `MOMENTRY_LOG_LEVEL` | 日誌級別（備選） | `info` |
| Variable | Description | Default |
|----------|-------------|---------|
| `QDRANT_URL` | Qdrant server URL | `http://localhost:6333` |
| `QDRANT_API_KEY` | Qdrant API key | `Test3200Test3200Test3200` |
| `QDRANT_COLLECTION` | Collection name | `momentry_rule1` (prod), `momentry_dev_rule1_v2` (dev) |
## 使用方式
### LLM
### 1. 首次設置
| Variable | Description | Default |
|----------|-------------|---------|
| `MOMENTRY_LLM_CHAT_URL` | Chat/function-calling endpoint | `http://127.0.0.1:8082/v1/chat/completions` |
| `MOMENTRY_LLM_CHAT_MODEL` | Chat model name | `google_gemma-4-26B-A4B-it-Q5_K_M.gguf` |
| `MOMENTRY_LLM_VISION_URL` | Vision LLM endpoint (E4B) | falls back to CHAT_URL |
| `MOMENTRY_LLM_VISION_MODEL` | Vision model name (E4B) | falls back to CHAT_MODEL |
| `MOMENTRY_LLM_SUMMARY_URL` | Summary LLM endpoint (5W1H) | falls back to CHAT_URL |
| `MOMENTRY_LLM_SUMMARY_MODEL` | Summary model name | falls back to CHAT_MODEL |
| `MOMENTRY_LLM_SUMMARY_ENABLED` | Toggle 5W1H summary generation | `true` |
| `MOMENTRY_LLM_SUMMARY_TIMEOUT` | 5W1H timeout in seconds | `120` |
| `MOMENTRY_LLM_CHAT_TIMEOUT` | Chat LLM timeout in seconds | `120` |
| `MOMENTRY_LLM_VISION_TIMEOUT` | Vision LLM timeout in seconds | `120` |
### Embedding
| Variable | Description | Default |
|----------|-------------|---------|
| `MOMENTRY_EMBED_URL` | Embedding server URL | `http://localhost:11436` |
### TMDb Integration
| Variable | Description | Default |
|----------|-------------|---------|
| `TMDB_API_KEY` | TMDb API key (required for probe) | (none) |
| `MOMENTRY_TMDB_PROBE_ENABLED` | Enable TMDb probe during register | `false` |
### Paths
| Variable | Description | Default |
|----------|-------------|---------|
| `MOMENTRY_OUTPUT_DIR` | Output directory for processing | `/Users/accusys/momentry/output` |
| `MOMENTRY_BACKUP_DIR` | Backup directory | `/Users/accusys/momentry/backup/momentry` |
| `MOMENTRY_SCRIPTS_DIR` | Python scripts directory | `/Users/accusys/momentry_core_0.1/scripts` |
| `MOMENTRY_PYTHON_PATH` | Python interpreter path | `/opt/homebrew/bin/python3.11` |
| `MOMENTRY_MEDIA_BASE_URL` | Base URL for media serving | (none) |
### Processor Timeouts
| Variable | Description | Default |
|----------|-------------|---------|
| `MOMENTRY_ASR_TIMEOUT` | ASR timeout in seconds | `3600` |
| `MOMENTRY_CUT_TIMEOUT` | CUT timeout in seconds | `3600` |
| `MOMENTRY_DEFAULT_TIMEOUT` | Default timeout in seconds | `7200` |
### Logging
| Variable | Description | Default |
|----------|-------------|---------|
| `RUST_LOG` | Rust log level (tracing) | `info` |
| `MOMENTRY_LOG_LEVEL` | Fallback log level | `info` |
### Worker
| Variable | Description | Default |
|----------|-------------|---------|
| `MOMENTRY_WORKER_ENABLED` | Enable background worker | `true` |
| `MOMENTRY_MAX_CONCURRENT` | Max concurrent jobs | `6` |
| `MOMENTRY_POLL_INTERVAL` | Poll interval in seconds | `10` |
| `MOMENTRY_WORKER_BATCH_SIZE` | Batch size | `5` |
### Synonym Expansion
| Variable | Description | Default |
|----------|-------------|---------|
| `MOMENTRY_SYNONYM_FILES` | Comma-separated paths to synonym JSON files | (none) |
| `MOMENTRY_SYNONYM_FILE` | Single synonym file (deprecated) | (none) |
### Encryption
| Variable | Description | Default |
|----------|-------------|---------|
| `AUDIT_ENCRYPTION_KEY` | 32-byte hex encryption key (64 hex chars) | (none) |
## Port Registry
See `config/port_registry.tsv` for the authoritative list of all ports and their owners.
| Port | Service | Owner | Config Key |
|------|---------|-------|------------|
| 5432 | PostgreSQL | postgres | `DATABASE_URL` |
| 6379 | Redis | redis-server | `REDIS_URL` |
| 6333 | Qdrant | qdrant | `QDRANT_URL` |
| 8082 | LLM Chat (A4B) | llama-server | `MOMENTRY_LLM_CHAT_URL` |
| 8083 | LLM Vision (E4B) | llama-server | `MOMENTRY_LLM_VISION_URL` |
| 11434 | Ollama | ollama | `MOMENTRY_OLLAMA_URL` |
| 11436 | Embedding | embeddinggemma_server.py | `MOMENTRY_EMBED_URL` |
| 27017 | MongoDB | mongod | `MONGODB_URL` |
| 3002 | Production API | momentry | `MOMENTRY_SERVER_PORT` |
| 3003 | Playground API | momentry_playground | `MOMENTRY_SERVER_PORT` |
## Quick Start
```bash
# 複製模板
# 1. Copy template
cp .env.example .env
# 編輯配置
nano .env
# 2. Edit .env for production or use .env.development for playground
# 3. Start all services
./scripts/start_momentry.sh
```
### 2. 本地覆蓋
## Version Control
創建 `.env.local` 設置僅本地適用的配置：
```bash
# .env.local 示例
DATABASE_URL=postgres://local:password@localhost:5432/momentry_dev
MOMENTRY_LOG_LEVEL=debug
```
### 3. 運行應用
```bash
# 加載配置並運行
source .env && cargo run
# 或使用 direnv
direnv allow
```
## 版本控制策略
| 文件 | 版本控制 | 說明 |
|------|---------|------|
| `.env.example` | ✅ 追蹤 | 模板，包含所有選項 |
| `.env` | ❌ 忽略 | 本地敏感配置 |
| `.env.local` | ❌ 忽略 | 本地覆蓋配置 |
## 部署檢查清單
- [ ] 複製 `.env.example` 到 `.env`
- [ ] 設置數據庫連接
- [ ] 設置 Redis 密碼
- [ ] 配置目錄路徑
- [ ] 確認日誌級別
| File | Tracked | Purpose |
|------|---------|---------|
| `.env.example` | ✅ Yes | Template with all options documented |
| `.env` | ❌ No | Local sensitive config |
| `.env.development` | ❌ No | Dev-specific overrides |
| `.env.local` | ❌ No | Local overrides (highest priority) |
config/development.env
+47
View File
@@ -0,0 +1,47 @@
# Development Environment Configuration
# Used by: momentry_playground binary on port 3003
#
# This file extracts development-specific variables from .env.development
# Startup scripts must export these variables for Python subprocess inheritance
# Server Configuration
MOMENTRY_SERVER_PORT=3003
MOMENTRY_REDIS_PREFIX=momentry_dev:
# Database Schema
DATABASE_SCHEMA=dev
# Output Directory (CRITICAL for Python scripts)
MOMENTRY_OUTPUT_DIR=/Users/accusys/momentry/output_dev
# Backup Directory
MOMENTRY_BACKUP_DIR=/Users/accusys/momentry/backup/momentry_dev
# Storage
MOMENTRY_SFTP_ROOT=/Users/accusys/momentry/var/sftpgo/data/demo/
# Python Path (venv for development)
MOMENTRY_PYTHON_PATH=/Users/accusys/momentry_core/venv/bin/python
MOMENTRY_SCRIPTS_DIR=/Users/accusys/momentry_core/scripts
# Logging
RUST_LOG=info
MOMENTRY_LOG_LEVEL=info
# Worker Configuration
MOMENTRY_WORKER_ENABLED=true
MOMENTRY_MAX_CONCURRENT=6
MOMENTRY_POLL_INTERVAL=10
MOMENTRY_WORKER_BATCH_SIZE=5
# TMDb Integration
TMDB_API_KEY=e9cde52197f6f8df4d9db99da93db1fb
MOMENTRY_TMDB_PROBE_ENABLED=true
# LLM Configuration
MOMENTRY_LLM_SUMMARY_URL=http://127.0.0.1:8000/v1/chat/completions
MOMENTRY_LLM_SUMMARY_MODEL=gemma-4-E4B
MOMENTRY_LLM_SUMMARY_ENABLED=true
# Embedding
MOMENTRY_EMBED_URL=http://localhost:11436
config/port_registry.tsv
+3 -1
View File
@@ -16,7 +16,9 @@
6379 redis redis-server REDIS_URL redis://...:6379 start_momentry.sh
6333 qdrant qdrant QDRANT_URL http://...:6333 start_momentry.sh
8081 wordpress Caddy - - Caddyfile
8082 llm llama-server MOMENTRY_LLM_CHAT_URL http://...:8082 start_momentry.sh
8082 llm-chat llama-server MOMENTRY_LLM_CHAT_URL http://...:8082 start_momentry.sh
8083 llm-vision llama-server MOMENTRY_LLM_VISION_URL http://...:8083 start_momentry.sh
9000 php-fpm php-fpm - 9000 brew services
11434 ollama ollama MOMENTRY_OLLAMA_URL http://...:11434 start_momentry.sh
11436 embedding embeddinggemma MOMENTRY_EMBED_URL http://...:11436 start_momentry.sh
27017 mongodb mongod MONGODB_URL mongodb://...:27017 start_momentry.sh
1 # Port Registry - Momentry Core
16 6379
17 6333
18 8081
19 8082
20 8083
21 9000
22 11434
23 11436
24 27017
config/production.env
+39
View File
@@ -0,0 +1,39 @@
# Production Environment Configuration
# Used by: momentry binary on port 3002
#
# This file extracts production-specific variables from .env
# Startup scripts must export these variables for Python subprocess inheritance
# Server Configuration
MOMENTRY_SERVER_PORT=3002
MOMENTRY_REDIS_PREFIX=momentry:
# Database Schema
DATABASE_SCHEMA=public
# Output Directory (CRITICAL for Python scripts)
MOMENTRY_OUTPUT_DIR=/Users/accusys/momentry/output
# Backup Directory
MOMENTRY_BACKUP_DIR=/Users/accusys/momentry/backup/momentry
# Storage
MOMENTRY_STORAGE_ROOT=/Users/accusys/momentry/var/sftpgo/data
# Python Path
MOMENTRY_PYTHON_PATH=/opt/homebrew/bin/python3.11
# Logging
RUST_LOG=debug
MOMENTRY_LOG_LEVEL=debug
# Worker Configuration
MOMENTRY_WORKER_ENABLED=true
MOMENTRY_MAX_CONCURRENT=6
MOMENTRY_POLL_INTERVAL=10
MOMENTRY_WORKER_BATCH_SIZE=5
MOMENTRY_FORCE_RETRY=true
# TMDb Integration
TMDB_API_KEY=e9cde52197f6f8df4d9db99da93db1fb
MOMENTRY_TMDB_PROBE_ENABLED=true
deliverable_v1.1.0/AGENTS.md
+761
View File
@@ -0,0 +1,761 @@
# AGENTS.md - Momentry Core
Rust-based digital asset management system with video analysis and RAG capabilities.
---
## ⚠️ CRITICAL: 開發隔離原則
### 絕對禁止事項
- **絕對不可修改 `/Users/accusys/wordpress/` 目錄下的任何檔案**
- **絕對不可修改 n8n 工作流或設定**
- **絕對不可修改 WordPress 或 n8n 的資料庫 table**
- **除非是 release 作業，絕對不可動 port 3002 (production)**
- **🔴 DELETE / REMOVE / DROP / CLEAR 任何資料前必須先問使用者「要刪嗎？」獲得明確同意後才能執行**
- **🔴 Qdrant collection 刪除、DB truncate、檔案刪除、資料清空 — 一律要先問**
- **🔴 不確定是否該刪 → 先問，不要自己決定**
### 開發範圍界定
| 範圍 | 狀態 | 說明 |
|------|------|------|
| `momentry_core_0.1/` | ✅ **可開發** | Momentry Core 主要開發目錄 |
| `momentry_core_0.1/portal/` | ✅ **可開發** | Tauri Portal 前端 |
| `momentry_core_0.1/src/` | ✅ **可開發** | Rust 後端程式碼 |
| `/Users/accusys/wordpress/` | ❌ **禁止修改** | WordPress/Marcom 團隊負責 |
| n8n 工作流 | ❌ **禁止修改** | 自動化流程，與 dev 無關 |
| WordPress/n8n 資料庫 table | ❌ **禁止修改** | Marcom 團隊管理，與 dev 無關 |
### 開發環境
| 服務 | Port | 用途 | 命令 |
|------|------|------|------|
| Playground | 3003 | **唯一開發環境** | `cargo run --bin momentry_playground -- server` |
| Production | 3002 | ❌ 禁止修改 | `cargo run -- server` (僅 release 時) |
| Portal (Tauri) | 1420 | 前端開發 | `npm run tauri dev` |
## ⚠️ 交叉污染防制 (Cross-Contamination Prevention)
**每個執行前必須評估是否會汙染其他獨立作業。**
### Scope Isolation Matrix
| 執行內容 | 允許的 Scope | 禁止影響 | 檢查事項 |
|----------|-------------|----------|----------|
| M4 delivery binary | `target/release/momentry` | Playground (3003), Production (3002) | 確認舊 process 未被誤殺 |
| Playground server | `localhost:3003`, `dev.*` schema | Production (3002), `public.*` schema | `DATABASE_SCHEMA=dev` |
| Production deploy | `localhost:3002`, `public.*` schema | Playground (3003), `dev.*` schema | 先停 production，不影響 playground |
| Git commit | 只包含意圖修改的檔案 | 無關的 untracked files | `git status` 確認 stage 內容正確 |
| CI / packaged tests | 測試環境 | 正式資料 | 測試用 DB 不能連到 production |
| Doc changes | 指定文件 | 其他文件、程式碼 | `git diff --stat` 檢查 scope |
| SQL migration | 目標 schema | 其他 schema、無關 table | `WHERE` clause 要精準 |
| `sed` / `grep` / mass edit | 目標檔案集 | 非目標檔案 | 先用 `grep -c` 確認只有目標檔案匹配 |
### Recent Violations / Near-Misses
| 事件 | 問題 | 防止方式 |
|------|------|----------|
| `sed` API doc 編號 | `sed -i '' 's/.../.../g'` 改到所有行 | 先 `grep -c` 確認匹配，`git diff` 再提交 |
| 亂加 `/api/v1/register` route | 不必要的 API 別名，汙染路由表 | 角色切換：路由設計不該由實作方決定 |
| `API_WORKSPACE/` vs `GUIDES/` vs `REFERENCE/` vs `DESIGN/` vs `OPERATIONS/` vs `INTEGRATIONS/` | 文件放到錯誤分類 | API 文件改在 API_WORKSPACE/modules/ 編輯，`make deploy` 生成到 GUIDES/ |
| Build release binary in plan mode | 浪費時間，無意義 | 嚴格遵守 plan/build mode 規定 |
### ⛔ 嚴格測試隔離規則 (Strict Test Isolation)
- **所有測試 (Test) 必須在 Dev (3003) 進行**。
- **絕對禁止 (ABSOLUTELY FORBIDDEN)** 在任何測試指令、Demo 流程或 API 檢查中使用 `localhost:3002`。
- 即使是「測試 Unregister」或「檢查版本」，若未明確標示為 "Production Deployment"，一律視為違規。
- **預設行為**: 所有 curl, CLI, 或程式碼測試指令，預設 URL 必須為 `http://localhost:3003`。
### 違反後果
- 修改 WordPress/n8n 可能影響 marcom 團隊工作與生產環境
- 修改 WordPress/n8n 資料庫 table 可能破壞自動化流程與資料完整性
- 修改 port 3002 可能中斷正在使用的服務 (這是非常嚴重的錯誤)
- 所有 dev 測試必須在 playground (3003) 進行
---
## AI Coding Principles (Karpathy-Inspired)
Behavioral guidelines to reduce common LLM coding mistakes.
Source: [andrej-karpathy-skills](https://github.com/forrestchang/andrej-karpathy-skills) (94K stars)
**Tradeoff:** These guidelines bias toward caution over speed. For trivial tasks, use judgment.
### 1. Think Before Coding
**Don't assume. Don't hide confusion. Surface tradeoffs.**
- State your assumptions explicitly. If uncertain, ask.
- If multiple interpretations exist, present them - don't pick silently.
- If a simpler approach exists, say so. Push back when warranted.
- If something is unclear, stop. Name what's confusing. Ask.
### 2. Simplicity First
**Minimum code that solves the problem. Nothing speculative.**
- No features beyond what was asked.
- No abstractions for single-use code.
- No "flexibility" or "configurability" that wasn't requested.
- No error handling for impossible scenarios.
- If you write 200 lines and it could be 50, rewrite it.
Ask yourself: "Would a senior engineer say this is overcomplicated?" If yes, simplify.
### 3. Surgical Changes
**Touch only what you must. Clean up only your own mess.**
When editing existing code:
- Don't "improve" adjacent code, comments, or formatting.
- Don't refactor things that aren't broken.
- Match existing style, even if you'd do it differently.
- If you notice unrelated dead code, mention it - don't delete it.
When your changes create orphans:
- Remove imports/variables/functions that YOUR changes made unused.
- Don't remove pre-existing dead code unless asked.
The test: Every changed line should trace directly to the user's request.
### 4. Goal-Driven Execution
**Define success criteria. Loop until verified.**
Transform tasks into verifiable goals:
- "Add validation" -> "Write tests for invalid inputs, then make them pass"
- "Fix the bug" -> "Write a test that reproduces it, then make it pass"
- "Refactor X" -> "Ensure tests pass before and after"
For multi-step tasks, state a brief plan:
```
1. [Step] -> verify: [check]
2. [Step] -> verify: [check]
3. [Step] -> verify: [check]
```
Strong success criteria let you loop independently. Weak criteria ("make it work") require constant clarification.
---
These guidelines are working if: fewer unnecessary changes in diffs, fewer rewrites due to overcomplication, and clarifying questions come before implementation rather than after mistakes.
---
## Terminology (V4.0)
| Term | Scope | Description | Example |
|------|-------|-------------|---------|
| **file_uuid** | Video file | Video file identifier (renamed from `video_uuid`) | `384b0ff44aaaa1f1` |
| **identity_uuid** | Global identity | Global person identity (cross-file) | `a9a90105-6d6b-46ff-92da-0c3c1a57dff4` |
| **face_id** | Single detection | Single face detection (frame-level) | `face_100` |
| **trace_id** | Face tracking | Face tracking ID (Face Tracker output) | `2` |
| **chunk_id** | Sentence chunk | Sentence chunk (from pre_chunks via rules) | `chunk_1` |
| **speaker_id** | Speaker segment | Speaker ID (from ASRX) | `SPEAKER_0` |
| **person_id** | ❌ **Deprecated** | Video-local person ID (removed in V4.0) | - |
### Architecture (V4.0)
```
Face → Identity (Two-layer, direct binding)
↓
person_identities table: REMOVED
file_identities table: ADDED (N:N relationship)
```
### Key Changes (V3.x → V4.0)
| Change | V3.x | V4.0 |
|--------|------|------|
| **video_uuid** | Used everywhere | **file_uuid** |
| **person_identities** | Required (303 records) | **Removed** |
| **person_id APIs** | 28 endpoints | **Removed** (except register/bind) |
| **Face binding** | Person → Identity | **Face → Identity** (direct) |
| **Chunk binding** | Manual | **Auto** (time alignment) |
---
## Build & Run Commands
```bash
# Build project (use debug builds for development/testing)
cargo build
cargo build --bin momentry
cargo build --bin momentry_playground
# Build all binaries
cargo build --bins
# Run CLI
cargo run -- --help
cargo run -- register /path/to/video.mp4
cargo run -- server --host 0.0.0.0 --port 3002
# Run playground (development binary)
cargo run --bin momentry_playground -- server
cargo run --bin momentry_playground -- --help
```
### ⚠️ CRITICAL: `cargo build --release` PROHIBITION
- **NEVER run `cargo build --release` unless the user explicitly says "release the binary" or "正式 release"**
- `cargo build --release` is SLOW and only needed when producing a production binary for deployment
- For all development, testing, debugging, and linting: use `cargo build` or `cargo check`
- If uncertain, ALWAYS ask the user first
## Binaries
| Binary | Purpose | Port | Redis Prefix | Environment |
|--------|---------|------|--------------|-------------|
| `momentry` | Production | 3002 | `momentry:` | `.env` |
| `momentry_playground` | Development | 3003 | `momentry_dev:` | `.env.development` |
| `momentry_player` | Video player | - | - | - |
## Testing
```bash
# Run all tests
cargo test
# Run single test by name
cargo test test_name
# Run with output
cargo test -- --nocapture
# Doc tests
cargo test --doc
```
## Linting & Formatting
```bash
# Format code (edition=2021, max_width=100, tab_spaces=4)
cargo fmt
cargo fmt -- --check
# Lint
cargo clippy
cargo clippy --all-features
# Check for errors
cargo check
cargo check --all-features
```
## Code Style
### General
- Use Rust 2021 edition
- Use tracing for logging (not println!)
- Keep lines under 100 characters
### Imports (order: std → external → local)
```rust
use std::path::Path;
use anyhow::{Context, Result};
use async_trait::async_trait;
use serde::{Deserialize, Serialize};
use crate::core::chunk::Chunk;
```
### Error Handling
- Use `anyhow::Result<T>` for application code
- Use `thiserror` for library code
- Use `.context()` for error context
- Use `anyhow::bail!()` for early returns
```rust
fn example() -> Result<SomeType> {
let output = Command::new("ffprobe")
.args([...])
.output()
.context("Failed to run ffprobe")?;
if !output.status.success() {
anyhow::bail!("Command failed");
}
Ok(result)
}
```
### Naming
- Types/Enums: PascalCase (`VideoRecord`, `ChunkType`)
- Functions/Variables: snake_case (`get_video_by_uuid`)
- Traits: PascalCase with -er suffix (`Database`, `ChunkStore`)
- Files: snake_case (`postgres_db.rs`)
### Types
- Use `serde::{Deserialize, Serialize}` for serializable types
- Use `#[serde(rename_all = "snake_case")]` for enum variants
- Use explicit numeric types (i64, u32, f64)
```rust
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct VideoRecord {
pub id: i64,
pub uuid: String,
pub duration: f64,
pub width: u32,
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq)]
#[serde(rename_all = "snake_case")]
pub enum ChunkType {
TimeBased,
Sentence,
Cut,
}
```
### Async Programming
- Use `tokio` runtime with full features
- Use `#[async_trait]` for async trait methods
```rust
#[async_trait]
pub trait Database: Send + Sync {
async fn init() -> Result<Self>
where Self: Sized;
}
```
## Code Structure
```
src/
├── main.rs # CLI entry point
├── lib.rs # Library exports
├── core/
│ ├── api_key/ # API key management (anomaly, blacklist, encryption, etc.)
│ ├── chunk/ # Chunking logic
│ ├── config.rs # Centralized configuration (env vars)
│ ├── db/ # Database (PostgreSQL, MongoDB, Redis, Qdrant)
│ ├── embedding/ # Vector embeddings
│ ├── overlay/ # Video overlay
│ ├── probe/ # ffprobe integration
│ ├── processor/ # ASR, OCR, YOLO, Face, Pose, CUT, ASRX
│ │ └── executor.rs # Unified Python script executor
│ ├── storage/ # File management
│ └── thumbnail/ # Thumbnail extraction
├── api/ # HTTP API (axum)
├── player/ # Video player
├── ui/ # TUI components
└── watcher/ # File system watcher
```
## Key Dependencies
- **Error handling**: `anyhow`, `thiserror`
- **Async**: `tokio` (full features), `async-trait`
- **CLI**: `clap` (derive)
- **Serialization**: `serde`, `serde_json`, `chrono`
- **Database**: `sqlx`, `mongodb`, `redis` (1.0), `qdrant-client`
- **HTTP**: `axum`, `tower`
- **Logging**: `tracing`, `tracing-subscriber`
- **Config**: `once_cell` (lazy static config)
## Environment Variables
### Server
- `MOMENTRY_SERVER_PORT` - API server port (default: `3002` for production, `3003` for playground)
- `MOMENTRY_REDIS_PREFIX` - Redis key prefix (default: `momentry:` for production, `momentry_dev:` for playground)
- `MOMENTRY_API_KEY` - API key for Player online mode testing
### Testing API Key
```bash
export MOMENTRY_API_KEY="muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69"
# Test Player online mode
cargo run --features player --bin momentry_player -- -o
```
### Database
- `DATABASE_URL` - PostgreSQL (default: `postgres://accusys@localhost:5432/momentry`)
### Redis
- `REDIS_URL` - Redis URL (default: `redis://:accusys@localhost:6379`)
- `REDIS_PASSWORD` - Redis password (default: `accusys`)
### Paths
- `MOMENTRY_OUTPUT_DIR` - Output directory (default: `/Users/accusys/momentry/output`)
- `MOMENTRY_BACKUP_DIR` - Backup directory
- `MOMENTRY_PYTHON_PATH` - Python path (default: `/opt/homebrew/bin/python3.11`)
- `MOMENTRY_SCRIPTS_DIR` - Scripts directory
### Processor Timeouts
- `MOMENTRY_ASR_TIMEOUT` - ASR timeout in seconds (default: 3600)
- `MOMENTRY_CUT_TIMEOUT` - CUT timeout in seconds (default: 3600)
- `MOMENTRY_DEFAULT_TIMEOUT` - Default timeout (default: 7200)
### TMDb Integration (Face Clustering)
- `TMDB_API_KEY` - TMDb API key for movie metadata lookup (required for `MOMENTRY_TMDB_PROBE_ENABLED=true`)
- `MOMENTRY_TMDB_PROBE_ENABLED` - Enable TMDb probe during registration (default: `false`)
- Register phase: searches TMDb by filename, creates identities with tmdb_id/tmdb_profile
- Post-process phase: matches detected faces against TMDb identities via cosine similarity
### Synonym Expansion
- `MOMENTRY_SYNONYM_FILES` - Comma-separated paths to synonym JSON files (e.g., `data/english_synonyms.json,data/llm_synonyms.json`)
- `MOMENTRY_SYNONYM_FILE` - Single synonym JSON file path (deprecated, use above)
### Logging
- `RUST_LOG` or `MOMENTRY_LOG_LEVEL` - Log level (default: `info`)
## Notes
- Unit tests exist (86 library tests)
- Video processing uses external tools (ffprobe, Python scripts)
- Multi-database architecture (PostgreSQL, MongoDB, Redis, Qdrant)
- Monitor directory is a separate system (not Rust)
- PythonExecutor provides unified script execution with timeout support
- Redis 1.0.x for improved performance
- FaceNet CoreML model (`models/facenet512.mlpackage`) replaces InsightFace for embedding extraction (MIT license, ANE-accelerated)
### LLM Synonym Generation
Generate synonym database using llama.cpp (Gemma4):
```bash
# Generate full database (162 entries, ~5 minutes)
python3 scripts/generate_synonyms_llamacpp.py
# Quick test
python3 scripts/generate_synonyms_llamacpp.py --test
# Resume from existing file
python3 scripts/generate_synonyms_llamacpp.py --resume
# Output: data/llm_synonyms.json (27 Chinese + 135 English words)
```
## Task Management
### 使用 todowrite 追蹤任務
```bash
# 創建任務清單
/todo 建立配置模組 [in_progress]
/todo 添加單元測試 [pending]
# 更新狀態
/todo 完成標記 [completed]
```
### 任務批次建議
- 一次處理 1-2 個功能
- 每個功能完成後驗證 (clippy + test)
- 驗證通過後再繼續下一個
## Code Review Checklist
完成任務後檢查：
- [ ] `cargo clippy --lib` 通過
- [ ] `cargo test --lib` 通過
- [ ] `cargo fmt -- --check` 通過
- [ ] 文檔已更新 (如需要)
- [ ] 新功能有單元測試
## Commit Guidelines
```bash
# feat: 新功能
git commit -m "feat: add monitor_jobs table"
# fix: 錯誤修復
git commit -m "fix: resolve SQL injection in store_vector"
# refactor: 重構
git commit -m "refactor: use parameterized queries"
# docs: 文檔更新
git commit -m "docs: update AGENTS.md with new modules"
```
## Pre-commit Hook
專案已配置 `.git/hooks/pre-commit`，提交前自動檢查：
```bash
# 檢查內容
1. cargo fmt --check # Rust 格式化檢查
2. cargo clippy --lib # Rust Lint 檢查
3. cargo test --lib # Rust 單元測試
4. ruff check # Python Lint 檢查
5. ruff format --check # Python 格式化檢查
6. markdownlint # Markdown 格式檢查
7. shellcheck # Shell 腳本檢查
# 跳過檢查（不建議）
git commit --no-verify
# 跳過特定檢查
git commit --skip-checks
```
**注意**: Hook 僅檢查已暫存的 Rust/Python/Markdown 文件。
### Python 環境設置
```bash
# 安裝 ruff
pip install ruff==0.11.2
# 格式化 Python 文件
ruff format scripts/
# Lint Python 文件
ruff check scripts/
```
### Markdown 環境設置
```bash
# 安裝 markdownlint-cli (使用系統 Node.js)
npm install -g markdownlint-cli
# 檢查 Markdown 文件
markdownlint docs/
# 配置檔案
.markdownlint.json
```
### Shell 環境設置
```bash
# 安裝 shellcheck
brew install shellcheck
# 檢查 Shell 腳本
shellcheck scripts/*.sh monitor/**/*.sh
```
**注意**: Hook 只檢查 error 等級的 shellcheck 問題，style 警告會顯示但不阻擋提交。
## Release Workflow
### Release 前準備
每次 release production binary 前，必須：
1. **建立 Release Tag**
```bash
git tag -a v0.X.X -m "Release vX.X.X - YYYY-MM-DD"
git push origin v0.X.X
```
2. **備份獨立 Source Code**
```bash
# 建立 release 獨立目錄
RELEASE_DIR="/Users/accusys/momentry_core_releases/v0.X.X"
mkdir -p "$RELEASE_DIR"
# 複製完整原始碼（排除不必要的檔案）
rsync -av --exclude='.git' --exclude='target' --exclude='node_modules' \
/Users/accusys/momentry_core_0.1/ "$RELEASE_DIR/"
# 記錄 release 資訊
echo "Release: v0.X.X" > "$RELEASE_DIR/RELEASE_INFO.txt"
echo "Date: $(date)" >> "$RELEASE_DIR/RELEASE_INFO.txt"
echo "Git Commit: $(git rev-parse HEAD)" >> "$RELEASE_DIR/RELEASE_INFO.txt"
echo "Binary: $(ls -la target/release/momentry)" >> "$RELEASE_DIR/RELEASE_INFO.txt"
```
3. **備份 Binary**
```bash
cp target/release/momentry "$RELEASE_DIR/momentry_v0.X.X"
cp target/release/momentry_playground "$RELEASE_DIR/momentry_playground_v0.X.X" 2>/dev/null
```
4. **記錄資料庫 Schema**
```bash
pg_dump -U accusys -d momentry --schema-only > "$RELEASE_DIR/schema_v0.X.X.sql"
```
### 重要性
- 避免 release binary 與 current source code 不一致
- 方便追蹤特定 release 的程式碼狀態
- 必要時可快速復原或比對差異
- 確保資料庫 schema 與程式碼版本對應
## Reference Documents
| 文件 | 用途 |
|------|------|
| `docs/OPENCODE_GUIDE.md` | OpenCode 使用規範 |
| `docs/ARCHITECTURE_EVALUATION.md` | 架構優化待評估項目 (含 GraphRAG) |
| `docs/PENDING_ISSUES.md` | 待解決問題追蹤 |
| `docs/MOMENTRY_CORE_MONITORING.md` | 監控系統規範 |
| `docs/MOMENTRY_CORE_REDIS_KEYS.md` | Redis Key 設計規範 |
| `docs/PYTHON.md` | Python 腳本規範 |
| `docs/FILE_CHANGE_MANAGEMENT.md` | 文件修改管理規範 |
| `docs/YOLO_RESUME_INTEGRATION.md` | YOLO Resume 功能整合記錄 |
| `docs/DOCUMENT_EMBEDDING_STRATEGY.md` | Parent-Child 嵌入策略 |
| `docs/PROCESSING_PIPELINE.md` | 處理流程文檔 |
| `docs/N8N_DEMO_WORKFLOW.md` | n8n 工作流文檔 |
| `docs/FRESH_MAC_INSTALLATION.md` | 全新 Mac 安裝指南 |
| `docs/SERVICES.md` | 服務總覽與管理 |
| `docs/SFTPGO_DEMO_USER.md` | SFTPGo 用戶指南 |
## Document Change Workflow
修改文件前請參考 `docs/FILE_CHANGE_MANAGEMENT.md`，確保：
1. **修改前**：完整閱讀文件、執行預檢清單
2. **修改中**：提供變更計畫、取得確認
3. **修改後**：展示 diff、更新版本歷史
4. **驗證**：執行 lint/test、提交前審查
### AI 工具修改規範
AI 工具修改文件時：
- 必須先完整閱讀文件（不可只讀取部分章節）
- 修改前先提出變更計畫供確認
- 修改後展示 diff 內容
- 更新版本歷史表
## PHP Development
WordPress 作為 Momentry Portal，負責 n8n 自動化與 sftpgo 檔案服務的頁面整合。
### 編輯器設定
| 編輯器 | LSP 方案 | 安裝方式 |
|--------|----------|----------|
| VS Code | Intelephense | Extension Marketplace (推薦) |
| Cursor | Intelephense | Extension Marketplace (推薦) |
| CLI | phpactor | `~/bin/phpactor` |
### Intelephense (VS Code/Cursor)
1. 安裝 Extension: 搜尋 "Intelephense"
2. 設定:
```json
{
"intelephense.stubs": ["wordpress"]
}
```
### phpactor (CLI)
```bash
# 安裝方式
brew install composer
curl -sSL https://github.com/phpactor/phpactor/releases/latest/download/phpactor.phar -o ~/bin/phpactor
chmod +x ~/bin/phpactor
# 安裝 WordPress Stubs
cd /Users/accusys/wordpress/web
composer require --dev php-stubs/wordpress-stubs
# 建立 WordPress 索引
cd /Users/accusys/wordpress/web
~/bin/phpactor index:build --reset
# 常用指令
~/bin/phpactor class:search "WP_User" # 搜尋類別
~/bin/phpactor index:query WP_User # 查看類別資訊
~/bin/phpactor navigate /path/to/file.php # 導航到定義
```
### WordPress 程式碼位置
| 類型 | 路徑 |
|------|------|
| 主題 | `/Users/accusys/wordpress/web/wp-content/themes/` |
| 插件 | `/Users/accusys/wordpress/web/wp-content/plugins/` |
### 與 marcom 團隊協作
| 角色 | 負責 |
|------|------|
| marcom 團隊 | Figma 設計 / Elementor 建構 |
| OpenCode | 程式碼實作 / 重構 |
### 開發時程
```
Phase 1: marcom 建構 (現在) → Elementor 頁面建構
Phase 2: 交付審視 (TBD) → 功能確認 / 重構評估
Phase 3: OpenCode 重構 → 純程式碼實作，交付無 Elementor 依賴版本
```
## M4 通知規範
### 固定通知方式
通知 M4 的唯一管道：**`M4_workspace/` 下建立回覆文件 + `git commit`**。不需口頭、即時訊息、郵件。
### 命名規則
```
docs_v1.0/M4_workspace/YYYY-MM-DD_<topic>_response.md (回覆 M4 問題)
docs_v1.0/M4_workspace/YYYY-MM-DD_<topic>.md (主動通報)
docs_v1.0/M4_workspace/YYYY-MM-DD_<topic>_test_report.md (測試報告)
```
### 觸發時機
| 情境 | 動作 |
|------|------|
| M4 提交問題報告到 `M4_workspace/` | 修復後，回覆 `*_response.md` |
| 完成 M4 要求的任務 | 回覆 `*_response.md` |
| 重大變更（模型替換、架構變更） | 主動通知 `*.md` |
| 新測試包產出 | `*_test_report.md` |
### 交付檢查
1. 文件寫入 `docs_v1.0/M4_workspace/`
2. `git add` 包含該文件
3. `git commit` 含相關變更
4. M4 透過 git log 查看
詳細規範見 `docs_v1.0/M4_workspace/M4_NOTIFICATION_PROTOCOL.md`。
## UUID Naming Rule
**Never use bare `uuid` in API route paths, query params, JSON keys, or code variable names. Always qualify:**
| Context | Must use | Never |
|---------|----------|-------|
| Video/file resource | `file_uuid` | `uuid` |
| Identity resource | `identity_uuid` | `uuid` |
| Query parameter | `file_uuid=`, `identity_uuid=` | `uuid=` |
| Route path | `:file_uuid`, `:identity_uuid` | `:uuid` |
| JSON key | `"file_uuid"`, `"identity_uuid"` | `"uuid"` |
This applies to docs, code, API responses, and curl examples. Exceptions: internal database primary key names (e.g. `identities.uuid` column).
## Document Compliance Checklist
Before creating any file in `docs_v1.0/` (API_WORKSPACE, GUIDES, REFERENCE, DESIGN, OPERATIONS, INTEGRATIONS), verify all items below.
**IMPORTANT**: API functional documents are generated from `API_WORKSPACE/modules/`. Edit modules there, then run `make deploy` in `API_WORKSPACE/` to update `GUIDES/`. Never edit generated files in `GUIDES/` directly. See `DESIGN/Modular_Doc_System_V1.0.md` for the full system design.
### P0 — Mandatory (7 items)
| # | Check | Rule |
|---|-------|------|
| 1 | YAML frontmatter | `title`, `version`, `date`, `author`, `status` present |
| 2 | Version history | Table at bottom of file tracking changes |
| 3 | Top info table | scope, status, applicable to, etc. |
| 4 | PascalCase filename | e.g. `DetectorRegistry.md`, not `detector_registry.md` |
| 5 | `_` separator | Within filenames use `_`, never spaces or other chars |
| 6 | English content | Entire file in English |
| 7 | Correct directory | File must reside in appropriate directory: `API_WORKSPACE/modules/` (API endpoint modules), `GUIDES/` (user docs, generated), `REFERENCE/` (data models), `DESIGN/` (architecture), `OPERATIONS/` (infra/release), `INTEGRATIONS/` (n8n/tests) |
### P0b — UUID Naming
| # | Check | Rule |
|---|-------|------|
| 8 | `file_uuid` not bare `uuid` | All file references use `file_uuid` (see UUID Naming Rule above) |
| 9 | `identity_uuid` not bare `uuid` | All identity references use `identity_uuid` |
### P1 — Suggested (3 items)
| # | Check | Note |
|---|-------|------|
| 1 | Cross-references | Link to related docs in API_WORKSPACE/, GUIDES/, REFERENCE/, DESIGN/, OPERATIONS/ |
| 2 | Glossary terms | Define non-obvious terms inline or link glossary |
| 3 | Diagrams | Include Mermaid/ASCII diagram for complex topics |
### Exception
`M4_workspace/` files are exempt from this checklist (free-format reply documents).
---
## Delivery Procedure
完整交付程序（M4_workspace → M5 → Release → Deploy → Public）見：
`docs_v1.0/OPERATIONS/DELIVERY_PROCEDURE.md`
deliverable_v1.1.0/SYSTEM_AUDIT_2026-05-17.md
+71
View File
@@ -0,0 +1,71 @@
# System Audit — 2026-05-17
## Current State
### Embedding Storage (三重冗余，無主)
| 資料類型 | PG pgvector | Qdrant | JSON 檔案 |
|---------|------------|--------|-----------|
| Sentence 向量 | `chunk.embedding` ✅ | `dev_v1` / `rule1_v2` / `sentence_*` ✅ | ❌ 無 |
| Story 向量 | `chunk.embedding` ✅ | `dev_v1` / `dev_stories` ✅ | `.story_llm.json` ✅ |
| Face 向量 | ❌ 已清除（依使用者指示） | `dev_faces` ✅ (97K) | `.face.json` ✅ |
| Voice 向量 | ❌ 無 | `dev_voice` ✅ (4K) | ❌ 無 |
### Pipeline 問題
| 問題 | 影響 |
|------|------|
| `processor_results.duration_secs` 全為 0 | 無法查各步驟耗時 |
| `processor_results.started_at/completed_at` 全 NULL | 時間線遺失 |
| Redis timing 在 job 完成後被清掉 | 唯一 timing 來源消失 |
| `get_chunk_by_chunk_id_and_uuid` 原本是 stub（已修） | Smart search 找不到 PG chunk |
| `server.rs::search()` 未 mount 但仍編譯 | Dead code，混淆 Qdrant 用途 |
| Face embedding 只寫 Qdrant 不寫 PG | 已刪除則全失 |
### Qdrant Collections 現況
| Collection | Points | 來源 | UUID |
|-----------|--------|------|------|
| `dev_v1` | 9,936 | PG rebuild | ✅ bd80fec... |
| `dev_faces` | 97,000 | face.json rebuild | ✅ bd80fec... |
| `dev_stories` | 560 | Snapshot | ✅ bd80fec... |
| `dev_voice` | 4,188 | Snapshot | ✅ bd80fec... |
| `dev_rule1_v2` | 3,417 | Snapshot | ✅ bd80fec... |
| `sentence_story` | 4,188 | Snapshot | ✅ bd80fec... |
| `sentence_summary` | 4,188 | Snapshot | ✅ bd80fec... |
## Safeguards & Fixes
### P0 — 必須修
| # | Fix | 做法 |
|---|-----|------|
| 1 | **Pipeline timing 寫入 DB** | `update_processor_result()` 加入 `started_at`、`completed_at`、`duration_secs` |
| 2 | **Qdrant 不當主要儲存** | Embedding 以 PG `chunk.embedding` 為 source of truth，Qdrant 唯讀 cache |
| 3 | **Smart search 只走 PG pgvector** | `search_parent_chunks_semantic` 已正確，無需 Qdrant |
| 4 | **移除 `server.rs::search()` dead code** | 或 mount 到正式 route 並確認可用 |
### P1 — 建議修
| # | Fix | 做法 |
|---|-----|------|
| 5 | **刪除 Qdrant 前先 snapshot** | 自動 snapshot script |
| 6 | **清理多餘 Qdrant collections** | `dev_voice` / `dev_stories` / `dev_rule1_v2` / `sentence_*` 無 server reader，可移除 |
| 7 | **Face embedding 寫入 PG 或移除 dead code** | 目前 face Qdrant write 無人讀取，可移除 `sync_face_embeddings` |
| 8 | **UUID 一致性檢查** | 同一 content 不應產生不同 UUID |
### P2 — 可選
| # | Fix | 做法 |
|---|-----|------|
| 9 | `chunk_selector.rs` （player binary）hardcode `momentry_rule1` | 改讀 env var 或 PG |
| 10 | AGENTS.md 已加入 delete 安全規則 | ✅ Done |
## Data Recovery Path
| 資料來源 | 可恢復到 | 方法 |
|---------|---------|------|
| `chunk.embedding` (PG) | Qdrant `dev_v1` | SQL → Qdrant upsert |
| `face.json` (磁碟) | Qdrant `dev_faces` | Python script |
| `story_llm.json` (磁碟) | Qdrant `dev_stories` | Python script |
| Qdrant snapshots (phase1) | Qdrant collections | Snapshot upload API |
deliverable_v1.1.0/html_docs/doc/01_auth.html
+388
View File
@@ -0,0 +1,388 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>01 Auth - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: auth -->
<!-- description: Authentication — login, logout, JWT, session cookie, API key -->
<!-- depends: -->
<h2>Base URL</h2>
<table class="table">
<thead>
<tr>
<th>Environment</th>
<th>URL</th>
<th>Purpose</th>
</tr>
</thead>
<tbody>
<tr>
<td>Production</td>
<td><code>http://localhost:3002</code></td>
<td>Production deployment</td>
</tr>
<tr>
<td>External (M5)</td>
<td><code>https://m5api.momentry.ddns.net</code></td>
<td>Remote access</td>
</tr>
</tbody>
</table>
<h2>Variables</h2>
<p>All examples in this documentation use these environment variables:</p>
<div class="codehilite"><pre><span></span><code><span class="nv">API</span><span class="o">=</span><span class="s2">&quot;http://localhost:3002&quot;</span>
<span class="nv">KEY</span><span class="o">=</span><span class="s2">&quot;your-api-key-here&quot;</span>
</code></pre></div>
<h2>Authentication</h2>
<p>All endpoints under <code>/api/v1/*</code> require authentication.
The following endpoints are public (no auth needed):</p>
<ul>
<li><code>GET /health</code></li>
<li><code>POST /api/v1/auth/login</code></li>
<li><code>POST /api/v1/auth/logout</code></li>
</ul>
<h3>Three Authentication Modes</h3>
<p>The system supports three authentication methods, checked in <strong>priority order</strong> by the middleware:</p>
<div class="codehilite"><pre><span></span><code>Middleware priority:
1. Session Cookie (Portal/browser)
2. JWT Bearer (API clients, CLI)
3. API Key Header (legacy compatibility)
4. API Key Query Param (?api_key=)
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Mode</th>
<th>Transport</th>
<th>Expiry</th>
<th>Scope</th>
<th>Best for</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Session Cookie</strong></td>
<td><code>Cookie: session_id=&lt;session_id&gt;</code></td>
<td>24h</td>
<td>per-browser session</td>
<td>Portal (browser)</td>
</tr>
<tr>
<td><strong>JWT</strong></td>
<td><code>Authorization: Bearer &lt;token&gt;</code></td>
<td>1h</td>
<td>per-login token</td>
<td>API clients, CLI, scripts</td>
</tr>
<tr>
<td><strong>API Key</strong></td>
<td><code>X-API-Key: &lt;key&gt;</code></td>
<td>90d</td>
<td>fixed key for automation</td>
<td>Legacy scripts, WordPress</td>
</tr>
</tbody>
</table>
<hr />
<h3>Login</h3>
<p><strong>Default accounts &amp; API keys:</strong></p>
<table class="table">
<thead>
<tr>
<th>Username</th>
<th>Password</th>
<th>API Key</th>
<th>Role</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>admin</code></td>
<td><code>admin</code></td>
<td>—</td>
<td>admin</td>
</tr>
<tr>
<td><code>demo</code></td>
<td><code>demo</code></td>
<td><code>muser_demo_key_32chars_abcdef1234567890</code></td>
<td>user</td>
</tr>
</tbody>
</table>
<p>The demo API key is set via <code>MOMENTRY_DEMO_API_KEY</code> env var and can be used in place of JWT for marcom integrations:</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># Using API key instead of JWT</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: muser_demo_key_32chars_abcdef1234567890&quot;</span>
</code></pre></div>
<div class="codehilite"><pre><span></span><code><span class="c1"># Login as admin</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/auth/login&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;username&quot;: &quot;admin&quot;, &quot;password&quot;: &quot;admin&quot;}&#39;</span>
<span class="c1"># Login as demo user</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/auth/login&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;username&quot;: &quot;demo&quot;, &quot;password&quot;: &quot;demo&quot;}&#39;</span>
</code></pre></div>
<h4>Success Response</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;jwt&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;eyJhbGciOiJIUzI1NiIs...&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;api_key&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;muser_...&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;user&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;username&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;admin&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;role&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;admin&quot;</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;expires_at&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;2026-05-18T13:00:00Z&quot;</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>jwt</code></td>
<td>string</td>
<td>JWT access token. Use as <code>Authorization: Bearer &lt;jwt&gt;</code>. Expires in 1 hour.</td>
</tr>
<tr>
<td><code>api_key</code></td>
<td>string</td>
<td>Legacy API key. Use as <code>X-API-Key: &lt;key&gt;</code>. Good for 90 days.</td>
</tr>
<tr>
<td><code>user.username</code></td>
<td>string</td>
<td>Username</td>
</tr>
<tr>
<td><code>user.role</code></td>
<td>string</td>
<td>Role: <code>admin</code>, <code>user</code>, or <code>readonly</code></td>
</tr>
<tr>
<td><code>expires_at</code></td>
<td>string</td>
<td>ISO8601 timestamp of JWT expiration</td>
</tr>
</tbody>
</table>
<p>The login endpoint also sets a <code>Set-Cookie</code> header for browser-based clients:</p>
<div class="codehilite"><pre><span></span><code><span class="nt">Set-Cookie</span><span class="o">:</span><span class="w"> </span><span class="nt">session_id</span><span class="o">=&lt;</span><span class="nt">session_id</span><span class="o">&gt;;</span><span class="w"> </span><span class="nt">Path</span><span class="o">=/;</span><span class="w"> </span><span class="nt">HttpOnly</span><span class="o">;</span><span class="w"> </span><span class="nt">SameSite</span><span class="o">=</span><span class="nt">Strict</span><span class="o">;</span><span class="w"> </span><span class="nt">Max-Age</span><span class="o">=</span><span class="nt">86400</span>
</code></pre></div>
<h4>Error Response (401)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Invalid username or password&quot;</span>
<span class="p">}</span>
</code></pre></div>
<hr />
<h3>Using JWT</h3>
<p>JWT is preferred for API clients (CLI scripts, WordPress). It is validated by the middleware without a database lookup (stateless).</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># Login and capture JWT</span>
<span class="nv">JWT</span><span class="o">=</span><span class="k">$(</span>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/auth/login&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;username&quot;:&quot;admin&quot;,&quot;password&quot;:&quot;admin&quot;}&#39;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>python3<span class="w"> </span>-c<span class="w"> </span><span class="s2">&quot;import json,sys;print(json.load(sys.stdin)[&#39;jwt&#39;])&quot;</span><span class="k">)</span>
<span class="c1"># Use JWT for all subsequent requests</span>
curl<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan&quot;</span>
curl<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/resource/tmdb&quot;</span>
</code></pre></div>
<p>JWT is short-lived (1 hour). When it expires, request a new one via login.</p>
<hr />
<h3>Using Session Cookie (Browser)</h3>
<p>Browser-based clients (Portal) get a session cookie automatically after login. The browser sends the cookie with every request—no manual header needed.</p>
<div class="codehilite"><pre><span></span><code><span class="c1"># Login captures the session cookie from Set-Cookie header</span>
curl<span class="w"> </span>-v<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/auth/login&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;username&quot;:&quot;admin&quot;,&quot;password&quot;:&quot;admin&quot;}&#39;</span><span class="w"> </span><span class="m">2</span>&gt;<span class="p">&amp;</span><span class="m">1</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">&quot;Set-Cookie&quot;</span>
<span class="c1"># Browser automatically sends: Cookie: session_id=&lt;session_id&gt;</span>
<span class="c1"># No manual header needed for subsequent requests</span>
</code></pre></div>
<p>The session cookie is HttpOnly (not accessible from JavaScript) and SameSite=Strict (protected against CSRF).</p>
<hr />
<h3>Using Legacy API Key</h3>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan&quot;</span>
<span class="c1"># Also accepted via Bearer header (non-JWT format) or query parameter:</span>
curl<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan&quot;</span>
curl<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan?api_key=</span><span class="nv">$KEY</span><span class="s2">&quot;</span>
</code></pre></div>
<p>API keys are validated via SHA256 hash lookup in the database. They are long-lived (90 days) and intended for automation.</p>
<h3>Obtaining an API Key (CLI)</h3>
<div class="codehilite"><pre><span></span><code>momentry<span class="w"> </span>api-key<span class="w"> </span>create<span class="w"> </span><span class="s2">&quot;My API Key&quot;</span><span class="w"> </span>--key-type<span class="w"> </span>user
</code></pre></div>
<hr />
<h3>Logout</h3>
<div class="codehilite"><pre><span></span><code><span class="c1"># Logout using the session cookie (browser)</span>
curl<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/auth/logout&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Cookie: session_id=&lt;uuid&gt;&quot;</span>
</code></pre></div>
<h4>What logout does</h4>
<table class="table">
<thead>
<tr>
<th>Auth mode</th>
<th>Effect</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Session Cookie</strong></td>
<td>Session deleted from database. Same cookie returns 401 on subsequent requests.</td>
</tr>
<tr>
<td><strong>JWT</strong></td>
<td>JWT remains valid until expiry. (JWT is stateless — logout adds JWT to a blacklist only if API key mode is used.)</td>
</tr>
<tr>
<td><strong>API Key</strong></td>
<td>API key remains valid. (Legacy keys are shared across sessions — revoking would break other clients.)</td>
</tr>
</tbody>
</table>
<h4>Example: full session lifecycle</h4>
<div class="codehilite"><pre><span></span><code><span class="c1"># 1. Login</span>
<span class="nv">SESSION_ID</span><span class="o">=</span><span class="k">$(</span>curl<span class="w"> </span>-s<span class="w"> </span>-D<span class="w"> </span>-<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/auth/login&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;username&quot;:&quot;admin&quot;,&quot;password&quot;:&quot;admin&quot;}&#39;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">&quot;Set-Cookie&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">&#39;s/.*session_id=\([^;]*\).*/\1/&#39;</span><span class="k">)</span>
<span class="c1"># 2. Use session (works)</span>
curl<span class="w"> </span>-s<span class="w"> </span>-o<span class="w"> </span>/dev/null<span class="w"> </span>-w<span class="w"> </span><span class="s2">&quot;HTTP %{http_code}\n&quot;</span><span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/resource/tmdb&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Cookie: session_id=</span><span class="nv">$SESSION_ID</span><span class="s2">&quot;</span>
<span class="c1"># → HTTP 200</span>
<span class="c1"># 3. Logout</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/auth/logout&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Cookie: session_id=</span><span class="nv">$SESSION_ID</span><span class="s2">&quot;</span>
<span class="c1"># → {&quot;success&quot;: true}</span>
<span class="c1"># 4. Use session again (rejected)</span>
curl<span class="w"> </span>-s<span class="w"> </span>-o<span class="w"> </span>/dev/null<span class="w"> </span>-w<span class="w"> </span><span class="s2">&quot;HTTP %{http_code}\n&quot;</span><span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/resource/tmdb&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Cookie: session_id=</span><span class="nv">$SESSION_ID</span><span class="s2">&quot;</span>
<span class="c1"># → HTTP 401</span>
</code></pre></div>
<hr />
<h3>Authentication Flow Summary</h3>
<div class="codehilite"><pre><span></span><code>Login Request
│
▼
┌──────────────────┐
│ 1. Check users │ ← users table (argon2 password verify)
│ table │
└──────┬───────────┘
│
┌───┴───┐
│ match │
└───┬───┘
│
▼
┌──────────────────┐
│ 2. Create JWT │ ← 1h expiry, signed with JWT_SECRET
├──────────────────┤
│ 3. Create │ ← 24h expiry, stored in sessions table
│ session │
├──────────────────┤
│ 4. Set-Cookie │ ← HttpOnly, SameSite=Strict, Path=/
├──────────────────┤
│ 5. Return │ ← JWT + api_key + user info to client
└──────────────────┘
</code></pre></div>
<div class="codehilite"><pre><span></span><code>Protected Request
│
▼
┌──────────────────────┐
│ Middleware checks: │
│ │
│ 1. Cookie session? │ → DB lookup session → get api_key → verify
│ │
│ 2. JWT Bearer? │ → verify JWT signature → decode claims
│ │
│ 3. X-API-Key? │ → SHA256 hash → DB lookup → verify
│ │
│ 4. ?api_key=? │ → same as #3
│ │
│ 5. None → 401 │
└──────────────────────┘
</code></pre></div>
<hr />
<h3>Error Responses</h3>
<table class="table">
<thead>
<tr>
<th>HTTP</th>
<th>When</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>401</code></td>
<td>Missing or invalid authentication</td>
</tr>
<tr>
<td><code>401</code></td>
<td>Session expired or logged out</td>
</tr>
<tr>
<td><code>401</code></td>
<td>JWT expired</td>
</tr>
<tr>
<td><code>401</code></td>
<td>API key revoked or inactive</td>
</tr>
</tbody>
</table>
<hr />
<h3>Related</h3>
<ul>
<li><code>POST /api/v1/resource/tmdb/check</code> — test authentication + TMDb API connectivity</li>
<li><code>GET /health/detailed</code> — view auth status (integrations section)</li>
</ul>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/02_health.html
+277
View File
@@ -0,0 +1,277 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>02 Health - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: health -->
<!-- description: Health check endpoints -->
<!-- depends: 01_auth -->
<h2>Health Check</h2>
<h3><code>GET /health</code></h3>
<p><strong>Auth</strong>: Public
<strong>Scope</strong>: system-level</p>
<p>Returns basic server health status — used by load balancers and monitoring.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/health&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{status, version}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;ok&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;version&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;1.0.0&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;build_git_hash&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;3a6c1865&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;build_timestamp&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;2026-05-16T13:38:15Z&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;uptime_ms&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">3015</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>status</code></td>
<td>string</td>
<td><code>ok</code> or <code>degraded</code></td>
</tr>
<tr>
<td><code>version</code></td>
<td>string</td>
<td>Semver version</td>
</tr>
<tr>
<td><code>build_git_hash</code></td>
<td>string</td>
<td>Git commit hash</td>
</tr>
<tr>
<td><code>build_timestamp</code></td>
<td>string</td>
<td>Binary build time</td>
</tr>
<tr>
<td><code>uptime_ms</code></td>
<td>integer</td>
<td>Milliseconds since server start</td>
</tr>
</tbody>
</table>
<hr />
<h3><code>GET /health/detailed</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: system-level</p>
<p>Returns full system health including each service status, resource utilization, pipeline readiness, schema migration status, identity file sync status, and external integrations.</p>
<blockquote>
<p>Requires authentication (JWT, session cookie, or API key). The basic <code>/health</code> endpoint remains public for load balancer checks.</p>
</blockquote>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/health/detailed&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{status, services, resources: {cpu: .resources.cpu_used_percent, memory: .resources.memory_used_percent}}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;ok&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;version&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;1.0.0&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;services&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;postgres&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;ok&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;latency_ms&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">3</span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;redis&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;ok&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;latency_ms&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;qdrant&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;ok&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;latency_ms&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">}</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;resources&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;cpu_used_percent&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">12.5</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;memory_available_mb&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">32768</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;memory_used_percent&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">31.7</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;pipeline&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;scripts_ready&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;scripts_count&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">345</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;processors&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;asr&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;yolo&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;face&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;pose&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;ocr&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;cut&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;scene&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;asrx&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;visual_chunk&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;models_ready&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;models_count&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">42</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;scripts_integrity&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">&quot;matched&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">332</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;total&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">345</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;ok&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;ffmpeg&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;schema&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;table_exists&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;applied&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="nt">&quot;filename&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;migrate_add_users_table.sql&quot;</span><span class="p">}],</span>
<span class="w"> </span><span class="nt">&quot;required&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span>
<span class="w"> </span><span class="nt">&quot;ok&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;identities&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;directory_exists&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;files_count&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">3481</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;index_ok&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;db_count&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">3481</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;synced&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;integrations&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;tmdb&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;api_key_configured&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;enabled&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;api_reachable&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h4>Response Fields</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>status</code></td>
<td>string</td>
<td><code>ok</code> if all essential services healthy</td>
</tr>
<tr>
<td><code>services</code></td>
<td>object</td>
<td>Per-service status (postgres, redis, qdrant)</td>
</tr>
<tr>
<td><code>services.*.status</code></td>
<td>string</td>
<td><code>ok</code>, <code>error</code>, or <code>degraded</code></td>
</tr>
<tr>
<td><code>services.*.latency_ms</code></td>
<td>int</td>
<td>Response time in milliseconds</td>
</tr>
<tr>
<td><code>resources</code></td>
<td>object</td>
<td>CPU, memory usage</td>
</tr>
<tr>
<td><code>pipeline.scripts_ready</code></td>
<td>boolean</td>
<td>Scripts directory accessible</td>
</tr>
<tr>
<td><code>pipeline.scripts_count</code></td>
<td>int</td>
<td>Number of Python processor scripts</td>
</tr>
<tr>
<td><code>pipeline.processors</code></td>
<td>object</td>
<td>Per-processor availability</td>
</tr>
<tr>
<td><code>pipeline.models_ready</code></td>
<td>boolean</td>
<td>Models directory accessible</td>
</tr>
<tr>
<td><code>pipeline.scripts_integrity</code></td>
<td>object</td>
<td>SHA256 checksum verification results</td>
</tr>
<tr>
<td><code>schema.ok</code></td>
<td>boolean</td>
<td>All required migrations applied</td>
</tr>
<tr>
<td><code>identities.synced</code></td>
<td>boolean</td>
<td>Identity file count matches DB count</td>
</tr>
<tr>
<td><code>integrations.tmdb</code></td>
<td>object</td>
<td>TMDB API key config and reachability</td>
</tr>
</tbody>
</table>
<h4>Health status rules</h4>
<table class="table">
<thead>
<tr>
<th>Condition</th>
<th>status</th>
</tr>
</thead>
<tbody>
<tr>
<td>All services ok</td>
<td><code>ok</code></td>
</tr>
<tr>
<td>Any service error</td>
<td><code>degraded</code></td>
</tr>
<tr>
<td>Postgres or Redis error</td>
<td><code>degraded</code> (server still responds)</td>
</tr>
</tbody>
</table>
<hr />
<h3>Stats Endpoints</h3>
<table class="table">
<thead>
<tr>
<th>Method</th>
<th>Endpoint</th>
<th>Auth</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>GET</td>
<td><code>/api/v1/stats/sftpgo</code></td>
<td>No</td>
<td>SFTPGo service status</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/03_register.html
+444
View File
@@ -0,0 +1,444 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>03 Register - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: register -->
<!-- description: File registration — register, scan -->
<!-- depends: 01_auth -->
<h2>File Registration</h2>
<h3><code>POST /api/v1/files/register</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Register a video file for processing. Returns the file's metadata and UUID.</p>
<p><strong>New in v0.1.2</strong>: Registration now <strong>automatically triggers the processing pipeline</strong> — no need to call <code>POST /api/v1/file/:file_uuid/process</code> separately. The system will:
1. Register the file and run ffprobe
2. Auto-run offline TMDb probe (reads local identity files, no API calls)
3. Create a monitor job for the worker
4. Worker starts all 10 processors (Cut → ASR → ASRX → YOLO → OCR → Face → Pose → VisualChunk → Story → 5W1H)</p>
<p>If the file already exists (same content hash), returns the existing record with <code>already_exists: true</code>.</p>
<h4>Request Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_path</code></td>
<td>string</td>
<td>Yes</td>
<td>—</td>
<td>Path to video file on disk</td>
</tr>
<tr>
<td><code>pattern</code></td>
<td>string</td>
<td>No</td>
<td>—</td>
<td>Regex pattern for batch register (requires <code>file_path</code> to be a directory)</td>
</tr>
<tr>
<td><code>user_id</code></td>
<td>integer</td>
<td>No</td>
<td>—</td>
<td>User ID to associate with registration</td>
</tr>
<tr>
<td><code>content_hash</code></td>
<td>string</td>
<td>No</td>
<td>—</td>
<td>Pre-computed SHA-256 hash (skips computation)</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code><span class="c1"># Register a single file</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/register&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_path&quot;: &quot;/path/to/video.mp4&quot;}&#39;</span>
<span class="c1"># Batch register files matching a pattern in a directory</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/register&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_path&quot;: &quot;/path/to/dir&quot;, &quot;pattern&quot;: &quot;.*\\.mp4$&quot;}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;3a6c1865...&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;video.mp4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_path&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;/path/to/video.mp4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;video&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;duration&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">120.5</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;width&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1920</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;height&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1080</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;fps&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">24.0</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;total_frames&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">2892</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;already_exists&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;File registered successfully&quot;</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>success</code></td>
<td>boolean</td>
<td>Always true on 200</td>
</tr>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>32-char hex UUID of the registered file</td>
</tr>
<tr>
<td><code>file_name</code></td>
<td>string</td>
<td>File name (auto-renamed if name conflict)</td>
</tr>
<tr>
<td><code>file_path</code></td>
<td>string</td>
<td>Canonical path on disk</td>
</tr>
<tr>
<td><code>file_type</code></td>
<td>string</td>
<td><code>"video"</code>, <code>"audio"</code>, or <code>"unknown"</code></td>
</tr>
<tr>
<td><code>duration</code></td>
<td>float</td>
<td>Duration in seconds</td>
</tr>
<tr>
<td><code>width</code></td>
<td>integer</td>
<td>Video width in pixels</td>
</tr>
<tr>
<td><code>height</code></td>
<td>integer</td>
<td>Video height in pixels</td>
</tr>
<tr>
<td><code>fps</code></td>
<td>float</td>
<td>Frames per second</td>
</tr>
<tr>
<td><code>total_frames</code></td>
<td>integer</td>
<td>Total frame count</td>
</tr>
<tr>
<td><code>already_exists</code></td>
<td>boolean</td>
<td>True if same content was already registered</td>
</tr>
<tr>
<td><code>message</code></td>
<td>string</td>
<td>Human-readable status</td>
</tr>
</tbody>
</table>
<h4>Error Responses</h4>
<table class="table">
<thead>
<tr>
<th>HTTP</th>
<th>When</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>401</code></td>
<td>Missing or invalid API key</td>
</tr>
<tr>
<td><code>400</code></td>
<td>Invalid request body</td>
</tr>
<tr>
<td><code>404</code></td>
<td>File path does not exist</td>
</tr>
</tbody>
</table>
<hr />
<h3><code>GET /api/v1/files/scan</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Scan the filesystem directory and list all media files, showing which are registered, processing, or unregistered.</p>
<h4>Query Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>page</code></td>
<td>integer</td>
<td>No</td>
<td>1</td>
<td>Page number (1-based)</td>
</tr>
<tr>
<td><code>page_size</code></td>
<td>integer</td>
<td>No</td>
<td>all</td>
<td>Items per page (alias: <code>limit</code>)</td>
</tr>
<tr>
<td><code>limit</code></td>
<td>integer</td>
<td>No</td>
<td>all</td>
<td>Max items (alias for <code>page_size</code>)</td>
</tr>
<tr>
<td><code>pattern</code></td>
<td>string</td>
<td>No</td>
<td>—</td>
<td>Regex filter on file name (e.g., <code>.*\\.mp4$</code>)</td>
</tr>
<tr>
<td><code>sort_by</code></td>
<td>string</td>
<td>No</td>
<td><code>name</code></td>
<td>Sort field: <code>name</code>, <code>size</code>, <code>modified</code>, <code>status</code></td>
</tr>
<tr>
<td><code>sort_order</code></td>
<td>string</td>
<td>No</td>
<td><code>asc</code></td>
<td>Sort direction: <code>asc</code> or <code>desc</code></td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code><span class="c1"># Full scan</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{total, registered_count, unregistered_count}&#39;</span>
<span class="c1"># Paginated (page 1, 5 per page)</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan?page=1&amp;page_size=5&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{page, total_pages, files: [.files[].file_name]}&#39;</span>
<span class="c1"># Regex filter: only mp4 files</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan?pattern=.*\\.mp4</span>$<span class="s2">&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{filtered_total, files: [.files[].file_name]}&#39;</span>
<span class="c1"># Sort by file size (largest first)</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan?sort_by=size&amp;sort_order=desc&amp;page_size=5&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;[.files[] | {file_name, file_size}]&#39;</span>
<span class="c1"># Sort by modified time (most recent first)</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan?sort_by=modified&amp;sort_order=desc&amp;page_size=5&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;[.files[] | {file_name, modified_time}]&#39;</span>
<span class="c1"># Sort by status</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/scan?sort_by=status&amp;page_size=5&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;[.files[] | {file_name, status}]&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;files&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;file_name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;video.mp4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_size&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">12345678</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;is_registered&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;3a6c1865...&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;completed&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;registration_time&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;2026-05-16T12:00:00Z&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;job_id&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">42</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;total&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">107</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;filtered_total&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">80</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;page&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;page_size&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">20</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;total_pages&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">4</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;registered_count&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">26</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;unregistered_count&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">81</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>files</code></td>
<td>array</td>
<td>Array of file info objects (paginated)</td>
</tr>
<tr>
<td><code>files[].file_name</code></td>
<td>string</td>
<td>File name</td>
</tr>
<tr>
<td><code>files[].relative_path</code></td>
<td>string</td>
<td>Path relative to scan root</td>
</tr>
<tr>
<td><code>files[].file_path</code></td>
<td>string</td>
<td>Absolute path on disk</td>
</tr>
<tr>
<td><code>files[].file_size</code></td>
<td>integer</td>
<td>File size in bytes</td>
</tr>
<tr>
<td><code>files[].modified_time</code></td>
<td>string</td>
<td>Last modified timestamp (ISO8601)</td>
</tr>
<tr>
<td><code>files[].is_registered</code></td>
<td>boolean</td>
<td>Whether file is registered in DB</td>
</tr>
<tr>
<td><code>files[].file_uuid</code></td>
<td>string</td>
<td>32-char hex UUID (only if registered)</td>
</tr>
<tr>
<td><code>files[].status</code></td>
<td>string</td>
<td><code>"completed"</code>, <code>"processing"</code>, <code>"registered"</code>, <code>"unregistered"</code>, or <code>null</code></td>
</tr>
<tr>
<td><code>files[].registration_time</code></td>
<td>string</td>
<td>DB registration timestamp (only if registered)</td>
</tr>
<tr>
<td><code>files[].job_id</code></td>
<td>integer</td>
<td>Processing job ID (only if a job exists)</td>
</tr>
<tr>
<td><code>total</code></td>
<td>integer</td>
<td>Total files found on disk (unfiltered)</td>
</tr>
<tr>
<td><code>filtered_total</code></td>
<td>integer</td>
<td>Files matching regex filter</td>
</tr>
<tr>
<td><code>page</code></td>
<td>integer</td>
<td>Current page number</td>
</tr>
<tr>
<td><code>page_size</code></td>
<td>integer</td>
<td>Items per page</td>
</tr>
<tr>
<td><code>total_pages</code></td>
<td>integer</td>
<td>Total pages</td>
</tr>
<tr>
<td><code>registered_count</code></td>
<td>integer</td>
<td>Files registered in DB</td>
</tr>
<tr>
<td><code>unregistered_count</code></td>
<td>integer</td>
<td>Files not yet registered</td>
</tr>
</tbody>
</table>
<h4>Notes</h4>
<table class="table">
<thead>
<tr>
<th>Feature</th>
<th>Behavior</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Regex</strong></td>
<td>Case-insensitive (<code>(?i)</code> prefix auto-applied). Applied to <code>file_name</code>.</td>
</tr>
<tr>
<td><strong>Sort order</strong></td>
<td>Default (<code>sort_by=name</code>): registered files first, then alphabetically. <code>sort_by=status</code>: alphabetical by status string.</td>
</tr>
<tr>
<td><strong>Pagination</strong></td>
<td><code>page_size</code> and <code>limit</code> are aliases. Default: show all results.</td>
</tr>
<tr>
<td><strong>Processing order</strong></td>
<td><code>pattern</code> regex filter → <code>sort_by</code>/<code>sort_order</code> → <code>page</code>/<code>page_size</code> slice.</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/04_lookup.html
+291
View File
@@ -0,0 +1,291 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>04 Lookup - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: lookup -->
<!-- description: File lookup by name and unregistration -->
<!-- depends: 01_auth, 03_register -->
<h2>File Lookup</h2>
<h3><code>GET /api/v1/files/lookup</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Search registered files by file name. Performs a case-insensitive LIKE search on the file name column. Returns basic info about matching files.</p>
<h4>Query Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_name</code></td>
<td>string</td>
<td>Yes</td>
<td>File name to search for (partial matches supported)</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code><span class="c1"># Look up a specific file</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/lookup?file_name=video.mp4&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span>
<span class="c1"># Partial name search</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/files/lookup?file_name=charade&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;.matches[].file_name&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;file_name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;video.mp4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;exists&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;matches&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;a03485a40b2df2d3&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;video.mp4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;video&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;completed&quot;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;next_name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;video (2).mp4&quot;</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_name</code></td>
<td>string</td>
<td>Searched name</td>
</tr>
<tr>
<td><code>exists</code></td>
<td>boolean</td>
<td>Exact name match exists</td>
</tr>
<tr>
<td><code>matches</code></td>
<td>array</td>
<td>Array of matching registered files</td>
</tr>
<tr>
<td><code>matches[].file_uuid</code></td>
<td>string</td>
<td>32-char hex UUID</td>
</tr>
<tr>
<td><code>matches[].file_name</code></td>
<td>string</td>
<td>Registered file name</td>
</tr>
<tr>
<td><code>matches[].file_type</code></td>
<td>string</td>
<td><code>"video"</code>, <code>"audio"</code>, or <code>null</code></td>
</tr>
<tr>
<td><code>matches[].status</code></td>
<td>string</td>
<td>Registration/processing status</td>
</tr>
<tr>
<td><code>next_name</code></td>
<td>string</td>
<td>Suggested name for avoiding conflicts</td>
</tr>
</tbody>
</table>
<hr />
<h2>Unregister</h2>
<h3><code>POST /api/v1/unregister</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Delete a registered file from the system. Supports single file by UUID, or batch by directory + regex pattern.</p>
<h4>What gets deleted</h4>
<table class="table">
<thead>
<tr>
<th>Removed (default)</th>
<th>Not removed</th>
</tr>
</thead>
<tbody>
<tr>
<td>Database records (videos, chunks, embeddings, processor_results, pre_chunks)</td>
<td>The original source video file on disk</td>
</tr>
<tr>
<td>Processor output JSON files (<code>{uuid}.*.json</code>) — unless <code>delete_output_files: false</code></td>
<td>Temp/working directories</td>
</tr>
<tr>
<td>In-memory cache entries</td>
<td></td>
</tr>
<tr>
<td>MongoDB cached lists</td>
<td></td>
</tr>
</tbody>
</table>
<blockquote>
<p>⚠️ Database deletion is <strong>irreversible</strong>. To keep output files, set <code>"delete_output_files": false</code>.</p>
</blockquote>
<h4>Request Parameters</h4>
<p>At least one mode must be specified: either <code>file_uuid</code> alone, or <code>file_path</code> + <code>pattern</code> together.</p>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>*</td>
<td>—</td>
<td>Single file UUID to delete</td>
</tr>
<tr>
<td><code>file_path</code></td>
<td>string</td>
<td>*</td>
<td>—</td>
<td>Directory path (for batch delete)</td>
</tr>
<tr>
<td><code>pattern</code></td>
<td>string</td>
<td>*</td>
<td>—</td>
<td>Regex pattern (requires <code>file_path</code>)</td>
</tr>
<tr>
<td><code>delete_output_files</code></td>
<td>boolean</td>
<td>No</td>
<td><code>true</code></td>
<td>If <code>true</code>, also delete processor output JSON files (<code>{uuid}.*.json</code>). Set to <code>false</code> to keep them.</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code><span class="c1"># Delete a single file by UUID (default: also deletes output JSON files)</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/unregister&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_uuid&quot;: &quot;&#39;</span><span class="s2">&quot;</span><span class="nv">$FILE_UUID</span><span class="s2">&quot;</span><span class="s1">&#39;&quot;}&#39;</span>
<span class="c1"># Keep output JSON files, only delete DB records</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/unregister&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_uuid&quot;: &quot;&#39;</span><span class="s2">&quot;</span><span class="nv">$FILE_UUID</span><span class="s2">&quot;</span><span class="s1">&#39;&quot;, &quot;delete_output_files&quot;: false}&#39;</span>
<span class="c1"># Batch delete all mp4 files in a directory</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/unregister&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_path&quot;: &quot;/path/to/dir&quot;, &quot;pattern&quot;: &quot;.*\\.mp4$&quot;}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;a03485a40b2df2d3&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Video unregistered successfully&quot;</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>success</code></td>
<td>boolean</td>
<td>True if deletion succeeded</td>
</tr>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>UUID of the deleted file (single mode)</td>
</tr>
<tr>
<td><code>message</code></td>
<td>string</td>
<td>Human-readable status</td>
</tr>
</tbody>
</table>
<h4>Error Responses</h4>
<table class="table">
<thead>
<tr>
<th>HTTP</th>
<th>When</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>400</code></td>
<td>Neither <code>file_uuid</code> nor <code>file_path</code>+<code>pattern</code> provided</td>
</tr>
<tr>
<td><code>404</code></td>
<td>File UUID not found</td>
</tr>
<tr>
<td><code>401</code></td>
<td>Missing or invalid API key</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/05_process.html
+505
View File
@@ -0,0 +1,505 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>05 Process - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: process -->
<!-- description: Processing pipeline — trigger, probe, progress, jobs -->
<!-- depends: 01_auth, 03_register -->
<h2>Processing Pipeline</h2>
<h3><code>POST /api/v1/file/:file_uuid/process</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Trigger the processing pipeline for a registered file. Creates a monitor job that the worker picks up and processes sequentially. Returns immediately with the job info—processing runs asynchronously in the background.</p>
<h4>Request Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>processors</code></td>
<td>string[]</td>
<td>No</td>
<td>all</td>
<td>Specific processors to run: <code>["cut","asr","asrx","yolo","ocr","face","pose","visual_chunk","story","5w1h"]</code></td>
</tr>
<tr>
<td><code>rules</code></td>
<td>string[]</td>
<td>No</td>
<td>all</td>
<td>Rule names to apply (currently unused)</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code><span class="c1"># Run all processors</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/file/</span><span class="nv">$FILE_UUID</span><span class="s2">/process&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{}&#39;</span>
<span class="c1"># Run specific processors only</span>
curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/file/</span><span class="nv">$FILE_UUID</span><span class="s2">/process&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;processors&quot;: [&quot;asr&quot;, &quot;face&quot;, &quot;yolo&quot;]}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;job_id&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">42</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;3a6c1865...&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;processing&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;pids&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="mi">12345</span><span class="p">,</span><span class="w"> </span><span class="mi">12346</span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Processing triggered for video.mp4&quot;</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>success</code></td>
<td>boolean</td>
<td>Always true on 200</td>
</tr>
<tr>
<td><code>job_id</code></td>
<td>integer</td>
<td>Monitor job ID (for job tracking)</td>
</tr>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>32-char hex UUID of the file</td>
</tr>
<tr>
<td><code>status</code></td>
<td>string</td>
<td><code>"processing"</code></td>
</tr>
<tr>
<td><code>pids</code></td>
<td>integer[]</td>
<td>Process IDs of started processors</td>
</tr>
<tr>
<td><code>message</code></td>
<td>string</td>
<td>Human-readable status</td>
</tr>
</tbody>
</table>
<h4>Error Responses</h4>
<table class="table">
<thead>
<tr>
<th>HTTP</th>
<th>When</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>404</code></td>
<td>File UUID not found</td>
</tr>
<tr>
<td><code>401</code></td>
<td>Missing or invalid API key</td>
</tr>
</tbody>
</table>
<hr />
<h3><code>GET /api/v1/file/:file_uuid/probe</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Get ffprobe metadata for a registered file. Returns video/audio stream info, codec details, duration, resolution, and frame rate.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/file/</span><span class="nv">$FILE_UUID</span><span class="s2">/probe&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;3a6c1865...&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;video.mp4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_size&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">794863677</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;duration&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">120.5</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;width&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1920</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;height&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1080</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;fps&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">24.0</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;total_frames&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">2892</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;cached&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;format&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;filename&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;/path/to/video.mp4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;format_name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;mov,mp4,m4a,3gp&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;duration&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;120.5&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;size&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;12345678&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;bit_rate&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;819200&quot;</span>
<span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="nt">&quot;streams&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;index&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;codec_name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;h264&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;codec_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;video&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;width&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1920</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;height&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1080</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;r_frame_rate&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;24/1&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;duration&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;120.5&quot;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>32-char hex UUID</td>
</tr>
<tr>
<td><code>file_name</code></td>
<td>string</td>
<td>File name</td>
</tr>
<tr>
<td><code>file_size</code></td>
<td>integer</td>
<td>File size in bytes (from filesystem)</td>
</tr>
<tr>
<td><code>duration</code></td>
<td>float</td>
<td>Duration in seconds</td>
</tr>
<tr>
<td><code>width</code></td>
<td>integer</td>
<td>Video width in pixels</td>
</tr>
<tr>
<td><code>height</code></td>
<td>integer</td>
<td>Video height in pixels</td>
</tr>
<tr>
<td><code>fps</code></td>
<td>float</td>
<td>Frames per second</td>
</tr>
<tr>
<td><code>total_frames</code></td>
<td>integer</td>
<td>Estimated total frames</td>
</tr>
<tr>
<td><code>cached</code></td>
<td>boolean</td>
<td>True if result was from cached probe JSON</td>
</tr>
<tr>
<td><code>format</code></td>
<td>object</td>
<td>Container format info (ffprobe format section)</td>
</tr>
<tr>
<td><code>streams</code></td>
<td>array</td>
<td>Array of stream info objects</td>
</tr>
</tbody>
</table>
<hr />
<h3><code>GET /api/v1/progress/:file_uuid</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Get real-time processing progress for a file via Redis pub/sub. Includes per-processor status, current/total frames, ETA, and system resource stats.</p>
<h4>Pipeline Order</h4>
<table class="table">
<thead>
<tr>
<th>Order</th>
<th>Processor</th>
<th>Dependencies</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td><code>cut</code></td>
<td>—</td>
<td>Scene detection</td>
</tr>
<tr>
<td>2</td>
<td><code>asr</code></td>
<td>cut</td>
<td>Speech-to-text (per scene)</td>
</tr>
<tr>
<td>3</td>
<td><code>asrx</code></td>
<td>asr</td>
<td>Speaker diarization</td>
</tr>
<tr>
<td>4</td>
<td><code>yolo</code></td>
<td>—</td>
<td>Object detection</td>
</tr>
<tr>
<td>5</td>
<td><code>ocr</code></td>
<td>—</td>
<td>Text recognition</td>
</tr>
<tr>
<td>6</td>
<td><code>face</code></td>
<td>—</td>
<td>Face detection &amp; embedding</td>
</tr>
<tr>
<td>7</td>
<td><code>pose</code></td>
<td>—</td>
<td>Pose estimation</td>
</tr>
<tr>
<td>8</td>
<td><code>visual_chunk</code></td>
<td>yolo</td>
<td>Visual scene chunks</td>
</tr>
<tr>
<td>9</td>
<td><code>story</code></td>
<td>asr, asrx, cut, yolo, face</td>
<td>Scene summaries (template)</td>
</tr>
<tr>
<td>10</td>
<td><code>5w1h</code></td>
<td>story</td>
<td>5W1H analysis (Gemma4 LLM)</td>
</tr>
</tbody>
</table>
<p>All processors except <code>story</code> and <code>5w1h</code> run concurrently when their dependencies are met. Story and 5W1H run sequentially after their prerequisites.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/progress/</span><span class="nv">$FILE_UUID</span><span class="s2">&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{overall_progress, processors: [.processors[] | {processor_type, status}]}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;3a6c1865...&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;overall_progress&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">71</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;cpu_percent&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">45.2</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;gpu_percent&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">30.1</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;memory_percent&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">62.4</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;processors&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="nt">&quot;processor_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;asr&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;complete&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;progress&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">100</span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="nt">&quot;processor_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;yolo&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;running&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;progress&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">65</span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="nt">&quot;processor_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;face&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;progress&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>32-char hex UUID</td>
</tr>
<tr>
<td><code>overall_progress</code></td>
<td>integer</td>
<td>Overall progress percentage (0–100)</td>
</tr>
<tr>
<td><code>processors</code></td>
<td>array</td>
<td>Per-processor status list</td>
</tr>
<tr>
<td><code>processors[].processor_type</code></td>
<td>string</td>
<td>Processor name (<code>asr</code>, <code>cut</code>, <code>yolo</code>, etc.)</td>
</tr>
<tr>
<td><code>processors[].status</code></td>
<td>string</td>
<td><code>"pending"</code>, <code>"running"</code>, <code>"complete"</code>, or <code>"failed"</code></td>
</tr>
<tr>
<td><code>processors[].progress</code></td>
<td>integer</td>
<td>Per-processor progress (0–100)</td>
</tr>
<tr>
<td><code>processors[].eta_seconds</code></td>
<td>integer</td>
<td>Estimated seconds remaining (running processors)</td>
</tr>
<tr>
<td><code>processors[].current</code></td>
<td>integer</td>
<td>Current frame count</td>
</tr>
<tr>
<td><code>processors[].total</code></td>
<td>integer</td>
<td>Total frame count</td>
</tr>
<tr>
<td><code>cpu_percent</code></td>
<td>float</td>
<td>Current CPU usage</td>
</tr>
<tr>
<td><code>gpu_percent</code></td>
<td>float</td>
<td>Current GPU utilization</td>
</tr>
<tr>
<td><code>memory_percent</code></td>
<td>float</td>
<td>Current memory usage</td>
</tr>
</tbody>
</table>
<hr />
<h3><code>GET /api/v1/jobs</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: system-level</p>
<p>List all processing jobs (monitor jobs) in the system. Shows job status, which file each job is processing, and current processor info.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/jobs&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{count, jobs: [.jobs[] | {uuid, status}]}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;jobs&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;id&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">42</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;3a6c1865...&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;running&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;current_processor&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;yolo&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;created_at&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;2026-05-16T12:00:00Z&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;started_at&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;2026-05-16T12:01:00Z&quot;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;count&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;page&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;page_size&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">20</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>jobs</code></td>
<td>array</td>
<td>Array of job info objects</td>
</tr>
<tr>
<td><code>jobs[].id</code></td>
<td>integer</td>
<td>Job ID</td>
</tr>
<tr>
<td><code>jobs[].uuid</code></td>
<td>string</td>
<td>File UUID being processed</td>
</tr>
<tr>
<td><code>jobs[].status</code></td>
<td>string</td>
<td><code>"pending"</code>, <code>"running"</code>, <code>"completed"</code>, <code>"failed"</code></td>
</tr>
<tr>
<td><code>jobs[].current_processor</code></td>
<td>string</td>
<td>Currently active processor, or null</td>
</tr>
<tr>
<td><code>count</code></td>
<td>integer</td>
<td>Total job count</td>
</tr>
<tr>
<td><code>page</code></td>
<td>integer</td>
<td>Current page number</td>
</tr>
<tr>
<td><code>page_size</code></td>
<td>integer</td>
<td>Jobs per page</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/06_search.html
+280
View File
@@ -0,0 +1,280 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>06 Search - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: search -->
<!-- description: Vector search, BM25, smart search, universal search, visual search -->
<!-- depends: 01_auth -->
<h2>Search APIs</h2>
<h3><code>POST /api/v1/search/smart</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Semantic vector search using EmbeddingGemma-300m. Generates a query embedding via EmbeddingGemma (port 11436), then searches pgvector <code>story_parent</code> and <code>llm_parent</code> chunks by cosine similarity.</p>
<h4>Request Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>Yes</td>
<td>—</td>
<td>File UUID to search within</td>
</tr>
<tr>
<td><code>query</code></td>
<td>string</td>
<td>Yes</td>
<td>—</td>
<td>Search text</td>
</tr>
<tr>
<td><code>limit</code></td>
<td>integer</td>
<td>No</td>
<td>5</td>
<td>Max results to return</td>
</tr>
<tr>
<td><code>page</code></td>
<td>integer</td>
<td>No</td>
<td>1</td>
<td>Page number</td>
</tr>
<tr>
<td><code>page_size</code></td>
<td>integer</td>
<td>No</td>
<td>5</td>
<td>Items per page</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/search/smart&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_uuid&quot;: &quot;&#39;</span><span class="s2">&quot;</span><span class="nv">$FILE_UUID</span><span class="s2">&quot;</span><span class="s1">&#39;&quot;, &quot;query&quot;: &quot;Audrey Hepburn&quot;}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;query&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Audrey Hepburn&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;results&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;parent_id&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1087822</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;scene_order&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1087822</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;start_frame&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">104438</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;end_frame&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">104538</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;fps&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">24.0</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;start_time&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">4351.6</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;end_time&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">4355.76</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;summary&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;[4352s-4356s, 4s] Cast: Audrey Hepburn. Total: 2 lines, 10 words. Speakers: Audrey Hepburn (2 lines)&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;similarity&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">0.67</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;page&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;page_size&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">5</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;strategy&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;semantic_vector_search&quot;</span>
<span class="p">}</span>
</code></pre></div>
<hr />
<h3><code>POST /api/v1/search/universal</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Multi-type BM25 full-text search across chunks, frames, and persons. Uses PostgreSQL <code>tsvector</code>.</p>
<h4>Request Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>query</code></td>
<td>string</td>
<td>Yes</td>
<td>—</td>
<td>Search text</td>
</tr>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>No</td>
<td>—</td>
<td>Restrict to specific file</td>
</tr>
<tr>
<td><code>types</code></td>
<td>string[]</td>
<td>No</td>
<td><code>["chunk","frame","person"]</code></td>
<td>Search types</td>
</tr>
<tr>
<td><code>limit</code></td>
<td>integer</td>
<td>No</td>
<td>10</td>
<td>Max results per type</td>
</tr>
<tr>
<td><code>page</code></td>
<td>integer</td>
<td>No</td>
<td>1</td>
<td>Page number</td>
</tr>
<tr>
<td><code>page_size</code></td>
<td>integer</td>
<td>No</td>
<td>20</td>
<td>Items per page</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/search/universal&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_uuid&quot;: &quot;&#39;</span><span class="s2">&quot;</span><span class="nv">$FILE_UUID</span><span class="s2">&quot;</span><span class="s1">&#39;&quot;, &quot;query&quot;: &quot;Cary Grant&quot;}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;results&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;chunk&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;chunk_id&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;bd80fec92b0b6963d177a2c55bf713e2_2&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;chunk_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;story_child&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;start_frame&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">5103</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;end_frame&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">5127</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;start_time&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">212.64</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;end_time&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">213.64</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;text&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;[213s-214s] Cary Grant: \&quot;Olá!\&quot;&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;score&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">0.9</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;total&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">20</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;took_ms&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">18</span>
<span class="p">}</span>
</code></pre></div>
<hr />
<h3><code>POST /api/v1/search/frames</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Search face detection frames by identity name or trace ID.</p>
<hr />
<h3><code>POST /api/v1/search/identity_text</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Search text chunks spoken by a specific identity.</p>
<hr />
<h3>Visual Search</h3>
<table class="table">
<thead>
<tr>
<th>Method</th>
<th>Endpoint</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>POST</td>
<td><code>/api/v1/search/visual</code></td>
<td>Search visual chunks</td>
</tr>
<tr>
<td>POST</td>
<td><code>/api/v1/search/visual/class</code></td>
<td>Search by object class</td>
</tr>
<tr>
<td>POST</td>
<td><code>/api/v1/search/visual/density</code></td>
<td>Search by object density</td>
</tr>
<tr>
<td>POST</td>
<td><code>/api/v1/search/visual/combination</code></td>
<td>Search by object combination</td>
</tr>
<tr>
<td>POST</td>
<td><code>/api/v1/search/visual/stats</code></td>
<td>Visual chunk statistics</td>
</tr>
</tbody>
</table>
<h4>Embedding Model</h4>
<table class="table">
<thead>
<tr>
<th>Detail</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Model</strong></td>
<td>EmbeddingGemma-300m</td>
</tr>
<tr>
<td><strong>Endpoint</strong></td>
<td><code>POST /api/v1/embeddings</code> on port 11436</td>
</tr>
<tr>
<td><strong>Dimension</strong></td>
<td>768</td>
</tr>
<tr>
<td><strong>Storage</strong></td>
<td>pgvector (<code>chunk.embedding</code> column)</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/07_identity.html
+510
View File
@@ -0,0 +1,510 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>07 Identity - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: identity -->
<!-- description: Global identities — CRUD, detail, files, faces, bind, unbind, search -->
<!-- depends: 01_auth -->
<h2>Global Identities</h2>
<h3><code>GET /api/v1/identities</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>List all registered identities with pagination.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identities?page=1&amp;page_size=20&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{count, identities: [.identities[] | {name}]}&#39;</span>
</code></pre></div>
<hr />
<h3><code>GET /api/v1/identity/:identity_uuid</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Get detailed information for a specific identity, including metadata and TMDb references.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identity/</span><span class="nv">$IDENTITY_UUID</span><span class="s2">&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;identity_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;a9a901056d6b46ff92da0c3c1a57dff4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Cary Grant&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;identity_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;people&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;source&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;tmdb&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;confirmed&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;tmdb_id&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">112</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;tmdb_profile&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;{output}/identities/{identity_uuid}/profile.jpg&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;metadata&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{},</span>
<span class="w"> </span><span class="nt">&quot;reference_data&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{},</span>
<span class="w"> </span><span class="nt">&quot;created_at&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;2026-05-16T12:00:00Z&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;updated_at&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>identity_uuid</code></td>
<td>string</td>
<td>Identity identifier</td>
</tr>
<tr>
<td><code>name</code></td>
<td>string</td>
<td>Identity name</td>
</tr>
<tr>
<td><code>identity_type</code></td>
<td>string</td>
<td><code>"people"</code> or null</td>
</tr>
<tr>
<td><code>source</code></td>
<td>string</td>
<td><code>.json</code>, <code>auto</code>, <code>tmdb</code>, <code>user_defined</code>, or <code>merged</code></td>
</tr>
<tr>
<td><code>status</code></td>
<td>string</td>
<td><code>"confirmed"</code>, <code>"pending"</code>, or <code>"inactive"</code></td>
</tr>
<tr>
<td><code>tmdb_id</code></td>
<td>integer</td>
<td>TMDb person ID (only if source = tmdb)</td>
</tr>
<tr>
<td><code>tmdb_profile</code></td>
<td>string</td>
<td>Local profile image path (<code>{output}/identities/{uuid}/profile.jpg</code>)</td>
</tr>
<tr>
<td><code>metadata</code></td>
<td>object</td>
<td>Metadata JSON (tmdb_character, cast_order, etc.)</td>
</tr>
<tr>
<td><code>created_at</code></td>
<td>string</td>
<td>Creation timestamp</td>
</tr>
</tbody>
</table>
<hr />
<h3><code>DELETE /api/v1/identity/:identity_uuid</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Delete an identity permanently.</p>
<hr />
<h3><code>GET /api/v1/identity/:identity_uuid/files</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Get all files where this identity appears. Returns per-file summary including face count, confidence, and appearance time range.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identity/</span><span class="nv">$IDENTITY_UUID</span><span class="s2">/files&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span>
</code></pre></div>
<hr />
<h3><code>GET /api/v1/identity/:identity_uuid/faces</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Get all face detection records associated with this identity.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identity/</span><span class="nv">$IDENTITY_UUID</span><span class="s2">/faces&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>File where face was detected</td>
</tr>
<tr>
<td><code>frame_number</code></td>
<td>integer</td>
<td>Frame number of detection</td>
</tr>
<tr>
<td><code>face_id</code></td>
<td>string</td>
<td>Face ID (format: <code>face_{frame_number}</code>)</td>
</tr>
<tr>
<td><code>confidence</code></td>
<td>float</td>
<td>Detection confidence</td>
</tr>
</tbody>
</table>
<hr />
<h3><code>GET /api/v1/identity/:identity_uuid/chunks</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Get all text chunks (sentences) spoken while this identity's face was on screen. Useful for finding what a person said.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identity/</span><span class="nv">$IDENTITY_UUID</span><span class="s2">/chunks&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;identity_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;a9a901056d6b46ff92da0c3c1a57dff4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;data&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;id&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">0</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;bd80fec92b0b6963d177a2c55bf713e2&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;chunk_id&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;bd80fec92b0b6963d177a2c55bf713e2_2&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;chunk_type&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;sentence&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;start_frame&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">5103</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;end_frame&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">5127</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;fps&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">24.0</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;start_time&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">212.64</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;end_time&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">213.64</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;text_content&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;[213s-214s] Cary Grant: \&quot;Olá!\&quot;&quot;</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>File identifier</td>
</tr>
<tr>
<td><code>chunk_id</code></td>
<td>string</td>
<td>Sentence chunk identifier</td>
</tr>
<tr>
<td><code>start_frame</code></td>
<td>integer</td>
<td>Frame-accurate start position</td>
</tr>
<tr>
<td><code>end_frame</code></td>
<td>integer</td>
<td>Frame-accurate end position</td>
</tr>
<tr>
<td><code>fps</code></td>
<td>float</td>
<td>Frames per second</td>
</tr>
<tr>
<td><code>start_time</code></td>
<td>float</td>
<td>Start time in seconds</td>
</tr>
<tr>
<td><code>end_time</code></td>
<td>float</td>
<td>End time in seconds</td>
</tr>
<tr>
<td><code>text_content</code></td>
<td>string</td>
<td>Spoken text content</td>
</tr>
</tbody>
</table>
<hr />
<h3><code>POST /api/v1/identity/:identity_uuid/bind</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Bind a face detection to an identity. Associates the face trace with the identity for future search and recognition.</p>
<h4>Request Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>Yes</td>
<td>File where face is detected</td>
</tr>
<tr>
<td><code>face_id</code></td>
<td>string</td>
<td>Yes</td>
<td>Face ID (format: <code>{frame}_{idx}</code>)</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identity/</span><span class="nv">$IDENTITY_UUID</span><span class="s2">/bind&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_uuid&quot;: &quot;&#39;</span><span class="s2">&quot;</span><span class="nv">$FILE_UUID</span><span class="s2">&quot;</span><span class="s1">&#39;&quot;, &quot;face_id&quot;: &quot;1_5&quot;}&#39;</span>
</code></pre></div>
<hr />
<h3><code>POST /api/v1/identity/:identity_uuid/unbind</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Unbind a face detection from an identity. Removes the identity association from the face record.</p>
<hr />
<h3><code>GET /api/v1/identities/search</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Search identities by name (ILIKE search). Returns matching identity records.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identities/search?q=Cary&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>name</code></td>
<td>string</td>
<td>Identity name</td>
</tr>
<tr>
<td><code>source</code></td>
<td>string</td>
<td>Identity source</td>
</tr>
<tr>
<td><code>tmdb_id</code></td>
<td>integer</td>
<td>TMDb ID (if source = tmdb)</td>
</tr>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>Associated file</td>
</tr>
</tbody>
</table>
<hr />
<hr />
<h3><code>POST /api/v1/identity/upload</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Upload an identity.json file to create or update an identity. Accepts the same format as the identity.json files stored on disk.</p>
<p>If an identity with the same <code>name</code> already exists, it will be updated with the new values.</p>
<h4>Request</h4>
<p>The request body is an <code>IdentityFile</code> object:</p>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>identity_uuid</code></td>
<td>string</td>
<td>Yes</td>
<td>Identity identifier</td>
</tr>
<tr>
<td><code>name</code></td>
<td>string</td>
<td>Yes</td>
<td>Identity display name</td>
</tr>
<tr>
<td><code>identity_type</code></td>
<td>string</td>
<td>No</td>
<td><code>"people"</code> or null</td>
</tr>
<tr>
<td><code>source</code></td>
<td>string</td>
<td>No</td>
<td><code>.json</code>, <code>auto</code>, <code>tmdb</code>, <code>user_defined</code>, or <code>merged</code></td>
</tr>
<tr>
<td><code>status</code></td>
<td>string</td>
<td>No</td>
<td><code>"confirmed"</code>, <code>"pending"</code>, or <code>"inactive"</code></td>
</tr>
<tr>
<td><code>tmdb_id</code></td>
<td>integer</td>
<td>No</td>
<td>TMDb person ID</td>
</tr>
<tr>
<td><code>tmdb_profile</code></td>
<td>string</td>
<td>No</td>
<td>TMDb profile image URL</td>
</tr>
<tr>
<td><code>metadata</code></td>
<td>object</td>
<td>No</td>
<td>Arbitrary metadata JSON</td>
</tr>
<tr>
<td><code>file_bindings</code></td>
<td>array</td>
<td>No</td>
<td>Array of <code>{ file_uuid, trace_ids, face_count }</code> (informational)</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identity/upload&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{</span>
<span class="s1"> &quot;version&quot;: 1,</span>
<span class="s1"> &quot;identity_uuid&quot;: &quot;a9a901056d6b46ff92da0c3c1a57dff4&quot;,</span>
<span class="s1"> &quot;name&quot;: &quot;Cary Grant&quot;,</span>
<span class="s1"> &quot;identity_type&quot;: &quot;people&quot;,</span>
<span class="s1"> &quot;source&quot;: &quot;.json&quot;,</span>
<span class="s1"> &quot;status&quot;: &quot;confirmed&quot;,</span>
<span class="s1"> &quot;metadata&quot;: {},</span>
<span class="s1"> &quot;file_bindings&quot;: []</span>
<span class="s1"> }&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;identity_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;a9a901056d6b46ff92da0c3c1a57dff4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Cary Grant&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Identity uploaded successfully&quot;</span>
<span class="p">}</span>
</code></pre></div>
<hr />
<hr />
<h3><code>POST /api/v1/identity/:identity_uuid/profile-image</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Upload a profile image (JPEG or PNG) for an identity. The image is saved to <code>{output}/identities/{uuid}/profile.{ext}</code>.</p>
<p>Uses <code>multipart/form-data</code> with field name <code>image</code>.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identity/</span><span class="nv">$IDENTITY_UUID</span><span class="s2">/profile-image&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-F<span class="w"> </span><span class="s2">&quot;image=@/path/to/photo.jpg&quot;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;identity_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;a9a901056d6b46ff92da0c3c1a57dff4&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;path&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;/path/to/output/identities/.../profile.jpg&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Profile image saved: profile.jpg&quot;</span>
<span class="p">}</span>
</code></pre></div>
<h4>Error Responses</h4>
<table class="table">
<thead>
<tr>
<th>HTTP</th>
<th>When</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>400</code></td>
<td>Missing image field or unsupported format</td>
</tr>
<tr>
<td><code>404</code></td>
<td>Identity not found</td>
</tr>
<tr>
<td><code>415</code></td>
<td>Unsupported image type (use JPEG or PNG)</td>
</tr>
</tbody>
</table>
<hr />
<h3><code>GET /api/v1/identity/:identity_uuid/profile-image</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: identity-level</p>
<p>Retrieve the profile image for an identity. Returns the raw image data with appropriate Content-Type header.</p>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/identity/</span><span class="nv">$IDENTITY_UUID</span><span class="s2">/profile-image&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span>-o<span class="w"> </span>profile.jpg
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Response Header</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>content-type</code></td>
<td><code>image/jpeg</code> or <code>image/png</code></td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/08_identity_agent.html
+97
View File
@@ -0,0 +1,97 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>08 Identity Agent - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: identity_agent -->
<!-- description: Identity agent — match from photo, match from trace -->
<!-- depends: 01_auth, 07_identity -->
<h2>Identity Agent</h2>
<h3><code>POST /api/v1/agents/identity/match-from-photo</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Upload a face photo to match against known identities. Detects face via InsightFace, extracts 512D embedding via CoreML FaceNet, then searches pgvector for the closest identity.</p>
<h4>Request</h4>
<p><code>multipart/form-data</code> with field <code>image</code> (JPEG/PNG) and optional <code>file_uuid</code>.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/agents/identity/match-from-photo&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-F<span class="w"> </span><span class="s2">&quot;image=@/path/to/face.jpg&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-F<span class="w"> </span><span class="s2">&quot;file_uuid=</span><span class="nv">$FILE_UUID</span><span class="s2">&quot;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;matches&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;identity_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;a9a90105...&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Cary Grant&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;similarity&quot;</span><span class="p">:</span><span class="w"> </span><span class="mf">0.87</span>
<span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<hr />
<h3><code>POST /api/v1/agents/identity/match-from-trace</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Match a face trace (tracked face across frames) against known identities. Samples 3 angles from the trace, generates embeddings, and searches pgvector.</p>
<h4>Request Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>Yes</td>
<td>File containing the trace</td>
</tr>
<tr>
<td><code>trace_id</code></td>
<td>integer</td>
<td>Yes</td>
<td>Face trace ID to match</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/agents/identity/match-from-trace&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_uuid&quot;: &quot;&#39;</span><span class="s2">&quot;</span><span class="nv">$FILE_UUID</span><span class="s2">&quot;</span><span class="s1">&#39;&quot;, &quot;trace_id&quot;: 10}&#39;</span>
</code></pre></div>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/08_media.html
+303
View File
@@ -0,0 +1,303 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>08 Media - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: media -->
<!-- description: Video streaming & frame extraction -->
<!-- depends: 01_auth -->
<h2>Video Streaming &amp; Frame Extraction</h2>
<p>All video streaming endpoints support the following common query parameters:</p>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>mode</code></td>
<td>string</td>
<td>No</td>
<td><code>normal</code></td>
<td><code>normal</code> or <code>debug</code> (draws detection overlays)</td>
</tr>
<tr>
<td><code>audio</code></td>
<td>string</td>
<td>No</td>
<td><code>on</code></td>
<td><code>on</code> or <code>off</code></td>
</tr>
</tbody>
</table>
<hr />
<h3><code>GET /api/v1/file/:file_uuid/video</code></h3>
<p>Stream the full video file with range support for seeking.</p>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<h4>Response</h4>
<ul>
<li><strong>200</strong>: Video stream (<code>Content-Type</code> based on file extension)</li>
<li><strong>206</strong>: Partial content (range request)</li>
<li>Supports <code>Range</code> header for seeking</li>
</ul>
<hr />
<h3><code>GET /api/v1/file/:file_uuid/trace/:trace_id/video</code></h3>
<p>Stream video with highlights for a specific face trace (follows a single person across frames with bounding box overlay).</p>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<hr />
<h3><code>GET /api/v1/file/:file_uuid/video/bbox</code></h3>
<p>Stream video with bounding box overlay for all detected objects/faces.</p>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Uses a built-in 5×7 bitmap font renderer to draw labels directly on video frames via FFmpeg <code>drawtext</code> filter.</p>
<hr />
<h3><code>GET /api/v1/file/:file_uuid/thumbnail</code></h3>
<p>Extract a single frame from a video as JPEG image. Uses FFmpeg <code>select</code> filter.</p>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<h4>Query Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>frame</code></td>
<td>integer</td>
<td>Yes</td>
<td>—</td>
<td>Zero-based frame number to extract</td>
</tr>
<tr>
<td><code>x</code></td>
<td>integer</td>
<td>No</td>
<td>—</td>
<td>Crop start X (left edge). Requires <code>y</code>, <code>w</code>, <code>h</code>.</td>
</tr>
<tr>
<td><code>y</code></td>
<td>integer</td>
<td>No</td>
<td>—</td>
<td>Crop start Y (top edge). Requires <code>x</code>, <code>w</code>, <code>h</code>.</td>
</tr>
<tr>
<td><code>w</code></td>
<td>integer</td>
<td>No</td>
<td>—</td>
<td>Crop width in pixels. Requires <code>x</code>, <code>y</code>, <code>h</code>.</td>
</tr>
<tr>
<td><code>h</code></td>
<td>integer</td>
<td>No</td>
<td>—</td>
<td>Crop height in pixels. Requires <code>x</code>, <code>y</code>, <code>w</code>.</td>
</tr>
</tbody>
</table>
<p>All four crop params (<code>x</code>, <code>y</code>, <code>w</code>, <code>h</code>) must be provided together or omitted.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code><span class="c1"># Extract frame 1000 (full frame)</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/file/bd80fec92b0b6963d177a2c55bf713e2/thumbnail?frame=1000&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span>-o<span class="w"> </span>frame_1000.jpg
<span class="c1"># Extract and crop face region (x=320, y=240, w=160, h=160)</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/file/bd80fec92b0b6963d177a2c55bf713e2/thumbnail?frame=1000&amp;x=320&amp;y=240&amp;w=160&amp;h=160&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span>-o<span class="w"> </span>face_crop.jpg
</code></pre></div>
<h4>Response</h4>
<ul>
<li><strong>200</strong>: <code>image/jpeg</code> binary data</li>
<li><strong>404</strong>: File not found</li>
<li><strong>500</strong>: FFmpeg error (e.g., frame number exceeds video duration)</li>
</ul>
<h3><code>GET /api/v1/file/:file_uuid/clip</code></h3>
<p>Extract a video clip (time range) as MPEG-TS stream. Uses FFmpeg <code>-ss</code> fast seek.</p>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<h4>Query Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Default</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>start_frame</code></td>
<td>integer</td>
<td>No*</td>
<td>—</td>
<td>Start frame (zero-based). <strong>Frame-accurate</strong> — use this for precision.</td>
</tr>
<tr>
<td><code>end_frame</code></td>
<td>integer</td>
<td>No*</td>
<td>—</td>
<td>End frame (zero-based, inclusive). Requires <code>start_frame</code>.</td>
</tr>
<tr>
<td><code>start_time</code></td>
<td>float</td>
<td>No*</td>
<td>—</td>
<td>Start time in seconds. Approximate (FPS-dependent). Fallback if frames not given.</td>
</tr>
<tr>
<td><code>end_time</code></td>
<td>float</td>
<td>No*</td>
<td>—</td>
<td>End time in seconds. Approximate (FPS-dependent). Fallback if frames not given.</td>
</tr>
<tr>
<td><code>fps</code></td>
<td>float</td>
<td>No</td>
<td>video FPS</td>
<td>Override frames-per-second for frame↔time calculation. Defaults to video's detected FPS.</td>
</tr>
<tr>
<td><code>mode</code></td>
<td>string</td>
<td>No</td>
<td><code>normal</code></td>
<td><code>normal</code> or <code>debug</code> (draws "CLIP" overlay)</td>
</tr>
<tr>
<td><code>audio</code></td>
<td>string</td>
<td>No</td>
<td><code>on</code></td>
<td><code>on</code> or <code>off</code></td>
</tr>
</tbody>
</table>
<p>Either (<code>start_frame</code>+<code>end_frame</code>) OR (<code>start_time</code>+<code>end_time</code>) must be provided.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code><span class="c1"># Clip by frame range (primary)</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/file/bd80fec92b0b6963d177a2c55bf713e2/clip?start_frame=0&amp;end_frame=47&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span>-o<span class="w"> </span>clip.ts
<span class="c1"># Clip by time range (fallback)</span>
curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/file/bd80fec92b0b6963d177a2c55bf713e2/clip?start_time=30&amp;end_time=45&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Authorization: Bearer </span><span class="nv">$JWT</span><span class="s2">&quot;</span><span class="w"> </span>-o<span class="w"> </span>clip.ts
</code></pre></div>
<h4>Response</h4>
<ul>
<li><strong>200</strong>: <code>video/mp2t</code> MPEG-TS stream</li>
<li><strong>400</strong>: Missing/invalid range parameters</li>
<li><strong>404</strong>: File not found</li>
<li><strong>500</strong>: FFmpeg error</li>
</ul>
<h4>Technical Notes</h4>
<table class="table">
<thead>
<tr>
<th>Detail</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Backend</strong></td>
<td>FFmpeg (<code>ffmpeg-full</code>)</td>
</tr>
<tr>
<td><strong>Seek</strong></td>
<td><code>-ss</code> before <code>-i</code> (fast keyframe seek)</td>
</tr>
<tr>
<td><strong>Format</strong></td>
<td>MPEG-TS (<code>mpegts</code> muxer, pipe-safe)</td>
</tr>
<tr>
<td><strong>Codec</strong></td>
<td>H.264 + AAC</td>
</tr>
<tr>
<td><strong>Cache</strong></td>
<td><code>Cache-Control: public, max-age=86400</code> (24h)</td>
</tr>
</tbody>
</table>
<hr />
<table class="table">
<thead>
<tr>
<th>Detail</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Backend</strong></td>
<td>FFmpeg (<code>ffmpeg-full</code>)</td>
</tr>
<tr>
<td><strong>Filter</strong></td>
<td><code>select=eq(n\,FRAME)</code> to select frame, optional <code>crop=W:H:X:Y</code></td>
</tr>
<tr>
<td><strong>Output</strong></td>
<td>Single JPEG via pipe (<code>image2pipe</code>, <code>mjpeg</code> codec)</td>
</tr>
<tr>
<td><strong>Cache</strong></td>
<td><code>Cache-Control: public, max-age=86400</code> (24h)</td>
</tr>
<tr>
<td><strong>Frame number</strong></td>
<td>Zero-based (<code>frame=0</code> = first frame of video)</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/09_tmdb.html
+123
View File
@@ -0,0 +1,123 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>09 Tmdb - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: tmdb -->
<!-- description: TMDb enrichment endpoints — prefetch, probe, resource, check -->
<!-- depends: 01_auth, 03_register -->
<h2>TMDb Enrichment</h2>
<blockquote>
<p><strong>Offline operation</strong>: TMDb prefetch now checks local identity files first (<code>identities/_index.json</code> + <code>*.tmdb.json</code>).
If local files exist, no external API call is made. Internet is only needed for initial data seeding.</p>
</blockquote>
<h3>Overview</h3>
<p>TMDb enrichment is an optional identity enrichment step that can be run after Pipeline face detection completes. The workflow is:</p>
<ol>
<li><strong>Prefetch</strong> (requires internet): Download movie cast data from TMDb API → cache to <code>{file_uuid}.tmdb.json</code></li>
<li><strong>Probe</strong>: Read local cache → create identities for <strong>all</strong> cast members (<code>source='tmdb'</code>) + save <code>identity.json</code> + download profile image to <code>{OUTPUT}/identities/{uuid}/profile.jpg</code></li>
<li><strong>Match</strong>: The worker automatically matches video faces against TMDb identities when <code>MOMENTRY_TMDB_PROBE_ENABLED=true</code></li>
</ol>
<h3><code>POST /api/v1/agents/tmdb/prefetch</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Fetch TMDb cast data for a registered file and cache it locally. This is the only step requiring internet access.</p>
<h4>Request Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>file_uuid</code></td>
<td>string</td>
<td>Yes</td>
<td>File UUID to enrich</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/agents/tmdb/prefetch&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;file_uuid&quot;: &quot;&#39;</span><span class="s2">&quot;</span><span class="nv">$FILE_UUID</span><span class="s2">&quot;</span><span class="s1">&#39;&quot;}&#39;</span>
</code></pre></div>
<h4>Response (200)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;...&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;cache_path&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;/output/...tmdb.json&quot;</span><span class="p">}</span>
</code></pre></div>
<h3><code>POST /api/v1/file/:file_uuid/tmdb-probe</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: file-level</p>
<p>Read local TMDb cache and create/update identities. Requires prefetch to have been run first.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/file/</span><span class="nv">$FILE_UUID</span><span class="s2">/tmdb-probe&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{identities_created, movie_title}&#39;</span>
</code></pre></div>
<h4>Response (200 — identities created)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;identities_created&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">15</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;movie_title&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Charade&quot;</span><span class="p">}</span>
</code></pre></div>
<h4>Response (200 — no cache)</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;No TMDb cache found. Run tmdb-prefetch first.&quot;</span><span class="p">}</span>
</code></pre></div>
<h3><code>GET /api/v1/resource/tmdb</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: system-level</p>
<p>View TMDb resource status including configuration, identity counts, and cache file count.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/resource/tmdb&quot;</span><span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;{identities_seeded, cache_files}&#39;</span>
</code></pre></div>
<h3><code>POST /api/v1/resource/tmdb/check</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: system-level</p>
<p>Ping the TMDb API to verify connectivity and measure latency.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/resource/tmdb/check&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;.status&#39;</span>
</code></pre></div>
<h4>Response</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;api_key_configured&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;enabled&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;api_reachable&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;api_latency_ms&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">120</span>
<span class="p">}</span>
</code></pre></div>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/10_pipeline.html
+364
View File
@@ -0,0 +1,364 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>10 Pipeline - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: pipeline -->
<!-- description: Pipeline processors, ingestion status, stats endpoints -->
<!-- depends: 01_auth -->
<h2>Pipeline</h2>
<h3>Dependency Graph</h3>
<div class="codehilite"><pre><span></span><code><span class="n">flowchart</span><span class="w"> </span><span class="n">TB</span>
<span class="w"> </span><span class="n">subgraph</span><span class="w"> </span><span class="n">Processors</span><span class="p">[</span><span class="s">&quot;10 Processors&quot;</span><span class="p">]</span>
<span class="w"> </span><span class="n">Cut</span><span class="p">[</span><span class="n">Cut</span><span class="p">]</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">ASR</span><span class="p">[</span><span class="n">ASR</span><span class="p">]</span>
<span class="w"> </span><span class="n">ASR</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">ASRX</span><span class="p">[</span><span class="n">ASRX</span><span class="p">]</span>
<span class="w"> </span><span class="n">ASRX</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Story</span><span class="p">[</span><span class="n">Story</span><span class="p">]</span>
<span class="w"> </span><span class="n">Cut</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Story</span>
<span class="w"> </span><span class="n">YOLO</span><span class="p">[</span><span class="n">YOLO</span><span class="p">]</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">VisualChunk</span><span class="p">[</span><span class="n">VisualChunk</span><span class="p">]</span>
<span class="w"> </span><span class="n">VisualChunk</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Story</span>
<span class="w"> </span><span class="n">Face</span><span class="p">[</span><span class="n">Face</span><span class="p">]</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Story</span>
<span class="w"> </span><span class="n">Story</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">FiveW1H</span><span class="p">[</span><span class="mi">5</span><span class="n">W1H</span><span class="p">]</span>
<span class="w"> </span><span class="n">OCR</span><span class="p">[</span><span class="n">OCR</span><span class="p">]</span>
<span class="w"> </span><span class="n">Pose</span><span class="p">[</span><span class="n">Pose</span><span class="p">]</span>
<span class="w"> </span><span class="n">end</span>
<span class="w"> </span><span class="n">subgraph</span><span class="w"> </span><span class="n">Ingestion</span><span class="p">[</span><span class="s">&quot;入庫 (Post-Processing)&quot;</span><span class="p">]</span>
<span class="w"> </span><span class="n">ASR</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Rule1</span><span class="p">[</span><span class="n">Rule</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Sentence</span><span class="p">]</span>
<span class="w"> </span><span class="n">ASRX</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Rule1</span>
<span class="w"> </span><span class="n">Rule1</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Vectorize</span><span class="p">[</span><span class="n">Auto</span><span class="o">-</span><span class="n">Vectorize</span><span class="p">]</span>
<span class="w"> </span><span class="n">Rule1</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Phase1</span><span class="p">[</span><span class="n">Phase</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="n">Pack</span><span class="p">]</span>
<span class="w"> </span><span class="n">Cut</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Rule3</span><span class="p">[</span><span class="n">Rule</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="n">Scene</span><span class="p">]</span>
<span class="w"> </span><span class="n">ASR</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Rule3</span>
<span class="w"> </span><span class="n">Face</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Trace</span><span class="p">[</span><span class="n">Face</span><span class="w"> </span><span class="n">Trace</span><span class="p">]</span>
<span class="w"> </span><span class="n">Trace</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Qdrant</span><span class="p">[</span><span class="n">Qdrant</span><span class="w"> </span><span class="n">Sync</span><span class="p">]</span>
<span class="w"> </span><span class="n">Trace</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">TraceChunks</span><span class="p">[</span><span class="n">Trace</span><span class="w"> </span><span class="n">Chunks</span><span class="p">]</span>
<span class="w"> </span><span class="n">Trace</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">TKG</span><span class="p">[</span><span class="n">TKG</span><span class="w"> </span><span class="n">Builder</span><span class="p">]</span>
<span class="w"> </span><span class="n">Face</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">TMDbMatch</span><span class="p">[</span><span class="n">TMDb</span><span class="w"> </span><span class="n">Match</span><span class="p">]</span>
<span class="w"> </span><span class="n">Face</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">SceneMeta</span><span class="p">[</span><span class="n">Scene</span><span class="w"> </span><span class="n">Metadata</span><span class="p">]</span>
<span class="w"> </span><span class="n">YOLO</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">SceneMeta</span>
<span class="w"> </span><span class="n">Face</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">IdentityAgent</span><span class="p">[</span><span class="n">Identity</span><span class="w"> </span><span class="n">Agent</span><span class="p">]</span>
<span class="w"> </span><span class="n">ASRX</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">IdentityAgent</span>
<span class="w"> </span><span class="n">Cut</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Agent5W1H</span><span class="p">[</span><span class="mi">5</span><span class="n">W1H</span><span class="w"> </span><span class="n">Agent</span><span class="p">]</span>
<span class="w"> </span><span class="n">ASR</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Agent5W1H</span>
<span class="w"> </span><span class="n">Agent5W1H</span><span class="w"> </span><span class="o">--&gt;</span><span class="w"> </span><span class="n">Phase2</span><span class="p">[</span><span class="n">Phase</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="n">Pack</span><span class="p">]</span>
<span class="w"> </span><span class="n">end</span>
<span class="w"> </span><span class="n">style</span><span class="w"> </span><span class="n">Processors</span><span class="w"> </span><span class="n">fill</span><span class="o">:</span><span class="err">#</span><span class="mi">1</span><span class="n">a1a2e</span><span class="p">,</span><span class="n">stroke</span><span class="o">:</span><span class="err">#</span><span class="n">e94560</span>
<span class="w"> </span><span class="n">style</span><span class="w"> </span><span class="n">Ingestion</span><span class="w"> </span><span class="n">fill</span><span class="o">:</span><span class="err">#</span><span class="mi">16213</span><span class="n">e</span><span class="p">,</span><span class="n">stroke</span><span class="o">:</span><span class="err">#</span><span class="mf">0f</span><span class="mi">3460</span>
</code></pre></div>
<h3>Pipeline Completion Flow</h3>
<p>The pipeline is <strong>not complete</strong> until both the 10 processors AND the 入庫 (ingestion) steps have finished. The worker polls every 3 seconds and only marks the job as <code>completed</code> when all ingestion steps verify OK.</p>
<div class="codehilite"><pre><span></span><code><span class="mf">10</span><span class="w"> </span><span class="n">processors</span><span class="w"> </span><span class="n">done</span>
<span class="w"> </span><span class="err">↓</span><span class="w"> </span><span class="p">(</span><span class="n">job</span><span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="n">stays</span><span class="w"> </span><span class="s">&quot;running&quot;</span><span class="p">)</span>
<span class="n">Algorithm</span><span class="w"> </span><span class="mf">1</span><span class="w"> </span><span class="n">Trigger</span><span class="p">:</span><span class="w"> </span><span class="n">Rule</span><span class="w"> </span><span class="mf">1</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Vectorize</span><span class="w"> </span><span class="o">+</span><span class="w"> </span><span class="n">Phase</span><span class="w"> </span><span class="mf">1</span><span class="w"> </span><span class="n">Pack</span>
<span class="w"> </span><span class="err">↓</span><span class="w"> </span><span class="p">(</span><span class="n">job</span><span class="w"> </span><span class="kr">run</span><span class="n">s</span><span class="w"> </span><span class="n">in</span><span class="w"> </span><span class="n">parallel</span><span class="p">)</span>
<span class="n">Algorithm</span><span class="w"> </span><span class="mf">2</span><span class="w"> </span><span class="n">Trigger</span><span class="p">:</span><span class="w"> </span><span class="n">Face</span><span class="w"> </span><span class="n">Trace</span><span class="w"> </span><span class="err">→</span><span class="w"> </span><span class="n">TKG</span><span class="p">,</span><span class="w"> </span><span class="n">Scene</span><span class="w"> </span><span class="n">Metadata</span><span class="p">,</span><span class="w"> </span><span class="n">Identity</span><span class="w"> </span><span class="n">Agent</span><span class="p">,</span><span class="w"> </span><span class="mf">5</span><span class="n">W1H</span><span class="w"> </span><span class="n">Agent</span>
<span class="w"> </span><span class="err">↓</span><span class="w"> </span><span class="p">(</span><span class="n">poll</span><span class="w"> </span><span class="n">checks</span><span class="w"> </span><span class="n">every</span><span class="w"> </span><span class="mf">3</span><span class="n">s</span><span class="p">)</span>
<span class="n">Ingestion</span><span class="w"> </span><span class="n">verification</span><span class="p">:</span><span class="w"> </span><span class="n">rule1</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="n">vectorize</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="n">rule3</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="n">face_trace</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="n">tkg</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="n">scene_meta</span><span class="w"> </span><span class="err">✓</span><span class="w"> </span><span class="mf">5</span><span class="n">w1h</span><span class="w"> </span><span class="err">✓</span>
<span class="w"> </span><span class="err">↓</span>
<span class="n">job</span><span class="w"> </span><span class="n">status</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&quot;completed&quot;</span>
</code></pre></div>
<h3>10 Processor Stages</h3>
<table class="table">
<thead>
<tr>
<th>#</th>
<th>Processor</th>
<th>Depends On</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td><code>Cut</code></td>
<td>—</td>
<td>Scene boundary detection (PySceneDetect)</td>
</tr>
<tr>
<td>2</td>
<td><code>ASR</code></td>
<td>Cut</td>
<td>Automatic speech recognition (faster-whisper)</td>
</tr>
<tr>
<td>3</td>
<td><code>ASRX</code></td>
<td>ASR</td>
<td>Speaker diarization + ASR refinement</td>
</tr>
<tr>
<td>4</td>
<td><code>YOLO</code></td>
<td>—</td>
<td>Object detection (YOLOv8)</td>
</tr>
<tr>
<td>5</td>
<td><code>OCR</code></td>
<td>—</td>
<td>Optical character recognition</td>
</tr>
<tr>
<td>6</td>
<td><code>Face</code></td>
<td>—</td>
<td>Face detection + recognition (InsightFace + CoreML)</td>
</tr>
<tr>
<td>7</td>
<td><code>Pose</code></td>
<td>—</td>
<td>Pose estimation</td>
</tr>
<tr>
<td>8</td>
<td><code>VisualChunk</code></td>
<td>YOLO</td>
<td>Visual object chunking</td>
</tr>
<tr>
<td>9</td>
<td><code>Story</code></td>
<td>ASRX + Cut + YOLO + Face</td>
<td>Narrative scene summarization (LLM, with embedding)</td>
</tr>
<tr>
<td>10</td>
<td><code>5W1H</code></td>
<td>Story</td>
<td>Who/What/When/Where/Why extraction (LLM, with embedding)</td>
</tr>
</tbody>
</table>
<h3>入庫 (Post-Processing / Ingestion)</h3>
<p>These steps run after the 10 processors and are <strong>required for pipeline completion</strong>. The worker checks all of them before marking the job as done.</p>
<table class="table">
<thead>
<tr>
<th>#</th>
<th>Step</th>
<th>Triggers When</th>
<th>Verification</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td><strong>Rule 1 Sentence Chunking</strong></td>
<td>ASR + ASRX done</td>
<td><code>chunk</code> table has rows with <code>chunk_type = 'sentence'</code></td>
</tr>
<tr>
<td>2</td>
<td><strong>Auto-Vectorize</strong></td>
<td>Rule 1 done</td>
<td><code>chunk.embedding</code> IS NOT NULL for sentence chunks</td>
</tr>
<tr>
<td>3</td>
<td><strong>Phase 1 Pack</strong></td>
<td>Rule 1 done</td>
<td><code>release_pack.py --phase 1</code> executed</td>
</tr>
<tr>
<td>4</td>
<td><strong>Rule 3 Scene Chunking</strong></td>
<td>All 10 processors done + Cut + ASR</td>
<td><code>chunk</code> table has rows with <code>chunk_type = 'cut'</code></td>
</tr>
<tr>
<td>5</td>
<td><strong>Face Trace</strong></td>
<td>All 10 processors done + Face</td>
<td><code>face_detections.trace_id</code> IS NOT NULL</td>
</tr>
<tr>
<td>6</td>
<td><strong>Qdrant Face Sync</strong></td>
<td>Face Trace done</td>
<td>Qdrant face_embedding collection populated</td>
</tr>
<tr>
<td>7</td>
<td><strong>Trace Chunks</strong></td>
<td>Face Trace done</td>
<td><code>chunk</code> table has rows with <code>chunk_type = 'trace'</code></td>
</tr>
<tr>
<td>8</td>
<td><strong>TKG Builder</strong></td>
<td>Face Trace done</td>
<td><code>tkg_nodes</code> + <code>tkg_edges</code> tables have rows</td>
</tr>
<tr>
<td>9</td>
<td><strong>TMDb Face Matching</strong></td>
<td>TMDb enabled + Face done</td>
<td><code>face_detections.identity_id</code> IS NOT NULL</td>
</tr>
<tr>
<td>10</td>
<td><strong>Heuristic Scene Metadata</strong></td>
<td>Face + YOLO done</td>
<td><code>{file_uuid}.scene_meta.json</code> exists on disk</td>
</tr>
<tr>
<td>11</td>
<td><strong>Identity Agent</strong></td>
<td>Face + ASRX done</td>
<td><code>identities</code> with <code>source = 'identity_agent'</code></td>
</tr>
<tr>
<td>12</td>
<td><strong>5W1H Agent</strong></td>
<td>Cut + ASR done</td>
<td><code>chunk.summary_text</code> IS NOT NULL for cut chunks</td>
</tr>
<tr>
<td>13</td>
<td><strong>Release Pack</strong></td>
<td>5W1H Agent done</td>
<td><code>release_pack.py --phase 2</code> executed</td>
</tr>
</tbody>
</table>
<h3>Ingestion Status</h3>
<p>Check real-time ingestion status for a file:</p>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/stats/ingestion-status/{file_uuid}&quot;</span>
</code></pre></div>
<p>Returns per-step <code>done</code> / <code>pending</code> status with detail counts.</p>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span><span class="s2">&quot;http://localhost:3003/api/v1/stats/ingestion-status/bd80fec9c42afb0307eb28f22c64c76a&quot;</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>jq<span class="w"> </span><span class="s1">&#39;.steps[] | {name, status, detail}&#39;</span>
</code></pre></div>
<h4>Response</h4>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;bd80fec9c42afb0307eb28f22c64c76a&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;steps&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;rule1_sentence&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;detail&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;0 sentence chunks&quot;</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;auto_vectorize&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;detail&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;0 embedded&quot;</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;rule3_scene&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;detail&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;0 scene chunks&quot;</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;face_trace&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;detail&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;0 traces&quot;</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;trace_chunks&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;detail&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;0 trace chunks&quot;</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;tkg&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;detail&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;0 nodes, 0 edges&quot;</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;identity_match&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;detail&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;0 identities&quot;</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;scene_metadata&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;detail&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="w"> </span><span class="p">},</span>
<span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nt">&quot;name&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;5w1h&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;status&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;pending&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;detail&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;0 scenes with 5W1H&quot;</span><span class="w"> </span><span class="p">}</span>
<span class="w"> </span><span class="p">]</span>
<span class="p">}</span>
</code></pre></div>
<h3>Stats Endpoints</h3>
<table class="table">
<thead>
<tr>
<th>Method</th>
<th>Endpoint</th>
<th>Auth</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>GET</td>
<td><code>/api/v1/stats/sftpgo</code></td>
<td>No</td>
<td>SFTPGo service status</td>
</tr>
<tr>
<td>GET</td>
<td><code>/api/v1/stats/ingestion-status/:file_uuid</code></td>
<td>No</td>
<td>Per-file ingestion checklist</td>
</tr>
</tbody>
</table>
<h3>Configuration</h3>
<h3><code>POST /api/v1/config/cache</code></h3>
<p><strong>Auth</strong>: Required
<strong>Scope</strong>: system-level</p>
<p>Toggle the Redis cache on or off.</p>
<h4>Request Parameters</h4>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>enabled</code></td>
<td>boolean</td>
<td>Yes</td>
<td><code>true</code> to enable, <code>false</code> to disable</td>
</tr>
</tbody>
</table>
<h4>Example</h4>
<div class="codehilite"><pre><span></span><code>curl<span class="w"> </span>-s<span class="w"> </span>-X<span class="w"> </span>POST<span class="w"> </span><span class="s2">&quot;</span><span class="nv">$API</span><span class="s2">/api/v1/config/cache&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;Content-Type: application/json&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-H<span class="w"> </span><span class="s2">&quot;X-API-Key: </span><span class="nv">$KEY</span><span class="s2">&quot;</span><span class="w"> </span><span class="se">\</span>
<span class="w"> </span>-d<span class="w"> </span><span class="s1">&#39;{&quot;enabled&quot;: false}&#39;</span>
</code></pre></div>
<h3>Unmounted Routes</h3>
<p>The following routes are defined in source code but are <strong>NOT</strong> currently mounted in the router:</p>
<table class="table">
<thead>
<tr>
<th>Endpoint</th>
<th>Source file</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>/api/v1/search/persons</code></td>
<td><code>universal_search.rs</code> (not mounted)</td>
</tr>
<tr>
<td><code>/api/v1/who</code></td>
<td><code>who.rs</code></td>
</tr>
<tr>
<td><code>/api/v1/who/candidates</code></td>
<td><code>who.rs</code></td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/12_agent.html
+207
View File
@@ -0,0 +1,207 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>12 Agent - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<h1>Agent Endpoints</h1>
<p>Agent endpoints provide AI-powered capabilities including translation, identity analysis, and 5W1H extraction.</p>
<h2>POST /api/v1/agents/translate</h2>
<p>Translate text between languages using Gemma4 (llama.cpp, port 8082).</p>
<h3>Request</h3>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;text&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Hello, welcome to Momentry Core.&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;target_language&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Traditional Chinese&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;source_language&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;English&quot;</span>
<span class="p">}</span>
</code></pre></div>
<table class="table">
<thead>
<tr>
<th>Field</th>
<th>Type</th>
<th>Required</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>text</code></td>
<td>string</td>
<td>✅</td>
<td>Text to translate</td>
</tr>
<tr>
<td><code>target_language</code></td>
<td>string</td>
<td>✅</td>
<td>Target language name (e.g. "Traditional Chinese", "Japanese")</td>
</tr>
<tr>
<td><code>source_language</code></td>
<td>string</td>
<td>❌</td>
<td>Source language (default: "auto")</td>
</tr>
</tbody>
</table>
<h3>Response</h3>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;translated_text&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;您好，歡迎使用 Momentry Core。&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;source_language_detected&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;English&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;model_used&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;google_gemma-4-26B-A4B-it-Q5_K_M.gguf&quot;</span>
<span class="p">}</span>
</code></pre></div>
<h3>Supported Language Pairs (tested)</h3>
<table class="table">
<thead>
<tr>
<th>Source</th>
<th>Target</th>
<th>Quality</th>
</tr>
</thead>
<tbody>
<tr>
<td>English</td>
<td>Traditional Chinese</td>
<td>✅</td>
</tr>
<tr>
<td>English</td>
<td>Japanese</td>
<td>✅</td>
</tr>
<tr>
<td>Chinese</td>
<td>English</td>
<td>✅</td>
</tr>
<tr>
<td>English</td>
<td>French</td>
<td>✅</td>
</tr>
<tr>
<td>Chinese</td>
<td>Japanese</td>
<td>✅</td>
</tr>
</tbody>
</table>
<h3>Model</h3>
<ul>
<li><strong>Model</strong>: Gemma4 26B (Q5_K_M)</li>
<li><strong>Engine</strong>: llama.cpp at <code>localhost:8082</code></li>
<li><strong>Endpoint</strong>: <code>/v1/chat/completions</code> (OpenAI-compatible)</li>
<li><strong>Temperature</strong>: 0.1</li>
<li><strong>Max tokens</strong>: 1024</li>
</ul>
<h3>Errors</h3>
<table class="table">
<thead>
<tr>
<th>Status</th>
<th>Condition</th>
</tr>
</thead>
<tbody>
<tr>
<td>500</td>
<td>LLM unreachable or response parse failure</td>
</tr>
<tr>
<td>401</td>
<td>Missing/invalid auth</td>
</tr>
</tbody>
</table>
<hr />
<h2>POST /api/v1/agents/5w1h/analyze</h2>
<p>Extract 5W1H (Who, What, When, Where, Why, How) from a scene. Uses Gemma4 LLM on port 8082.</p>
<h3>Request</h3>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;3abeee81d94597629ed8cb943f182e94&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;scene_id&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">42</span>
<span class="p">}</span>
</code></pre></div>
<h3>Response</h3>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;5w1h&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;who&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&quot;Cary Grant&quot;</span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;what&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&quot;discussing plans&quot;</span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;when&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&quot;1963&quot;</span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;where&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&quot;Paris&quot;</span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;why&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&quot;vacation&quot;</span><span class="p">],</span>
<span class="w"> </span><span class="nt">&quot;how&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">&quot;in person&quot;</span><span class="p">]</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h2>POST /api/v1/agents/5w1h/batch</h2>
<p>Batch analyze all scenes in a file for 5W1H extraction. Uses the pipeline's <code>parent_chunk_5w1h.py --mode llm</code>.</p>
<h3>Request</h3>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;file_uuid&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;3abeee81d94597629ed8cb943f182e94&quot;</span>
<span class="p">}</span>
</code></pre></div>
<h2>GET /api/v1/agents/5w1h/status</h2>
<p>Get status of the 5W1H agent pipeline for a file.</p>
<hr />
<h2>Embedding Model</h2>
<table class="table">
<thead>
<tr>
<th>Detail</th>
<th>Value</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>Model</strong></td>
<td>EmbeddingGemma-300m</td>
</tr>
<tr>
<td><strong>Endpoint</strong></td>
<td><code>POST /v1/embeddings</code> on port 11436</td>
</tr>
<tr>
<td><strong>Dimension</strong></td>
<td>768</td>
</tr>
<tr>
<td><strong>Used by</strong></td>
<td><code>parent_chunk_5w1h.py --embed</code>, story, 5W1H, search</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/index.html
+29
View File
@@ -0,0 +1,29 @@
<!DOCTYPE html>
<html lang="zh-TW">
<head>
<meta charset="UTF-8">
<title>Momentry API 文件</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 900px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 28px; margin-bottom: 8px; }
p.subtitle { color: #666; margin-bottom: 24px; }
table { width: 100%; border-collapse: collapse; }
tr { border-bottom: 1px solid #eee; }
tr:last-child { border: none; }
td { padding: 10px 0; }
td.cn { width: 140px; font-weight: 600; color: #333; }
td.en { color: #666; font-size: 14px; }
a { color: #0066cc; text-decoration: none; display: block; }
a:hover td { background: #f8f8f8; border-radius: 4px; }
</style>
</head>
<body>
<div class="container">
<h1>Momentry API 文件</h1>
<p class="subtitle">API 參考手冊 — 登入後可瀏覽各模組文件</p>
<table><tr onclick="window.location='01_auth.html'" style="cursor:pointer"><td class="cn">安全認證</td><td class="en">Authentication</td></tr><tr onclick="window.location='02_health.html'" style="cursor:pointer"><td class="cn">健康檢查</td><td class="en">Health</td></tr><tr onclick="window.location='03_register.html'" style="cursor:pointer"><td class="cn">檔案註冊</td><td class="en">File Registration</td></tr><tr onclick="window.location='04_lookup.html'" style="cursor:pointer"><td class="cn">檔案屬性查詢</td><td class="en">File Lookup</td></tr><tr onclick="window.location='05_process.html'" style="cursor:pointer"><td class="cn">處理流程</td><td class="en">Processing</td></tr><tr onclick="window.location='06_search.html'" style="cursor:pointer"><td class="cn">搜尋功能</td><td class="en">Search</td></tr><tr onclick="window.location='07_identity.html'" style="cursor:pointer"><td class="cn">身份識別</td><td class="en">Identity</td></tr><tr onclick="window.location='08_identity_agent.html'" style="cursor:pointer"><td class="cn">智能身份綁定</td><td class="en">Smart Identity Binding</td></tr><tr onclick="window.location='08_media.html'" style="cursor:pointer"><td class="cn">串流與截圖</td><td class="en">Streaming & Thumbnails</td></tr><tr onclick="window.location='09_tmdb.html'" style="cursor:pointer"><td class="cn">TMDb 整合</td><td class="en">TMDb Integration</td></tr><tr onclick="window.location='10_pipeline.html'" style="cursor:pointer"><td class="cn">生產線</td><td class="en">Pipeline</td></tr><tr onclick="window.location='12_agent.html'" style="cursor:pointer"><td class="cn">智慧代理</td><td class="en">AI Agents</td></tr></table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc/login.html
+46
View File
@@ -0,0 +1,46 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Login - Momentry Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; display: flex; justify-content: center; align-items: center; height: 100vh; }
.card { background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; width: 360px; }
h1 { font-size: 24px; margin-bottom: 24px; text-align: center; }
input { width: 100%; padding: 10px 12px; margin-bottom: 12px; border: 1px solid #ddd; border-radius: 6px; font-size: 14px; }
button { width: 100%; padding: 10px; background: #0066cc; color: white; border: none; border-radius: 6px; font-size: 16px; cursor: pointer; }
button:hover { background: #0052a3; }
.error { color: #cc0000; font-size: 13px; margin-bottom: 12px; display: none; }
</style>
</head>
<body>
<div class="card">
<h1>Momentry Docs</h1>
<form id="loginForm">
<input type="text" id="username" placeholder="Username" value="demo" required>
<input type="password" id="password" placeholder="Password" value="demo" required>
<div class="error" id="error">Invalid credentials</div>
<button type="submit">Login</button>
</form>
</div>
<script>
document.getElementById('loginForm').onsubmit = async function(e) {
e.preventDefault();
const resp = await fetch('/api/v1/auth/login', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({
username: document.getElementById('username').value,
password: document.getElementById('password').value
})
});
if (resp.ok) {
window.location.href = '/doc/index.html';
} else {
document.getElementById('error').style.display = 'block';
}
};
</script>
</body>
</html>
deliverable_v1.1.0/html_docs/doc_developer/11_error_codes.html
+180
View File
@@ -0,0 +1,180 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>11 Error Codes - Momentry API Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 24px; margin: 24px 0 12px; }
h2 { font-size: 20px; margin: 20px 0 10px; color: #222; }
h3 { font-size: 16px; margin: 16px 0 8px; color: #444; }
p { line-height: 1.6; margin: 8px 0; }
table { border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }
th, td { border: 1px solid #ddd; padding: 8px 12px; text-align: left; }
th { background: #f0f0f0; font-weight: 600; }
code { background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }
pre { background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }
pre code { background: none; padding: 0; }
a { color: #0066cc; }
.back { display: inline-block; margin-bottom: 20px; color: #666; }
.back:hover { color: #333; }
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
<!-- module: error_codes -->
<!-- description: Standard API error codes -->
<!-- depends: -->
<h2>Error Response Format</h2>
<p>All API errors follow this JSON structure:</p>
<div class="codehilite"><pre><span></span><code><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;success&quot;</span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;error&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nt">&quot;code&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;E001_NOT_FOUND&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;message&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;Resource not found&quot;</span><span class="p">,</span>
<span class="w"> </span><span class="nt">&quot;details&quot;</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="nt">&quot;resource&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;file_uuid&quot;</span><span class="p">,</span><span class="w"> </span><span class="nt">&quot;value&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;abc&quot;</span><span class="p">}</span>
<span class="w"> </span><span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h2>Error Code List</h2>
<h3>Generic Errors (E0xx)</h3>
<table class="table">
<thead>
<tr>
<th>Code</th>
<th>HTTP</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>E001_NOT_FOUND</code></td>
<td>404</td>
<td>Resource not found (file, identity, chunk)</td>
</tr>
<tr>
<td><code>E002_DUPLICATE</code></td>
<td>409</td>
<td>Resource already exists</td>
</tr>
<tr>
<td><code>E003_VALIDATION</code></td>
<td>400</td>
<td>Request parameter validation failed</td>
</tr>
<tr>
<td><code>E004_UNAUTHORIZED</code></td>
<td>401</td>
<td>Invalid API key or token</td>
</tr>
<tr>
<td><code>E005_INTERNAL</code></td>
<td>500</td>
<td>Internal server error</td>
</tr>
</tbody>
</table>
<h3>Processor Errors (E1xx)</h3>
<table class="table">
<thead>
<tr>
<th>Code</th>
<th>HTTP</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>E101_PROCESSOR_FAIL</code></td>
<td>500</td>
<td>Python script execution failed</td>
</tr>
<tr>
<td><code>E102_TIMEOUT</code></td>
<td>504</td>
<td>Processing timeout</td>
</tr>
<tr>
<td><code>E103_RESUME_FAIL</code></td>
<td>500</td>
<td>Resume failed (checkpoint not found)</td>
</tr>
<tr>
<td><code>E104_NO_VIDEO</code></td>
<td>400</td>
<td>Video file path not found</td>
</tr>
</tbody>
</table>
<h3>Identity Errors (E2xx)</h3>
<table class="table">
<thead>
<tr>
<th>Code</th>
<th>HTTP</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>E201_FACE_NOT_FOUND</code></td>
<td>404</td>
<td>Face detection not found</td>
</tr>
<tr>
<td><code>E202_MERGE_CONFLICT</code></td>
<td>409</td>
<td>Identity merge conflict</td>
</tr>
<tr>
<td><code>E203_CANDIDATE_EMPTY</code></td>
<td>404</td>
<td>No candidates available for confirmation</td>
</tr>
</tbody>
</table>
<h3>TMDb Errors (E3xx)</h3>
<table class="table">
<thead>
<tr>
<th>Code</th>
<th>HTTP</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>E301_TMDB_NO_KEY</code></td>
<td>400</td>
<td><code>TMDB_API_KEY</code> environment variable not set</td>
</tr>
<tr>
<td><code>E302_TMDB_UNREACHABLE</code></td>
<td>502</td>
<td>TMDb API unreachable or timed out</td>
</tr>
<tr>
<td><code>E303_TMDB_CACHE_NOT_FOUND</code></td>
<td>200</td>
<td>No local TMDb cache; run prefetch first</td>
</tr>
<tr>
<td><code>E304_TMDB_PROBE_FAILED</code></td>
<td>500</td>
<td>TMDb probe execution failed</td>
</tr>
<tr>
<td><code>E305_TMDB_MOVIE_NOT_FOUND</code></td>
<td>404</td>
<td>No matching TMDb movie found from filename</td>
</tr>
</tbody>
</table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc_developer/index.html
+29
View File
@@ -0,0 +1,29 @@
<!DOCTYPE html>
<html lang="zh-TW">
<head>
<meta charset="UTF-8">
<title>Momentry API 文件</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }
.container { max-width: 900px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }
h1 { font-size: 28px; margin-bottom: 8px; }
p.subtitle { color: #666; margin-bottom: 24px; }
table { width: 100%; border-collapse: collapse; }
tr { border-bottom: 1px solid #eee; }
tr:last-child { border: none; }
td { padding: 10px 0; }
td.cn { width: 140px; font-weight: 600; color: #333; }
td.en { color: #666; font-size: 14px; }
a { color: #0066cc; text-decoration: none; display: block; }
a:hover td { background: #f8f8f8; border-radius: 4px; }
</style>
</head>
<body>
<div class="container">
<h1>Momentry API 文件</h1>
<p class="subtitle">API 參考手冊 — 登入後可瀏覽各模組文件</p>
<table><tr onclick="window.location='11_error_codes.html'" style="cursor:pointer"><td class="cn">錯誤碼</td><td class="en">Error Codes</td></tr></table>
</div>
</body>
</html>
deliverable_v1.1.0/html_docs/doc_developer/login.html
+46
View File
@@ -0,0 +1,46 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Login - Momentry Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; display: flex; justify-content: center; align-items: center; height: 100vh; }
.card { background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; width: 360px; }
h1 { font-size: 24px; margin-bottom: 24px; text-align: center; }
input { width: 100%; padding: 10px 12px; margin-bottom: 12px; border: 1px solid #ddd; border-radius: 6px; font-size: 14px; }
button { width: 100%; padding: 10px; background: #0066cc; color: white; border: none; border-radius: 6px; font-size: 16px; cursor: pointer; }
button:hover { background: #0052a3; }
.error { color: #cc0000; font-size: 13px; margin-bottom: 12px; display: none; }
</style>
</head>
<body>
<div class="card">
<h1>Momentry Docs</h1>
<form id="loginForm">
<input type="text" id="username" placeholder="Username" value="demo" required>
<input type="password" id="password" placeholder="Password" value="demo" required>
<div class="error" id="error">Invalid credentials</div>
<button type="submit">Login</button>
</form>
</div>
<script>
document.getElementById('loginForm').onsubmit = async function(e) {
e.preventDefault();
const resp = await fetch('/api/v1/auth/login', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({
username: document.getElementById('username').value,
password: document.getElementById('password').value
})
});
if (resp.ok) {
window.location.href = '/doc/index.html';
} else {
document.getElementById('error').style.display = 'block';
}
};
</script>
</body>
</html>
deliverable_v1.1.0/modules/01_auth.md
+280
View File
@@ -0,0 +1,280 @@
<!-- module: auth -->
<!-- description: Authentication — login, logout, JWT, session cookie, API key -->
<!-- depends: -->
## Base URL
| Environment | URL | Purpose |
|-------------|-----|---------|
| Production | `http://localhost:3002` | Production deployment |
| External (M5) | `https://m5api.momentry.ddns.net` | Remote access |
## Variables
All examples in this documentation use these environment variables:
```bash
API="http://localhost:3002"
KEY="your-api-key-here"
```
## Authentication
All endpoints under `/api/v1/*` require authentication.
The following endpoints are public (no auth needed):
- `GET /health`
- `POST /api/v1/auth/login`
- `POST /api/v1/auth/logout`
### Three Authentication Modes
The system supports three authentication methods, checked in **priority order** by the middleware:
```
Middleware priority:
1. Session Cookie (Portal/browser)
2. JWT Bearer (API clients, CLI)
3. API Key Header (legacy compatibility)
4. API Key Query Param (?api_key=)
```
| Mode | Transport | Expiry | Scope | Best for |
|------|-----------|--------|-------|----------|
| **Session Cookie** | `Cookie: session_id=<session_id>` | 24h | per-browser session | Portal (browser) |
| **JWT** | `Authorization: Bearer <token>` | 1h | per-login token | API clients, CLI, scripts |
| **API Key** | `X-API-Key: <key>` | 90d | fixed key for automation | Legacy scripts, WordPress |
---
### Login
**Default accounts & API keys:**
| Username | Password | API Key | Role |
|----------|----------|---------|------|
| `admin` | `admin` | — | admin |
| `demo` | `demo` | `muser_demo_key_32chars_abcdef1234567890` | user |
The demo API key is set via `MOMENTRY_DEMO_API_KEY` env var and can be used in place of JWT for marcom integrations:
```bash
# Using API key instead of JWT
curl -s "$API/api/v1/files/scan" -H "X-API-Key: muser_demo_key_32chars_abcdef1234567890"
```
```bash
# Login as admin
curl -s -X POST "$API/api/v1/auth/login" \
-H "Content-Type: application/json" \
-d '{"username": "admin", "password": "admin"}'
# Login as demo user
curl -s -X POST "$API/api/v1/auth/login" \
-H "Content-Type: application/json" \
-d '{"username": "demo", "password": "demo"}'
```
#### Success Response
```json
{
"success": true,
"jwt": "eyJhbGciOiJIUzI1NiIs...",
"api_key": "muser_...",
"user": {
"username": "admin",
"role": "admin"
},
"expires_at": "2026-05-18T13:00:00Z"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `jwt` | string | JWT access token. Use as `Authorization: Bearer <jwt>`. Expires in 1 hour. |
| `api_key` | string | Legacy API key. Use as `X-API-Key: <key>`. Good for 90 days. |
| `user.username` | string | Username |
| `user.role` | string | Role: `admin`, `user`, or `readonly` |
| `expires_at` | string | ISO8601 timestamp of JWT expiration |
The login endpoint also sets a `Set-Cookie` header for browser-based clients:
```
Set-Cookie: session_id=<session_id>; Path=/; HttpOnly; SameSite=Strict; Max-Age=86400
```
#### Error Response (401)
```json
{
"success": false,
"message": "Invalid username or password"
}
```
---
### Using JWT
JWT is preferred for API clients (CLI scripts, WordPress). It is validated by the middleware without a database lookup (stateless).
```bash
# Login and capture JWT
JWT=$(curl -s -X POST "$API/api/v1/auth/login" \
-H "Content-Type: application/json" \
-d '{"username":"admin","password":"admin"}' | python3 -c "import json,sys;print(json.load(sys.stdin)['jwt'])")
# Use JWT for all subsequent requests
curl -H "Authorization: Bearer $JWT" "$API/api/v1/files/scan"
curl -H "Authorization: Bearer $JWT" "$API/api/v1/resource/tmdb"
```
JWT is short-lived (1 hour). When it expires, request a new one via login.
---
### Using Session Cookie (Browser)
Browser-based clients (Portal) get a session cookie automatically after login. The browser sends the cookie with every request—no manual header needed.
```bash
# Login captures the session cookie from Set-Cookie header
curl -v -X POST "$API/api/v1/auth/login" \
-H "Content-Type: application/json" \
-d '{"username":"admin","password":"admin"}' 2>&1 | grep "Set-Cookie"
# Browser automatically sends: Cookie: session_id=<session_id>
# No manual header needed for subsequent requests
```
The session cookie is HttpOnly (not accessible from JavaScript) and SameSite=Strict (protected against CSRF).
---
### Using Legacy API Key
```bash
curl -H "X-API-Key: $KEY" "$API/api/v1/files/scan"
# Also accepted via Bearer header (non-JWT format) or query parameter:
curl -H "Authorization: Bearer $KEY" "$API/api/v1/files/scan"
curl "$API/api/v1/files/scan?api_key=$KEY"
```
API keys are validated via SHA256 hash lookup in the database. They are long-lived (90 days) and intended for automation.
### Obtaining an API Key (CLI)
```bash
momentry api-key create "My API Key" --key-type user
```
---
### Logout
```bash
# Logout using the session cookie (browser)
curl -X POST "$API/api/v1/auth/logout" \
-H "Cookie: session_id=<uuid>"
```
#### What logout does
| Auth mode | Effect |
|-----------|--------|
| **Session Cookie** | Session deleted from database. Same cookie returns 401 on subsequent requests. |
| **JWT** | JWT remains valid until expiry. (JWT is stateless — logout adds JWT to a blacklist only if API key mode is used.) |
| **API Key** | API key remains valid. (Legacy keys are shared across sessions — revoking would break other clients.) |
#### Example: full session lifecycle
```bash
# 1. Login
SESSION_ID=$(curl -s -D - -X POST "$API/api/v1/auth/login" \
-H "Content-Type: application/json" \
-d '{"username":"admin","password":"admin"}' | grep "Set-Cookie" | sed 's/.*session_id=\([^;]*\).*/\1/')
# 2. Use session (works)
curl -s -o /dev/null -w "HTTP %{http_code}\n" "$API/api/v1/resource/tmdb" \
-H "Cookie: session_id=$SESSION_ID"
# → HTTP 200
# 3. Logout
curl -s -X POST "$API/api/v1/auth/logout" \
-H "Cookie: session_id=$SESSION_ID"
# → {"success": true}
# 4. Use session again (rejected)
curl -s -o /dev/null -w "HTTP %{http_code}\n" "$API/api/v1/resource/tmdb" \
-H "Cookie: session_id=$SESSION_ID"
# → HTTP 401
```
---
### Authentication Flow Summary
```
Login Request
│
▼
┌──────────────────┐
│ 1. Check users │ ← users table (argon2 password verify)
│ table │
└──────┬───────────┘
│
┌───┴───┐
│ match │
└───┬───┘
│
▼
┌──────────────────┐
│ 2. Create JWT │ ← 1h expiry, signed with JWT_SECRET
├──────────────────┤
│ 3. Create │ ← 24h expiry, stored in sessions table
│ session │
├──────────────────┤
│ 4. Set-Cookie │ ← HttpOnly, SameSite=Strict, Path=/
├──────────────────┤
│ 5. Return │ ← JWT + api_key + user info to client
└──────────────────┘
```
```
Protected Request
│
▼
┌──────────────────────┐
│ Middleware checks: │
│ │
│ 1. Cookie session? │ → DB lookup session → get api_key → verify
│ │
│ 2. JWT Bearer? │ → verify JWT signature → decode claims
│ │
│ 3. X-API-Key? │ → SHA256 hash → DB lookup → verify
│ │
│ 4. ?api_key=? │ → same as #3
│ │
│ 5. None → 401 │
└──────────────────────┘
```
---
### Error Responses
| HTTP | When |
|------|------|
| `401` | Missing or invalid authentication |
| `401` | Session expired or logged out |
| `401` | JWT expired |
| `401` | API key revoked or inactive |
---
### Related
- `POST /api/v1/resource/tmdb/check` — test authentication + TMDb API connectivity
- `GET /health/detailed` — view auth status (integrations section)
deliverable_v1.1.0/modules/02_health.md
+147
View File
@@ -0,0 +1,147 @@
<!-- module: health -->
<!-- description: Health check endpoints -->
<!-- depends: 01_auth -->
## Health Check
### `GET /health`
**Auth**: Public
**Scope**: system-level
Returns basic server health status — used by load balancers and monitoring.
#### Example
```bash
curl "$API/health" | jq '{status, version}'
```
#### Response (200)
```json
{
"status": "ok",
"version": "1.0.0",
"build_git_hash": "3a6c1865",
"build_timestamp": "2026-05-16T13:38:15Z",
"uptime_ms": 3015
}
```
| Field | Type | Description |
|-------|------|-------------|
| `status` | string | `ok` or `degraded` |
| `version` | string | Semver version |
| `build_git_hash` | string | Git commit hash |
| `build_timestamp` | string | Binary build time |
| `uptime_ms` | integer | Milliseconds since server start |
---
### `GET /health/detailed`
**Auth**: Required
**Scope**: system-level
Returns full system health including each service status, resource utilization, pipeline readiness, schema migration status, identity file sync status, and external integrations.
> Requires authentication (JWT, session cookie, or API key). The basic `/health` endpoint remains public for load balancer checks.
#### Example
```bash
curl "$API/health/detailed" | jq '{status, services, resources: {cpu: .resources.cpu_used_percent, memory: .resources.memory_used_percent}}'
```
#### Response (200)
```json
{
"status": "ok",
"version": "1.0.0",
"services": {
"postgres": {"status": "ok", "latency_ms": 3},
"redis": {"status": "ok", "latency_ms": 1},
"qdrant": {"status": "ok", "latency_ms": 5}
},
"resources": {
"cpu_used_percent": 12.5,
"memory_available_mb": 32768,
"memory_used_percent": 31.7
},
"pipeline": {
"scripts_ready": true,
"scripts_count": 345,
"processors": {
"asr": true,
"yolo": true,
"face": true,
"pose": true,
"ocr": true,
"cut": true,
"scene": true,
"asrx": true,
"visual_chunk": true
},
"models_ready": true,
"models_count": 42,
"scripts_integrity": {"matched": 332, "total": 345, "ok": false},
"ffmpeg": true
},
"schema": {
"table_exists": true,
"applied": [{"filename": "migrate_add_users_table.sql"}],
"required": [],
"ok": true
},
"identities": {
"directory_exists": true,
"files_count": 3481,
"index_ok": true,
"db_count": 3481,
"synced": true
},
"integrations": {
"tmdb": {
"api_key_configured": false,
"enabled": false,
"api_reachable": null
}
}
}
```
#### Response Fields
| Field | Type | Description |
|-------|------|-------------|
| `status` | string | `ok` if all essential services healthy |
| `services` | object | Per-service status (postgres, redis, qdrant) |
| `services.*.status` | string | `ok`, `error`, or `degraded` |
| `services.*.latency_ms` | int | Response time in milliseconds |
| `resources` | object | CPU, memory usage |
| `pipeline.scripts_ready` | boolean | Scripts directory accessible |
| `pipeline.scripts_count` | int | Number of Python processor scripts |
| `pipeline.processors` | object | Per-processor availability |
| `pipeline.models_ready` | boolean | Models directory accessible |
| `pipeline.scripts_integrity` | object | SHA256 checksum verification results |
| `schema.ok` | boolean | All required migrations applied |
| `identities.synced` | boolean | Identity file count matches DB count |
| `integrations.tmdb` | object | TMDB API key config and reachability |
#### Health status rules
| Condition | status |
|-----------|--------|
| All services ok | `ok` |
| Any service error | `degraded` |
| Postgres or Redis error | `degraded` (server still responds) |
---
### Stats Endpoints
| Method | Endpoint | Auth | Description |
|--------|----------|------|-------------|
| GET | `/api/v1/stats/sftpgo` | No | SFTPGo service status |
deliverable_v1.1.0/modules/03_register.md
+184
View File
@@ -0,0 +1,184 @@
<!-- module: register -->
<!-- description: File registration — register, scan -->
<!-- depends: 01_auth -->
## File Registration
### `POST /api/v1/files/register`
**Auth**: Required
**Scope**: file-level
Register a video file for processing. Returns the file's metadata and UUID.
**New in v0.1.2**: Registration now **automatically triggers the processing pipeline** — no need to call `POST /api/v1/file/:file_uuid/process` separately. The system will:
1. Register the file and run ffprobe
2. Auto-run offline TMDb probe (reads local identity files, no API calls)
3. Create a monitor job for the worker
4. Worker starts all 10 processors (Cut → ASR → ASRX → YOLO → OCR → Face → Pose → VisualChunk → Story → 5W1H)
If the file already exists (same content hash), returns the existing record with `already_exists: true`.
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `file_path` | string | Yes | — | Path to video file on disk |
| `pattern` | string | No | — | Regex pattern for batch register (requires `file_path` to be a directory) |
| `user_id` | integer | No | — | User ID to associate with registration |
| `content_hash` | string | No | — | Pre-computed SHA-256 hash (skips computation) |
#### Example
```bash
# Register a single file
curl -s -X POST "$API/api/v1/files/register" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"file_path": "/path/to/video.mp4"}'
# Batch register files matching a pattern in a directory
curl -s -X POST "$API/api/v1/files/register" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"file_path": "/path/to/dir", "pattern": ".*\\.mp4$"}'
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "3a6c1865...",
"file_name": "video.mp4",
"file_path": "/path/to/video.mp4",
"file_type": "video",
"duration": 120.5,
"width": 1920,
"height": 1080,
"fps": 24.0,
"total_frames": 2892,
"already_exists": false,
"message": "File registered successfully"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on 200 |
| `file_uuid` | string | 32-char hex UUID of the registered file |
| `file_name` | string | File name (auto-renamed if name conflict) |
| `file_path` | string | Canonical path on disk |
| `file_type` | string | `"video"`, `"audio"`, or `"unknown"` |
| `duration` | float | Duration in seconds |
| `width` | integer | Video width in pixels |
| `height` | integer | Video height in pixels |
| `fps` | float | Frames per second |
| `total_frames` | integer | Total frame count |
| `already_exists` | boolean | True if same content was already registered |
| `message` | string | Human-readable status |
#### Error Responses
| HTTP | When |
|------|------|
| `401` | Missing or invalid API key |
| `400` | Invalid request body |
| `404` | File path does not exist |
---
### `GET /api/v1/files/scan`
**Auth**: Required
**Scope**: file-level
Scan the filesystem directory and list all media files, showing which are registered, processing, or unregistered.
#### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `page` | integer | No | 1 | Page number (1-based) |
| `page_size` | integer | No | all | Items per page (alias: `limit`) |
| `limit` | integer | No | all | Max items (alias for `page_size`) |
| `pattern` | string | No | — | Regex filter on file name (e.g., `.*\\.mp4$`) |
| `sort_by` | string | No | `name` | Sort field: `name`, `size`, `modified`, `status` |
| `sort_order` | string | No | `asc` | Sort direction: `asc` or `desc` |
#### Example
```bash
# Full scan
curl -s "$API/api/v1/files/scan" -H "X-API-Key: $KEY" | jq '{total, registered_count, unregistered_count}'
# Paginated (page 1, 5 per page)
curl -s "$API/api/v1/files/scan?page=1&page_size=5" -H "X-API-Key: $KEY" | jq '{page, total_pages, files: [.files[].file_name]}'
# Regex filter: only mp4 files
curl -s "$API/api/v1/files/scan?pattern=.*\\.mp4$" -H "X-API-Key: $KEY" | jq '{filtered_total, files: [.files[].file_name]}'
# Sort by file size (largest first)
curl -s "$API/api/v1/files/scan?sort_by=size&sort_order=desc&page_size=5" -H "X-API-Key: $KEY" | jq '[.files[] | {file_name, file_size}]'
# Sort by modified time (most recent first)
curl -s "$API/api/v1/files/scan?sort_by=modified&sort_order=desc&page_size=5" -H "X-API-Key: $KEY" | jq '[.files[] | {file_name, modified_time}]'
# Sort by status
curl -s "$API/api/v1/files/scan?sort_by=status&page_size=5" -H "X-API-Key: $KEY" | jq '[.files[] | {file_name, status}]'
```
#### Response (200)
```json
{
"files": [
{
"file_name": "video.mp4",
"file_size": 12345678,
"is_registered": true,
"file_uuid": "3a6c1865...",
"status": "completed",
"registration_time": "2026-05-16T12:00:00Z",
"job_id": 42
}
],
"total": 107,
"filtered_total": 80,
"page": 1,
"page_size": 20,
"total_pages": 4,
"registered_count": 26,
"unregistered_count": 81
}
```
| Field | Type | Description |
|-------|------|-------------|
| `files` | array | Array of file info objects (paginated) |
| `files[].file_name` | string | File name |
| `files[].relative_path` | string | Path relative to scan root |
| `files[].file_path` | string | Absolute path on disk |
| `files[].file_size` | integer | File size in bytes |
| `files[].modified_time` | string | Last modified timestamp (ISO8601) |
| `files[].is_registered` | boolean | Whether file is registered in DB |
| `files[].file_uuid` | string | 32-char hex UUID (only if registered) |
| `files[].status` | string | `"completed"`, `"processing"`, `"registered"`, `"unregistered"`, or `null` |
| `files[].registration_time` | string | DB registration timestamp (only if registered) |
| `files[].job_id` | integer | Processing job ID (only if a job exists) |
| `total` | integer | Total files found on disk (unfiltered) |
| `filtered_total` | integer | Files matching regex filter |
| `page` | integer | Current page number |
| `page_size` | integer | Items per page |
| `total_pages` | integer | Total pages |
| `registered_count` | integer | Files registered in DB |
| `unregistered_count` | integer | Files not yet registered |
#### Notes
| Feature | Behavior |
|---------|----------|
| **Regex** | Case-insensitive (`(?i)` prefix auto-applied). Applied to `file_name`. |
| **Sort order** | Default (`sort_by=name`): registered files first, then alphabetically. `sort_by=status`: alphabetical by status string. |
| **Pagination** | `page_size` and `limit` are aliases. Default: show all results. |
| **Processing order** | `pattern` regex filter → `sort_by`/`sort_order` → `page`/`page_size` slice. |
deliverable_v1.1.0/modules/04_lookup.md
+138
View File
@@ -0,0 +1,138 @@
<!-- module: lookup -->
<!-- description: File lookup by name and unregistration -->
<!-- depends: 01_auth, 03_register -->
## File Lookup
### `GET /api/v1/files/lookup`
**Auth**: Required
**Scope**: file-level
Search registered files by file name. Performs a case-insensitive LIKE search on the file name column. Returns basic info about matching files.
#### Query Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_name` | string | Yes | File name to search for (partial matches supported) |
#### Example
```bash
# Look up a specific file
curl -s "$API/api/v1/files/lookup?file_name=video.mp4" \
-H "X-API-Key: $KEY"
# Partial name search
curl -s "$API/api/v1/files/lookup?file_name=charade" \
-H "X-API-Key: $KEY" | jq '.matches[].file_name'
```
#### Response (200)
```json
{
"file_name": "video.mp4",
"exists": true,
"matches": [
{
"file_uuid": "a03485a40b2df2d3",
"file_name": "video.mp4",
"file_type": "video",
"status": "completed"
}
],
"next_name": "video (2).mp4"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_name` | string | Searched name |
| `exists` | boolean | Exact name match exists |
| `matches` | array | Array of matching registered files |
| `matches[].file_uuid` | string | 32-char hex UUID |
| `matches[].file_name` | string | Registered file name |
| `matches[].file_type` | string | `"video"`, `"audio"`, or `null` |
| `matches[].status` | string | Registration/processing status |
| `next_name` | string | Suggested name for avoiding conflicts |
---
## Unregister
### `POST /api/v1/unregister`
**Auth**: Required
**Scope**: file-level
Delete a registered file from the system. Supports single file by UUID, or batch by directory + regex pattern.
#### What gets deleted
| Removed (default) | Not removed |
|---------|-------------|
| Database records (videos, chunks, embeddings, processor_results, pre_chunks) | The original source video file on disk |
| Processor output JSON files (`{uuid}.*.json`) — unless `delete_output_files: false` | Temp/working directories |
| In-memory cache entries | |
| MongoDB cached lists | |
> ⚠️ Database deletion is **irreversible**. To keep output files, set `"delete_output_files": false`.
#### Request Parameters
At least one mode must be specified: either `file_uuid` alone, or `file_path` + `pattern` together.
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `file_uuid` | string | * | — | Single file UUID to delete |
| `file_path` | string | * | — | Directory path (for batch delete) |
| `pattern` | string | * | — | Regex pattern (requires `file_path`) |
| `delete_output_files` | boolean | No | `true` | If `true`, also delete processor output JSON files (`{uuid}.*.json`). Set to `false` to keep them. |
#### Example
```bash
# Delete a single file by UUID (default: also deletes output JSON files)
curl -s -X POST "$API/api/v1/unregister" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"file_uuid": "'"$FILE_UUID"'"}'
# Keep output JSON files, only delete DB records
curl -s -X POST "$API/api/v1/unregister" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"file_uuid": "'"$FILE_UUID"'", "delete_output_files": false}'
# Batch delete all mp4 files in a directory
curl -s -X POST "$API/api/v1/unregister" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"file_path": "/path/to/dir", "pattern": ".*\\.mp4$"}'
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "a03485a40b2df2d3",
"message": "Video unregistered successfully"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | True if deletion succeeded |
| `file_uuid` | string | UUID of the deleted file (single mode) |
| `message` | string | Human-readable status |
#### Error Responses
| HTTP | When |
|------|------|
| `400` | Neither `file_uuid` nor `file_path`+`pattern` provided |
| `404` | File UUID not found |
| `401` | Missing or invalid API key |
deliverable_v1.1.0/modules/05_process.md
+236
View File
@@ -0,0 +1,236 @@
<!-- module: process -->
<!-- description: Processing pipeline — trigger, probe, progress, jobs -->
<!-- depends: 01_auth, 03_register -->
## Processing Pipeline
### `POST /api/v1/file/:file_uuid/process`
**Auth**: Required
**Scope**: file-level
Trigger the processing pipeline for a registered file. Creates a monitor job that the worker picks up and processes sequentially. Returns immediately with the job info—processing runs asynchronously in the background.
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `processors` | string[] | No | all | Specific processors to run: `["cut","asr","asrx","yolo","ocr","face","pose","visual_chunk","story","5w1h"]` |
| `rules` | string[] | No | all | Rule names to apply (currently unused) |
#### Example
```bash
# Run all processors
curl -s -X POST "$API/api/v1/file/$FILE_UUID/process" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" -d '{}'
# Run specific processors only
curl -s -X POST "$API/api/v1/file/$FILE_UUID/process" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"processors": ["asr", "face", "yolo"]}'
```
#### Response (200)
```json
{
"success": true,
"job_id": 42,
"file_uuid": "3a6c1865...",
"status": "processing",
"pids": [12345, 12346],
"message": "Processing triggered for video.mp4"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on 200 |
| `job_id` | integer | Monitor job ID (for job tracking) |
| `file_uuid` | string | 32-char hex UUID of the file |
| `status` | string | `"processing"` |
| `pids` | integer[] | Process IDs of started processors |
| `message` | string | Human-readable status |
#### Error Responses
| HTTP | When |
|------|------|
| `404` | File UUID not found |
| `401` | Missing or invalid API key |
---
### `GET /api/v1/file/:file_uuid/probe`
**Auth**: Required
**Scope**: file-level
Get ffprobe metadata for a registered file. Returns video/audio stream info, codec details, duration, resolution, and frame rate.
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/probe" -H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"file_uuid": "3a6c1865...",
"file_name": "video.mp4",
"file_size": 794863677,
"duration": 120.5,
"width": 1920,
"height": 1080,
"fps": 24.0,
"total_frames": 2892,
"cached": true,
"format": {
"filename": "/path/to/video.mp4",
"format_name": "mov,mp4,m4a,3gp",
"duration": "120.5",
"size": "12345678",
"bit_rate": "819200"
},
"streams": [
{
"index": 0,
"codec_name": "h264",
"codec_type": "video",
"width": 1920,
"height": 1080,
"r_frame_rate": "24/1",
"duration": "120.5"
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | 32-char hex UUID |
| `file_name` | string | File name |
| `file_size` | integer | File size in bytes (from filesystem) |
| `duration` | float | Duration in seconds |
| `width` | integer | Video width in pixels |
| `height` | integer | Video height in pixels |
| `fps` | float | Frames per second |
| `total_frames` | integer | Estimated total frames |
| `cached` | boolean | True if result was from cached probe JSON |
| `format` | object | Container format info (ffprobe format section) |
| `streams` | array | Array of stream info objects |
---
### `GET /api/v1/progress/:file_uuid`
**Auth**: Required
**Scope**: file-level
Get real-time processing progress for a file via Redis pub/sub. Includes per-processor status, current/total frames, ETA, and system resource stats.
#### Pipeline Order
| Order | Processor | Dependencies | Description |
|-------|-----------|-------------|-------------|
| 1 | `cut` | — | Scene detection |
| 2 | `asr` | cut | Speech-to-text (per scene) |
| 3 | `asrx` | asr | Speaker diarization |
| 4 | `yolo` | — | Object detection |
| 5 | `ocr` | — | Text recognition |
| 6 | `face` | — | Face detection & embedding |
| 7 | `pose` | — | Pose estimation |
| 8 | `visual_chunk` | yolo | Visual scene chunks |
| 9 | `story` | asr, asrx, cut, yolo, face | Scene summaries (template) |
| 10 | `5w1h` | story | 5W1H analysis (Gemma4 LLM) |
All processors except `story` and `5w1h` run concurrently when their dependencies are met. Story and 5W1H run sequentially after their prerequisites.
#### Example
```bash
curl -s "$API/api/v1/progress/$FILE_UUID" -H "X-API-Key: $KEY" | jq '{overall_progress, processors: [.processors[] | {processor_type, status}]}'
```
#### Response (200)
```json
{
"file_uuid": "3a6c1865...",
"overall_progress": 71,
"cpu_percent": 45.2,
"gpu_percent": 30.1,
"memory_percent": 62.4,
"processors": [
{"processor_type": "asr", "status": "complete", "progress": 100},
{"processor_type": "yolo", "status": "running", "progress": 65},
{"processor_type": "face", "status": "pending", "progress": 0}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | 32-char hex UUID |
| `overall_progress` | integer | Overall progress percentage (0–100) |
| `processors` | array | Per-processor status list |
| `processors[].processor_type` | string | Processor name (`asr`, `cut`, `yolo`, etc.) |
| `processors[].status` | string | `"pending"`, `"running"`, `"complete"`, or `"failed"` |
| `processors[].progress` | integer | Per-processor progress (0–100) |
| `processors[].eta_seconds` | integer | Estimated seconds remaining (running processors) |
| `processors[].current` | integer | Current frame count |
| `processors[].total` | integer | Total frame count |
| `cpu_percent` | float | Current CPU usage |
| `gpu_percent` | float | Current GPU utilization |
| `memory_percent` | float | Current memory usage |
---
### `GET /api/v1/jobs`
**Auth**: Required
**Scope**: system-level
List all processing jobs (monitor jobs) in the system. Shows job status, which file each job is processing, and current processor info.
#### Example
```bash
curl -s "$API/api/v1/jobs" -H "X-API-Key: $KEY" | jq '{count, jobs: [.jobs[] | {uuid, status}]}'
```
#### Response (200)
```json
{
"jobs": [
{
"id": 42,
"uuid": "3a6c1865...",
"status": "running",
"current_processor": "yolo",
"created_at": "2026-05-16T12:00:00Z",
"started_at": "2026-05-16T12:01:00Z"
}
],
"count": 15,
"page": 1,
"page_size": 20
}
```
| Field | Type | Description |
|-------|------|-------------|
| `jobs` | array | Array of job info objects |
| `jobs[].id` | integer | Job ID |
| `jobs[].uuid` | string | File UUID being processed |
| `jobs[].status` | string | `"pending"`, `"running"`, `"completed"`, `"failed"` |
| `jobs[].current_processor` | string | Currently active processor, or null |
| `count` | integer | Total job count |
| `page` | integer | Current page number |
| `page_size` | integer | Jobs per page |
deliverable_v1.1.0/modules/06_search.md
+145
View File
@@ -0,0 +1,145 @@
<!-- module: search -->
<!-- description: Vector search, BM25, smart search, universal search, visual search -->
<!-- depends: 01_auth -->
## Search APIs
### `POST /api/v1/search/smart`
**Auth**: Required
**Scope**: file-level
Semantic vector search using EmbeddingGemma-300m. Generates a query embedding via EmbeddingGemma (port 11436), then searches pgvector `story_parent` and `llm_parent` chunks by cosine similarity.
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `file_uuid` | string | Yes | — | File UUID to search within |
| `query` | string | Yes | — | Search text |
| `limit` | integer | No | 5 | Max results to return |
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 5 | Items per page |
#### Example
```bash
curl -s -X POST "$API/api/v1/search/smart" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $JWT" \
-d '{"file_uuid": "'"$FILE_UUID"'", "query": "Audrey Hepburn"}'
```
#### Response (200)
```json
{
"query": "Audrey Hepburn",
"results": [
{
"parent_id": 1087822,
"scene_order": 1087822,
"start_frame": 104438,
"end_frame": 104538,
"fps": 24.0,
"start_time": 4351.6,
"end_time": 4355.76,
"summary": "[4352s-4356s, 4s] Cast: Audrey Hepburn. Total: 2 lines, 10 words. Speakers: Audrey Hepburn (2 lines)",
"similarity": 0.67
}
],
"page": 1,
"page_size": 5,
"strategy": "semantic_vector_search"
}
```
---
### `POST /api/v1/search/universal`
**Auth**: Required
**Scope**: file-level
Multi-type BM25 full-text search across chunks, frames, and persons. Uses PostgreSQL `tsvector`.
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `query` | string | Yes | — | Search text |
| `file_uuid` | string | No | — | Restrict to specific file |
| `types` | string[] | No | `["chunk","frame","person"]` | Search types |
| `limit` | integer | No | 10 | Max results per type |
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 20 | Items per page |
#### Example
```bash
curl -s -X POST "$API/api/v1/search/universal" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $JWT" \
-d '{"file_uuid": "'"$FILE_UUID"'", "query": "Cary Grant"}'
```
#### Response (200)
```json
{
"results": [
{
"type": "chunk",
"chunk_id": "bd80fec92b0b6963d177a2c55bf713e2_2",
"chunk_type": "story_child",
"start_frame": 5103,
"end_frame": 5127,
"start_time": 212.64,
"end_time": 213.64,
"text": "[213s-214s] Cary Grant: \"Olá!\"",
"score": 0.9
}
],
"total": 20,
"took_ms": 18
}
```
---
### `POST /api/v1/search/frames`
**Auth**: Required
**Scope**: file-level
Search face detection frames by identity name or trace ID.
---
### `POST /api/v1/search/identity_text`
**Auth**: Required
**Scope**: file-level
Search text chunks spoken by a specific identity.
---
### Visual Search
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/v1/search/visual` | Search visual chunks |
| POST | `/api/v1/search/visual/class` | Search by object class |
| POST | `/api/v1/search/visual/density` | Search by object density |
| POST | `/api/v1/search/visual/combination` | Search by object combination |
| POST | `/api/v1/search/visual/stats` | Visual chunk statistics |
#### Embedding Model
| Detail | Value |
|--------|-------|
| **Model** | EmbeddingGemma-300m |
| **Endpoint** | `POST /api/v1/embeddings` on port 11436 |
| **Dimension** | 768 |
| **Storage** | pgvector (`chunk.embedding` column) |
deliverable_v1.1.0/modules/08_identity_agent.md
+65
View File
@@ -0,0 +1,65 @@
<!-- module: identity_agent -->
<!-- description: Identity agent — match from photo, match from trace -->
<!-- depends: 01_auth, 07_identity -->
## Identity Agent
### `POST /api/v1/agents/identity/match-from-photo`
**Auth**: Required
**Scope**: file-level
Upload a face photo to match against known identities. Detects face via InsightFace, extracts 512D embedding via CoreML FaceNet, then searches pgvector for the closest identity.
#### Request
`multipart/form-data` with field `image` (JPEG/PNG) and optional `file_uuid`.
#### Example
```bash
curl -s -X POST "$API/api/v1/agents/identity/match-from-photo" \
-H "Authorization: Bearer $JWT" \
-F "image=@/path/to/face.jpg" \
-F "file_uuid=$FILE_UUID"
```
#### Response (200)
```json
{
"success": true,
"matches": [
{
"identity_uuid": "a9a90105...",
"name": "Cary Grant",
"similarity": 0.87
}
]
}
```
---
### `POST /api/v1/agents/identity/match-from-trace`
**Auth**: Required
**Scope**: file-level
Match a face trace (tracked face across frames) against known identities. Samples 3 angles from the trace, generates embeddings, and searches pgvector.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File containing the trace |
| `trace_id` | integer | Yes | Face trace ID to match |
#### Example
```bash
curl -s -X POST "$API/api/v1/agents/identity/match-from-trace" \
-H "Authorization: Bearer $JWT" \
-H "Content-Type: application/json" \
-d '{"file_uuid": "'"$FILE_UUID"'", "trace_id": 10}'
```
deliverable_v1.1.0/modules/09_tmdb.md
+109
View File
@@ -0,0 +1,109 @@
<!-- module: tmdb -->
<!-- description: TMDb enrichment endpoints — prefetch, probe, resource, check -->
<!-- depends: 01_auth, 03_register -->
## TMDb Enrichment
> **Offline operation**: TMDb prefetch now checks local identity files first (`identities/_index.json` + `*.tmdb.json`).
> If local files exist, no external API call is made. Internet is only needed for initial data seeding.
### Overview
TMDb enrichment is an optional identity enrichment step that can be run after Pipeline face detection completes. The workflow is:
1. **Prefetch** (requires internet): Download movie cast data from TMDb API → cache to `{file_uuid}.tmdb.json`
2. **Probe**: Read local cache → create identities for **all** cast members (`source='tmdb'`) + save `identity.json` + download profile image to `{OUTPUT}/identities/{uuid}/profile.jpg`
3. **Match**: The worker automatically matches video faces against TMDb identities when `MOMENTRY_TMDB_PROBE_ENABLED=true`
### `POST /api/v1/agents/tmdb/prefetch`
**Auth**: Required
**Scope**: file-level
Fetch TMDb cast data for a registered file and cache it locally. This is the only step requiring internet access.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File UUID to enrich |
#### Example
```bash
curl -s -X POST "$API/api/v1/agents/tmdb/prefetch" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"file_uuid": "'"$FILE_UUID"'"}'
```
#### Response (200)
```json
{"success": true, "file_uuid": "...", "cache_path": "/output/...tmdb.json"}
```
### `POST /api/v1/file/:file_uuid/tmdb-probe`
**Auth**: Required
**Scope**: file-level
Read local TMDb cache and create/update identities. Requires prefetch to have been run first.
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tmdb-probe" \
-H "X-API-Key: $KEY" | jq '{identities_created, movie_title}'
```
#### Response (200 — identities created)
```json
{"success": true, "identities_created": 15, "movie_title": "Charade"}
```
#### Response (200 — no cache)
```json
{"success": false, "message": "No TMDb cache found. Run tmdb-prefetch first."}
```
### `GET /api/v1/resource/tmdb`
**Auth**: Required
**Scope**: system-level
View TMDb resource status including configuration, identity counts, and cache file count.
#### Example
```bash
curl -s "$API/api/v1/resource/tmdb" -H "X-API-Key: $KEY" \
| jq '{identities_seeded, cache_files}'
```
### `POST /api/v1/resource/tmdb/check`
**Auth**: Required
**Scope**: system-level
Ping the TMDb API to verify connectivity and measure latency.
#### Example
```bash
curl -s -X POST "$API/api/v1/resource/tmdb/check" \
-H "X-API-Key: $KEY" | jq '.status'
```
#### Response
```json
{
"api_key_configured": true,
"enabled": false,
"api_reachable": true,
"api_latency_ms": 120
}
```
deliverable_v1.1.0/modules/10_pipeline.md
+178
View File
@@ -0,0 +1,178 @@
<!-- module: pipeline -->
<!-- description: Pipeline processors, ingestion status, stats endpoints -->
<!-- depends: 01_auth -->
## Pipeline
### Dependency Graph
```mermaid
flowchart TB
subgraph Processors["10 Processors"]
Cut[Cut] --> ASR[ASR]
ASR --> ASRX[ASRX]
ASRX --> Story[Story]
Cut --> Story
YOLO[YOLO] --> VisualChunk[VisualChunk]
VisualChunk --> Story
Face[Face] --> Story
Story --> FiveW1H[5W1H]
OCR[OCR]
Pose[Pose]
end
subgraph Ingestion["入庫 (Post-Processing)"]
ASR --> Rule1[Rule 1 Sentence]
ASRX --> Rule1
Rule1 --> Vectorize[Auto-Vectorize]
Rule1 --> Phase1[Phase 1 Pack]
Cut --> Rule3[Rule 3 Scene]
ASR --> Rule3
Face --> Trace[Face Trace]
Trace --> Qdrant[Qdrant Sync]
Trace --> TraceChunks[Trace Chunks]
Trace --> TKG[TKG Builder]
Face --> TMDbMatch[TMDb Match]
Face --> SceneMeta[Scene Metadata]
YOLO --> SceneMeta
Face --> IdentityAgent[Identity Agent]
ASRX --> IdentityAgent
Cut --> Agent5W1H[5W1H Agent]
ASR --> Agent5W1H
Agent5W1H --> Phase2[Phase 2 Pack]
end
style Processors fill:#1a1a2e,stroke:#e94560
style Ingestion fill:#16213e,stroke:#0f3460
```
### Pipeline Completion Flow
The pipeline is **not complete** until both the 10 processors AND the 入庫 (ingestion) steps have finished. The worker polls every 3 seconds and only marks the job as `completed` when all ingestion steps verify OK.
```
10 processors done
↓ (job status stays "running")
Algorithm 1 Trigger: Rule 1 + Vectorize + Phase 1 Pack
↓ (job runs in parallel)
Algorithm 2 Trigger: Face Trace → TKG, Scene Metadata, Identity Agent, 5W1H Agent
↓ (poll checks every 3s)
Ingestion verification: rule1 ✓ vectorize ✓ rule3 ✓ face_trace ✓ tkg ✓ scene_meta ✓ 5w1h ✓
↓
job status = "completed"
```
### 10 Processor Stages
| # | Processor | Depends On | Description |
|---|-----------|------------|-------------|
| 1 | `Cut` | — | Scene boundary detection (PySceneDetect) |
| 2 | `ASR` | Cut | Automatic speech recognition (faster-whisper) |
| 3 | `ASRX` | ASR | Speaker diarization + ASR refinement |
| 4 | `YOLO` | — | Object detection (YOLOv8) |
| 5 | `OCR` | — | Optical character recognition |
| 6 | `Face` | — | Face detection + recognition (InsightFace + CoreML) |
| 7 | `Pose` | — | Pose estimation |
| 8 | `VisualChunk` | YOLO | Visual object chunking |
| 9 | `Story` | ASRX + Cut + YOLO + Face | Narrative scene summarization (LLM, with embedding) |
| 10 | `5W1H` | Story | Who/What/When/Where/Why extraction (LLM, with embedding) |
### 入庫 (Post-Processing / Ingestion)
These steps run after the 10 processors and are **required for pipeline completion**. The worker checks all of them before marking the job as done.
| # | Step | Triggers When | Verification |
|---|------|--------------|-------------|
| 1 | **Rule 1 Sentence Chunking** | ASR + ASRX done | `chunk` table has rows with `chunk_type = 'sentence'` |
| 2 | **Auto-Vectorize** | Rule 1 done | `chunk.embedding` IS NOT NULL for sentence chunks |
| 3 | **Phase 1 Pack** | Rule 1 done | `release_pack.py --phase 1` executed |
| 4 | **Rule 3 Scene Chunking** | All 10 processors done + Cut + ASR | `chunk` table has rows with `chunk_type = 'cut'` |
| 5 | **Face Trace** | All 10 processors done + Face | `face_detections.trace_id` IS NOT NULL |
| 6 | **Qdrant Face Sync** | Face Trace done | Qdrant face_embedding collection populated |
| 7 | **Trace Chunks** | Face Trace done | `chunk` table has rows with `chunk_type = 'trace'` |
| 8 | **TKG Builder** | Face Trace done | `tkg_nodes` + `tkg_edges` tables have rows |
| 9 | **TMDb Face Matching** | TMDb enabled + Face done | `face_detections.identity_id` IS NOT NULL |
| 10 | **Heuristic Scene Metadata** | Face + YOLO done | `{file_uuid}.scene_meta.json` exists on disk |
| 11 | **Identity Agent** | Face + ASRX done | `identities` with `source = 'identity_agent'` |
| 12 | **5W1H Agent** | Cut + ASR done | `chunk.summary_text` IS NOT NULL for cut chunks |
| 13 | **Release Pack** | 5W1H Agent done | `release_pack.py --phase 2` executed |
### Ingestion Status
Check real-time ingestion status for a file:
```bash
curl "$API/api/v1/stats/ingestion-status/{file_uuid}"
```
Returns per-step `done` / `pending` status with detail counts.
#### Example
```bash
curl "http://localhost:3003/api/v1/stats/ingestion-status/bd80fec9c42afb0307eb28f22c64c76a" | jq '.steps[] | {name, status, detail}'
```
#### Response
```json
{
"file_uuid": "bd80fec9c42afb0307eb28f22c64c76a",
"steps": [
{ "name": "rule1_sentence", "status": "pending", "detail": "0 sentence chunks" },
{ "name": "auto_vectorize", "status": "pending", "detail": "0 embedded" },
{ "name": "rule3_scene", "status": "pending", "detail": "0 scene chunks" },
{ "name": "face_trace", "status": "pending", "detail": "0 traces" },
{ "name": "trace_chunks", "status": "pending", "detail": "0 trace chunks" },
{ "name": "tkg", "status": "pending", "detail": "0 nodes, 0 edges" },
{ "name": "identity_match", "status": "pending", "detail": "0 identities" },
{ "name": "scene_metadata", "status": "pending", "detail": null },
{ "name": "5w1h", "status": "pending", "detail": "0 scenes with 5W1H" }
]
}
```
### Stats Endpoints
| Method | Endpoint | Auth | Description |
|--------|----------|------|-------------|
| GET | `/api/v1/stats/sftpgo` | No | SFTPGo service status |
| GET | `/api/v1/stats/ingestion-status/:file_uuid` | No | Per-file ingestion checklist |
### Configuration
### `POST /api/v1/config/cache`
**Auth**: Required
**Scope**: system-level
Toggle the Redis cache on or off.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `enabled` | boolean | Yes | `true` to enable, `false` to disable |
#### Example
```bash
curl -s -X POST "$API/api/v1/config/cache" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"enabled": false}'
```
### Unmounted Routes
The following routes are defined in source code but are **NOT** currently mounted in the router:
| Endpoint | Source file |
|----------|-------------|
| `/api/v1/search/persons` | `universal_search.rs` (not mounted) |
| `/api/v1/who` | `who.rs` |
| `/api/v1/who/candidates` | `who.rs` |
deliverable_v1.1.0/modules/11_error_codes.md
+57
View File
@@ -0,0 +1,57 @@
<!-- module: error_codes -->
<!-- description: Standard API error codes -->
<!-- depends: -->
## Error Response Format
All API errors follow this JSON structure:
```json
{
"success": false,
"error": {
"code": "E001_NOT_FOUND",
"message": "Resource not found",
"details": {"resource": "file_uuid", "value": "abc"}
}
}
```
## Error Code List
### Generic Errors (E0xx)
| Code | HTTP | Description |
|------|------|-------------|
| `E001_NOT_FOUND` | 404 | Resource not found (file, identity, chunk) |
| `E002_DUPLICATE` | 409 | Resource already exists |
| `E003_VALIDATION` | 400 | Request parameter validation failed |
| `E004_UNAUTHORIZED` | 401 | Invalid API key or token |
| `E005_INTERNAL` | 500 | Internal server error |
### Processor Errors (E1xx)
| Code | HTTP | Description |
|------|------|-------------|
| `E101_PROCESSOR_FAIL` | 500 | Python script execution failed |
| `E102_TIMEOUT` | 504 | Processing timeout |
| `E103_RESUME_FAIL` | 500 | Resume failed (checkpoint not found) |
| `E104_NO_VIDEO` | 400 | Video file path not found |
### Identity Errors (E2xx)
| Code | HTTP | Description |
|------|------|-------------|
| `E201_FACE_NOT_FOUND` | 404 | Face detection not found |
| `E202_MERGE_CONFLICT` | 409 | Identity merge conflict |
| `E203_CANDIDATE_EMPTY` | 404 | No candidates available for confirmation |
### TMDb Errors (E3xx)
| Code | HTTP | Description |
|------|------|-------------|
| `E301_TMDB_NO_KEY` | 400 | `TMDB_API_KEY` environment variable not set |
| `E302_TMDB_UNREACHABLE` | 502 | TMDb API unreachable or timed out |
| `E303_TMDB_CACHE_NOT_FOUND` | 200 | No local TMDb cache; run prefetch first |
| `E304_TMDB_PROBE_FAILED` | 500 | TMDb probe execution failed |
| `E305_TMDB_MOVIE_NOT_FOUND` | 404 | No matching TMDb movie found from filename |
deliverable_v1.1.0/modules/12_agent.md
+118
View File
@@ -0,0 +1,118 @@
# Agent Endpoints
Agent endpoints provide AI-powered capabilities including translation, identity analysis, and 5W1H extraction.
## POST /api/v1/agents/translate
Translate text between languages using Gemma4 (llama.cpp, port 8082).
### Request
```json
{
"text": "Hello, welcome to Momentry Core.",
"target_language": "Traditional Chinese",
"source_language": "English"
}
```
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `text` | string | ✅ | Text to translate |
| `target_language` | string | ✅ | Target language name (e.g. "Traditional Chinese", "Japanese") |
| `source_language` | string | ❌ | Source language (default: "auto") |
### Response
```json
{
"success": true,
"translated_text": "您好，歡迎使用 Momentry Core。",
"source_language_detected": "English",
"model_used": "google_gemma-4-26B-A4B-it-Q5_K_M.gguf"
}
```
### Supported Language Pairs (tested)
| Source | Target | Quality |
|--------|--------|---------|
| English | Traditional Chinese | ✅ |
| English | Japanese | ✅ |
| Chinese | English | ✅ |
| English | French | ✅ |
| Chinese | Japanese | ✅ |
### Model
- **Model**: Gemma4 26B (Q5_K_M)
- **Engine**: llama.cpp at `localhost:8082`
- **Endpoint**: `/v1/chat/completions` (OpenAI-compatible)
- **Temperature**: 0.1
- **Max tokens**: 1024
### Errors
| Status | Condition |
|--------|-----------|
| 500 | LLM unreachable or response parse failure |
| 401 | Missing/invalid auth |
---
## POST /api/v1/agents/5w1h/analyze
Extract 5W1H (Who, What, When, Where, Why, How) from a scene. Uses Gemma4 LLM on port 8082.
### Request
```json
{
"file_uuid": "3abeee81d94597629ed8cb943f182e94",
"scene_id": 42
}
```
### Response
```json
{
"success": true,
"5w1h": {
"who": ["Cary Grant"],
"what": ["discussing plans"],
"when": ["1963"],
"where": ["Paris"],
"why": ["vacation"],
"how": ["in person"]
}
}
```
## POST /api/v1/agents/5w1h/batch
Batch analyze all scenes in a file for 5W1H extraction. Uses the pipeline's `parent_chunk_5w1h.py --mode llm`.
### Request
```json
{
"file_uuid": "3abeee81d94597629ed8cb943f182e94"
}
```
## GET /api/v1/agents/5w1h/status
Get status of the 5W1H agent pipeline for a file.
---
## Embedding Model
| Detail | Value |
|--------|-------|
| **Model** | EmbeddingGemma-300m |
| **Endpoint** | `POST /v1/embeddings` on port 11436 |
| **Dimension** | 768 |
| **Used by** | `parent_chunk_5w1h.py --embed`, story, 5W1H, search |
deliverable_v1.1.0/modules/_template.md
+63
View File
@@ -0,0 +1,63 @@
# {Module Name} — API Workspace Module
> Use this template when adding or editing API endpoint documentation modules.
## Module Metadata
Every module MUST start with:
```markdown
<!-- module: <short_name> -->
<!-- description: One-line description of what this module covers -->
<!-- depends: <comma-separated list of dependency module names> -->
```
## Endpoint Template
Each endpoint MUST use this structure:
### `METHOD /path/to/endpoint`
**Auth**: Required / Optional / Public
**Scope**: file-level / identity-level / system-level
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `param1` | string | Yes | — | Description |
#### Example
```bash
# brief description of what this example demonstrates
curl -s -X METHOD "$API/path" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"param1": "value"}'
```
#### Response (200)
```json
{ "success": true }
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on 200 |
#### Error Codes
| Code | HTTP | When |
|------|------|------|
| E0xx | 4xx | Description |
## Rules
1. Each module file covers ONE topic group (e.g., `09_tmdb.md` = all TMDb endpoints)
2. Use `$API` and `$KEY` in all curl examples
3. Use `$FILE_UUID`, `$IDENTITY_UUID` variables for UUID examples
4. Module filename = `NN_topic.md` (NN = execution order, 01-99)
5. `depends` metadata = which modules must be assembled before this one
deliverable_v1.1.0/scripts/build_docs.py
+225
View File
@@ -0,0 +1,225 @@
#!/opt/homebrew/bin/python3.11
"""Build HTML documentation from module source files."""
import os, markdown, re, glob, shutil
MODULES_DIR = os.path.join(os.path.dirname(__file__), "..", "docs_v1.0", "API_WORKSPACE", "modules")
DOC_DIR = os.path.join(os.path.dirname(__file__), "..", "docs_v1.0", "doc")
DOC_DEV_DIR = os.path.join(os.path.dirname(__file__), "..", "docs_v1.0", "doc_developer")
# User-facing modules (no developer content)
USER_MODULES = {
"01_auth", "02_health", "03_register", "04_lookup", "05_process",
"06_search", "07_identity", "08_identity_agent", "08_media",
"09_tmdb", "10_pipeline", "12_agent",
}
def md_to_html(md_text: str) -> str:
"""Convert Markdown to HTML."""
html = markdown.markdown(md_text, extensions=['fenced_code', 'tables', 'codehilite'])
# Wrap tables
html = re.sub(r'<table>', '<table class="table">', html)
return html
def build_index(files, dev=False):
"""Build index.html."""
links = []
for fname in sorted(files):
name = os.path.splitext(fname)[0]
label = MODULE_LABELS.get(name, name.replace("_", " ").title())
if "｜" in label:
cn, en = label.split("｜", 1)
else:
cn, en = label, ""
html_name = fname.replace(".md", ".html")
links.append(f'<tr onclick="window.location=\'{html_name}\'" style="cursor:pointer"><td class="cn">{cn}</td><td class="en">{en}</td></tr>')
title = "Momentry API 開發者文件" if dev else "Momentry API 文件"
subtitle = "開發者專用" if dev else "API 參考手冊 — 登入後可瀏覽各模組文件"
return f"""<!DOCTYPE html>
<html lang="zh-TW">
<head>
<meta charset="UTF-8">
<title>{title}</title>
<style>
* {{ margin: 0; padding: 0; box-sizing: border-box; }}
body {{ font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }}
.container {{ max-width: 900px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }}
h1 {{ font-size: 28px; margin-bottom: 8px; }}
p.subtitle {{ color: #666; margin-bottom: 24px; }}
table {{ width: 100%; border-collapse: collapse; }}
tr {{ border-bottom: 1px solid #eee; }}
tr:last-child {{ border: none; }}
td {{ padding: 10px 0; }}
td.cn {{ width: 140px; font-weight: 600; color: #333; }}
td.en {{ color: #666; font-size: 14px; }}
a {{ color: #0066cc; text-decoration: none; display: block; }}
a:hover td {{ background: #f8f8f8; border-radius: 4px; }}
</style>
</head>
<body>
<div class="container">
<h1>{title}</h1>
<p class="subtitle">{subtitle}</p>
<table>{"".join(links)}</table>
</div>
</body>
</html>"""
MODULE_LABELS = {
"01_auth": "安全認證｜Authentication",
"02_health": "健康檢查｜Health",
"03_register": "檔案註冊｜File Registration",
"04_lookup": "檔案屬性查詢｜File Lookup",
"05_process": "處理流程｜Processing",
"06_search": "搜尋功能｜Search",
"07_identity": "身份識別｜Identity",
"08_identity_agent": "智能身份綁定｜Smart Identity Binding",
"08_media": "串流與截圖｜Streaming & Thumbnails",
"09_tmdb": "TMDb 整合｜TMDb Integration",
"10_pipeline": "生產線｜Pipeline",
"11_error_codes": "錯誤碼｜Error Codes",
"12_agent": "智慧代理｜AI Agents",
}
def build_html(md_text: str, title: str) -> str:
"""Wrap MD content in HTML page."""
content = md_to_html(md_text)
return f"""<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>{title} - Momentry API Docs</title>
<style>
* {{ margin: 0; padding: 0; box-sizing: border-box; }}
body {{ font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; color: #333; padding: 40px; }}
.container {{ max-width: 960px; margin: 0 auto; background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; }}
h1 {{ font-size: 24px; margin: 24px 0 12px; }}
h2 {{ font-size: 20px; margin: 20px 0 10px; color: #222; }}
h3 {{ font-size: 16px; margin: 16px 0 8px; color: #444; }}
p {{ line-height: 1.6; margin: 8px 0; }}
table {{ border-collapse: collapse; width: 100%; margin: 12px 0; font-size: 14px; }}
th, td {{ border: 1px solid #ddd; padding: 8px 12px; text-align: left; }}
th {{ background: #f0f0f0; font-weight: 600; }}
code {{ background: #f0f0f0; padding: 2px 6px; border-radius: 3px; font-size: 13px; }}
pre {{ background: #f8f8f8; border: 1px solid #ddd; border-radius: 6px; padding: 12px; overflow-x: auto; margin: 12px 0; }}
pre code {{ background: none; padding: 0; }}
a {{ color: #0066cc; }}
.back {{ display: inline-block; margin-bottom: 20px; color: #666; }}
.back:hover {{ color: #333; }}
</style>
</head>
<body>
<div class="container">
<a class="back" href="index.html">&larr; Back to index</a>
{content}
</div>
</body>
</html>"""
def login_page() -> str:
return """<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Login - Momentry Docs</title>
<style>
* { margin: 0; padding: 0; box-sizing: border-box; }
body { font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; background: #f5f5f5; display: flex; justify-content: center; align-items: center; height: 100vh; }
.card { background: white; border-radius: 12px; box-shadow: 0 2px 12px rgba(0,0,0,0.08); padding: 40px; width: 360px; }
h1 { font-size: 24px; margin-bottom: 24px; text-align: center; }
input { width: 100%; padding: 10px 12px; margin-bottom: 12px; border: 1px solid #ddd; border-radius: 6px; font-size: 14px; }
button { width: 100%; padding: 10px; background: #0066cc; color: white; border: none; border-radius: 6px; font-size: 16px; cursor: pointer; }
button:hover { background: #0052a3; }
.error { color: #cc0000; font-size: 13px; margin-bottom: 12px; display: none; }
</style>
</head>
<body>
<div class="card">
<h1>Momentry Docs</h1>
<form id="loginForm">
<input type="text" id="username" placeholder="Username" value="demo" required>
<input type="password" id="password" placeholder="Password" value="demo" required>
<div class="error" id="error">Invalid credentials</div>
<button type="submit">Login</button>
</form>
</div>
<script>
document.getElementById('loginForm').onsubmit = async function(e) {
e.preventDefault();
const resp = await fetch('/api/v1/auth/login', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({
username: document.getElementById('username').value,
password: document.getElementById('password').value
})
});
if (resp.ok) {
window.location.href = '/doc/index.html';
} else {
document.getElementById('error').style.display = 'block';
}
};
</script>
</body>
</html>"""
def main():
# Clean and recreate doc dirs
for d in [DOC_DIR, DOC_DEV_DIR]:
if os.path.exists(d):
shutil.rmtree(d)
os.makedirs(d)
md_files = sorted(glob.glob(os.path.join(MODULES_DIR, "*.md")))
if not md_files:
print(f"No MD files found in {MODULES_DIR}")
return
user_html = []
dev_html = []
for md_path in md_files:
with open(md_path) as f:
md_text = f.read()
fname = os.path.basename(md_path)
stem = os.path.splitext(fname)[0]
# Skip template
if stem == "_template":
continue
# Skip error codes (developer-only)
if stem == "11_error_codes":
dev_only = True
else:
dev_only = stem not in USER_MODULES
title = stem.replace("_", " ").title()
html = build_html(md_text, title)
if dev_only:
out_path = os.path.join(DOC_DEV_DIR, fname.replace(".md", ".html"))
with open(out_path, "w") as f:
f.write(html)
dev_html.append(fname)
print(f" [dev] {fname}")
else:
out_path = os.path.join(DOC_DIR, fname.replace(".md", ".html"))
with open(out_path, "w") as f:
f.write(html)
user_html.append(fname)
print(f" [doc] {fname}")
# Build indexes + login page
for d, files, label in [(DOC_DIR, user_html, "User"), (DOC_DEV_DIR, dev_html, "Dev")]:
index = build_index(files)
with open(os.path.join(d, "index.html"), "w") as f:
f.write(index)
with open(os.path.join(d, "login.html"), "w") as f:
f.write(login_page())
print(f" {label}: {len(files)} pages -> {d}")
if __name__ == "__main__":
main()
deliverable_v1.1.0/scripts/sync_dev_to_public.sh
Executable
+148
View File
@@ -0,0 +1,148 @@
#!/bin/bash
# sync_dev_to_public.sh — 比對 dev/public schema，同步 pipeline 資料
# Usage: ./sync_dev_to_public.sh [check|sync] [file_uuid]
PSQL="/opt/homebrew/opt/libpq/bin/psql"
set -euo pipefail
SCHEMA="${MOMENTRY_DB_SCHEMA:-dev}"
DB_URL="${DATABASE_URL:-postgres://accusys@localhost:5432/momentry}"
MODE="${1:-check}"
FILE_UUID="${2:-}"
TABLES=("videos" "chunk" "face_detections" "processor_results" "monitor_jobs"
"identities" "identity_bindings" "tkg_nodes" "tkg_edges")
TARGET="public"
if [ -z "$FILE_UUID" ]; then
echo "Usage: $0 [check|sync] <file_uuid>"
echo ""
echo "Examples:"
echo " $0 check bd80fec92b0b6963d177a2c55bf713e2"
echo " $0 sync bd80fec92b0b6963d177a2c55bf713e2"
exit 1
fi
echo "=== Schema Sync: $SCHEMA → $TARGET ==="
echo "File UUID: $FILE_UUID"
echo "Mode: $MODE"
echo ""
check_table() {
local table=$1
local col=$2
local src_count dev_count pub_count
dev_count=$($PSQL -At "$DB_URL" -c "SELECT COUNT(*) FROM ${SCHEMA}.${table} WHERE ${col} = '${FILE_UUID}';" 2>/dev/null || echo "ERROR")
pub_count=$($PSQL -At "$DB_URL" -c "SELECT COUNT(*) FROM ${TARGET}.${table} WHERE ${col} = '${FILE_UUID}';" 2>/dev/null || echo "ERROR")
if [ "$dev_count" = "ERROR" ] || [ "$pub_count" = "ERROR" ]; then
echo " ⚠️ $table — query error (table may not exist in $TARGET)"
return 1
fi
if [ "$dev_count" -eq "$pub_count" ]; then
echo " ✅ $table — $dev_count rows (match)"
return 0
else
echo " ❌ $table — dev=$dev_count pub=$pub_count (MISMATCH)"
return 1
fi
}
sync_table() {
local table=$1
local col=$2
local src_count dev_count pub_count
dev_count=$($PSQL -At "$DB_URL" -c "SELECT COUNT(*) FROM ${SCHEMA}.${table} WHERE ${col} = '${FILE_UUID}';" 2>/dev/null || echo "0")
pub_count=$($PSQL -At "$DB_URL" -c "SELECT COUNT(*) FROM ${TARGET}.${table} WHERE ${col} = '${FILE_UUID}';" 2>/dev/null || echo "0")
if [ "$dev_count" = "0" ]; then
echo " ⏭️ $table — dev has 0 rows, skipping"
return
fi
if [ "$dev_count" -eq "$pub_count" ]; then
echo " ✅ $table — already synced ($dev_count rows)"
return
fi
echo " 🔄 Syncing $table: dev=$dev_count → pub=$pub_count ..."
# Delete existing public rows, insert from dev
$PSQL "$DB_URL" -q -c "DELETE FROM ${TARGET}.${table} WHERE ${col} = '${FILE_UUID}';" 2>/dev/null || true
# Get columns list (excluding id for SERIAL)
COLS=$($PSQL -At "$DB_URL" -c "
SELECT string_agg(column_name, ', ' ORDER BY ordinal_position)
FROM information_schema.columns
WHERE table_schema='${SCHEMA}' AND table_name='${table}'
AND column_name != 'id'
AND is_updatable='YES';
")
$PSQL "$DB_URL" -q -c "
INSERT INTO ${TARGET}.${table} (${COLS})
SELECT ${COLS}
FROM ${SCHEMA}.${table}
WHERE ${col} = '${FILE_UUID}';
" 2>/dev/null && echo " ✅ $table synced" || echo " ❌ $table sync FAILED"
}
echo "=== Checking Tables ==="
echo ""
MISMATCH=0
for table in "${TABLES[@]}"; do
# Determine the UUID column name for each table
case "$table" in
videos) col="file_uuid" ;;
chunk) col="file_uuid" ;;
face_detections) col="file_uuid" ;;
processor_results) col="file_uuid" ;;
monitor_jobs) col="uuid" ;;
identities) col="uuid" ;; # identities.uuid is UUID type
identity_bindings) col="uuid" ;;
tkg_nodes) col="file_uuid" ;;
tkg_edges) col="file_uuid" ;;
*) col="file_uuid" ;;
esac
if ! check_table "$table" "$col"; then
MISMATCH=$((MISMATCH + 1))
fi
done
echo ""
if [ "$MISMATCH" -eq 0 ]; then
echo "✅ All tables in sync"
exit 0
fi
if [ "$MODE" != "sync" ]; then
echo "⚠️ $MISMATCH table(s) have mismatches. Run '$0 sync $FILE_UUID' to fix."
exit 1
fi
echo "=== Syncing Tables ==="
echo ""
for table in "${TABLES[@]}"; do
case "$table" in
videos) col="file_uuid" ;;
chunk) col="file_uuid" ;;
face_detections) col="file_uuid" ;;
processor_results) col="file_uuid" ;;
monitor_jobs) col="uuid" ;;
identities) col="uuid" ;;
identity_bindings) col="uuid" ;;
tkg_nodes) col="file_uuid" ;;
tkg_edges) col="file_uuid" ;;
*) col="file_uuid" ;;
esac
sync_table "$table" "$col"
done
echo ""
echo "✅ Sync complete"
deliverable_v1.1.0/scripts/update_qdrant_uuid.py
+174
View File
@@ -0,0 +1,174 @@
#!/usr/bin/env python3
"""批量更新 Qdrant collection 中的 file_uuid (舊→新)"""
import json
import subprocess
import sys
QDRANT_URL = "http://localhost:6333"
# UUID mapping: 舊 → 新
UUID_MAP = {
"aeed71342a899fe4b4c57b7d41bcb692": [
"bd80fec92b0b6963d177a2c55bf713e2",
],
}
# Collections to process
COLLECTIONS = [
"momentry_dev_v1",
"momentry_dev_stories",
"momentry_dev_voice",
"momentry_dev_rule1_v2",
"momentry_dev_faces",
"sentence_story",
"sentence_summary",
]
def qdrant_get(path: str) -> dict:
res = subprocess.run(
["curl", "-s", "-X", "GET", f"{QDRANT_URL}{path}"],
capture_output=True, text=True
)
return json.loads(res.stdout) if res.stdout.strip() else {}
def qdrant_post(path: str, body: dict) -> dict:
tmp = "/tmp/qdrant_post.json"
with open(tmp, "w") as f:
json.dump(body, f)
res = subprocess.run(
["curl", "-s", "-X", "POST", f"{QDRANT_URL}{path}",
"-H", "Content-Type: application/json", "-d", f"@{tmp}"],
capture_output=True, text=True
)
return json.loads(res.stdout) if res.stdout.strip() else {}
def qdrant_put(path: str, body: dict) -> dict:
tmp = "/tmp/qdrant_update.json"
with open(tmp, "w") as f:
json.dump(body, f)
res = subprocess.run(
["curl", "-s", "-X", "PUT", f"{QDRANT_URL}{path}",
"-H", "Content-Type: application/json", "-d", f"@{tmp}"],
capture_output=True, text=True
)
return json.loads(res.stdout) if res.stdout.strip() else {}
def scroll_all(collection: str, filter_old: dict) -> list:
"""Scroll all matching points from a collection"""
points = []
offset = None
while True:
body = {
"limit": 1000,
"with_payload": True,
"with_vector": True,
"filter": filter_old,
}
if offset:
body["offset"] = offset
result = qdrant_post(f"/collections/{collection}/points/scroll", body)
batch = result.get("result", {}).get("points", [])
points.extend(batch)
next_offset = result.get("result", {}).get("next_page_offset")
if next_offset is None:
break
offset = next_offset
return points
def update_points(collection: str, points: list, old_uuid: str, new_uuid: str):
"""Update file_uuid in payload for the given points"""
if not points:
return 0
updated = []
for p in points:
pl = p.get("payload", {})
# Check both 'uuid' and 'file_uuid' fields
changed = False
if pl.get("uuid") == old_uuid:
pl["uuid"] = new_uuid
changed = True
if pl.get("file_uuid") == old_uuid:
pl["file_uuid"] = new_uuid
changed = True
if changed:
updated.append({
"id": p["id"],
"vector": p["vector"],
"payload": pl,
})
if not updated:
return 0
# Update in batches of 500
total = len(updated)
for i in range(0, total, 500):
batch = updated[i:i+500]
result = qdrant_put(
f"/collections/{collection}/points?wait=true",
{"points": batch}
)
if result.get("status") != "ok":
print(f" Error at {i}: {result}")
return i
return total
def main():
for collection in COLLECTIONS:
# Check if collection exists
info = qdrant_get(f"/collections/{collection}")
if "result" not in info:
continue
for old_uuid, new_uuids in UUID_MAP.items():
for new_uuid in new_uuids:
# Scroll all points with this old UUID
filter_body = {
"must": [
{"should": [
{"key": "uuid", "match": {"value": old_uuid}},
{"key": "file_uuid", "match": {"value": old_uuid}},
]}
]
}
points = scroll_all(collection, filter_body)
if not points:
continue
print(f"{collection}: {len(points)} points with UUID {old_uuid[:8]}...")
updated = update_points(collection, points, old_uuid, new_uuid)
print(f" → {updated} points updated to {new_uuid[:8]}...")
# Verify
print("\n=== Verification ===")
for collection in COLLECTIONS:
for old_uuid, new_uuids in UUID_MAP.items():
for what, uuid in [("old", old_uuid), ("new", new_uuids[0])]:
filter_body = {
"must": [
{"should": [
{"key": "uuid", "match": {"value": uuid}},
{"key": "file_uuid", "match": {"value": uuid}},
]}
]
}
result = qdrant_post(
f"/collections/{collection}/points/count",
{"filter": filter_body}
)
cnt = result.get("result", {}).get("count", 0)
if cnt > 0:
print(f" {collection}: {cnt} points with {what} UUID")
print("✅ Done")
if __name__ == "__main__":
main()
docs/3002_3003_SEPARATION_STATUS.md
+70
View File
@@ -0,0 +1,70 @@
# 3002/3003 Schema Separation Status
Date: 2026-05-17
Status: ✅ Pipeline tables created in `public`; schema incompatibilities remain
## Summary
| Schema | Has pipeline tables | Has auth tables | Used by |
|--------|-------------------|-----------------|---------|
| `public` | ✅ (newly created) | ✅ (original) | 3002 (production) — currently using `dev` as workaround |
| `dev` | ✅ (full, working) | ✅ (synced) | 3003 (playground) |
## What Was Done
### Pipeline tables created in `public` schema (11 tables)
- `videos`, `chunk`, `chunk_vectors`, `cuts`, `frames`
- `monitor_jobs`, `processor_results`, `processor_versions`
- `parent_chunks`, `tkg_edges`, `tkg_nodes`
All include proper sequences, indexes, and constraints matching the `dev` schema.
## Remaining Blockers
### Schema incompatibilities between `dev` and `public`
| Table | dev cols | public cols | Status |
|-------|---------|------------|--------|
| identities | 17 | 16 | ⚠️ Different columns (e.g. `name` vs `real_name`/`actor_name`) |
| face_detections | 16 | 17 | ⚠️ Column count mismatch |
| identity_bindings | 7 | 8 | ⚠️ Column count mismatch |
| person_identities | 16 | 15 | ⚠️ Column count mismatch |
| pre_chunks | 19 | 10 | ⚠️ Significantly different |
| api_keys | 19 | 19 | ✅ Match |
| resources | 9 | 9 | ✅ Match |
| users | 8 | 8 | ✅ Match |
### Identities table key differences
- `public.identities` uses `real_name` + `actor_name` (old schema)
- `dev.identities` uses `name` (new unified schema)
- `dev.identities` has `tmdb_poster`, `file_uuid`, `face_embedding`, `voice_embedding`, `identity_embedding`
- `public.identities` only has `face_embedding`, `voice_embedding` (no `identity_embedding`)
## Options
### Option A: Full data migration (recommended for later)
1. Dump data from old public tables
2. Drop old public tables
3. Recreate from dev schema DDL
4. Migrate data with column mapping
5. Switch 3002 to `DATABASE_SCHEMA=public`
### Option B: Keep current workaround (simplest for now)
- 3002 continues with `DATABASE_SCHEMA=dev`
- 3003 uses `DATABASE_SCHEMA=dev`
- Both share the same schema, but have separate Redis key prefixes + ports
### Option C: Rename dev → public (requires downtime)
1. Stop all services
2. Rename `dev` schema to something else
3. Rename `public` to `public_old`
4. Rename `dev` to `public`
5. Update references
## Current Status
✅ Pipeline tables exist in both schemas
✅ auth tables (users, sessions, jwt_blacklist) exist in both
✅ Redis key prefixes separate (`momentry:` vs `momentry_dev:`)
⚠️ 3002 still uses `DATABASE_SCHEMA=dev` workaround
⛔ Shared tables need migration before 3002 can use `public` schema
docs/SEARCH_SCORE_IMPROVEMENT.md
+134
View File
@@ -0,0 +1,134 @@
# Search Scoring Improvement: Score-based Merge for search/smart
## 發現者
WordPress 前端專案（search-chat 頁面）
## 問題描述
### 症狀
跨語言搜尋結果不一致：
- 搜尋「槍」（中文）→ 回傳無關結果（如「讓T-shirt」、「靠直的後製神器」）
- 搜尋 `gun`（英文）→ 回傳 "So where's your gun?"、"He has a gun"
- 兩者應該找到相同語意主題的結果（武器相關片段），但實際回傳完全不同的集合
### 影響範圍
`GET/POST /api/v1/search/smart` endpoint
## 根因分析
### 1. Qdrant 語意搜尋本身是正確的
直接查詢 Qdrant 驗證：
```
cos(search_query: 槍, search_document: "So where's your gun?") = 0.6905
cos(search_query: 槍, search_document: "這是一把槍") = 0.8256
cos(search_query: gun, search_document: "So where's your gun?") = 0.7435
```
**embedding model (EmbeddingGemma-300m) 的 cross-lingual 對齊正常。**
### 2. 問題在 RRF 合併邏輯
`search/smart` 用 **RRF (Reciprocal Rank Fusion)** 合併三組結果：
```rust
let rrf_k = 60.0;
// RRF 貢獻 = 1 / (60 + rank + 1)
// Semantic rank 0: 貢獻 1/61 = 0.016
// Keyword rank 0: 貢獻 1/61 = 0.016
```
RRF 的權重只看**排名位置**，不看**實際相似度分數**。
- cosine similarity = 0.69 的語意結果 → RRF 貢獻 0.016
- ILIKE 隨便撈到的 keyword 匹配 → RRF 貢獻也是 0.016
- 兩者在排序中權重完全相等
### 3. Keyword (ILIKE) 對跨語言有害
- `ILIKE '%槍%'` 只找到中文文字包含「槍」的 chunks
- `ILIKE '%gun%'` 只找到英文文字包含 "gun" 的 chunks
- 這兩組結果在語意上完全不同，卻透過 RRF 被提升到與語意結果同權重
- 導致「槍」和 `gun` 的結果各自被自己的 ILIKE 匹配汙染
## 建議方案
### 核心原則
向量高信心度時應該優先。
### 合併方式
將 RRF 改為 score-based merge，各來源分數定義：
| 來源 | 分數 | 說明 |
|---|---|---|
| **Semantic (Qdrant)** | `cosine_similarity` (0~1) | 原始 Qdrant 分數，不加權 |
| **Identity** | 固定 `0.85` | 人名精準匹配，維持高度信心 |
| **Keyword (ILIKE)** | 固定 `0.5` | 降權至低分，只作為語意找不到時的補底 |
最終分數 = `max(semantic, keyword, identity)`
依最終分數降冪排序。
### 預期效果
| 情況 | 排序行為 |
|---|---|
| cosine > 0.5 的語意結果 | 排在 keyword 前面 ✅ |
| cosine 在 0.3~0.5 | 與 keyword 穿插（都不太確定，合理） |
| cosine < 0.3 | keyword 補底（語意沒找到，靠文字比對） |
| 跨語言查詢（槍 vs gun） | 各自的高分 cross-lingual 結果優先呈現 ✅ |
### 不建議的方案
- **不要用 weight-based average**（如 `0.7*semantic + 0.3*keyword`）：兩種模型的 score scale 不同，加權無法通用
- **不要保留 RRF 只調 k 值**：k 值調再高也無法區分品質，只能稀釋影響
## 修改範圍
### 檔案
`src/api/search.rs` 中的 `smart_search()` 函數
### 需要修改的區塊
1. **移除 RRF 常數**（`rrf_k = 60.0`）
2. **Semantic 結果**：保留 Qdrant 回傳的 `score`（已在 `h.score as f64` 取得）
3. **Keyword 結果**：固定設為 `0.5_f64`（忽略原本 `combined_score`）
4. **Identity 結果**：固定設為 `0.85_f64`（忽略原本硬編碼的 `0.85` 但保留值）
5. **排序邏輯**：改為 `max(semantic, keyword, identity)` 降冪
6. **輸出 similarity**：改為回傳最終分數，而非 `rrf_score`
### 注意事項
- Qdrant 回傳的 `score` 是 `f32`，需 cast 為 `f64`
- `keyword_results` 的 `combined_score` 實際上是 `1.0`（`search_bm25` 固定值），不應使用
- 修改後需 **`cargo build --release`** 再重啟 server
## 驗證測試
### 手動測試
```bash
# 1. 槍 vs gun 應該回傳相似主題
curl -X POST 'http://localhost:3002/api/v1/search/smart' \
-H 'X-API-Key: {KEY}' -H 'Content-Type: application/json' \
-d '{"query":"槍","limit":10}'
curl -X POST 'http://localhost:3002/api/v1/search/smart' \
-H 'X-API-Key: {KEY}' -H 'Content-Type: application/json' \
-d '{"query":"gun","limit":10}'
# 2. 確認 similarity 值為實際 cosine (e.g. 0.6~0.9) 而非 RRF 值 (~0.016)
```
### 預期結果
| Query | Top 結果應包含 |
|---|---|
| `槍` | gun 相關片段、「這是一把槍」、武器相關語意匹配 |
| `gun` | 與 `槍` 主題一致（都是武器） |
| `車` / `car` | 行車相關片段，非姓名含「車」的人物 |
| `So where's your gun?` | 自身為 top-1（self-match cosine ≈ 1.0） |
## 附錄：前端處理
WordPress 側 (`snippet #37`) 已配合修正：`mode=semantic` 不再疊加 `search/universal`（ILIKE）結果，僅回傳 `search/smart` 的輸出。這部分無需 backend 配合。
docs_v1.0/API_WORKSPACE/.gitignore
+2
View File
@@ -0,0 +1,2 @@
_build/
.DS_Store
docs_v1.0/API_WORKSPACE/README.md
+60
View File
@@ -0,0 +1,60 @@
# API Workspace
## Purpose
This directory is the **single source of truth** for all API documentation modules.
Generated outputs go to `../GUIDES/` as assembled deliverable documents.
## Workflow
```bash
# 1. Edit a module
vim modules/09_tmdb.md
# 2. Preview the generated output
make _build/API_ENDPOINTS.md
# 3. Check diff against current GUIDES/ content
make check
# 4. Deploy to GUIDES/
make deploy
# 5. Regenerate all
make all
```
## Directory Structure
```
API_WORKSPACE/
├── modules/ ← 11 module files (01_auth ... 11_error_codes)
├── configs/ ← 7 assembly recipies (.toml)
├── narratives/ ← narrative intros for specific output files
├── _build/ ← generated output (gitignored)
├── Makefile ← build targets
├── assemble_docs.sh ← assembly engine
└── README.md
```
## Available `make` Targets
| Target | Output |
|--------|--------|
| `make reference` | `_build/API_REFERENCE.md` |
| `make endpoints` | `_build/API_ENDPOINTS.md` |
| `make quickref` | `_build/API_QUICK_REFERENCE.md` |
| `make errors` | `_build/API_ERROR_CODES.md` |
| `make index` | `_build/API_INDEX.md` |
| `make marcom` | `_build/API_TRAINING_MARCOM.md` |
| `make tmdb` | `_build/TMDb_User_Guide.md` |
| `make all` | All of the above |
| `make deploy` | Copy `_build/*` → `../GUIDES/` |
| `make check` | `diff` against existing `../GUIDES/` files |
## Adding a New Endpoint
1. Add the endpoint to the appropriate module (e.g., `modules/XX_files.md`)
2. Follow the template in `modules/_template.md`
3. `make all && make check`
4. `make deploy`
docs_v1.0/API_WORKSPACE/modules/04_lookup.md
+281 -2
View File
@@ -1,5 +1,5 @@
<!-- module: lookup -->
<!-- description: File lookup by name and unregistration -->
<!-- description: File listing, lookup by name, file detail, faces, identities, JSON download, unregistration -->
<!-- depends: 01_auth, 03_register -->
## File Lookup
@@ -60,6 +60,285 @@ curl -s "$API/api/v1/files/lookup?file_name=charade" \
---
---
## File Listing
### `GET /api/v1/files`
**Auth**: Required
**Scope**: system-level
List all registered files with pagination. Optionally filter by status or fetch a specific file by UUID.
#### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 20 | Items per page |
| `status` | string | No | — | Filter by status: `registered`, `processing`, `completed`, `failed`, `indexed`, `checked_out` |
| `file_uuid` | string | No | — | Fetch a specific file (returns as single-item list) |
#### Example
```bash
# List all files (paginated)
curl -s "$API/api/v1/files?page=1&page_size=10" \
-H "X-API-Key: $KEY"
# Filter by status
curl -s "$API/api/v1/files?status=completed" \
-H "X-API-Key: $KEY"
# Fetch specific file
curl -s "$API/api/v1/files?file_uuid=$FILE_UUID" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"total": 42,
"page": 1,
"page_size": 10,
"data": [
{
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"file_name": "video.mp4",
"file_path": "/path/to/video.mp4",
"status": "completed"
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on 200 |
| `total` | integer | Total file count |
| `page` | integer | Current page |
| `page_size` | integer | Items per page |
| `data` | array | Array of file items |
| `data[].file_uuid` | string | 32-char hex UUID |
| `data[].file_name` | string | Registered file name |
| `data[].file_path` | string | Full filesystem path |
| `data[].status` | string | Processing status |
---
### `GET /api/v1/file/:file_uuid`
**Auth**: Required
**Scope**: file-level
Get detailed info for a specific registered file including metadata, duration, FPS, and probe data.
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"file_name": "video.mp4",
"file_path": "/path/to/video.mp4",
"status": "completed",
"duration": 120.5,
"fps": 24.0,
"metadata": {
"format": {"duration": "120.5", "size": "794863677"},
"streams": [{"codec_name": "h264", "width": 1920, "height": 1080}]
},
"created_at": "2026-05-16T12:00:00Z"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on 200 |
| `file_uuid` | string | 32-char hex UUID |
| `file_name` | string | Registered file name |
| `file_path` | string | Full filesystem path |
| `status` | string | Processing status |
| `duration` | float | Duration in seconds |
| `fps` | float | Frames per second |
| `metadata` | object | Full ffprobe metadata (probe.json) |
| `created_at` | string | Registration timestamp (ISO 8601) |
#### Error Codes
| HTTP | When |
|------|------|
| `404` | File UUID not found |
---
### `GET /api/v1/file/:file_uuid/identities`
**Auth**: Required
**Scope**: file-level
Get all identities present in a specific file with pagination.
#### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 20 | Items per page |
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/identities?page=1&page_size=50" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"fps": 24.0,
"total": 5,
"page": 1,
"page_size": 20,
"data": [
{
"identity_id": 1,
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"name": "Audrey Hepburn",
"metadata": {"source": "tmdb", "tmdb_id": 1234},
"face_count": 142,
"speaker_count": 8,
"start_frame": 100,
"end_frame": 5000,
"start_time": 4.17,
"end_time": 208.33,
"confidence": 0.87
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `data[].identity_id` | integer | Database identity ID |
| `data[].identity_uuid` | string/null | Global identity UUID (null if unbound) |
| `data[].name` | string | Identity name |
| `data[].metadata` | object | Source metadata (TMDb, etc.) |
| `data[].face_count` | integer/null | Number of face detections |
| `data[].speaker_count` | integer/null | Number of speaker segments |
| `data[].start_frame` | integer/null | First appearance frame |
| `data[].end_frame` | integer/null | Last appearance frame |
| `data[].start_time` | float/null | First appearance time (seconds) |
| `data[].end_time` | float/null | Last appearance time (seconds) |
| `data[].confidence` | float/null | Average detection confidence |
---
### `GET /api/v1/file/:file_uuid/faces`
**Auth**: Required
**Scope**: file-level
List all face detections in a specific file with pagination.
#### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 50 | Items per page |
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/faces?page=1&page_size=100" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"total": 1420,
"page": 1,
"page_size": 50,
"data": [
{
"face_id": "face_100",
"frame_number": 1200,
"timestamp": 50.0,
"bbox": [100, 50, 300, 400],
"confidence": 0.95,
"identity_id": 1,
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"trace_id": 2
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `data[].face_id` | string | Face detection ID |
| `data[].frame_number` | integer | Frame number in video |
| `data[].timestamp` | float | Timestamp in seconds |
| `data[].bbox` | array | Bounding box `[x1, y1, x2, y2]` |
| `data[].confidence` | float | Detection confidence |
| `data[].identity_id` | integer/null | Bound identity ID (null if unbound) |
| `data[].identity_uuid` | string/null | Bound identity UUID (null if unbound) |
| `data[].trace_id` | integer/null | Face trace ID (null if not traced) |
---
### `POST /api/v1/file/:file_uuid/json/:processor`
**Auth**: Required
**Scope**: file-level
Download raw JSON output for a specific processor.
#### Path Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File UUID |
| `processor` | string | Yes | Processor name: `cut`, `asrx`, `yolo`, `ocr`, `face`, `pose`, `story`, etc. |
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/json/face" \
-H "X-API-Key: $KEY" | jq '.frames | length'
```
#### Response (200)
Returns the raw JSON output of the specified processor. Structure varies by processor type.
#### Error Codes
| HTTP | When |
|------|------|
| `404` | JSON file not found |
| `500` | Failed to parse JSON |
---
## Unregister
### `POST /api/v1/unregister`
@@ -138,4 +417,4 @@ curl -s -X POST "$API/api/v1/unregister" \
| `401` | Missing or invalid API key |
---
*Updated: 2026-05-19 12:49:24*
*Updated: 2026-06-20 — Added file listing, file detail, file identities, file faces, and JSON download endpoints*
docs_v1.0/API_WORKSPACE/modules/05_process.md
+299 -10
View File
@@ -51,8 +51,8 @@ curl -s -X POST "$API/api/v1/file/$FILE_UUID/process" \
| `success` | boolean | Always true on 200 |
| `job_id` | integer | Monitor job ID (for job tracking) |
| `file_uuid` | string | 32-char hex UUID of the file |
| `status` | string | `"processing"` |
| `pids` | integer[] | Process IDs of started processors |
| `status` | string | `"queued"` — file enters the FIFO queue |
| `pids` | integer[] | Process IDs of started processors (empty for queued) |
| `message` | string | Human-readable status |
#### Error Responses
@@ -127,13 +127,15 @@ curl -s "$API/api/v1/file/$FILE_UUID/probe" -H "X-API-Key: $KEY"
---
### `GET /api/v1/progress/:file_uuid`
### `POST /api/v1/progress/:file_uuid`
**Auth**: Required
**Scope**: file-level
Get real-time processing progress for a file via Redis pub/sub. Includes per-processor status, current/total frames, ETA, and system resource stats.
**Note**: This endpoint uses **POST** method, not GET. The progress data is stored in Redis as a hash, and POST is used to retrieve the latest state.
#### Pipeline Order
| Order | Processor | Dependencies | Description |
@@ -143,18 +145,37 @@ Get real-time processing progress for a file via Redis pub/sub. Includes per-pro
| 3 | `asrx` | asr | Speaker diarization |
| 4 | `yolo` | — | Object detection |
| 5 | `ocr` | — | Text recognition |
| 6 | `face` | — | Face detection & embedding |
| 7 | `pose` | — | Pose estimation |
| 8 | `visual_chunk` | yolo | Visual scene chunks |
| 9 | `story` | asr, asrx, cut, yolo, face | Scene summaries (template) |
| 10 | `5w1h` | story | 5W1H analysis (Gemma4 LLM) |
| 6 | `face` | — | Face detection & embedding (8Hz sampling) |
| 7 | `face_trace` | face | Face tracking (IoU + embedding, assigns trace_id) |
| 8 | `pose` | face_trace | Pose expansion from face traces, inherits trace_id |
| 9 | `appearance` | pose | Appearance expansion from pose traces, inherits trace_id |
**Key Concepts:**
- **Face** = Identity anchor (who is this person?) — requires high-quality embedding
- **Pose** = Tracking (where is this person?) — extends tracking when face is occluded
- **Appearance** = Tracking (what do they look like?) — extends tracking when pose is occluded
**Trace ID Inheritance:**
```
Face trace (identity anchor)
↓ inherits trace_id
Pose expansion (tracking continuity)
↓ inherits trace_id
Appearance expansion (tracking continuity)
```
**Frame Count Relationship:**
```
face frames ≤ pose frames ≤ appearance frames
```
(Each level expands outward from the previous level's traces)
All processors except `story` and `5w1h` run concurrently when their dependencies are met. Story and 5W1H run sequentially after their prerequisites.
#### Example
```bash
curl -s "$API/api/v1/progress/$FILE_UUID" -H "X-API-Key: $KEY" | jq '{overall_progress, processors: [.processors[] | {processor_type, status}]}'
curl -s -X POST "$API/api/v1/progress/$FILE_UUID" -H "X-API-Key: $KEY" | jq '{overall_progress, processors: [.processors[] | {name, status}]}'
```
#### Response (200)
@@ -235,5 +256,273 @@ curl -s "$API/api/v1/jobs" -H "X-API-Key: $KEY" | jq '{count, jobs: [.jobs[] | {
| `page` | integer | Current page number |
| `page_size` | integer | Jobs per page |
### `GET /api/v1/job/:uuid`
**Auth**: Required
**Scope**: file-level
Get detailed information about a specific processing job, including its queue position.
#### Response (200)
```json
{
"id": 51,
"uuid": "c36f35685177c981aa139b66bbbccc5b",
"status": "queued",
"current_processor": null,
"progress_current": 0,
"progress_total": 0,
"processors": [],
"created_at": "2026-06-22 23:08:48.497018",
"started_at": null,
"updated_at": null,
"queue_position": 3
}
```
| Field | Type | Description |
|-------|------|-------------|
| `id` | integer | Monitor job ID |
| `uuid` | string | File UUID |
| `status` | string | `"pending"`, `"queued"`, `"running"`, `"completed"`, `"failed"` |
| `current_processor` | string | Currently active processor, or null |
| `progress_current` | integer | Current progress count |
| `progress_total` | integer | Total progress count |
| `processors` | array | Processor list |
| `created_at` | string | Job creation timestamp |
| `started_at` | string | Processing start timestamp, or null |
| `updated_at` | string | Last update timestamp, or null |
| `queue_position` | integer | Position in FIFO queue (null if not pending/queued) |
---
*Updated: 2026-05-19 12:49:24*
### Status Lifecycle
```
register ──→ pending
│
trigger (POST /process)
│
queued ←── queue_position counts jobs ahead
│
worker picks up
│
processing
│
┌────────┴────────┐
▼ ▼
completed failed
│
checkin ──→ indexed
checkout ──→ checked_out
```
| Status | Meaning |
|--------|---------|
| `pending` | File registered, not yet triggered |
| `queued` | Triggered, waiting for worker in FIFO queue |
| `processing` | Worker actively processing |
| `completed` | All processors finished successfully |
| `failed` | One or more essential processors failed |
| `indexed` | Post-processing checkin complete |
| `checked_out` | User checked out the file |
Queue order is FIFO (`created_at ASC`). The `GET /api/v1/job/:uuid` endpoint returns `queue_position` showing how many jobs are ahead.
### Frontend Status Mapping
When displaying file status in the frontend list (e.g. after `GET /api/v1/files/scan`), map the `status` field as follows:
| DB Status | Status Label | Filter: 待處理 | Filter: 處理中 | Count: pendingCount | Count: processingCount |
|-----------|-------------|----------------|----------------|---------------------|-----------------------|
| `unregistered` | 未註冊 | No | No | No | No |
| `registered` | 待處理 | **Yes** | No | **Yes** | No |
| `pending` | 待處理 | **Yes** | No | **Yes** | No |
| `queued` | 排隊中 | **Yes** | **Yes** | **Yes** | **Yes** |
| `processing` | 處理中 | No | **Yes** | No | **Yes** |
| `completed` | 已完成 | No | No | No | No |
| `failed` | 處理失敗 | No | No | No | No |
| `indexed` | 已入庫 | No | No | No | No |
**`queued` 的特殊處理**：
- `statusLabel` → 顯示「排隊中」，加 `ms-badge-warn` 樣式（黃色）
- `filterPending` → 應包含 `queued`，讓它在「待處理」filter 可見
- `pendingCount` + `processingCount` → 兩者都應包含 `queued`，因它既是「待處理」也是「正在排隊」
- 在 `refreshAllStatus` / `loadFiles` 中，如果檔案狀態是 `queued`，應顯示簡單的排隊訊息（無需 polling progress）
- 當 worker pickup 後，狀態會變為 `processing`，此時 `refreshAllStatus` 會自動偵測到並開始 polling progress
- 也可以提供一個「queue_position」顯示：呼叫 `GET /api/v1/job/:uuid` 取得排在第幾位
---
### `GET /api/v1/file/:file_uuid/processor-counts`
**Auth**: Required
**Scope**: file-level
Get counts of processor JSON output files. See `15_tkg.md` for full documentation.
---
## Pipeline Steps (Manual)
These endpoints execute individual pipeline steps. They are typically called by the worker automatically, but can be invoked manually for debugging or re-processing.
### `POST /api/v1/file/:file_uuid/store-asrx`
**Auth**: Required
**Scope**: file-level
Store ASRX diarization results as chunk records in the database. Converts ASRX segments into searchable chunk entries.
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/store-asrx" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"message": "ASRX chunks stored",
"file_uuid": "3a6c1865..."
}
```
---
### `POST /api/v1/file/:file_uuid/rule1`
**Auth**: Required
**Scope**: file-level
Execute Rule 1 pipeline step. Applies rule-based chunking to create structured chunk records from processor outputs.
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/rule1" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"message": "Rule 1 complete: 45 chunks",
"file_uuid": "3a6c1865...",
"chunks": 45
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on 200 |
| `message` | string | Human-readable completion message |
| `file_uuid` | string | 32-char hex UUID |
| `chunks` | integer | Number of chunks produced |
---
### `POST /api/v1/file/:file_uuid/vectorize`
**Auth**: Required
**Scope**: file-level
Generate vector embeddings for all chunks of a file and store them in Qdrant for semantic search.
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/vectorize" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"message": "Vectorization complete",
"file_uuid": "3a6c1865..."
}
```
---
### `POST /api/v1/file/:file_uuid/phase1`
**Auth**: Required
**Scope**: file-level
Execute Phase 1 of the post-processing pipeline. Combines store-asrx, rule1, and vectorize into a single step.
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/phase1" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"message": "Phase 1 complete",
"file_uuid": "3a6c1865..."
}
```
---
### `POST /api/v1/file/:file_uuid/complete`
**Auth**: Required
**Scope**: file-level
Mark a video as fully processed. Updates the video status to `completed` and finalizes all pipeline state.
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/complete" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"message": "Video marked as completed",
"file_uuid": "3a6c1865..."
}
```
---
### Pipeline Step Order
```
process (trigger)
│
├─→ cut, yolo, ocr, face, pose, asrx (parallel processors)
│
├─→ store-asrx (store diarization as chunks)
│
├─→ rule1 (rule-based chunking)
│
├─→ vectorize (embed chunks to Qdrant)
│
└─→ complete (mark done)
```
Phase 1 (`/phase1`) combines store-asrx + rule1 + vectorize into one call.
---
*Updated: 2026-06-23 — Added queued status, FIFO queue order, queue_position in job detail, frontend status mapping table*
docs_v1.0/API_WORKSPACE/modules/06_search.md
+244 -20
View File
@@ -1,5 +1,5 @@
<!-- module: search -->
<!-- description: Vector search, BM25, smart search, universal search, visual search -->
<!-- description: Vector search, BM25, smart search, universal search, LLM reranked search, frame search -->
<!-- depends: 01_auth -->
## Search APIs
@@ -7,7 +7,7 @@
### `POST /api/v1/search/smart`
**Auth**: Required
**Scope**: file-level
**Scope**: global / file-level
Semantic vector search using EmbeddingGemma-300m. Generates a query embedding via EmbeddingGemma (port 11436), then searches pgvector `story_parent` and `llm_parent` chunks by cosine similarity.
@@ -15,13 +15,22 @@ Semantic vector search using EmbeddingGemma-300m. Generates a query embedding vi
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `file_uuid` | string | Yes | — | File UUID to search within |
| `query` | string | Yes | — | Search text |
| `file_uuid` | string | No | — | File UUID to search within. If omitted, searches all files (global search) |
| `limit` | integer | No | 5 | Max results to return |
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 5 | Items per page |
#### Example
#### Example (Global Search)
```bash
curl -s -X POST "$API/api/v1/search/smart" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $JWT" \
-d '{"query": "Audrey Hepburn"}'
```
#### Example (File-specific Search)
```bash
curl -s -X POST "$API/api/v1/search/smart" \
@@ -37,6 +46,7 @@ curl -s -X POST "$API/api/v1/search/smart" \
"query": "Audrey Hepburn",
"results": [
{
"file_uuid": "a6fb22eebefaef17e62af874997c5944",
"parent_id": 1087822,
"scene_order": 1087822,
"start_frame": 104438,
@@ -54,12 +64,16 @@ curl -s -X POST "$API/api/v1/search/smart" \
}
```
| Field | Type | Description |
|-------|------|-------------|
| `results[].file_uuid` | string | File UUID where result was found |
---
### `POST /api/v1/search/universal`
**Auth**: Required
**Scope**: file-level
**Scope**: global / file-level
Multi-type BM25 full-text search across chunks, frames, and persons. Uses PostgreSQL `tsvector`.
@@ -68,13 +82,22 @@ Multi-type BM25 full-text search across chunks, frames, and persons. Uses Postgr
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `query` | string | Yes | — | Search text |
| `file_uuid` | string | No | — | Restrict to specific file |
| `file_uuid` | string | No | — | Restrict to specific file. If omitted, searches all files (global search) |
| `types` | string[] | No | `["chunk","frame","person"]` | Search types |
| `limit` | integer | No | 10 | Max results per type |
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 20 | Items per page |
#### Example
#### Example (Global Search)
```bash
curl -s -X POST "$API/api/v1/search/universal" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $JWT" \
-d '{"query": "Cary Grant"}'
```
#### Example (File-specific Search)
```bash
curl -s -X POST "$API/api/v1/search/universal" \
@@ -90,6 +113,7 @@ curl -s -X POST "$API/api/v1/search/universal" \
"results": [
{
"type": "chunk",
"file_uuid": "a6fb22eebefaef17e62af874997c5944",
"chunk_id": "bd80fec92b0b6963d177a2c55bf713e2_2",
"chunk_type": "story_child",
"start_frame": 5103,
@@ -98,6 +122,25 @@ curl -s -X POST "$API/api/v1/search/universal" \
"end_time": 213.64,
"text": "[213s-214s] Cary Grant: \"Olá!\"",
"score": 0.9
},
{
"type": "frame",
"file_uuid": "a6fb22eebefaef17e62af874997c5944",
"frame_number": 5105,
"timestamp": 212.72,
"score": 0.7,
"objects": null,
"ocr_texts": null,
"faces": null
},
{
"type": "person",
"file_uuid": "a6fb22eebefaef17e62af874997c5944",
"identity_id": 12,
"identity_uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"name": "Cary Grant",
"appearance_count": 542,
"score": 0.95
}
],
"total": 20,
@@ -105,35 +148,216 @@ curl -s -X POST "$API/api/v1/search/universal" \
}
```
| Field | Type | Description |
|-------|------|-------------|
| `results[].type` | string | Result type: `chunk`, `frame`, or `person` |
| `results[].file_uuid` | string | File UUID where result was found (all types) |
---
### `POST /api/v1/search/frames`
**Auth**: Required
**Scope**: file-level
**Scope**: global / file-level
Search face detection frames by identity name or trace ID.
Search frames by YOLO objects, OCR text, face IDs, or pose detections. Filters frames based on visual content detected during processing.
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `file_uuid` | string | No | — | Restrict to specific file |
| `object_class` | string | No | — | Filter by YOLO object class (e.g., `person`, `car`, `dog`) |
| `ocr_text` | string | No | — | Filter by OCR text content (ILIKE match) |
| `face_id` | string | No | — | Filter by face detection ID |
| `time_range` | [float, float] | No | — | Filter by time range `[start_secs, end_secs]` |
| `limit` | integer | No | 100 | Max results |
#### Example
```bash
# Search for frames containing "person" objects
curl -s -X POST "$API/api/v1/search/frames" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"file_uuid": "'"$FILE_UUID"'", "object_class": "person", "limit": 20}'
# Search for frames with specific OCR text
curl -s -X POST "$API/api/v1/search/frames" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"file_uuid": "'"$FILE_UUID"'", "ocr_text": "hello", "time_range": [10.0, 30.0]}'
```
#### Response (200)
```json
{
"frames": [
{
"frame_number": 1200,
"timestamp": 50.0,
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"objects": [{"class": "person", "confidence": 0.95, "bbox": [100, 50, 300, 400]}],
"ocr_texts": ["Hello World"],
"faces": [{"face_id": "face_42", "confidence": 0.88}],
"pose_persons": [{"trace_id": 2, "bbox": [120, 60, 280, 380]}]
}
],
"total": 15
}
```
| Field | Type | Description |
|-------|------|-------------|
| `frames` | array | Array of matching frame objects |
| `frames[].frame_number` | integer | Frame number in video |
| `frames[].timestamp` | float | Timestamp in seconds |
| `frames[].file_uuid` | string | File UUID |
| `frames[].objects` | array/null | YOLO detections in this frame |
| `frames[].ocr_texts` | array/null | OCR text strings in this frame |
| `frames[].faces` | array/null | Face detections in this frame |
| `frames[].pose_persons` | array/null | Pose-detected persons in this frame |
| `total` | integer | Total matching frame count |
---
### `POST /api/v1/search/identity_text`
### `POST /api/v1/search/llm-smart`
**Auth**: Required
**Scope**: file-level
**Scope**: global / file-level
Search text chunks spoken by a specific identity.
Smart search with LLM re-ranking. First fetches candidate results via RRF (Reciprocal Rank Fusion) using the existing smart search, then uses an LLM (Gemma4 on port 8000) to re-rank candidates by relevance to the query.
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `query` | string | Yes | — | Search text |
| `file_uuid` | string | No | — | File UUID to search within |
| `limit` | integer | No | 10 | Max results to return |
#### Pipeline
```
1. smart_search → fetch N candidates (limit × 3, clamped 10-20)
2. LLM rerank → re-order by relevance using Gemma4
3. trim → return top `limit` results
```
#### Example
```bash
curl -s -X POST "$API/api/v1/search/llm-smart" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"query": "two people having a conversation about business", "limit": 5}'
```
#### Response (200)
```json
{
"query": "two people having a conversation about business",
"results": [
{
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"parent_id": 1234,
"scene_order": 1234,
"start_frame": 5000,
"end_frame": 5200,
"fps": 24.0,
"start_time": 208.3,
"end_time": 216.7,
"summary": "[208s-217s, 9s] Two people discussing project timeline...",
"similarity": 0.72
}
],
"page": 1,
"page_size": 5,
"strategy": "llm_reranked"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `strategy` | string | Always `"llm_reranked"` for this endpoint |
| `results` | array | Re-ranked search results (same format as smart search) |
#### Fallback
If LLM reranking fails (model unavailable, timeout), falls back to RRF order without error.
---
### Visual Search
| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/v1/search/visual` | Search visual chunks |
| POST | `/api/v1/search/visual/class` | Search by object class |
| POST | `/api/v1/search/visual/density` | Search by object density |
| POST | `/api/v1/search/visual/combination` | Search by object combination |
| POST | `/api/v1/search/visual/stats` | Visual chunk statistics |
**Auth**: Required
**Scope**: global / file-level
Search text chunks → find associated identities. Returns chunks where face detections overlap with text content.
#### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `q` | string | Yes | — | Search text (ILIKE match) |
| `file_uuid` | string | No | — | Restrict to specific file. If omitted, searches all files (global search) |
| `limit` | integer | No | 50 | Max results |
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 50 | Items per page |
#### Example (Global Search)
```bash
curl -s "$API/api/v1/search/identity_text?q=love" -H "X-API-Key: $KEY"
```
#### Example (File-specific Search)
```bash
curl -s "$API/api/v1/search/identity_text?file_uuid=$FILE_UUID&q=love" -H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"total": 5,
"results": [
{
"file_uuid": "a6fb22eebefaef17e62af874997c5944",
"chunk_id": "llm_parent_..._256_270",
"start_time": 256.256,
"end_time": 270.228,
"text_content": "...lack of affection...",
"identity_id": 9,
"identity_name": "Audrey Hepburn",
"identity_source": "tmdb",
"trace_id": 94
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `results[].file_uuid` | string | File UUID where chunk was found |
| `results[].identity_id` | integer | Identity ID if face was detected |
| `results[].trace_id` | integer | Face trace ID |
---
### Visual Search (Planned)
| Method | Endpoint | Status | Description |
|--------|----------|--------|-------------|
| POST | `/api/v1/search/visual` | Not implemented | Search visual chunks |
| POST | `/api/v1/search/visual/class` | Not implemented | Search by object class |
| POST | `/api/v1/search/visual/density` | Not implemented | Search by object density |
| POST | `/api/v1/search/visual/combination` | Not implemented | Search by object combination |
| POST | `/api/v1/search/visual/stats` | Not implemented | Visual chunk statistics |
#### Embedding Model
@@ -145,4 +369,4 @@ Search text chunks spoken by a specific identity.
| **Storage** | pgvector (`chunk.embedding` column) |
---
*Updated: 2026-05-19 12:49:24*
*Updated: 2026-06-20 — Added llm-smart search, completed frames search documentation, marked visual search as planned*
docs_v1.0/API_WORKSPACE/modules/07_identity.md
+584 -109
View File
@@ -70,7 +70,16 @@ curl -s "$API/api/v1/identity/$IDENTITY_UUID" -H "X-API-Key: $KEY"
**Auth**: Required
**Scope**: identity-level
Delete an identity permanently.
Delete an identity permanently. All face detections bound to this identity are unbound (`identity_id` set to `NULL`). The identity JSON file is deleted from disk.
#### History & Undo/Redo
Every DELETE records a full snapshot of the identity and its unbound faces. See [`14_identity_history.md`](14_identity_history.md#4-delete-history--undoredo) for:
- Undo via `POST /api/v1/identity/:identity_uuid/undo` — recreates identity and re-binds faces
- Redo via `POST /api/v1/identity/:identity_uuid/redo` — re-deletes the identity
**Note**: Delete undo/redo reuses the same endpoints as PATCH undo/redo. The endpoint automatically detects whether the identity was deleted (undo) or needs to be re-deleted (redo) based on the history record.
---
@@ -129,124 +138,75 @@ curl -s -X PATCH "$API/api/v1/identity/$IDENTITY_UUID" \
| HTTP | When |
|------|------|
| `400` | No fields to update or invalid UUID format |
| `404` | Identity not found |
| `500` | Database error |
#### History & Undo/Redo
Every bind records a before/after snapshot. See [`14_identity_history.md`](14_identity_history.md#2-bindunbindtrace-history--undoredo) for:
- `POST /api/v1/identity/:identity_uuid/bind/undo` — Revert a bind
- `POST /api/v1/identity/:identity_uuid/bind/redo` — Reapply an undone bind
- `GET /api/v1/identity/:identity_uuid/bind/history` — Query bind operations
---
### `GET /api/v1/identity/:identity_uuid/files`
## Metadata (Embedded JSON)
**Auth**: Required
**Scope**: identity-level
The `identities.metadata` column is a **JSONB** field that stores arbitrary structured data alongside the identity's core fields (name, status, identity_type). No schema is enforced — any valid JSON object is accepted.
Get all files where this identity appears. Returns per-file summary including face count, confidence, and appearance time range.
### Merge Behavior
#### Example
| Operation | Strategy | Example |
|-----------|----------|---------|
| **PATCH** | Shallow top-level merge: `COALESCE(metadata,'{}'::jsonb) \|\| $1::jsonb` | Sending `{"tmdb_rating": 8.5}` only adds/overwrites `tmdb_rating`; all other existing keys are preserved. |
| **mergeinto** | Recursive deep merge — nested sub-keys are merged individually, not replaced wholesale | Target has `{"tmdb": {"biography": "..."}}`, source has `{"tmdb": {"birthday": "1904-01-18"}}` → result is `{"tmdb": {"biography": "...", "birthday": "1904-01-18"}}`. |
| **Upload (`POST`)** | Direct overwrite — the entire `metadata` field is replaced with the request value. | |
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/files" -H "X-API-Key: $KEY"
```
### Validation
---
| Scenario | Result |
|----------|--------|
| PATCH with non-object metadata (`string`, `array`, `number`, `null`) | `400 Bad Request: "metadata must be a JSON object"` |
| mergeinto with non-object metadata | Accepted (mergeinto validates at application level) |
| Upload with non-object metadata | Accepted (upload replaces directly) |
### `GET /api/v1/identity/:identity_uuid/faces`
### Conventional Keys
**Auth**: Required
**Scope**: identity-level
| Key | Type | Writer | Purpose |
|-----|------|--------|---------|
| `aliases` | `[{locale, name}]` | PATCH, mergeinto | Multilingual display names (see [Alias System](#alias-system-bcp-47-locale-tags)) |
| `merged_into` | `{uuid, at}` | mergeinto | Marks an identity as merged (undo mechanism reads this) |
| `tmdb_*` | various | TMDb probe | Movie metadata (biography, birthday, known_for, etc.). Written only when `MOMENTRY_TMDB_PROBE_ENABLED=true`. |
| `source` | string | mergeinto | Tagged on aliases/metadata when added by merge (`"merge"` value) |
Get all face detection records associated with this identity.
Custom keys are fully supported — no registration required.
#### Example
### Search Coverage
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/faces" -H "X-API-Key: $KEY"
```
The identity search endpoint (`GET /api/v1/identity/search`) matches across three scopes:
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | File where face was detected |
| `frame_number` | integer | Frame number of detection |
| `face_id` | string | Face ID (format: `face_{frame_number}`) |
| `confidence` | float | Detection confidence |
1. `i.name` — exact and ILIKE against display name
2. `jsonb_array_elements(i.metadata->'aliases')->>'name'` — locale-tagged alias names
3. `i.metadata::text ILIKE $1` — raw string search across the entire JSON blob (all keys, all values)
---
This means searching for `"1904-01-18"` or `"biography"` will match identities whose metadata contains those strings anywhere.
### `GET /api/v1/identity/:identity_uuid/chunks`
### History Snapshots
**Auth**: Required
**Scope**: identity-level
Every `identity_history` record captures the **full metadata** in both `before_snapshot` and `after_snapshot` (as part of the complete identity JSONB dump). Undo restores the identity row — including metadata — to the `before_snapshot` state.
Get all text chunks (sentences) spoken while this identity's face was on screen. Useful for finding what a person said.
For merge operations, the MongoDB merge history records `metadata_fields_added` and `metadata_fields_added_paths` (dot-separated paths like `"tmdb.biography"`). Merge undo removes only those specific paths, preserving subsequent manual edits to other metadata keys.
#### Example
### Best Practices
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/chunks" -H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"data": [
{
"id": 0,
"file_uuid": "bd80fec92b0b6963d177a2c55bf713e2",
"chunk_id": "bd80fec92b0b6963d177a2c55bf713e2_2",
"chunk_type": "sentence",
"start_frame": 5103,
"end_frame": 5127,
"fps": 24.0,
"start_time": 212.64,
"end_time": 213.64,
"text_content": "[213s-214s] Cary Grant: \"Olá!\""
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | File identifier |
| `chunk_id` | string | Sentence chunk identifier |
| `start_frame` | integer | Frame-accurate start position |
| `end_frame` | integer | Frame-accurate end position |
| `fps` | float | Frames per second |
| `start_time` | float | Start time in seconds |
| `end_time` | float | End time in seconds |
| `text_content` | string | Spoken text content |
---
### `POST /api/v1/identity/:identity_uuid/bind`
**Auth**: Required
**Scope**: identity-level
Bind a face detection to an identity. Associates the face trace with the identity for future search and recognition.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File where face is detected |
| `face_id` | string | Yes | Face ID (format: `{frame}_{idx}`) |
#### Side Effects
- 清除該 face detection row 的 `stranger_id`（設為 NULL）
- 不影響 `identities` 表中原有的 stranger auto-identity 記錄
#### Example
```bash
curl -s -X POST "$API/api/v1/identity/$IDENTITY_UUID/bind" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"file_uuid": "'"$FILE_UUID"'", "face_id": "1_5"}'
```
| Guideline | Reason |
|-----------|--------|
| Deep nesting is allowed in metadata | All metadata merge operations use `jsonb_deep_merge()` — nested sub-keys are merged recursively, not replaced wholesale |
| Use `aliases` for display names | Frontend has built-in locale fallback logic (see [Alias System](#alias-system-bcp-47-locale-tags)) |
| Avoid >1MB per identity | Metadata is included in search indexing (`metadata::text ILIKE`); large blobs degrade query performance |
| Don't rely on metadata ordering | JSONB preserves insertion order but PostgreSQL does not guarantee it across operations |
| No LLM/Gemma4 agent writes to metadata | Only API endpoints (PATCH, mergeinto, upload) and TMDb probe modify `identities.metadata` |
---
@@ -295,6 +255,10 @@ curl -s -X POST "$API/api/v1/identity/$IDENTITY_UUID/bind/trace" \
| `404` | Identity not found |
| `500` | Database error |
#### History & Undo/Redo
Trace bind operations share the same history/undo/redo system as single-face binds. See [`14_identity_history.md`](14_identity_history.md#2-bindunbindtrace-history--undoredo) for endpoints.
---
### `GET /api/v1/identity/:identity_uuid/traces`
@@ -382,6 +346,13 @@ Unbind a face detection from an identity. Removes the identity association from
- 被 unbind 的 face 不會自動成為 stranger
- 要重新標記為 stranger 需重新跑 Agent API（`identity/analyze`）
#### History & Undo/Redo
Unbind records a before/after snapshot. See [`14_identity_history.md`](14_identity_history.md#2-bindunbindtrace-history--undoredo) for:
- `POST /api/v1/identity/:identity_uuid/bind/undo` — Revert an unbind
- `POST /api/v1/identity/:identity_uuid/bind/redo` — Reapply an undone unbind
---
### `POST /api/v1/identity/:identity_uuid/mergeinto`
@@ -391,6 +362,13 @@ Unbind a face detection from an identity. Removes the identity association from
Transfer all face bindings from this identity to another identity, then optionally delete or mark the source as merged.
#### Two Merge Cases
| Case | Description | Undo/Redo Support |
|------|-------------|-------------------|
| **stranger → identity** | Merge an auto-generated stranger identity into a known identity (TMDb or user-defined) | ✅ 24hr undo/redo |
| **identity A → identity B** | Merge two known identities (e.g., duplicate entries) | ✅ 24hr undo/redo |
#### Request Parameters
| Field | Type | Required | Default | Description |
@@ -402,8 +380,12 @@ Transfer all face bindings from this identity to another identity, then optional
- 轉移所有 `face_detections.identity_id` 到目標 identity
- 同時清除所有被轉移 rows 的 `stranger_id`
- 將 source name 加入 target aliases (with `source: "merge"` tag)
- 將 source aliases 加入 target aliases (if not already present)
- 將 source metadata fields 加入 target metadata (if not already present)
- `keep_history: true`（預設）：source identity 設為 `status='merged'`，保留記錄
- `keep_history: false`：**刪除** source identity 及其 identity JSON 檔案
- **記錄 merge history 到 MongoDB**（支援 undo/redo）
#### Example
@@ -411,7 +393,7 @@ Transfer all face bindings from this identity to another identity, then optional
curl -s -X POST "$API/api/v1/identity/$SOURCE_UUID/mergeinto" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"into_uuid": "'"$TARGET_UUID"'", "keep_history": false}'
-d '{"into_uuid": "'"$TARGET_UUID"'", "keep_history": true}'
```
#### Response (200)
@@ -419,11 +401,23 @@ curl -s -X POST "$API/api/v1/identity/$SOURCE_UUID/mergeinto" \
```json
{
"success": true,
"message": "Merged 'stranger_13894' into 'Louis Viret' (52 faces transferred, source deleted)",
"data": { "faces_transferred": 52 }
"message": "Merged 'stranger_13894' into 'Louis Viret' (52 faces transferred, history kept)",
"data": {
"merge_id": "550e8400-e29b-41d4-a716-446655440000",
"faces_transferred": 52,
"aliases_added": 1,
"metadata_fields_added": 2
}
}
```
| Field | Type | Description |
|-------|------|-------------|
| `merge_id` | string | Unique merge operation ID (for undo) |
| `faces_transferred` | integer | Number of face detections transferred |
| `aliases_added` | integer | Number of aliases added to target |
| `metadata_fields_added` | integer | Number of metadata fields added to target |
#### Error Responses
| HTTP | When |
@@ -433,25 +427,189 @@ curl -s -X POST "$API/api/v1/identity/$SOURCE_UUID/mergeinto" \
---
### `GET /api/v1/identities/search`
### `POST /api/v1/identity/merge/:merge_id/undo`
**Auth**: Required
**Scope**: identity-level
Search identities by name (ILIKE search). Returns matching identity records.
Undo a merge operation within 24 hours. Restores the source identity and reverts face bindings.
#### Undo Behavior
| Action | Description |
|--------|-------------|
| Restore source identity | If `keep_history=true`: restore status to `confirmed`<br>If `keep_history=false`: recreate identity from MongoDB snapshot |
| Restore faces | Transfer faces back to source identity |
| Remove aliases from target | Remove aliases with `source: "merge"` tag |
| Remove metadata fields from target | Remove fields that were added from source |
| **Preserve manual changes** | Keep aliases/metadata manually added after merge |
#### Example
```bash
curl -s "$API/api/v1/identities/search?q=Cary" -H "X-API-Key: $KEY"
curl -s -X POST "$API/api/v1/identity/merge/550e8400-e29b-41d4-a716-446655440000/undo" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"message": "Undo merge completed: 'stranger_13894' restored, 52 faces reverted",
"data": {
"source_identity_restored": {
"uuid": "a9a90105...",
"name": "stranger_13894",
"status": "confirmed"
},
"faces_reverted": 52,
"aliases_removed_from_target": 1,
"metadata_fields_removed_from_target": 2
}
}
```
#### Error Responses
| HTTP | When |
|------|------|
| `400` | Undo deadline expired (>24hr) or already undone |
| `404` | Merge record not found |
| `500` | Database error |
---
### `POST /api/v1/identity/merge/:merge_id/redo`
**Auth**: Required
**Scope**: identity-level
Redo a previously undone merge operation. See [`14_identity_history.md`](14_identity_history.md#post-apiv1identitymergemerge_idredo) for full details.
---
### `GET /api/v1/identity/merge/history`
**Auth**: Required
**Scope**: identity-level
Query merge history records from MongoDB.
#### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `source_uuid` | string | No | — | Filter by source identity UUID |
| `target_uuid` | string | No | — | Filter by target identity UUID |
| `merge_id` | string | No | — | Filter by specific merge ID |
| `undone` | bool | No | — | Filter by undone status |
| `page` | int | No | 1 | Page number |
| `page_size` | int | No | 20 | Items per page |
#### Example
```bash
curl -s "$API/api/v1/identity/merge/history?page=1&page_size=10" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"total": 5,
"page": 1,
"page_size": 10,
"results": [
{
"merge_id": "550e8400-e29b-41d4-a716-446655440000",
"source_name": "stranger_13894",
"target_name": "Louis Viret",
"faces_transferred": 52,
"merged_at": "2026-05-27T10:00:00Z",
"undo_deadline": "2026-05-28T10:00:00Z",
"undone": false,
"undo_expired": false
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `name` | string | Identity name |
| `source` | string | Identity source |
| `tmdb_id` | integer | TMDb ID (if source = tmdb) |
| `file_uuid` | string | Associated file |
| `merge_id` | string | Unique merge operation ID |
| `source_name` | string | Source identity name |
| `target_name` | string | Target identity name |
| `faces_transferred` | integer | Number of faces transferred |
| `merged_at` | datetime | When merge occurred |
| `undo_deadline` | datetime | 24hr deadline for undo |
| `undone` | bool | Whether merge was undone |
| `undo_expired` | bool | Whether undo deadline passed |
---
### `GET /api/v1/identities/search`
**Auth**: Required
**Scope**: global / file-level
Search identity name → find associated chunks. Searches identity name and aliases, returns identities with their associated text chunks.
#### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `q` | string | Yes | — | Search text (ILIKE match on name and aliases) |
| `file_uuid` | string | No | — | Restrict to specific file. If omitted, searches all files (global search) |
| `limit` | integer | No | 50 | Max results |
#### Example (Global Search)
```bash
curl -s "$API/api/v1/identities/search?q=Audrey" -H "X-API-Key: $KEY"
```
#### Example (File-specific Search)
```bash
curl -s "$API/api/v1/identities/search?q=Audrey&file_uuid=$FILE_UUID" -H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"total": 5,
"results": [
{
"identity_id": 9,
"name": "Audrey Hepburn",
"source": "tmdb",
"tmdb_id": 1932,
"file_uuid": "a6fb22eebefaef17e62af874997c5944",
"trace_id": 41,
"chunk_id": "llm_parent_..._204_207",
"start_time": 204.162,
"text_content": "...confrontation..."
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `results[].identity_id` | integer | Identity ID |
| `results[].name` | string | Identity name |
| `results[].source` | string | Identity source (`tmdb`, `user_defined`, etc.) |
| `results[].tmdb_id` | integer | TMDb person ID (if source = tmdb) |
| `results[].file_uuid` | string | File where identity appears |
| `results[].trace_id` | integer | Face trace ID |
| `results[].chunk_id` | string | Associated chunk ID |
| `results[].start_time` | float | Chunk start time |
| `results[].text_content` | string | Chunk text content |
---
@@ -571,6 +729,322 @@ curl -s "$API/api/v1/identity/$IDENTITY_UUID/profile-image" \
---
## Identity Related Data
### `GET /api/v1/identity/:identity_uuid/files`
**Auth**: Required
**Scope**: identity-level
List all files containing this identity.
#### Example
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/files" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"total": 3,
"files": [
{
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"file_name": "video1.mp4",
"face_count": 142,
"first_appearance": 4.17,
"last_appearance": 208.33
}
]
}
```
---
### `GET /api/v1/identity/:identity_uuid/chunks`
**Auth**: Required
**Scope**: identity-level
List all chunks associated with this identity (chunks where the identity's face appears).
#### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 20 | Items per page |
#### Example
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/chunks?page=1&page_size=50" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"total": 45,
"page": 1,
"page_size": 20,
"chunks": [
{
"chunk_id": "chunk_1",
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"start_time": 4.17,
"end_time": 8.33,
"text": "[4s-8s] Hello, how are you?",
"chunk_type": "story_child"
}
]
}
```
---
### `GET /api/v1/identity/:identity_uuid/faces`
**Auth**: Required
**Scope**: identity-level
List all face detections for this identity.
#### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 50 | Items per page |
#### Example
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/faces?page=1&page_size=100" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"total": 1420,
"page": 1,
"page_size": 50,
"faces": [
{
"face_id": "face_100",
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"frame_number": 1200,
"timestamp": 50.0,
"bbox": [100, 50, 300, 400],
"confidence": 0.95,
"trace_id": 2
}
]
}
```
---
### `GET /api/v1/identity/:identity_uuid/status`
**Auth**: Required
**Scope**: identity-level
Get processing/status info for an identity.
#### Example
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/status" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"name": "Audrey Hepburn",
"status": "confirmed",
"face_count": 1420,
"file_count": 3,
"has_embedding": true,
"has_profile_image": true
}
```
---
### `GET /api/v1/identity/:identity_uuid/json`
**Auth**: Required
**Scope**: identity-level
Get the raw identity JSON file (same format as identity.json on disk).
#### Example
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/json" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"version": 1,
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"name": "Audrey Hepburn",
"identity_type": "people",
"source": "tmdb",
"status": "confirmed",
"tmdb_id": 1234,
"tmdb_profile": "https://image.tmdb.org/...",
"metadata": {},
"file_bindings": [
{"file_uuid": "d3f9ae8e...", "trace_ids": [0, 1, 2], "face_count": 142}
]
}
```
---
---
### `POST /api/v1/file/:file_uuid/pending-person`
**Auth**: Required
**Scope**: file-level
Create a manually managed "pending person" under a specific file. A pending person is an identity with `status='pending'` and `source='manual'`, used for unmatched traces that the user wants to manually label before a full identity resolution.
Optionally binds a list of trace IDs to this new identity.
#### Request
```json
{
"trace_ids": [100, 150, 200],
"name": "Mystery Man #1"
}
```
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `trace_ids` | array[int] | No | `[]` | Trace IDs to bind to this pending person |
| `name` | string | No | `"Person N"` | Human-readable name. Auto-generated if omitted |
#### Example
```bash
# Create pending person with name and no traces
curl -s -X POST "$API/api/v1/file/$FILE_UUID/pending-person" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"name": "Unknown Woman #2", "trace_ids": []}'
# Create pending person with auto-name and bind traces
curl -s -X POST "$API/api/v1/file/$FILE_UUID/pending-person" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"trace_ids": [100, 150, 200]}'
```
#### Response (200)
```json
{
"success": true,
"message": "Created pending person: Mystery Man #1 (uuid: 4d96b25b-68f0-4c52-b238-d69f7dfd588b)",
"data": {
"identity_uuid": "4d96b25b-68f0-4c52-b238-d69f7dfd588b",
"identity_id": 55,
"name": "Mystery Man #1",
"bound_traces": 0
}
}
```
| Field | Type | Description |
|-------|------|-------------|
| `identity_uuid` | string | UUID of the newly created pending identity |
| `identity_id` | integer | Internal ID of the new identity |
| `name` | string | Display name |
| `bound_traces` | integer | Number of traces bound |
#### Side Effects
- Creates an `identities` row with `status='pending'`, `source='manual'`, `file_uuid=<file_uuid>`
- If `trace_ids` provided: `UPDATE face_detections SET identity_id = ...` for matching traces
- If `trace_ids` provided: TKG face_track nodes get `identity_id` / `identity_name` in properties
- Identity JSON file synced to disk
---
### `GET /api/v1/file/:file_uuid/pending-persons`
**Auth**: Required
**Scope**: file-level
List all pending persons for a file.
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/pending-persons" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"message": "Found 2 pending persons for c36f35685177c981aa139b66bbbccc5b",
"data": [
{
"identity_uuid": "232ecd08-a2bf-4bd0-bd25-0bd8fb7a7dae",
"identity_id": 56,
"name": "Person 2",
"created_at": "2026-06-23 17:13:23",
"trace_count": 3,
"bound_traces": null
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `identity_uuid` | string | Identity UUID |
| `identity_id` | integer | Internal identity ID |
| `name` | string | Display name |
| `created_at` | string | Creation timestamp |
| `trace_count` | integer | Number of face traces bound to this pending person |
| `bound_traces` | array[int] | List of bound trace IDs (currently null, reserved for future expansion) |
#### Notes
- Pending persons are normal `identities` rows with `status='pending'` — they can be promoted to confirmed via `PATCH /api/v1/identity/:identity_uuid` (`{"status": "confirmed"}`)
- They can be merged into known identities via `POST /api/v1/identity/:identity_uuid/mergeinto`
- Use `GET /api/v1/identity/:identity_uuid/traces` to get detailed trace info for each pending person
---
## Alias System (BCP 47 Locale Tags)
Identity aliases support multilingual display names. Aliases are stored in `metadata.aliases` as an array of `{locale, name}` objects.
@@ -628,4 +1102,5 @@ PATCH /api/v1/identity/:identity_uuid
This **replaces** the entire `aliases` array. To add to existing aliases, include all existing entries in the request.
---
*Updated: 2026-05-25
*Updated: 2026-07-21 — Fixed bind/unbind TKG update to match both trace_N and face_track_N external_id formats*
*Updated: 2026-06-20 — Added identity files, chunks, faces, status, and JSON endpoints*
docs_v1.0/API_WORKSPACE/modules/08_identity_agent.md
+60 -1
View File
@@ -65,4 +65,63 @@ curl -s -X POST "$API/api/v1/agents/identity/match-from-trace" \
```
---
*Updated: 2026-05-19 12:49:24*
### `POST /api/v1/agents/identity/confirm`
**Auth**: Required
**Scope**: file-level
Confirm identity binding for a trace. This marks the trace as confirmed in TKG, updates face_detections, adds to _seeds, and optionally triggers Round 2 propagation.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | Video file UUID |
| `trace_id` | integer | Yes | Face trace ID to confirm |
| `identity_id` | integer | Yes | Identity internal ID |
| `identity_uuid` | string | Yes | Identity UUID |
| `name` | string | Yes | Identity name |
| `propagate` | boolean | No | Auto-trigger Round 2 matching (default: true) |
#### Example
```bash
curl -s -X POST "$API/api/v1/agents/identity/confirm" \
-H "Authorization: Bearer $JWT" \
-H "Content-Type: application/json" \
-d '{"file_uuid": "'"$FILE_UUID"'", "trace_id": 10, "identity_id": 42, "identity_uuid": "'"$IDENTITY_UUID"'", "name": "Cary Grant", "propagate": false}'
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "384b0ff44aaaa1f1",
"trace_id": 10,
"identity_uuid": "a9a90105...",
"name": "Cary Grant",
"steps": {
"tkg_updated": true,
"qdrant_updated": 150,
"pg_updated": 150,
"seed_added": true
},
"propagation": {
"matched": 5,
"message": "Propagation completed"
}
}
```
#### Side Effects
1. TKG face_track node status → 'confirmed'
2. Qdrant _faces: identity_uuid added to payload
3. PG face_detections: identity_id set
4. Trace centroid added to _seeds (source='propagation')
5. Round 2 matching triggered (if propagate=true)
---
*Updated: 2026-06-26 00:30:00*
docs_v1.0/API_WORKSPACE/modules/08_media.md
+108 -1
View File
@@ -427,4 +427,111 @@ Both endpoints support time range extraction, but serve different use cases:
| **Frame number** | Zero-based (`frame=0` = first frame of video) |
---
*Updated: 2026-05-19 12:49:24*
### `GET /api/v1/file/:file_uuid/stranger/:stranger_id/representative-face`
**Auth**: Required
**Scope**: file-level
Get the representative face for a stranger (unidentified face trace).
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/stranger/1/representative-face" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"stranger_id": 1,
"face_count": 85,
"representative": {
"frame_number": 5000,
"timestamp_secs": 208.33,
"bbox": {"x": 200, "y": 100, "width": 150, "height": 150},
"confidence": 0.92,
"quality_score": 20700,
"blur_score": 8.5
}
}
```
---
### `GET /api/v1/file/:file_uuid/stranger/:stranger_id/thumbnail`
**Auth**: Required
**Scope**: file-level
Extract the best face image for a stranger as JPEG (320×320).
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/stranger/1/thumbnail" \
-H "X-API-Key: $KEY" -o stranger_1_face.jpg
```
#### Response
- **200**: `image/jpeg` binary data (320×320 cropped face)
- **404**: File or stranger not found
---
### `GET /api/v1/file/:file_uuid/chunk/:chunk_id/thumbnail`
**Auth**: Required
**Scope**: file-level
Get thumbnail for a specific chunk. Extracts the representative frame for the chunk's time range.
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/chunk/chunk_1/thumbnail" \
-H "X-API-Key: $KEY" -o chunk_1.jpg
```
#### Response
- **200**: `image/jpeg` binary data
- **404**: File or chunk not found
---
### `GET /api/v1/media-proxy`
**Auth**: Required
**Scope**: system-level
Proxy request to fetch media from external URLs. Useful for loading profile images or thumbnails from external services (TMDb, etc.) without exposing the external URL to the client.
#### Query Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `url` | string | Yes | External URL to proxy |
#### Example
```bash
curl -s "$API/api/v1/media-proxy?url=https://image.tmdb.org/t/p/w500/abc123.jpg" \
-H "X-API-Key: $KEY" -o tmdb_profile.jpg
```
#### Response
- **200**: Proxied media data (Content-Type from external source)
- **400**: Missing or invalid URL parameter
- **500**: External request failed
---
---
*Updated: 2026-06-20 — Added stranger endpoints, chunk thumbnail, and media proxy*
docs_v1.0/API_WORKSPACE/modules/09_tmdb.md
+90 -1
View File
@@ -108,5 +108,94 @@ curl -s -X POST "$API/api/v1/resource/tmdb/check" \
}
```
### `POST /api/v1/tmdb/fetch`
**Auth**: Required
**Scope**: system-level
Fetch TMDb data by filename, create identities with profile images and embeddings. Similar to prefetch+probe combined, but also downloads profile images and generates embeddings.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `filename` | string | Yes | Movie filename to search TMDb for |
#### Example
```bash
curl -s -X POST "$API/api/v1/tmdb/fetch" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"filename": "charade.mp4"}'
```
#### Response (200)
```json
{
"success": true,
"movie_title": "Charade (1963)",
"tmdb_id": 1234,
"identities_created": 15,
"profile_images_downloaded": 12
}
```
---
*Updated: 2026-05-19 12:49:24*
### `POST /api/v1/agents/tmdb/match/:file_uuid`
**Auth**: Required
**Scope**: file-level
Match TMDb identities to face traces using Qdrant vector similarity. Compares face embeddings against TMDb identity embeddings to find the best matches.
#### Example
```bash
curl -s -X POST "$API/api/v1/agents/tmdb/match/$FILE_UUID" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"matches": [
{
"trace_id": 0,
"identity_uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"identity_name": "Audrey Hepburn",
"confidence": 0.92,
"tmdb_id": 1234
}
],
"total_matches": 5
}
```
| Field | Type | Description |
|-------|------|-------------|
| `matches[].trace_id` | integer | Face trace ID |
| `matches[].identity_uuid` | string | Matched TMDb identity UUID |
| `matches[].identity_name` | string | Identity display name |
| `matches[].confidence` | float | Cosine similarity score (0.0–1.0) |
| `matches[].tmdb_id` | integer | TMDb person ID |
| `total_matches` | integer | Total successful matches |
---
### TMDb Auto-Match
When `MOMENTRY_TMDB_PROBE_ENABLED=true`, the worker automatically runs TMDb matching during the post-process phase:
1. **Register phase**: Searches TMDb by filename, creates identities with `tmdb_id`/`tmdb_profile`
2. **Post-process phase**: Matches detected faces against TMDb identities via cosine similarity using Qdrant
No manual API call needed if auto-match is enabled.
---
*Updated: 2026-06-20 — Added tmdb/fetch and tmdb/match endpoints*
docs_v1.0/API_WORKSPACE/modules/10_pipeline.md
+13 -10
View File
@@ -42,6 +42,7 @@ These steps run after the 10 processors and are **required for pipeline completi
| # | Step | Triggers When | Verification |
|---|------|--------------|-------------|
| 1 | **Rule 1 Sentence Chunking** | ASR + ASRX done | `chunk` table has rows with `chunk_type = 'sentence'` |
| 1.1 | **Rule 1 OCR Chunks** | OCR done | OCR pre_chunks grouped into sentence chunks |
| 2 | **Auto-Vectorize** | Rule 1 done | `chunk.embedding` IS NOT NULL for sentence chunks |
| 3 | **Phase 1 Pack** | Rule 1 done | `release_pack.py --phase 1` executed |
| 4 | **Rule 3 Scene Chunking** | All 10 processors done + Cut + ASR | `chunk` table has rows with `chunk_type = 'cut'` |
@@ -81,15 +82,17 @@ curl "$API/api/v1/stats/ingestion-status/bd80fec9c42afb0307eb28f22c64c76a" | jq
{
"file_uuid": "bd80fec9c42afb0307eb28f22c64c76a",
"steps": [
{ "name": "rule1_sentence", "status": "pending", "detail": "0 sentence chunks" },
{ "name": "auto_vectorize", "status": "pending", "detail": "0 embedded" },
{ "name": "rule3_scene", "status": "pending", "detail": "0 scene chunks" },
{ "name": "face_trace", "status": "pending", "detail": "0 traces" },
{ "name": "trace_chunks", "status": "pending", "detail": "0 trace chunks" },
{ "name": "tkg", "status": "pending", "detail": "0 nodes, 0 edges" },
{ "name": "identity_match", "status": "pending", "detail": "0 identities" },
{ "name": "scene_metadata", "status": "pending", "detail": null },
{ "name": "5w1h", "status": "pending", "detail": "0 scenes with 5W1H" }
{ "name": "rule1_sentence", "status": "done", "detail": "35 sentence chunks" },
{ "name": "rule1_ocr", "status": "done", "detail": "30 OCR frames" },
{ "name": "rule1_ocr_chunks", "status": "done", "detail": "3 OCR-only chunks" },
{ "name": "auto_vectorize", "status": "pending", "detail": "0 embedded" },
{ "name": "rule3_scene", "status": "pending", "detail": "0 scene chunks" },
{ "name": "face_trace", "status": "pending", "detail": "0 traces" },
{ "name": "trace_chunks", "status": "pending", "detail": "0 trace chunks" },
{ "name": "tkg", "status": "pending", "detail": "0 nodes, 0 edges" },
{ "name": "identity_match", "status": "pending", "detail": "0 identities" },
{ "name": "scene_metadata", "status": "pending", "detail": null },
{ "name": "5w1h", "status": "pending", "detail": "0 scenes with 5W1H" }
]
}
```
@@ -117,7 +120,7 @@ The following routes are defined in source code but are **NOT** currently mounte
| Endpoint | Source file |
|----------|-------------|
| `/api/v1/search/persons` | `universal_search.rs` (not mounted) |
| `/api/v1/search/people` | `universal_search.rs` (mounted) |
| `/api/v1/who` | `who.rs` |
| `/api/v1/who/candidates` | `who.rs` |
docs_v1.0/API_WORKSPACE/modules/14_identity_history.md
+696
View File
@@ -0,0 +1,696 @@
<!-- module: identity_history -->
<!-- description: Identity operation history, undo, and redo (PATCH, bind, unbind, bind_trace, mergeinto) -->
<!-- depends: 01_auth, 07_identity -->
## Identity Operation History
Every mutation on an identity automatically records a before/after snapshot. Use undo/redo to revert or reapply changes, and history to inspect the operation log.
Three independent undo/redo systems exist:
| System | Storage | Operations Covered |
|--------|---------|-------------------|
| **PATCH** | PostgreSQL `identity_history` | `update` |
| **Bind** | PostgreSQL `identity_history` | `bind`, `unbind`, `bind_trace` |
| **Merge** | MongoDB `identity_merge_history` | mergeinto |
| **Delete** | PostgreSQL `identity_history` | `delete` |
---
### 1. PATCH History & Undo/Redo
#### Overview
| Property | Value |
|----------|-------|
| Storage | PostgreSQL `identity_history` table |
| Snapshot | Full identity record (all fields) before and after each PATCH |
| Max records | 256 per identity (oldest auto-deleted when limit exceeded) |
| Undo steps | Unlimited (no expiry, no step limit) |
| Redo stack | Cleared on new PATCH (`is_undone=true` + `operation='update'` records are deleted) |
##### Stack Model
```
PATCH 1 → PATCH 2 → PATCH 3 (undo stack, is_undone=false)
↓ undo
PATCH 1 → PATCH 2 (undo stack)
PATCH 3 (redo stack, is_undone=true)
↓ redo
PATCH 1 → PATCH 2 → PATCH 3 (undo stack)
```
A new PATCH after undo clears only the operation='update' redo stack (PATCH 3 is lost). Bind/merge redo stacks are not affected.
---
#### `POST /api/v1/identity/:identity_uuid/undo`
**Auth**: Required
**Scope**: identity-level
Undo the most recent PATCH operations. Restores the identity's `before_snapshot` and marks the history records as undone.
##### Request (JSON)
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `steps` | integer | No | `1` | Number of undo steps to apply (max records undone in one call) |
##### Behavior
- Queries `is_undone=false` records with `operation='update'`, ordered by `created_at DESC`
- Restores `name`, `identity_type`, `source`, `status`, `metadata`, `tmdb_id`, `tmdb_profile` from the last record's `before_snapshot`
- Marks the undone records as `is_undone=true` with `undone_at=NOW()`
- Syncs `identity.json` to disk
- Updates `_index.json` if name changed
##### Example
```bash
curl -s -X POST "$API/api/v1/identity/$IDENTITY_UUID/undo" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"steps": 1}'
```
##### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"undone_count": 1,
"current_state": {
"id": 9,
"uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"name": "Cary Grant",
"identity_type": "people",
"source": "tmdb",
"status": "confirmed",
"metadata": {},
"tmdb_id": 112,
"tmdb_profile": null
}
}
```
| Field | Type | Description |
|-------|------|-------------|
| `undone_count` | integer | Number of history records undone |
| `current_state` | object | Full identity state after undo |
##### Error Responses
| HTTP | When |
|------|------|
| `400` | No undo operations available |
| `404` | Identity not found |
| `500` | Database error |
---
#### `POST /api/v1/identity/:identity_uuid/redo`
**Auth**: Required
**Scope**: identity-level
Redo previously undone PATCH operations. Restores the identity's `after_snapshot` and marks the history records as no longer undone.
##### Request (JSON)
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `steps` | integer | No | `1` | Number of redo steps to apply |
##### Behavior
- Queries `is_undone=true` records with `operation='update'`, ordered by `created_at DESC`
- Restores all identity fields from the last record's `after_snapshot`
- Marks records as `is_undone=false` with `undone_at=NULL`
- Syncs `identity.json` to disk
- Updates `_index.json` if name changed
##### Example
```bash
curl -s -X POST "$API/api/v1/identity/$IDENTITY_UUID/redo" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"steps": 1}'
```
##### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"redone_count": 1,
"current_state": {
"id": 9,
"uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"name": "John Smith",
"identity_type": "people",
"source": "tmdb",
"status": "confirmed",
"metadata": { "aliases": [...] },
"tmdb_id": 112,
"tmdb_profile": null
}
}
```
| Field | Type | Description |
|-------|------|-------------|
| `redone_count` | integer | Number of history records redone |
| `current_state` | object | Full identity state after redo |
##### Error Responses
| HTTP | When |
|------|------|
| `400` | No redo operations available |
| `404` | Identity not found |
| `500` | Database error |
---
#### `GET /api/v1/identity/:identity_uuid/history`
**Auth**: Required
**Scope**: identity-level
Query the PATCH operation history for an identity. Returns paginated records with undo/redo stack counts (filtered to `operation='update'`).
##### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `page` | integer | No | `1` | Page number (1-indexed) |
| `limit` | integer | No | `20` | Items per page (max 100) |
##### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"total": 5,
"undo_stack_count": 3,
"redo_stack_count": 2,
"results": [
{
"history_id": 42,
"operation": "update",
"is_undone": false,
"created_at": "2026-05-27T12:00:00Z",
"undone_at": null
},
{
"history_id": 41,
"operation": "update",
"is_undone": true,
"created_at": "2026-05-27T11:30:00Z",
"undone_at": "2026-05-27T13:00:00Z"
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `total` | integer | Total PATCH history records for this identity |
| `undo_stack_count` | integer | Records available for undo (`is_undone=false`) |
| `redo_stack_count` | integer | Records available for redo (`is_undone=true`) |
| `results[].history_id` | integer | History record ID |
| `results[].operation` | string | Operation type (`"update"` for PATCH) |
| `results[].is_undone` | boolean | Whether the operation has been undone |
| `results[].created_at` | string | When the PATCH was applied |
| `results[].undone_at` | string | When the undo occurred (null if not undone) |
##### Example
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/history?page=1&limit=10" \
-H "X-API-Key: $KEY"
```
##### Error Responses
| HTTP | When |
|------|------|
| `404` | Identity not found |
| `500` | Database error |
---
### 2. Bind/Unbind/Trace History & Undo/Redo
All three operations (`bind`, `unbind`, `bind_trace`) share a single history table and undo/redo stack.
#### Bind Operation Overview
| Property | Value |
|----------|-------|
| Storage | PostgreSQL `identity_history` table (same table as PATCH) |
| Snapshot | `{"file_uuid", "face_id" (or "trace_id"), "identity_id_before/after"}` |
| Max records | 256 per identity (shared limit across all operation types) |
| Undo steps | Unlimited (`steps` param) |
| Redo stack | Cleared on new bind/unbind/bind_trace (`operation IN ('bind','unbind','bind_trace')` + `is_undone=true` records deleted) |
| Stack isolation | Bind redo stack is **independent** from PATCH redo stack — clearing one does not affect the other |
##### Stack Model
```
bind face_1 (to id=9) → unbind face_1 → bind trace 906 (to id=9)
(undo stack, is_undone=false) (undo stack) (undo stack)
↓ undo (first undone: bind_trace)
bind trace 906 (is_undone=true)
(redo stack)
↓ redo
bind face_1 → unbind face_1 → bind trace 906
(undo stack)
```
A new bind/unbind/trace after undo clears only the bind redo stack (operations with `IN ('bind','unbind','bind_trace')`).
##### Snapshot Format
**Before (bind):**
```json
{
"file_uuid": "aeed71342a899fe4b4c57b7d41bcb692",
"face_id": "1_5",
"identity_id_before": null
}
```
**After (bind):**
```json
{
"file_uuid": "aeed71342a899fe4b4c57b7d41bcb692",
"face_id": "1_5",
"identity_id_after": 9
}
```
**Before (unbind) — binding existed before:**
```json
{
"file_uuid": "aeed71342a899fe4b4c57b7d41bcb692",
"face_id": "1_5",
"identity_id_before": 9
}
```
**After (unbind):**
```json
{
"file_uuid": "aeed71342a899fe4b4c57b7d41bcb692",
"face_id": "1_5",
"identity_id_after": null
}
```
For `bind_trace`, the snapshot uses `trace_id` instead of `face_id`, with `identity_id_before` capturing the first face's identity in that trace.
---
#### `POST /api/v1/identity/:identity_uuid/bind/undo`
**Auth**: Required
**Scope**: identity-level
Undo the most recent bind/unbind/bind_trace operations. Restores `identity_id_before` from the snapshot and marks records as undone.
##### Request (JSON)
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `steps` | integer | No | `1` | Number of undo steps to apply |
##### Behavior
- Queries `is_undone=false` records with `operation IN ('bind','unbind','bind_trace')`, ordered by `created_at DESC`
- Restores `identity_id_before` — for bind this is `null` (face was unbound), for unbind this is the original identity (face goes back), for bind_trace this is the trace's previous identity
- Marks the undone records as `is_undone=true` with `undone_at=NOW()`
##### Example
```bash
curl -s -X POST "$API/api/v1/identity/$IDENTITY_UUID/bind/undo" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"steps": 1}'
```
##### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"operation": "bind",
"undone_count": 1,
"affected_rows": 53
}
```
| Field | Type | Description |
|-------|------|-------------|
| `operation` | string | The actual operation undone (`bind`, `unbind`, or `bind_trace`) |
| `undone_count` | integer | Number of history records undone |
| `affected_rows` | integer | Number of `face_detections` rows updated |
##### Error Responses
| HTTP | When |
|------|------|
| `400` | No bind undo operations available |
| `404` | Identity not found |
| `500` | Database error |
---
#### `POST /api/v1/identity/:identity_uuid/bind/redo`
**Auth**: Required
**Scope**: identity-level
Redo previously undone bind/unbind/bind_trace operations. Restores `identity_id_after` from the snapshot.
##### Request (JSON)
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `steps` | integer | No | `1` | Number of redo steps to apply |
##### Behavior
- Queries `is_undone=true` records with `operation IN ('bind','unbind','bind_trace')`, ordered by `created_at DESC`
- Restores `identity_id_after` — for bind this is the identity the face was bound to, for unbind this is `null`
- Marks records as `is_undone=false` with `undone_at=NULL`
##### Example
```bash
curl -s -X POST "$API/api/v1/identity/$IDENTITY_UUID/bind/redo" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"steps": 1}'
```
##### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"operation": "unbind",
"redone_count": 1,
"affected_rows": 1
}
```
| Field | Type | Description |
|-------|------|-------------|
| `operation` | string | The actual operation redone (`bind`, `unbind`, or `bind_trace`) |
| `redone_count` | integer | Number of history records redone |
| `affected_rows` | integer | Number of `face_detections` rows updated |
##### Error Responses
| HTTP | When |
|------|------|
| `400` | No bind redo operations available |
| `404` | Identity not found |
| `500` | Database error |
---
#### `GET /api/v1/identity/:identity_uuid/bind/history`
**Auth**: Required
**Scope**: identity-level
Query the bind/unbind/bind_trace operation history for an identity. Returns paginated records with undo/redo stack counts.
##### Query Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `page` | integer | No | `1` | Page number (1-indexed) |
| `limit` | integer | No | `20` | Items per page (max 100) |
##### Response (200)
```json
{
"success": true,
"identity_uuid": "a9a901056d6b46ff92da0c3c1a57dff4",
"total": 3,
"undo_stack_count": 2,
"redo_stack_count": 1,
"results": [
{
"history_id": 52,
"operation": "bind_trace",
"is_undone": false,
"created_at": "2026-05-27T14:00:00Z",
"undone_at": null
},
{
"history_id": 51,
"operation": "unbind",
"is_undone": true,
"created_at": "2026-05-27T13:00:00Z",
"undone_at": "2026-05-27T14:30:00Z"
},
{
"history_id": 50,
"operation": "bind",
"is_undone": false,
"created_at": "2026-05-27T12:00:00Z",
"undone_at": null
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `total` | integer | Total bind history records for this identity |
| `undo_stack_count` | integer | Records available for undo (`is_undone=false`) |
| `redo_stack_count` | integer | Records available for redo (`is_undone=true`) |
| `results[].history_id` | integer | History record ID |
| `results[].operation` | string | Operation type (`bind`, `unbind`, or `bind_trace`) |
| `results[].is_undone` | boolean | Whether the operation has been undone |
| `results[].created_at` | string | When the operation was applied |
| `results[].undone_at` | string | When the undo occurred (null if not undone) |
##### Example
```bash
curl -s "$API/api/v1/identity/$IDENTITY_UUID/bind/history?page=1&limit=10" \
-H "X-API-Key: $KEY"
```
##### Error Responses
| HTTP | When |
|------|------|
| `404` | Identity not found |
| `500` | Database error |
---
### 3. Merge History & Undo/Redo
Merge operations use MongoDB for richer record-keeping, with a 24-hour undo deadline.
#### Merge Operation Overview
| Property | Value |
|----------|-------|
| Storage | MongoDB `identity_merge_history` collection |
| Snapshot | Full source identity state + target identity state + aliases/metadata diffs |
| Trigger | Every mergeinto with `keep_history=true` |
| Undo deadline | 24 hours (renewed on redo) |
| Redo support | Yes — restores undone merges with new 24hr deadline |
| Max records | Unlimited |
---
#### `POST /api/v1/identity/merge/:merge_id/undo`
Already documented in [`07_identity.md`](07_identity.md#post-apiv1identitymergemerge_idundo). See that document for full details.
---
#### `POST /api/v1/identity/merge/:merge_id/redo`
**Auth**: Required
**Scope**: identity-level
Redo a previously undone merge operation within the renewed 24-hour deadline.
##### Request
No body required. The merge ID is taken from the URL path.
##### Behavior
1. Validates the merge record exists and `undone=true` (not already active)
2. Checks the 24-hour undo deadline (if expired, the redo is rejected)
3. Restores face bindings: moves all faces from `target_identity` back to `source_identity`
4. Re-adds aliases that were removed by the undo (aliases with `source: "merge"` tag)
5. Re-adds metadata fields that were removed by the undo
6. If `keep_history=true`: sets `source_identity.status = 'merged'` again
7. If `keep_history=false`: recreates source identity from the `undone_snapshot` stored at undo time
8. Syncs both identity JSON files to disk
9. Sets `undone=false`, clears `undone_snapshot`, renews `undo_deadline = NOW() + 24h`
10. Records `redone_by` user for audit
##### Example
```bash
curl -s -X POST "$API/api/v1/identity/merge/550e8400-e29b-41d4-a716-446655440000/redo" \
-H "X-API-Key: $KEY"
```
##### Response (200)
```json
{
"success": true,
"message": "Redo merge completed: merged 'stranger_13894' into 'Louis Viret' (52 faces transferred)",
"data": {
"merge_id": "550e8400-e29b-41d4-a716-446655440000",
"faces_transferred": 52,
"aliases_re_added": 1,
"metadata_fields_re_added": 2
}
}
```
| Field | Type | Description |
|-------|------|-------------|
| `merge_id` | string | The merge operation ID |
| `faces_transferred` | integer | Number of faces transferred from source to target |
| `aliases_re_added` | integer | Number of aliases restored to target |
| `metadata_fields_re_added` | integer | Number of metadata fields restored to target |
##### Error Responses
| HTTP | When |
|------|------|
| `400` | Merge not undone, deadline expired, or cannot redo |
| `404` | Merge record not found |
| `500` | Database error |
---
### 4. Delete History & Undo/Redo
#### Delete Operation Overview
| Property | Value |
|----------|-------|
| Storage | PostgreSQL `identity_history` table |
| Snapshot | `{"identity": {...full row...}, "unbound_faces": [{file_uuid, face_id, trace_id}, ...]}` |
| Max records | 1 active delete record per identity (redo stack cleared on new delete) |
| Undo support | Yes — recreates identity row, re-binds faces |
| Redo support | Yes — re-deletes the identity |
| Identity file | Deleted on delete, recreated on undo |
#### Snapshot Format
```json
{
"identity": {
"id": 9,
"uuid": "a9a90105-6d6b-46ff-92da-0c3c1a57dff4",
"name": "Cary Grant",
"identity_type": "people",
"source": "tmdb",
"status": "confirmed",
"metadata": {},
"tmdb_id": 112,
"tmdb_profile": null
},
"unbound_faces": [
{
"file_uuid": "aeed71342a899fe4b4c57b7d41bcb692",
"face_id": "1_5",
"trace_id": null
},
{
"file_uuid": "aeed71342a899fe4b4c57b7d41bcb692",
"face_id": "1_6",
"trace_id": 906
}
]
}
```
#### Stack Model
```
DELETE identity (undo stack, is_undone=false)
↓ undo
Identity recreated, faces re-bound
→ delete history marked is_undone=true
↓ redo (re-delete)
Identity deleted again, faces unbound
→ delete history marked is_undone=false
```
A new delete after an undo clears the delete redo stack (no redo possible for the old delete).
#### Undo Behavior (via existing `POST /api/v1/identity/:identity_uuid/undo`)
1. Normal identity lookup fails (row was deleted)
2. Checks `identity_history` for `operation='delete' AND is_undone=false` matching the UUID in the snapshot
3. Recreates the identity row (new internal `id`, same UUID)
4. Re-binds all faces listed in `unbound_faces` to the new identity
5. Deletes the `identity_history` delete record as `is_undone=true` with `undone_at=NOW()`
6. Syncs `identity.json` to disk
7. Updates `_index.json`
#### Redo Behavior (via existing `POST /api/v1/identity/:identity_uuid/redo`)
1. Identity lookup succeeds (identity was restored by prior undo)
2. Checks `identity_history` for `operation='delete' AND is_undone=true` matching the identity_id
3. Deletes `identity.json` from disk
4. Unbinds all faces (`identity_id = NULL`)
5. Deletes the identity row
6. Marks the delete history record as `is_undone=false`
7. Returns success
#### Error Responses (delete undo/redo)
| HTTP | Scenario |
|------|----------|
| `400` | No delete history available (either no delete or already undone/redone) |
| `404` | Identity not found (for redo — identity wasn't restored) |
| `500` | Database error |
---
### Comparison: PATCH vs Bind vs Merge vs Delete Undo/Redo
| Aspect | PATCH Undo/Redo | Bind Undo/Redo | Merge Undo/Redo | Delete Undo/Redo |
|--------|----------------|----------------|-----------------|------------------|
| Storage | PostgreSQL `identity_history` | PostgreSQL `identity_history` | MongoDB `identity_merge_history` | PostgreSQL `identity_history` |
| Operation filter | `operation='update'` | `operation IN ('bind','unbind','bind_trace')` | — | `operation='delete'` |
| Trigger | Every PATCH | Every bind/unbind/bind_trace | Every mergeinto with `keep_history=true` | Every DELETE |
| Undo deadline | None (unlimited) | None (unlimited) | 24 hours (renewed on redo) | None (unlimited) |
| Redo support | Yes | Yes | Yes | Yes |
| Step undo | Yes (`steps` param) | Yes (`steps` param) | No (full undo/redo only) | No (single record) |
| Max records | 256 per identity | 256 per identity (shared) | Unlimited | 256 per identity (shared) |
| User tracking | `user_id` + `user_source` | `user_id` + `user_source` | `performed_by_user` + `undone_by` / `redone_by` | `user_id` + `user_source` |
---
*Updated: 2026-05-28*
docs_v1.0/API_WORKSPACE/modules/15_tkg.md
+526
View File
@@ -0,0 +1,526 @@
<!-- module: tkg -->
<!-- description: Temporal Knowledge Graph — rebuild, nodes, edges, processor counts -->
<!-- depends: 05_process, 07_identity -->
## Temporal Knowledge Graph (TKG)
TKG is a time-aligned knowledge graph built from multi-processor outputs (face, yolo, ocr, pose, asrx, gaze, lip, appearance). It produces 9 node types and 14 edge types stored in `dev.tkg_nodes` and `dev.tkg_edges`.
**Node naming convention:** All trace types use `_track` suffix. Text uses `_region` (non-temporal).
**See also:** `docs_v1.0/DESIGN/TKG_FORMATION_V1.0.md` for formation phases, flow diagrams, and query examples.
### Node Types
| Node Type | External ID Format | Description | Key Properties |
|-----------|-------------------|-------------|----------------|
| `face_track` | `trace_{trace_id}` | A tracked face identity over time | `trace_id`, `frame_count`, `status`, `avg_bbox`, `avg_yaw`, `avg_pitch`, `avg_roll`, `start_frame`, `end_frame`, `pose_count` |
| `gaze_track` | `gaze_track_{id}` | Gaze direction over time | `direction` (frontal/left/right/up/down + diagonals) |
| `lip_track` | `lip_track_{id}` | Lip movement synced with speech | `speaker_id`, `lip_area_range` |
| `text_region` | `text_region_{id}` | Spoken text aligned to time | `speaker_id`, `text`, `start_time`, `end_time` |
| `appearance_trace` | `appearance_{trace_id}` | Human appearance (clothing) over time | `clothing_color`, `upper_cloth`, `lower_cloth` |
| `accessory` | `accessory_{id}` | Detected accessories | `type` (glasses/hat/etc.), `confidence` |
| `object` | `object_{class}_{id}` | YOLO-detected object | `class`, `confidence`, `frame_count` |
| `speaker` | `speaker_{speaker_id}` | ASRX speaker segment | `speaker_id`, `segment_count`, `total_duration` |
---
### Identity Agent Integration (face_track nodes)
Identity Agent marks face_track nodes with identity binding status.
#### face_track Status Values
| Status | Description | Properties |
|--------|-------------|------------|
| `pending` | No identity suggestion | Default state |
| `suggested` | Identity Agent suggested | `pending_identity_name`, `pending_identity_uuid`, `suggested_by`, `confidence` |
| `confirmed` | User confirmed binding | `identity_uuid`, `identity_id`, `identity_ref`, `identity_name` |
| `stranger` | Stranger cluster member | `stranger_id`, `stranger_ref` |
#### Suggested By Values
| Value | Description |
|-------|-------------|
| `tmdb` | TMDb seed matched |
| `propagation` | Confirmed trace propagation |
| `manual` | User manual selection |
#### Example face_track Node
```json
{
"node_type": "face_track",
"external_id": "face_track_1",
"label": "Face Track 1",
"properties": {
"trace_id": 1,
"frame_count": 45,
"start_frame": 100,
"end_frame": 300,
"avg_bbox": {"x": 100, "y": 200, "width": 80, "height": 100},
"status": "suggested",
"pending_identity_name": "Tom Hanks",
"pending_identity_uuid": "xxx-xxx",
"suggested_by": "tmdb",
"confidence": 0.91
}
}
```
---
### Edge Types
| Edge Type | Storage Name | Source → Target | Description |
|-----------|--------------|-----------------|-------------|
| `co_occurs` | `CO_OCCURS_WITH` | object ↔ object | Two objects appear together in same frame |
| `speaker_face` | `SPEAKS_AS` | speaker → face_track | Speaker matched to face track via lip sync |
| `face_face` | `INTERACTS_WITH` | face_track ↔ face_track | Two face tracks interact (mutual gaze) |
| `mutual_gaze` | `MUTUAL_GAZE` | gaze_track ↔ gaze_track | Two people looking at each other |
| `lip_sync` | `LIP_SYNC` | lip_track → text_region | Lip movement aligned with spoken text |
| `has_appearance` | `HAS_APPEARANCE` | face_track → appearance_trace | Face has specific appearance |
| `wears` | `WEARS` | face_track → accessory | Face wears an accessory |
| `hand_object` | `HOLDS` | hand → object | Hand holding object |
---
### `POST /api/v1/file/:file_uuid/tkg/rebuild`
**Auth**: Required
**Scope**: file-level
Rebuild the Temporal Knowledge Graph for a file. Reads processor JSON outputs (face, yolo, ocr, pose, asrx, gaze, lip, appearance) and generates TKG nodes and edges. Clears existing nodes/edges for the file first, then rebuilds from scratch.
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/rebuild" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"result": {
"face_track_nodes": 16,
"gaze_track_nodes": 16,
"lip_track_nodes": 12,
"text_region_nodes": 24,
"appearance_trace_nodes": 8,
"skin_tone_trace_nodes": 5,
"accessory_nodes": 3,
"object_nodes": 26,
"speaker_nodes": 4,
"co_occurrence_edges": 94,
"speaker_face_edges": 12,
"face_face_edges": 8,
"mutual_gaze_edges": 2,
"lip_sync_edges": 10,
"has_appearance_edges": 16,
"wears_edges": 3
},
"error": null
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | True if rebuild completed |
| `file_uuid` | string | 32-char hex UUID |
| `result` | object | Node and edge counts by type |
| `error` | string/null | Error message if failed |
---
### `POST /api/v1/file/:file_uuid/tkg/nodes`
**Auth**: Required
**Scope**: file-level
Query TKG nodes with pagination and optional type filter.
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `node_type` | string | No | all | Filter by node type: `face_track`, `gaze_track`, `lip_track`, `text_region`, `appearance_trace`, `skin_tone_trace`, `accessory`, `object`, `speaker` |
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 100 | Items per page (max 500) |
#### Example
```bash
# Get all face_track nodes
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/nodes" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"node_type": "face_track", "page": 1, "page_size": 50}'
# Get all nodes
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/nodes" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{}'
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"total": 16,
"page": 1,
"page_size": 50,
"nodes": [
{
"id": 1,
"node_type": "face_track",
"external_id": "face_track_0",
"label": "Face Track 0",
"properties": {
"trace_id": 0,
"frame_count": 142,
"avg_confidence": 0.87
}
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on 200 |
| `file_uuid` | string | 32-char hex UUID |
| `total` | integer | Total matching node count |
| `page` | integer | Current page |
| `page_size` | integer | Items per page |
| `nodes` | array | Array of node objects |
| `nodes[].id` | integer | Database primary key |
| `nodes[].node_type` | string | Node type (see table above) |
| `nodes[].external_id` | string | External identifier (e.g., `trace_0`, `gaze_1`) |
| `nodes[].label` | string | Human-readable label |
| `nodes[].properties` | object | Type-specific properties as JSON |
---
### `POST /api/v1/file/:file_uuid/tkg/edges`
**Auth**: Required
**Scope**: file-level
Query TKG edges with pagination and optional filters.
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `edge_type` | string | No | all | Filter by edge type: `co_occurs`, `speaker_face`, `face_face`, `mutual_gaze`, `lip_sync`, `has_appearance`, `wears` |
| `source_type` | string | No | — | Filter by source node type |
| `target_type` | string | No | — | Filter by target node type |
| `page` | integer | No | 1 | Page number |
| `page_size` | integer | No | 100 | Items per page (max 500) |
#### Example
```bash
# Get all co_occurs edges
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/edges" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"edge_type": "co_occurs"}'
# Get edges between face_track and speaker nodes
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/edges" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{"source_type": "speaker", "target_type": "face_track"}'
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"total": 94,
"page": 1,
"page_size": 100,
"edges": [
{
"id": 1,
"edge_type": "co_occurs",
"source_node_id": 10,
"target_node_id": 15,
"properties": {
"frame_count": 45,
"confidence": 0.92
}
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on 200 |
| `file_uuid` | string | 32-char hex UUID |
| `total` | integer | Total matching edge count |
| `page` | integer | Current page |
| `page_size` | integer | Items per page |
| `edges` | array | Array of edge objects |
| `edges[].id` | integer | Database primary key |
| `edges[].edge_type` | string | Edge type |
| `edges[].source_node_id` | integer | Source node ID (FK to tkg_nodes) |
| `edges[].target_node_id` | integer | Target node ID (FK to tkg_nodes) |
| `edges[].properties` | object | Edge-specific properties as JSON |
---
### `GET /api/v1/file/:file_uuid/tkg/node/:node_id`
**Auth**: Required
**Scope**: file-level
Get detail for a specific TKG node including its connected edges.
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/tkg/node/1" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"success": true,
"node": {
"id": 1,
"node_type": "face_track",
"external_id": "face_track_0",
"label": "Face Track 0",
"properties": {
"trace_id": 0,
"frame_count": 142,
"avg_confidence": 0.87
}
},
"connected_edges": [
{
"id": 5,
"edge_type": "co_occurs",
"source_node_id": 1,
"target_node_id": 10,
"properties": {"frame_count": 45}
}
],
"edge_count": 3
}
```
| Field | Type | Description |
|-------|------|-------------|
| `success` | boolean | Always true on 200 |
| `node` | object | Node detail (same format as nodes query) |
| `connected_edges` | array | Edges connected to this node |
| `edge_count` | integer | Total connected edge count |
#### Error Codes
| HTTP | When |
|------|------|
| `404` | Node not found |
---
### `GET /api/v1/file/:file_uuid/processor-counts`
**Auth**: Required
**Scope**: file-level
Get counts of processor JSON output files for a file. Scans the output directory for `{file_uuid}.{processor}.json` files and extracts frame counts, segment counts, and chunk counts from each file.
Supports short UUID prefix matching (e.g., `d3f9ae8e` → resolves to full `d3f9ae8e471a1fc4d47022c66091b920`).
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/processor-counts" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"output_dir": "/Users/accusys/momentry/output_dev",
"processors": [
{
"processor": "cut",
"has_json": true,
"frame_count": 5391,
"segment_count": null,
"chunk_count": null,
"last_modified": "2026-06-16T18:48:01.987241061+00:00"
},
{
"processor": "face",
"has_json": true,
"frame_count": 1112,
"segment_count": null,
"chunk_count": null,
"last_modified": "2026-06-18T17:21:37.408383765+00:00"
},
{
"processor": "asrx",
"has_json": true,
"frame_count": null,
"segment_count": 6,
"chunk_count": null,
"last_modified": "2026-06-18T17:21:40.872063642+00:00"
},
{
"processor": "story",
"has_json": true,
"frame_count": null,
"segment_count": null,
"chunk_count": 12,
"last_modified": "2026-06-18T17:22:00.000000000+00:00"
},
{
"processor": "mediapipe",
"has_json": false,
"frame_count": null,
"segment_count": null,
"chunk_count": null,
"last_modified": null
}
]
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | Full 32-char hex UUID (resolved from prefix) |
| `output_dir` | string | Output directory scanned |
| `processors` | array | Per-processor output info |
| `processors[].processor` | string | Processor name |
| `processors[].has_json` | boolean | Whether JSON file exists |
| `processors[].frame_count` | integer/null | Total frames processed (frame-based processors) |
| `processors[].segment_count` | integer/null | Segment count (ASRX segments, etc.) |
| `processors[].chunk_count` | integer/null | Chunk count (Story chunks, etc.) |
| `processors[].last_modified` | string/null | ISO 8601 timestamp of last modification |
#### Error Codes
| HTTP | When |
|------|------|
| `404` | File UUID not found in database |
---
### Trace Management
Endpoints for managing face traces: list, delete, restore, and merge.
#### `DELETE /api/v1/file/:file_uuid/trace/:trace_id`
**Auth**: Required
Soft-delete a face trace (default) or hard-delete with `{"hard_delete": true}`.
Soft delete marks Qdrant points with `status: "deleted"` and TKG nodes with `status: "deleted"` in properties. Deleted traces are excluded from the traces list.
Hard delete permanently removes Qdrant points and TKG nodes.
**Request Body** (optional):
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `hard_delete` | boolean | `false` | Permanently delete instead of marking |
**Example**:
```bash
# Soft delete
curl -X DELETE "$API/api/v1/file/$FILE_UUID/trace/8" \
-H "X-API-Key: $KEY" -H "Content-Type: application/json" -d '{}'
# Hard delete
curl -X DELETE "$API/api/v1/file/$FILE_UUID/trace/8" \
-H "X-API-Key: $KEY" -H "Content-Type: application/json" \
-d '{"hard_delete": true}'
```
**Response**:
```json
{
"success": true,
"file_uuid": "9f6a9cd55a5809f977f5a6589b9045c5",
"trace_id": 8,
"hard_delete": false,
"qdrant_marked": true,
"tkg_nodes_marked": 1
}
```
---
#### `POST /api/v1/file/:file_uuid/trace/:trace_id/restore`
**Auth**: Required
Undo a soft-deleted trace. Clears `status: "deleted"` from Qdrant points and TKG node properties.
**Example**:
```bash
curl -X POST "$API/api/v1/file/$FILE_UUID/trace/8/restore" \
-H "X-API-Key: $KEY"
```
**Response**:
```json
{
"success": true,
"file_uuid": "9f6a9cd55a5809f977f5a6589b9045c5",
"trace_id": 8,
"qdrant_restored": true,
"tkg_nodes_restored": 1
}
```
---
#### `POST /api/v1/file/:file_uuid/trace/:source_trace_id/merge/:target_trace_id`
**Auth**: Required
Merge all face points from source trace into target trace. Updates Qdrant `trace_id` and deletes source TKG node.
**Example**:
```bash
curl -X POST "$API/api/v1/file/$FILE_UUID/trace/16/merge/3" \
-H "X-API-Key: $KEY"
```
**Response**:
```json
{
"success": true,
"file_uuid": "9f6a9cd55a5809f977f5a6589b9045c5",
"source_trace_id": 16,
"target_trace_id": 3,
"points_moved": 58,
"tkg_nodes_deleted": 1
}
```
---
*Updated: 2026-07-21 01:00:00*
docs_v1.0/API_WORKSPACE/modules/16_workspace.md
+148
View File
@@ -0,0 +1,148 @@
<!-- module: workspace -->
<!-- description: Workspace checkout/checkin — lock, clear, restore file data -->
<!-- depends: 04_lookup, 05_process -->
## Workspace Checkin/Checkout
Workspace checkin/checkout provides a transactional editing model for file data:
- **Checkout**: Clears PG tables (face_detections, speaker_detections, pre_chunks) and Qdrant vectors, creating an isolated workspace SQLite for editing.
- **Checkin**: Restores data from the workspace SQLite back to PG and Qdrant, marking the file as `Indexed`.
This allows safe concurrent editing — while a file is checked out, its main database records are cleared, preventing conflicts.
---
### `POST /api/v1/file/:file_uuid/checkout`
**Auth**: Required
**Scope**: file-level
Checkout a file workspace. Clears face detections, speaker detections, pre_chunks from PostgreSQL, deletes Qdrant vectors, and creates a workspace SQLite database for isolated editing.
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/checkout" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"rows_deleted": 1523,
"status": "checked_out"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | 32-char hex UUID |
| `rows_deleted` | integer | Total rows cleared from PG tables |
| `status` | string | `"checked_out"` |
#### Error Responses
| HTTP | When |
|------|------|
| `500` | Checkout failed (DB error, workspace creation error) |
---
### `POST /api/v1/file/:file_uuid/checkin`
**Auth**: Required
**Scope**: file-level
Checkin a file workspace. Restores face detections, speaker detections, pre_chunks from workspace SQLite back to PostgreSQL, re-indexes vectors to Qdrant, and sets video status to `Indexed`.
#### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/checkin" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"pre_chunks_moved": 45,
"face_detections_moved": 1200,
"speaker_detections_moved": 320,
"vectors_moved": 45,
"status": "indexed"
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | 32-char hex UUID |
| `pre_chunks_moved` | integer | Pre-chunks restored from workspace |
| `face_detections_moved` | integer | Face detections restored from workspace |
| `speaker_detections_moved` | integer | Speaker detections restored from workspace |
| `vectors_moved` | integer | Vectors re-indexed to Qdrant |
| `status` | string | `"indexed"` |
#### Error Responses
| HTTP | When |
|------|------|
| `500` | Checkin failed (DB error, workspace not found, vector index error) |
---
### `GET /api/v1/file/:file_uuid/workspace`
**Auth**: Required
**Scope**: file-level
Check if a workspace SQLite database exists for a file.
#### Example
```bash
curl -s "$API/api/v1/file/$FILE_UUID/workspace" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"file_uuid": "d3f9ae8e471a1fc4d47022c66091b920",
"exists": true
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | 32-char hex UUID |
| `exists` | boolean | True if workspace SQLite exists |
---
### Workflow
```
REGISTERED ──→ CHECKED_OUT ──→ INDEXED
│ │ │
│ checkout checkin
│ │ │
│ clear PG + Qdrant restore from SQLite
│ create workspace re-index vectors
│ set status set status
```
1. **Register** file → status: `REGISTERED`
2. **Process** file → processors run, data stored in PG + Qdrant
3. **Checkout** file → clear editable data, create workspace SQLite → status: `CHECKED_OUT`
4. **Edit** workspace via Agent Search / identity binding
5. **Checkin** file → restore from workspace SQLite → status: `INDEXED`
6. **Rebuild TKG** if needed after checkin
---
*Updated: 2026-06-20 12:00:00*
docs_v1.0/API_WORKSPACE/modules/17_progress.md
+545
View File
@@ -0,0 +1,545 @@
<!-- module: progress -->
<!-- description: Real-time progress tracking for processing pipeline, TKG build, and identity agent -->
<!-- depends: 01_auth, 03_register, 05_process -->
# Progress Tracking — API Workspace Module
## Overview
The progress tracking system provides real-time visibility into all processing stages:
| System | Redis Key | Coverage |
|--------|-----------|----------|
| **Processor Progress** | `{prefix}progress:{file_uuid}` | 7 main processors (cut, asr, asrx, ocr, face, pose, appearance) |
| **TKG Progress** | `{prefix}progress:{file_uuid}:tkg` | 18 TKG build phases (9 node types + 8 edge types + face_tracing) |
| **Agent Progress** | `{prefix}progress:{file_uuid}:agent` | 5 Identity Agent phases |
---
## `POST /api/v1/progress/:file_uuid`
**Auth**: Required
**Scope**: file-level
Get real-time processing progress including processor status, TKG build phases, and identity agent phases.
### Example
```bash
curl -s -X POST "$API/api/v1/progress/$FILE_UUID" \
-H "X-API-Key: $KEY" | jq '.'
```
### Response (200)
```json
{
"file_uuid": "3a6c1865...",
"overall_progress": 71,
"cpu_percent": 45.2,
"gpu_percent": 30.1,
"memory_percent": 62.4,
"processors": [
{"name": "asr", "status": "complete", "progress": 100, "current": 0, "total": 0, "message": "done"},
{"name": "face", "status": "complete", "progress": 100, "current": 0, "total": 0, "message": "done"},
{"name": "pose", "status": "complete", "progress": 100, "current": 0, "total": 0, "message": "done"}
],
"tkg_progress": {
"file_uuid": "3a6c1865...",
"phase": "mutual_gaze_edges",
"phase_index": 13,
"total_phases": 18,
"phase_progress": 0.8,
"overall_progress": 0.72,
"stats": {
"total_faces": 1250,
"traced_faces": 1250,
"total_traces": 45,
"face_track_nodes": 45,
"gaze_track_nodes": 45,
"lip_track_nodes": 12,
"text_region_nodes": 8,
"appearance_nodes": 38,
"accessory_nodes": 5,
"object_nodes": 156,
"hand_nodes": 22,
"speaker_nodes": 14,
"co_occurrence_edges": 890,
"speaker_face_edges": 120,
"face_face_edges": 234,
"mutual_gaze_edges": 67,
"total_nodes": 345,
"total_edges": 1311
},
"message": "67 mutual gaze edges",
"updated_at": "2026-07-02T10:30:00Z"
},
"agent_progress": {
"file_uuid": "3a6c1865...",
"phase": "completed",
"phase_index": 5,
"total_phases": 5,
"phase_progress": 1.0,
"overall_progress": 1.0,
"stats": {
"total_faces": 1250,
"total_traces": 45,
"clusters": 18,
"identities_created": 18,
"tmdb_matches": 5,
"speaker_bindings": 12,
"confirmations": 18
},
"message": "Identity Agent processing completed",
"updated_at": "2026-07-02T10:28:00Z"
}
}
```
### Field Descriptions
#### Top Level
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | 32-char hex UUID |
| `overall_progress` | integer | Overall processor progress (0–100) |
| `processors` | array | Per-processor status |
| `tkg_progress` | object | TKG build progress (null if not started) |
| `agent_progress` | object | Identity Agent progress (null if not started) |
#### TKG Progress Fields
| Field | Type | Description |
|-------|------|-------------|
| `phase` | string | Current phase name (see TKG Phases below) |
| `phase_index` | integer | Current phase index (0–17) |
| `total_phases` | integer | Total phases: 18 |
| `phase_progress` | float | Progress within current phase (0.0–1.0) |
| `overall_progress` | float | Overall TKG progress (0.0–1.0) |
| `stats` | object | Counts for all node and edge types |
| `message` | string | Human-readable status message |
#### TKG Phases (18 total)
| Index | Phase | Description |
|-------|-------|-------------|
| 0 | `face_tracing` | Populate trace_id from face.json |
| 1 | `face_track_nodes` | Build face_track nodes |
| 2 | `gaze_track_nodes` | Build gaze_track nodes |
| 3 | `lip_track_nodes` | Build lip_track nodes |
| 4 | `text_region_nodes` | Build text_region nodes |
| 5 | `appearance_nodes` | Build appearance_trace nodes |
| 6 | `accessory_nodes` | Build accessory nodes |
| 7 | `object_nodes` | Build yolo_object nodes |
| 8 | `hand_nodes` | Build hand nodes |
| 9 | `speaker_nodes` | Build speaker nodes |
| 10 | `co_occurrence_edges` | Build co_occurrence edges |
| 11 | `speaker_face_edges` | Build speaker_face edges |
| 12 | `face_face_edges` | Build face_face edges |
| 13 | `mutual_gaze_edges` | Build mutual_gaze edges |
| 14 | `lip_sync_edges` | Build lip_sync edges |
| 15 | `has_appearance_edges` | Build has_appearance edges |
| 16 | `wears_edges` | Build wears edges |
| 17 | `hand_object_edges` | Build hand_object edges |
#### TKG Stats Fields
| Field | Type | Description |
|-------|------|-------------|
| `total_faces` | integer | Total face detections |
| `traced_faces` | integer | Faces with trace_id assigned |
| `total_traces` | integer | Unique trace count |
| `face_track_nodes` | integer | Face track nodes created |
| `gaze_track_nodes` | integer | Gaze track nodes created |
| `lip_track_nodes` | integer | Lip track nodes created |
| `text_region_nodes` | integer | Text region nodes created |
| `appearance_nodes` | integer | Appearance trace nodes created |
| `accessory_nodes` | integer | Accessory nodes created |
| `object_nodes` | integer | YOLO object nodes created |
| `hand_nodes` | integer | Hand nodes created |
| `speaker_nodes` | integer | Speaker nodes created |
| `co_occurrence_edges` | integer | Co-occurrence edges created |
| `speaker_face_edges` | integer | Speaker-face edges created |
| `face_face_edges` | integer | Face-face edges created |
| `mutual_gaze_edges` | integer | Mutual gaze edges created |
| `lip_sync_edges` | integer | Lip sync edges created |
| `has_appearance_edges` | integer | Has-appearance edges created |
| `wears_edges` | integer | Wears edges created |
| `hand_object_edges` | integer | Hand-object edges created |
| `total_nodes` | integer | Total nodes (sum of all node types) |
| `total_edges` | integer | Total edges (sum of all edge types) |
---
## `GET /api/v1/stats/ingestion-status/:file_uuid`
**Auth**: Required
**Scope**: file-level
Get detailed ingestion status showing completion of all 24 processing steps.
### Example
```bash
curl -s "$API/api/v1/stats/ingestion-status/$FILE_UUID" \
-H "X-API-Key: $KEY" | jq '.steps[] | {name, status, detail}'
```
### Response (200)
```json
{
"file_uuid": "3a6c1865...",
"steps": [
{"name": "rule1_sentence", "status": "done", "detail": "156 sentence chunks"},
{"name": "auto_vectorize", "status": "done", "detail": "156 embedded"},
{"name": "face_track", "status": "done", "detail": "45 traces / 1250 detections"},
{"name": "trace_chunks", "status": "done", "detail": "45 trace chunks"},
{"name": "tkg_face_track", "status": "done", "detail": "45 nodes"},
{"name": "tkg_gaze_track", "status": "done", "detail": "45 nodes"},
{"name": "tkg_lip_track", "status": "done", "detail": "12 nodes"},
{"name": "tkg_text_region", "status": "done", "detail": "8 nodes"},
{"name": "tkg_appearance", "status": "done", "detail": "38 nodes"},
{"name": "tkg_accessory", "status": "done", "detail": "5 nodes"},
{"name": "tkg_object", "status": "done", "detail": "156 nodes"},
{"name": "tkg_hand", "status": "done", "detail": "22 nodes"},
{"name": "tkg_speaker", "status": "done", "detail": "14 nodes"},
{"name": "tkg_co_occurrence", "status": "done", "detail": "890 edges"},
{"name": "tkg_speaker_face", "status": "done", "detail": "120 edges"},
{"name": "tkg_face_face", "status": "done", "detail": "234 edges"},
{"name": "tkg_mutual_gaze", "status": "done", "detail": "67 edges"},
{"name": "tkg_lip_sync", "status": "done", "detail": "12 edges"},
{"name": "tkg_has_appearance", "status": "done", "detail": "38 edges"},
{"name": "tkg_wears", "status": "done", "detail": "22 edges"},
{"name": "tkg_hand_object", "status": "done", "detail": "18 edges"},
{"name": "rule2_relationship", "status": "done", "detail": "1331 relationship chunks"},
{"name": "identity_match", "status": "done", "detail": "18 identities matched"},
{"name": "scene_metadata", "status": "done", "detail": null}
],
"related_identities": [
{"uuid": "a9a901056d6b46ff92da0c3c1a57dff4", "name": "John Smith"}
],
"strangers": 3
}
```
### Step Descriptions
| Step | Status When Done |
|------|-----------------|
| `rule1_sentence` | sentence_count > 0 |
| `auto_vectorize` | sentence_embedded > 0 |
| `face_track` | trace_count > 0 |
| `trace_chunks` | trace_chunks > 0 |
| `tkg_face_track` → `tkg_speaker` | Node count > 0 (9 steps) |
| `tkg_co_occurrence` → `tkg_hand_object` | Edge count > 0 (8 steps) |
| `rule2_relationship` | relationship_chunks > 0 |
| `identity_match` | identity_count > 0 |
| `scene_metadata` | scene_meta.json exists |
---
## `POST /api/v1/file/:file_uuid/tkg/rebuild`
**Auth**: Required
**Scope**: file-level
Manually trigger TKG rebuild. Automatically triggers Rule 2 ingestion after TKG completes.
### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/tkg/rebuild" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" -d '{}'
```
### Response (200)
```json
{
"success": true,
"message": "TKG rebuild started",
"nodes": 345,
"edges": 1311
}
```
---
## `POST /api/v1/file/:file_uuid/rule2`
**Auth**: Required
**Scope**: file-level
Manually trigger Rule 2 ingestion (TKG edges → relationship chunks).
### Example
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/rule2" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" -d '{}'
```
### Response (200)
```json
{
"success": true,
"message": "Rule 2 ingestion: 1331 relationship chunks created",
"rule2_count": 1331
}
```
---
## Processing Pipeline Flow
```
1. Processors (concurrent)
├── cut, asr, ocr, face, pose, appearance → complete
└── asrx → after cut+asr
2. Post-Processor Triggers (automatic)
├── Rule 1 Ingestion (ASR+OCR → sentence chunks)
├── Face Trace + DB Store (face_traced.json → Qdrant trace_id)
├── TMDb Face Matching (if enabled)
├── Heuristic Scene Metadata
├── Identity Agent (face + ASRX)
└── TKG Build (automatic after processors complete)
└── Rule 2 Ingestion (automatic after TKG)
└── Relationship chunks vectorized
3. Completion
└── Job marked completed when all ingestion steps done
```
## Error Codes
| Code | HTTP | When |
|------|------|------|
| E001 | 400 | Invalid file_uuid format |
| E002 | 404 | File not found |
| E003 | 404 | No TKG data available |
| E010 | 500 | Qdrant connection failed |
| E011 | 500 | Database connection failed |
---
## `GET /api/v1/stats/pipeline/:file_uuid`
**Auth**: Required
**Scope**: file-level
Get segmented pipeline progress with weighted stage breakdown. Shows overall progress as weighted sum of all pipeline stages.
### Pipeline Stages and Weights
| Stage | Weight | Description |
|-------|--------|-------------|
| `processors` | 30% | 7 concurrent processors (cut, asr, asrx, ocr, face, pose, appearance) |
| `rule1_ingestion` | 5% | ASR+OCR → sentence chunks |
| `face_tracing` | 5% | Face trace_id assignment |
| `identity_agent` | 10% | Identity creation, TMDb matching, speaker binding |
| `tkg_nodes` | 20% | TKG node building (9 node types) |
| `tkg_edges` | 15% | TKG edge building (8 edge types) |
| `rule2_ingestion` | 15% | TKG edges → relationship chunks |
### Example
```bash
curl -s "$API/api/v1/stats/pipeline/$FILE_UUID" \
-H "X-API-Key: $KEY" | jq '.'
```
### Response (200)
```json
{
"file_uuid": "3a6c1865...",
"overall_progress": 0.65,
"stages": [
{"name": "processors", "weight": 0.30, "progress": 1.0, "status": "completed", "detail": "7/7 complete"},
{"name": "rule1_ingestion", "weight": 0.05, "progress": 1.0, "status": "completed", "detail": "156 chunks"},
{"name": "face_tracing", "weight": 0.05, "progress": 1.0, "status": "completed", "detail": "45 traces"},
{"name": "identity_agent", "weight": 0.10, "progress": 1.0, "status": "completed", "detail": "18 identities"},
{"name": "tkg_nodes", "weight": 0.20, "progress": 1.0, "status": "completed", "detail": "345 nodes"},
{"name": "tkg_edges", "weight": 0.15, "progress": 0.5, "status": "running", "detail": "mutual_gaze_edges: 67/8 expected"},
{"name": "rule2_ingestion", "weight": 0.15, "progress": 0.0, "status": "pending", "detail": null}
],
"updated_at": "2026-07-02T10:30:00Z"
}
```
### Field Descriptions
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | 32-char hex UUID |
| `overall_progress` | float | Weighted sum of all stage progress (0.0–1.0) |
| `stages` | array | Per-stage progress breakdown |
| `stages[].name` | string | Stage name |
| `stages[].weight` | float | Stage weight in overall progress |
| `stages[].progress` | float | Stage completion (0.0–1.0) |
| `stages[].status` | string | `"pending"`, `"running"`, `"completed"`, `"failed"` |
| `stages[].detail` | string | Human-readable detail (optional) |
| `updated_at` | string | ISO 8601 timestamp |
### Overall Progress Calculation
```
overall_progress = Σ(stage.weight × stage.progress) for all stages
```
Example calculation:
- processors: 0.30 × 1.0 = 0.30
- rule1_ingestion: 0.05 × 1.0 = 0.05
- face_tracing: 0.05 × 1.0 = 0.05
- identity_agent: 0.10 × 1.0 = 0.10
- tkg_nodes: 0.20 × 1.0 = 0.20
- tkg_edges: 0.15 × 0.5 = 0.075
- rule2_ingestion: 0.15 × 0.0 = 0.0
- **Total: 0.775 (77.5%)**
---
## `GET /api/v1/stats/file/:file_uuid`
**Auth**: Required
**Scope**: file-level
Get comprehensive file statistics from all data sources: JSON processing status, PostgreSQL counts, Qdrant collections, TKG nodes/edges, and Identity Agent stats.
### Example
```bash
curl -s "$API/api/v1/stats/file/$FILE_UUID" \
-H "X-API-Key: $KEY" | jq '.'
```
### Response (200)
```json
{
"file_uuid": "3a6c1865...",
"file_name": "video.mp4",
"status": "processing",
"processors": [
{"name": "asr", "status": "complete", "progress": 100, "message": "done"},
{"name": "face", "status": "complete", "progress": 100, "message": "done"}
],
"postgres": {
"sentence_chunks": 156,
"trace_chunks": 45,
"relationship_chunks": 1331,
"identities": 18,
"file_identities": 18
},
"qdrant": {
"faces": 1250,
"face_traces": 45,
"face_identities": 18,
"text_chunks": 4562,
"speakers": 434
},
"tkg": {
"total_nodes": 345,
"total_edges": 1311,
"face_track_nodes": 45,
"gaze_track_nodes": 45,
"lip_track_nodes": 12,
"text_region_nodes": 8,
"appearance_nodes": 38,
"accessory_nodes": 5,
"object_nodes": 156,
"hand_nodes": 22,
"speaker_nodes": 14,
"co_occurrence_edges": 890,
"speaker_face_edges": 120,
"face_face_edges": 234,
"mutual_gaze_edges": 67,
"lip_sync_edges": 12,
"has_appearance_edges": 38,
"wears_edges": 22,
"hand_object_edges": 18
},
"identity_agent": {
"clusters": 18,
"identities_created": 18,
"tmdb_matches": 5,
"speaker_bindings": 12,
"confirmations": 18
}
}
```
### Field Descriptions
#### Top Level
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | 32-char hex UUID |
| `file_name` | string | Original filename |
| `status` | string | File status: `registered`, `processing`, `completed`, `failed` |
| `processors` | array | Per-processor status from processing_status JSONB |
| `postgres` | object | PostgreSQL table counts |
| `qdrant` | object | Qdrant collection point counts |
| `tkg` | object | TKG node and edge counts by type |
| `identity_agent` | object | Identity Agent statistics |
#### PostgreSQL Stats
| Field | Type | Description |
|-------|------|-------------|
| `sentence_chunks` | integer | Rule 1 sentence chunks count |
| `trace_chunks` | integer | Face trace chunks count |
| `relationship_chunks` | integer | Rule 2 relationship chunks count |
| `identities` | integer | Unique identities bound to this file |
| `file_identities` | integer | File-identity mapping records |
#### Qdrant Stats
| Field | Type | Description |
|-------|------|-------------|
| `faces` | integer | Total face points in `_faces` collection |
| `face_traces` | integer | Unique trace IDs in `_faces` |
| `face_identities` | integer | Unique identity IDs bound in `_faces` |
| `text_chunks` | integer | Text chunk vectors in `momentry_*_rule1_v2` |
| `speakers` | integer | Speaker segments in `momentry_*_speaker` |
#### TKG Stats
| Field | Type | Description |
|-------|------|-------------|
| `total_nodes` | integer | Sum of all node types |
| `total_edges` | integer | Sum of all edge types |
| `face_track_nodes` | integer | Face track nodes |
| `gaze_track_nodes` | integer | Gaze track nodes |
| `lip_track_nodes` | integer | Lip track nodes |
| `text_region_nodes` | integer | Text region nodes |
| `appearance_nodes` | integer | Appearance trace nodes |
| `accessory_nodes` | integer | Accessory nodes |
| `object_nodes` | integer | YOLO object nodes |
| `hand_nodes` | integer | Hand nodes |
| `speaker_nodes` | integer | Speaker nodes |
| `co_occurrence_edges` | integer | Co-occurrence edges |
| `speaker_face_edges` | integer | Speaker-face edges |
| `face_face_edges` | integer | Face-face edges |
| `mutual_gaze_edges` | integer | Mutual gaze edges |
| `lip_sync_edges` | integer | Lip sync edges |
| `has_appearance_edges` | integer | Has-appearance edges |
| `wears_edges` | integer | Wears edges |
| `hand_object_edges` | integer | Hand-object edges |
#### Identity Agent Stats
| Field | Type | Description |
|-------|------|-------------|
| `clusters` | integer | Face clusters from face_clustered.json |
| `identities_created` | integer | Identities created from clusters |
| `tmdb_matches` | integer | TMDb identity matches |
| `speaker_bindings` | integer | Speaker-to-identity bindings |
| `confirmations` | integer | Confirmed identity bindings |
docs_v1.0/API_WORKSPACE/modules/18_profile.md
+476
View File
@@ -0,0 +1,476 @@
<!-- module: profile -->
<!-- description: Trace profile and file profile management — read/update face trace names, key frames, aliases, and file paths -->
<!-- depends: 01_auth, 07_identity, 15_tkg -->
## Profile Management
Endpoints for managing trace profiles (face track metadata stored in TKG) and file profiles (video metadata stored in PostgreSQL).
---
## Terminology
### Core Concepts
| Term | Definition | Identifier | Display Format |
|------|------------|------------|----------------|
| **Face** | A single human face detection on one frame | `face_id` | Usually not displayed |
| **Face Sequence (Trace)** | A collection of faces across multiple frames representing the same person | `trace_id` | `FS#233` |
| **Face Group** | A collection of traces sharing the same name/label | `label` (string) | `"Person A"` |
### Data Model Hierarchy
```
Video
└── Frame (F#233, F#234, ...) ← Single frame number
└── Face ← Single detection (bbox + confidence)
└── Face Sequence / Trace ← Same person across frames (FS#233)
└── Face Group ← Multiple traces with same name
```
### Example
```
Video: "interview.mp4"
├── Frame F#100
│ └── Face (bbox: {100, 200, 50, 50}, confidence: 0.95)
├── Frame F#105
│ └── Face (bbox: {110, 205, 48, 48}, confidence: 0.92)
├── Frame F#110
│ └── Face (bbox: {115, 210, 52, 52}, confidence: 0.88)
...
Face Sequence FS#233 = [Face@F#100, Face@F#105, Face@F#110, ...]
↓
Face Group "Person A" = [FS#233, FS#234, FS#235]
```
### Storage
| Entity | Storage | Table/Collection |
|--------|---------|------------------|
| Face | Qdrant | `_faces` collection |
| Face Sequence / Trace | PostgreSQL (TKG) | `tkg_nodes` where `node_type='face_track'` |
| Face Group | PostgreSQL (TKG) | `tkg_nodes.label` field |
### Operations
| Operation | Level | API |
|-----------|-------|-----|
| View faces | Face | Internal (embedded in trace data) |
| Merge traces | Trace | `POST /api/v1/trace/:file_uuid/merge` |
| Rename group | Group | `PUT /api/v1/trace-profile/group` |
| Merge groups | Group | `POST /api/v1/file/:file_uuid/groups/merge` |
### Naming Convention
- **Frame**: `F#{number}` — e.g., `F#233`
- **Face Sequence / Trace**: `FS#{number}` — e.g., `FS#233`
- **Face Group**: String name — e.g., `"Person A"`, `"Speaker 1"`
### `GET /api/v1/trace-profile`
**Auth**: Required
**Scope**: file-level
Read a single face trace's profile including name, key frame, key face, and multi-language aliases.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File UUID |
| `trace_id` | integer | Yes | Trace ID (numeric) |
#### Example
```bash
curl -s "$API/api/v1/trace-profile?file_uuid=$FILE_UUID&trace_id=7" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"file_uuid": "9f6a9cd55a5809f977f5a6589b9045c5",
"trace_id": 7,
"name": "John Doe",
"key_frame": 640,
"key_face": "face_12345",
"aliases": {
"en": "John Doe",
"zh": "約翰"
},
"properties": {
"status": "bound",
"avg_bbox": {"x": 899, "y": 212, "width": 342, "height": 342},
"start_frame": 624,
"end_frame": 669,
"frame_count": 5
}
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | File UUID |
| `trace_id` | integer | Trace ID |
| `name` | string | Display name (from `tkg_nodes.label`) |
| `key_frame` | integer | Representative frame number, or null |
| `key_face` | string | Representative face ID, or null |
| `aliases` | object | Multi-language name aliases |
| `properties` | object | Full TKG node properties (bbox, frames, etc.) |
#### Error Responses
| HTTP | When |
|------|------|
| `404` | Trace not found |
| `401` | Missing or invalid API key |
---
### `PUT /api/v1/trace-profile`
**Auth**: Required
**Scope**: file-level
Update a single face trace's profile fields. Only provided fields are updated; others remain unchanged.
#### Request Body
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File UUID |
| `trace_id` | integer | Yes | Trace ID to update |
| `name` | string | No | New display name |
| `key_frame` | integer | No | Representative frame number |
| `key_face` | string | No | Representative face ID |
| `aliases` | object | No | Multi-language aliases `{"en": "...", "zh": "..."}` |
| `properties` | object | No | Additional properties to merge into existing JSONB |
#### Example
```bash
curl -s -X PUT "$API/api/v1/trace-profile" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "9f6a9cd55a5809f977f5a6589b9045c5",
"trace_id": 7,
"name": "John Doe",
"key_frame": 640,
"key_face": "face_12345",
"aliases": {"en": "John Doe", "zh": "約翰"}
}'
```
#### Response (200)
```json
{
"success": true,
"message": "Trace profile updated",
"file_uuid": "9f6a9cd55a5809f977f5a6589b9045c5",
"trace_id": 7
}
```
---
### `PUT /api/v1/trace-profile/group`
**Auth**: Required
**Scope**: file-level
Batch update the `name` (label) for multiple traces in a face group. Used when renaming a group.
#### Request Body
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File UUID |
| `trace_ids` | integer[] | Yes | List of trace IDs to update |
| `name` | string | Yes | New group name for all traces |
#### Example
```bash
curl -s -X PUT "$API/api/v1/trace-profile/group" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "9f6a9cd55a5809f977f5a6589b9045c5",
"trace_ids": [7, 2, 13],
"name": "Group A"
}'
```
#### Response (200)
```json
{
"success": true,
"message": "Updated 3 traces in group",
"file_uuid": "9f6a9cd55a5809f977f5a6589b9045c5",
"updated_count": 3
}
```
---
### `POST /api/v1/file/:file_uuid/groups/merge`
**Auth**: Required
**Scope**: file-level
Merge multiple face groups into one target group. All traces from source groups are reassigned to the target group name.
#### Request Body
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File UUID |
| `source_groups` | string[] | Yes | List of group names to merge from |
| `target_group_name` | string | Yes | Target group name to merge into |
#### Examples
**Merge 2 groups**:
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/groups/merge" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "c36f35685177c981aa139b66bbbccc5b",
"source_groups": ["Person A"],
"target_group_name": "Person B"
}'
```
**Merge 4 groups**:
```bash
curl -s -X POST "$API/api/v1/file/$FILE_UUID/groups/merge" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "c36f35685177c981aa139b66bbbccc5b",
"source_groups": ["Person A", "Person C", "Person D"],
"target_group_name": "Person B"
}'
```
#### Response (200)
```json
{
"success": true,
"file_uuid": "c36f35685177c981aa139b66bbbccc5b",
"source_groups": ["Person A", "Person C", "Person D"],
"target_group_name": "Person B",
"traces_merged": 12,
"message": "Merged 3 group(s) into 'Person B'"
}
```
#### Error Responses
| HTTP | Condition |
|------|-----------|
| `400` | Target group in source_groups list |
| `400` | Empty source_groups array |
| `500` | Database error |
#### Behavior
1. Find all traces with `label IN (source_groups)`
2. Update their labels to `target_group_name`
3. All source groups disappear (no traces left)
4. Target group contains all traces from merged groups
---
### Merging Groups: Two Methods
#### Method 1: Use Merge Groups API (Recommended)
```bash
POST /api/v1/file/:file_uuid/groups/merge
{
"file_uuid": "...",
"source_groups": ["Person A", "Person C"],
"target_group_name": "Person B"
}
```
**Pros**: Simple, only requires group names, supports multi-group merge
**Cons**: New API (requires backend update)
#### Method 2: Use Batch Update API (Existing)
```bash
PUT /api/v1/trace-profile/group
{
"file_uuid": "...",
"trace_ids": [13, 14, 15, 43, 44],
"name": "Person B"
}
```
**Pros**: Works with existing API
**Cons**: Frontend must collect all trace_ids from both groups
#### Example: Merge Group A and C into Group B
**Before**:
```
Group A: [13, 14, 15]
Group B: [43, 44]
Group C: [67, 68]
```
**Method 1 (Recommended)**:
```bash
curl -X POST "$API/api/v1/file/$FILE_UUID/groups/merge" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "c36f35685177c981aa139b66bbbccc5b",
"source_groups": ["Person A", "Person C"],
"target_group_name": "Person B"
}'
```
**Method 2 (Existing API)**:
```bash
curl -X PUT "$API/api/v1/trace-profile/group" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "c36f35685177c981aa139b66bbbccc5b",
"trace_ids": [13, 14, 15, 43, 44, 67, 68],
"name": "Person B"
}'
```
**After**:
```
Group A: [] (disappeared)
Group B: [43, 44, 13, 14, 15, 67, 68]
Group C: [] (disappeared)
```
---
### `GET /api/v1/file-profile`
**Auth**: Required
**Scope**: file-level
Read a file's metadata including path, name, status, and technical details.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File UUID |
#### Example
```bash
curl -s "$API/api/v1/file-profile?file_uuid=$FILE_UUID" \
-H "X-API-Key: $KEY"
```
#### Response (200)
```json
{
"file_uuid": "49884ce1c341953d1ad7bf67a77c30cc",
"file_name": "Dedicatoria.mp4",
"file_path": "/Users/accusys/momentry/var/sftpgo/data/demo/Dedicatoria.mp4",
"status": "completed",
"duration": 93.33,
"width": 1280,
"height": 720,
"fps": 30.0,
"total_frames": 0
}
```
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | File UUID |
| `file_name` | string | File name |
| `file_path` | string | Full filesystem path |
| `status` | string | `pending`, `processing`, `completed`, `failed` |
| `duration` | float | Duration in seconds |
| `width` | integer | Video width in pixels |
| `height` | integer | Video height in pixels |
| `fps` | float | Frames per second |
| `total_frames` | integer | Total frame count |
---
### `PUT /api/v1/file-profile`
**Auth**: Required
**Scope**: file-level
Update file metadata, typically used when a file is moved to a new location.
#### Request Body
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File UUID |
| `file_path` | string | No | New filesystem path |
| `file_name` | string | No | New file name |
#### Example
```bash
curl -s -X PUT "$API/api/v1/file-profile" \
-H "X-API-Key: $KEY" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "49884ce1c341953d1ad7bf67a77c30cc",
"file_path": "/new/location/Dedicatoria.mp4"
}'
```
#### Response (200)
```json
{
"success": true,
"message": "File profile updated",
"file_uuid": "49884ce1c341953d1ad7bf67a77c30cc"
}
```
---
## Data Storage
| Profile Type | Storage | Table |
|-------------|---------|-------|
| **trace_profile** | PostgreSQL (TKG) | `tkg_nodes` where `node_type='face_track'` |
| **file_profile** | PostgreSQL | `videos` |
### Trace Profile Fields
| Field | TKG Column | Description |
|-------|-----------|-------------|
| `name` | `label` | Display name for the trace |
| `key_frame` | `properties->'key_frame'` | Representative frame number |
| `key_face` | `properties->'key_face'` | Representative face ID |
| `aliases` | `properties->'aliases'` | Multi-language name aliases |
---
*Updated: 2026-07-25 — Added Merge Groups API (POST /groups/merge), added Terminology section (Face, Face Sequence, Face Group)*
*Updated: 2026-07-21 — Fixed external_id matching (trace_N + face_track_N formats), fixed parameter ordering in UPDATE query*
*Updated: 2026-07-18 — New profile module: trace-profile (GET, PUT, PUT group) and file-profile (GET, PUT)*
docs_v1.0/API_WORKSPACE/modules/19_people_api.md
+396
View File
@@ -0,0 +1,396 @@
# People API
**Version**: 2.0
**Date**: 2026-07-26
**Base URL**: `http://localhost:3002`
**Auth**: Requires API key header (`Authorization: Bearer <api_key>`)
**Doc Path**: `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/19_people_api.md`
---
## 資料架構
### trace_profiles（PostgreSQL — People Search 專用）
每筆 `file_uuid + trace_id` 對應一個 face trace profile：
| 欄位 | 類型 | 說明 |
|------|------|------|
| `file_uuid` | string | 影片 UUID |
| `trace_id` | integer | Face trace ID（8Hz 取樣追蹤） |
| `name` | string | FS name（Face 頁面顯示 + People Search 搜尋） |
| `start_frame` | integer | 起始 frame |
| `end_frame` | integer | 結束 frame |
| `frame_count` | integer | 追蹤 frame 數（8Hz 取樣，多數 > 1） |
| `key_frame` | string | 關鍵幀圖片路徑 |
| `key_face` | string | 關鍵人臉裁切路徑 |
| `avg_confidence` | float | 平均偵測信心值 |
| `status` | string | 狀態（pending/confirmed） |
### 可搜尋註記（metadata / VLM 欄位）
`trace_profiles` 包含 VLM 產生的註記，可用於進階搜尋和統計：
| 欄位 | 類型 | 說明 |
|------|------|------|
| `vlm_description` | text | VLM 人物外貌描述 |
| `vlm_clothing` | text | VLM 衣著描述 |
| `vlm_tags` | text[] | VLM 標籤陣列 |
| `vlm_location` | string | VLM 地點 |
| `vlm_setting` | string | VLM 場景設定 |
| `vlm_lighting` | string | VLM 光線 |
| `vlm_weather` | string | VLM 天氣 |
| `vlm_background` | text | VLM 背景描述 |
| `vlm_bg_tags` | text[] | VLM 背景標籤 |
### tkg_nodes（PostgreSQL — TKG 圖譜專用，與 profile 獨立）
| 欄位 | 說明 |
|------|------|
| `external_id` | 如 "trace_9"，對應 `trace_profiles.trace_id` |
| `label` | TKG 圖譜節點標籤（與 `trace_profiles.name` 獨立） |
| `properties` | TKG 節點屬性 |
### _faces（Qdrant — Face 向量比對）
Face embedding 向量存在 Qdrant `_faces` collection，可用作 seed 比對：
| Payload 欄位 | 類型 | 說明 |
|-------------|------|------|
| `file_uuid` | string | 影片 UUID |
| `trace_id` | integer | Face trace ID |
| `identity_id` | integer \| null | 已綁定的 identity ID |
| `frame` | integer | Frame 編號 |
| `embedding` | vector[512] | FaceNet 512-d embedding |
| `bbox` | object | 人臉 bbox（x, y, width, height） |
| `confidence` | float | 偵測信心值 |
---
## 1. GET /api/v1/search/people
搜尋已命名的 face trace profiles。回傳符合 name 的卡片列表。
### Query Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `query` | string | ✅ Yes | 搜尋關鍵字（ILIKE 模糊比對 `trace_profiles.name`） |
| `file_uuid` | string | ❌ Optional | 限制搜尋範圍 |
| `limit` | integer | ❌ Optional | 回傳筆數上限（預設 10） |
### Example Request
```bash
# 搜尋名字
curl -X GET "http://localhost:3002/api/v1/search/people?query=Susan"
# 搜尋 VLM 註記（衣著、地點、標籤）
curl -X GET "http://localhost:3002/api/v1/search/people?query=blue+shirt"
# 限定影片
curl -X GET "http://localhost:3002/api/v1/search/people?file_uuid=abc123&query=Tom"
```
### 搜尋範圍
People Search 會搜尋以下欄位（ILIKE 模糊比對）：
| 欄位 | 說明 |
|------|------|
| `name` | FS name |
| `vlm_description` | VLM 人物外貌描述 |
| `vlm_clothing` | VLM 衣著描述 |
| `vlm_tags` | VLM 標籤陣列 |
| `vlm_location` | VLM 地點 |
| `vlm_setting` | VLM 場景設定 |
### Response
```json
{
"people": [
{
"file_uuid": "4655a0ab3c077e30c12b2298c2750650",
"file_name": "",
"trace_id": 9,
"external_id": "trace_9",
"name": "Susan",
"start_frame": 118,
"end_frame": 384,
"frame_count": 39,
"start_time": null,
"end_time": null,
"key_frame": "key_frame.jpg",
"key_face": null,
"avg_confidence": 0.694
}
],
"total": 18
}
```
### Response Fields
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | 影片 UUID |
| `trace_id` | integer | Face trace ID |
| `external_id` | string \| null | TKG node external_id（如 "trace_9"） |
| `name` | string \| null | 使用者命名的 FS name |
| `start_frame` | integer \| null | 起始 frame |
| `end_frame` | integer \| null | 結束 frame |
| `frame_count` | integer \| null | 追蹤 frame 數（8Hz 取樣） |
| `start_time` | float \| null | 起始時間（秒） |
| `end_time` | float \| null | 結束時間（秒） |
| `key_frame` | string \| null | 關鍵幀圖片路徑 |
| `key_face` | string \| null | 關鍵人臉裁切路徑 |
| `avg_confidence` | float \| null | 平均偵測信心值 |
---
## 2. POST /api/v1/search/universal
統一搜尋。設定 `types: ["people"]` 搜尋人物，或組合 `"chunk"`、`"frame"`。
People Search 會搜尋 `trace_profiles` 的 `name` + `vlm_*` 欄位：
- `name` — FS name
- `vlm_description` — VLM 人物外貌描述
- `vlm_clothing` — VLM 衣著描述
- `vlm_tags` — VLM 標籤陣列
- `vlm_location` — VLM 地點
- `vlm_setting` — VLM 場景設定
### Request Body
```json
{
"query": "Susan",
"file_uuid": "abc123",
"types": ["people"],
"page": 1,
"page_size": 20
}
```
### Request Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `query` | string | ✅ Yes | 搜尋關鍵字 |
| `file_uuid` | string | ❌ Optional | 限制搜尋範圍 |
| `types` | string[] | ❌ Optional | `["chunk", "frame", "people"]` — 預設全部 |
| `page` | integer | ❌ Optional | 頁碼（預設 1） |
| `page_size` | integer | ❌ Optional | 每頁筆數（預設 20，最大 200） |
### Response
```json
{
"query": "Susan",
"results": [
{
"type": "person",
"file_uuid": "4655a0ab3c077e30c12b2298c2750650",
"trace_id": 9,
"external_id": "trace_9",
"name": "Susan",
"frame_count": 39,
"score": 0.95,
"start_time": null,
"end_time": null,
"key_frame": "key_frame.jpg",
"key_face": null
}
],
"total": 18,
"page": 1,
"page_size": 20,
"has_more": false
}
```
---
## 3. PUT /api/v1/trace-profile（單筆改名）
更新單一 face trace profile 的 name 和 metadata。
### Request Body
```json
{
"file_uuid": "4655a0ab3c077e30c12b2298c2750650",
"trace_id": 9,
"name": "Susan",
"properties": { "custom_key": "custom_value" },
"key_frame": "path/to/key_frame.jpg",
"key_face": "path/to/key_face.jpg",
"aliases": ["Susie", "Sue"]
}
```
### Request Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | ✅ Yes | 影片 UUID |
| `trace_id` | integer | ✅ Yes | Face trace ID |
| `name` | string | ❌ Optional | 新的 FS name |
| `properties` | object | ❌ Optional | 自訂屬性（合併更新） |
| `key_frame` | string | ❌ Optional | 關鍵幀路徑 |
| `key_face` | string | ❌ Optional | 關鍵人臉路徑 |
| `aliases` | string[] | ❌ Optional | 別名列表 |
---
## 4. PUT /api/v1/trace-profile/group（批次改名）
批次更新多個 face trace profiles 的 name。
### Request Body
```json
{
"file_uuid": "4655a0ab3c077e30c12b2298c2750650",
"trace_ids": [9, 22, 5],
"name": "Susan"
}
```
---
## 5. POST /api/v1/groups/merge（合併群組）
將多個 group 的 traces 合併為一個 group name。
### Request Body
```json
{
"file_uuid": "4655a0ab3c077e30c12b2298c2750650",
"source_groups": ["Person_0", "Person_1"],
"target_group_name": "Susan"
}
```
---
## Error Responses
### 400 Bad Request
```json
{ "error": "name is required" }
```
### 401 Unauthorized
```
HTTP 401 (no body) — Missing or invalid API key
```
### 404 Not Found
```json
{ "error": "Profile not found" }
```
### 500 Internal Server Error
```json
{ "error": "DB error: connection refused" }
```
---
## 如何命名 Face Trace
使用 profile API 為 face trace 命名：
```bash
curl -X PUT "http://localhost:3002/api/v1/trace-profile" \
-H "Authorization: Bearer <api_key>" \
-H "Content-Type: application/json" \
-d '{
"file_uuid": "4655a0ab3c077e30c12b2298c2750650",
"trace_id": 9,
"name": "Susan"
}'
```
命名後即可透過 People Search API 搜尋：
```bash
curl "http://localhost:3002/api/v1/search/people?query=Susan" \
-H "Authorization: Bearer <api_key>"
```
---
## Pipeline 狀態定義
所有檔案的 status 由 `/api/v1/file/{file_uuid}/sync-status` 自動更新，依據 processor outputs 和功能就緒狀態判定。
| Status | 前端顯示 | 條件 | 啟用功能 |
|--------|---------|------|---------|
| `registered` | 📋 已註冊 | 0 processor outputs | 檔案註冊 |
| `scanning` | 🔍 掃描中 | 1-4/5 processor outputs | — |
| `keyword_ready` | 🔎 關鍵字可用 | sentence chunks > 0 | Smart Search |
| `semantic_ready` | 🧠 語意可用 | embedded chunks > 0 | Semantic Search |
| `face_mgmt_ready` | 🎭 臉部管理可用 | face_trace + trace_profiles + tkg_nodes(face_track) | People Search, Face Profile 讀寫, Face Group |
| `agent_ready` | 🤖 Agent 可用 | face_mgmt_ready + tkg_edges | Agent TKG Tools |
| `completed` | ✅ 完成 | 5/5 processors + 全部功能就緒 | — |
### Face Management Ready 明確定義
`face_mgmt_ready` 表示檔案已具備完整臉部管理功能，需同時滿足以下四個條件：
| 依賴項 | 檢查條件 | 說明 | 提供功能 |
|--------|---------|------|---------|
| **Face Qdrant Ready** | `_faces` collection 有 face points | Face embeddings 已存入 Qdrant | 臉部比對、相似度搜尋 |
| **Face Trace Ready** | `face_traced.json` 存在且有 traces | Face tracking 已完成 | 人臉軌跡資料 |
| **Face Profile Ready** | `trace_profiles` 有紀錄 | Profile 資料已建立 | 名字、VLM 註記、key_frame/key_face |
| **Face TKG Node Ready** | `tkg_nodes` 有 `face_track` nodes | TKG 圖譜節點已建立 | 臉部群組合併、關係查詢 |
#### 檢查 SQL
```sql
-- Face Qdrant Ready
SELECT COUNT(*) FROM _faces WHERE file_uuid = $1; -- > 0
-- Face Trace Ready
-- 檢查 face_traced.json 檔案存在且有 traces
-- Face Profile Ready
SELECT COUNT(*) FROM trace_profiles WHERE file_uuid = $1; -- > 0
-- Face TKG Node Ready
SELECT COUNT(*) FROM tkg_nodes WHERE file_uuid = $1 AND node_type = 'face_track'; -- > 0
```
#### 啟用功能清單
| 功能 | API Endpoint | 說明 |
|------|-------------|------|
| People Search | `GET /api/v1/search/people` | 搜尋已命名人臉 |
| Face Profile 讀寫 | `PUT /api/v1/trace-profile` | 更新名字、VLM 註記 |
| Face Group 合併 | `POST /api/v1/groups/merge` | 合併多個群組 |
| Face Trace 改名 | `PUT /api/v1/trace-profile` | 為單個 trace 命名 |
| Agent Face Search | Agent Tool: `face_profile_search` | Agent 搜尋人臉 |
---
### 狀態依賴關係
```
registered → scanning → keyword_ready → semantic_ready
↓
face_mgmt_ready → agent_ready → completed
```
### 每個階段對應的 Processor
| 階段 | 必要 Processor | 產出 |
|------|---------------|------|
| `keyword_ready` | ASR, ASRX | sentence chunks |
| `semantic_ready` | keyword_ready + Vectorize | embedded chunks |
| `face_mgmt_ready` | Face, FaceCluster | trace_profiles (People Search + Face Management) |
| `agent_ready` | face_mgmt_ready + TKG Edges | tkg_edges |
docs_v1.0/API_WORKSPACE/modules/21_studio_team_guide.md
+199
View File
@@ -0,0 +1,199 @@
# Studio Team API 變更指南
**Version**: 1.0
**Date**: 2026-07-26
**Doc Path**: `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/21_studio_team_guide.md`
**目標**: 通知 Studio team 後端 API 狀態欄位變更，Momentry Studio 前端需配合更新
---
## 1. 概述
後端已將檔案狀態從技術術語改為**功能導向**名稱。Studio team 的 `~/momentry_studio/` 前端顯示邏輯需要更新。
**影響範圍：**
- 後端 API：`/Users/accusys/momentry_core/src/api/files.rs`（Core team 已修改）
- Studio 前端：`~/momentry_studio/`（需 Studio team 更新）
---
## 2. 團隊職責
| 團隊 | 負責專案 | 目錄 |
|------|---------|------|
| **Core team** | Momentry Core 後端 | `~/momentry_core/` |
| **Studio team** | Momentry Studio 前端 | `~/momentry_studio/` |
| **Marcom team** | WordPress 網站 | `/Users/accusys/wordpress/` |
---
## 3. API 變更
### `/api/v1/file/{file_uuid}/sync-status` 回傳格式
**POST** `/api/v1/file/{file_uuid}/sync-status`
```json
{
"success": true,
"file_uuid": "4655a0ab3c077e30c12b2298c2750650",
"status": "face_mgmt_ready",
"processors_complete": 4,
"processors_total": 5,
"worker": {
"has_job": true,
"job_status": "pending",
"current_processor": null,
"progress_total": 7,
"progress_current": 7,
"completed_processors": ["cut", "asr", "face", "ocr", "pose", "asrx", "face_cluster"],
"failed_processors": [],
"updated_at": "2026-07-24T21:26:06"
}
}
```
**無 job 的檔案：**
```json
{
"success": true,
"file_uuid": "...",
"status": "registered",
"processors_complete": 0,
"processors_total": 5,
"worker": {
"has_job": false
}
}
```
### Worker 狀態說明
| 欄位 | 說明 |
|------|------|
| `has_job` | 是否有 monitor_job 紀錄 |
| `job_status` | pending / running / completed / failed |
| `current_processor` | 目前正在執行的 processor |
| `progress_total` | 總 processor 數量 |
| `progress_current` | 已完成的 processor 數量 |
| `completed_processors` | 已完成的 processor 列表 |
| `failed_processors` | 失敗的 processor 列表 |
| `updated_at` | 最後更新時間 |
### 使用者可區分的情境
| 情境 | `job_status` | `current_processor` | 顯示建議 |
|------|-------------|-------------------|---------|
| **正在處理** | `running` | `face` | 🔄 正在執行 Face |
| **排隊等待** | `pending` | `null` | ⏳ 等待處理 |
| **已完成** | `completed` | `null` | ✅ Processor 完成 |
| **處理失敗** | `failed` | `face` | ❌ Face 失敗 |
| **從未提交** | `has_job=false` | - | 📋 僅註冊 |
### 新狀態清單
| Status | 中文顯示 | 圖示 | 說明 |
|--------|---------|------|------|
| `registered` | 已註冊 | 📋 | 0 processor outputs |
| `scanning` | 掃描中 | 🔍 | 1-4/5 processor outputs |
| `keyword_ready` | 關鍵字可用 | 🔎 | sentence chunks > 0 |
| `semantic_ready` | 語意可用 | 🧠 | embedded chunks > 0 |
| `face_mgmt_ready` | 臉部管理可用 | 🎭 | trace_profiles + tkg_nodes > 0 |
| `agent_ready` | Agent 可用 | 🤖 | tkg_edges > 0 |
| `completed` | 已完成 | ✅ | 全部就緒 |
### 已移除的舊狀態
| 舊狀態 | 新狀態 |
|--------|--------|
| `pending` | `registered` |
| `processing` | `scanning` |
---
## 4. Studio 前端需修改項目
### 4.1 狀態顯示邏輯
**位置：** `~/momentry_studio/` 中處理檔案狀態顯示的元件
**修改前：**
```javascript
switch (file.status) {
case 'completed': return '✅ 已完成';
case 'processing': return '🔄 處理中';
case 'pending': return '⏳ 待處理';
default: return '📦 未註冊';
}
```
**修改後：**
```javascript
const statusMap = {
completed: { label: '已完成', icon: '✅', color: 'green' },
agent_ready: { label: 'Agent 可用', icon: '🤖', color: 'orange' },
face_mgmt_ready: { label: '臉部管理可用', icon: '🎭', color: 'cyan' },
semantic_ready: { label: '語意可用', icon: '🧠', color: 'purple' },
keyword_ready: { label: '關鍵字可用', icon: '🔎', color: 'green' },
scanning: { label: '掃描中', icon: '🔍', color: 'blue' },
registered: { label: '已註冊', icon: '📋', color: 'gray' },
};
const display = statusMap[file.status] || { label: '未註冊', icon: '📦', color: 'gray' };
return `${display.icon} ${display.label}`;
```
### 4.2 狀態過濾選項
**位置：** `~/momentry_studio/` 中檔案列表過濾元件
**移除：** `pending`, `processing`
**新增：** `registered`, `scanning`, `keyword_ready`, `semantic_ready`, `face_mgmt_ready`, `agent_ready`
---
## 5. Face Management Ready 明確定義
`face_mgmt_ready` 表示檔案已具備完整臉部管理功能，需同時滿足以下四個條件：
| 依賴項 | 檢查條件 | 說明 |
|--------|---------|------|
| **Face Qdrant Ready** | `_faces` collection 有 face points | Face embeddings 已存入 Qdrant |
| **Face Trace Ready** | `face_traced.json` 存在且有 traces | Face tracking 已完成 |
| **Face Profile Ready** | `trace_profiles` 有紀錄 | Profile 資料已建立 |
| **Face TKG Node Ready** | `tkg_nodes` 有 `face_track` nodes | TKG 圖譜節點已建立 |
---
## 6. 狀態依賴關係
```
registered → scanning → keyword_ready → semantic_ready
↓
face_mgmt_ready → agent_ready → completed
```
---
## 7. 測試 API
```bash
# 取得所有檔案
curl http://localhost:3002/api/v1/files \
-H "Authorization: Bearer <api_key>"
# 觸發狀態同步
curl -X POST http://localhost:3002/api/v1/file/{file_uuid}/sync-status \
-H "Authorization: Bearer <api_key>"
```
---
## 8. 參考文件
| 文件 | 完整路徑 |
|------|---------|
| People API | `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/19_people_api.md` |
| Status Unification | `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/20_status_unification.md` |
| Studio Team Guide | `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/21_studio_team_guide.md` |
| AGENTS.md | `/Users/accusys/momentry_core/AGENTS.md` |
docs_v1.0/API_WORKSPACE/modules/22_studio_pipeline_changes.md
+207
View File
@@ -0,0 +1,207 @@
# Studio Team 配套修改指南
**Version**: 1.0
**Date**: 2026-07-26
**Doc Path**: `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/22_studio_pipeline_changes.md`
**目標**: 通知 Studio team 後端 Pipeline 狀態變更，前端需配合更新
---
## 1. 概述
後端已完成以下修改：
1. 移除 `identity_agent`，替換為 `face_dedup`（Face Deduplication）
2. 重構 `sync_file_status` 邏輯：檢查 JSON 存在 + DB 一致性
3. Pipeline 進度階段重新分配
**影響範圍：**
- 檔案狀態顯示（status）
- Pipeline 進度顯示（progress stages）
- 統計 API 回傳格式
---
## 2. 檔案狀態變更
### 新狀態清單
| Status | 中文顯示 | 圖示 | 說明 |
|--------|---------|------|------|
| `registered` | 已註冊 | 📋 | 尚未開始處理 |
| `scanning` | 掃描中 | 🔍 | 處理中（JSON 存在但 DB 不一致） |
| `completed` | 已完成 | ✅ | 所有 processor 完成且 DB 一致 |
| `agent_ready` | Agent 可用 | 🤖 | TKG edges 存在 |
### 已移除的狀態
| 舊狀態 | 新狀態 | 說明 |
|--------|--------|------|
| `processing` | `scanning` | 更明確表示正在處理 |
| `pending` | `registered` | 尚未開始 |
| `face_mgmt_ready` | `completed` | 最終狀態統一為 completed |
### 狀態判斷邏輯
| 條件 | 狀態 |
|------|------|
| 所有 JSON 存在且 DB 一致 | `completed` |
| 部分 JSON 存在但不一致 | `scanning` |
| 無 JSON 存在 | `registered` |
| TKG edges 存在 | `agent_ready` |
---
## 3. Pipeline 進度階段變更
### 修改前（7 個 stage）
```
Processors (30%) → Rule1 (5%) → Face Tracing (5%) → identity_agent (10%) → TKG Nodes (20%) → TKG Edges (15%) → Rule2 (15%)
```
### 修改後（7 個 stage）
```
Processors (30%) → Rule1 (5%) → Face Tracing (5%) → face_dedup (10%) → TKG Nodes (20%) → TKG Edges (15%) → Rule2 (15%)
```
### 需要修改的前端元件
| 元件 | 修改內容 |
|------|---------|
| Pipeline Progress Bar | 移除 `identity_agent`，新增 `face_dedup` |
| Stage Labels | 更新 stage 名稱 |
| Overall Progress | 重新計算權重 |
---
## 4. API 變更
### `/api/v1/file/{file_uuid}/sync-status`
**回傳格式變更：**
**修改前：**
```json
{
"status": "face_mgmt_ready",
"processors_complete": 5,
"processors_total": 5,
"worker": { ... }
}
```
**修改後：**
```json
{
"success": true,
"file_uuid": "...",
"status": "completed",
"processors": {
"asr": { "json_exists": true, "consistent": true },
"asrx": { "json_exists": true, "consistent": true },
"ocr": { "json_exists": true, "consistent": true },
"pose": { "json_exists": true, "consistent": true },
"cut": { "json_exists": true, "consistent": true },
"face": { "json_exists": true, "consistent": true },
"face_cluster": { "json_exists": true, "consistent": true }
},
"worker": { ... }
}
```
### `/api/v1/file/{file_uuid}/stats`
**回傳格式變更：**
**修改前：**
```json
{
"identity_agent": {
"clusters": 0,
"identities_created": 0,
"tmdb_matches": 0,
"speaker_bindings": 0,
"confirmations": 0
}
}
```
**修改後：**
```json
{
"face_dedup": {
"clusters": 5,
"face_tracks": 24,
"consistent": true
}
}
```
---
## 5. Studio 前端需修改項目
### 5.1 狀態顯示邏輯
**位置：** Studio 前端檔案列表元件
**修改前：**
```javascript
const statusLabels = {
'completed': '✅ 已完成',
'processing': '🔄 處理中',
'pending': '⏳ 待處理',
'face_mgmt_ready': '🎭 臉部管理可用',
'agent_ready': '🤖 Agent 可用'
};
```
**修改後：**
```javascript
const statusLabels = {
'completed': '✅ 已完成',
'scanning': '🔍 掃描中',
'registered': '📋 已註冊',
'agent_ready': '🤖 Agent 可用'
};
```
### 5.2 Pipeline 進度顯示
**位置：** Studio 前端 Pipeline Progress 元件
**修改項目：**
1. 移除 `identity_agent` stage
2. 新增 `face_dedup` stage
3. 更新 stage 名稱映射
### 5.3 統計面板
**位置：** Studio 前端檔案統計面板
**修改項目：**
1. 移除 `Identity Agent` 區塊
2. 新增 `Face Deduplication` 區塊
3. 顯示欄位：`clusters`, `face_tracks`, `consistent`
---
## 6. 測試清單
- [ ] 檔案列表狀態顯示正確
- [ ] Pipeline 進度階段正確顯示
- [ ] Overall Progress 計算正確
- [ ] 統計面板 Face Dedup 區塊顯示正確
- [ ] sync-status API 回傳格式解析正確
- [ ] stats API 回傳格式解析正確
---
## 7. 參考文件
| 文件 | 完整路徑 |
|------|---------|
| Studio Team Guide | `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/21_studio_team_guide.md` |
| People API | `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/19_people_api.md` |
| Pipeline Changes | `/Users/accusys/momentry_core/docs_v1.0/API_WORKSPACE/modules/22_studio_pipeline_changes.md` |
docs_v1.0/API_WORKSPACE/modules/23_studio_face_group_name_fix.md
+114
View File
@@ -0,0 +1,114 @@
# Studio Fix: Face Group Name 讀寫不一致
**Date**: 2026-07-27
**Author**: Studio Team
**Status**: ✅ 已修復並部署
---
## 問題描述
Studio 前端在 Face 頁面 rename group name 後，離開再回來時名稱未更新。但 People Search 可以查到新名稱。
### 根因
**寫入**和**讀取**走了不同的資料來源：
| 操作 | API Endpoint | 實際讀寫欄位 |
|------|-------------|------------|
| **寫入 (rename)** | `PUT /api/v1/trace-profile/group` | `trace_profiles.name` |
| **讀取 (卡片顯示)** | `GET /api/v1/file/:uuid/face-groups` | `tkg_nodes.label` |
`trace_profiles.name` 和 `tkg_nodes.label` 是兩張獨立的表/欄位（見 `19_people_api.md`），rename 只更新了 `trace_profiles.name`，但卡片顯示讀的是 `tkg_nodes.label`。
---
## 修復內容
### 修改檔案
`/Users/accusys/momentry_core/src/api/profile.rs` — `get_face_groups_handler` (line 552-591)
### 改前 SQL
```sql
SELECT label, properties FROM tkg_nodes
WHERE file_uuid = $1 AND node_type = 'face_track'
ORDER BY (properties->>'trace_id')::int
```
### 改後 SQL
```sql
SELECT COALESCE(tp.name, tn.label) as name, tn.properties
FROM tkg_nodes tn
LEFT JOIN trace_profiles tp ON tp.file_uuid = tn.file_uuid
AND tp.trace_id = (tn.properties->>'trace_id')::int
WHERE tn.file_uuid = $1 AND tn.node_type = 'face_track'
ORDER BY (tn.properties->>'trace_id')::int
```
### Rust 程式碼變更
```diff
pub async fn get_face_groups_handler(...) {
let tkg_table = schema::table_name("tkg_nodes");
+ let tp_table = schema::table_name("trace_profiles");
let rows: Vec<(String, serde_json::Value)> = sqlx::query_as(&format!(
- "SELECT label, properties FROM {} \
- WHERE file_uuid = $1 AND node_type = 'face_track' \
- ORDER BY (properties->>'trace_id')::int",
- tkg_table
+ "SELECT COALESCE(tp.name, tn.label) as name, tn.properties \
+ FROM {} tn \
+ LEFT JOIN {} tp ON tp.file_uuid = tn.file_uuid \
+ AND tp.trace_id = (tn.properties->>'trace_id')::int \
+ WHERE tn.file_uuid = $1 AND tn.node_type = 'face_track' \
+ ORDER BY (tn.properties->>'trace_id')::int",
+ tkg_table, tp_table
))
...
- for (label, properties) in rows {
+ for (name, properties) in rows {
...
- if label.starts_with("Face Trace ") || label.starts_with("Trace ") {
+ if name.starts_with("Face Trace ") || name.starts_with("Trace ") {
unassigned.push(trace_id);
} else {
- groups.entry(label).or_default().push(trace_id);
+ groups.entry(name).or_default().push(trace_id);
}
}
}
```
---
## 影響範圍
| 項目 | 影響 |
|------|------|
| **Face 頁面卡片顯示** | ✅ 現在從 `trace_profiles.name` 讀取，rename 後立即生效 |
| **People Search** | 不受影響（本來就查 `trace_profiles.name`） |
| **未命名 traces** | 不受影響（`COALESCE` fallback 到 `tkg_nodes.label`，如 `Person_0`） |
| **Cluster Agent 初始化** | 不受影響（初始化時兩邊都寫入相同值） |
---
## 注意事項（給 Core Team）
1. **未來如果有需要同時更新 `tkg_nodes.label` 的情境**（例如 graph 顯示需要），請注意 `PUT /api/v1/trace-profile/group` 目前只更新 `trace_profiles.name`
2. **建議**：如果 `tkg_nodes.label` 和 `trace_profiles.name` 應該保持一致，可以考慮：
- 方案 A（目前做法）：讀取端用 `COALESCE` 優先取 `trace_profiles.name`
- 方案 B：寫入端同時更新兩個欄位（需要改 `update_trace_profile_group_handler`）
3. **`merge_groups_handler`** 目前同時更新 `trace_profiles.name` 和 `tkg_nodes.label`（line 278-310），行為正確，不需要改
---
## 驗證
```bash
# 測試 face-groups endpoint 正確回傳 rename 後的名稱
curl -s "http://localhost:3002/api/v1/file/4655a0ab3c077e30c12b2298c2750650/face-groups" \
-H "X-API-Key: <key>" | jq '.face_groups[] | {name, trace_count}'
# 預期輸出包含 "Susan"（已 rename 的 group）
# { "name": "Susan", "trace_count": 7 }
```
docs_v1.0/API_WORKSPACE/modules/99_incomplete.md
+188
View File
@@ -0,0 +1,188 @@
<!-- module: incomplete -->
<!-- description: Incomplete, stub, or undocumented API endpoints — tracking list -->
<!-- depends: 01_auth -->
## Incomplete / Undocumented APIs
This module tracks API endpoints that exist in the codebase but are either undocumented, partially documented, or stubs.
> **Note**: Endpoints listed here should be fully documented and moved to their appropriate module once implemented.
---
## Identity Binding
### `POST /api/v1/identity/:identity_uuid/bind`
**Auth**: Required
**Scope**: identity-level
Bind a single face detection to an identity. Unlike `bind/trace` which binds all faces in a trace, this binds one specific face.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuid` | string | Yes | File containing the face |
| `face_id` | string | Yes | Face detection ID to bind |
#### Status
⚠️ **Undocumented** — exists in code but no full request/response documentation.
---
## Resource Management
### `POST /api/v1/resource/register`
**Auth**: Required
**Scope**: system-level
Register an external resource (e.g., storage backend, API service).
#### Status
⚠️ **Undocumented** — endpoint exists but no documentation.
---
### `POST /api/v1/resource/heartbeat`
**Auth**: Required
**Scope**: system-level
Send heartbeat for a registered resource to verify it's still alive.
#### Status
⚠️ **Undocumented** — endpoint exists but no documentation.
---
### `GET /api/v1/resources`
**Auth**: Required
**Scope**: system-level
List all registered resources with their status.
#### Status
⚠️ **Undocumented** — endpoint exists but no documentation.
---
## 5W1H Agent
### `POST /api/v1/agents/5w1h/analyze`
**Auth**: Required
**Scope**: file-level
Run 5W1H analysis on all cut scenes for a file. Uses LLM (Gemma4) to summarize each scene with who/what/where/when/why/how.
#### Status
⚠️ **Partially documented** — listed in `12_agent.md` but missing full request/response examples.
---
### `POST /api/v1/agents/5w1h/batch`
**Auth**: Required
**Scope**: system-level
Run 5W1H analysis on multiple files at once.
#### Request Parameters
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `file_uuids` | string[] | Yes | Array of file UUIDs to analyze |
#### Status
⚠️ **Partially documented** — listed in `12_agent.md` but missing full request/response examples.
---
### `GET /api/v1/agents/5w1h/status`
**Auth**: Required
**Scope**: system-level
Get 5W1H analysis status across all videos (which files have been analyzed, which are pending).
#### Status
⚠️ **Partially documented** — listed in `12_agent.md` but missing full response schema.
---
## Identity Agent
### `POST /api/v1/agents/identity/match-from-photo`
**Auth**: Required
**Scope**: system-level
Match an identity using an uploaded photo. Extracts face embedding, finds best trace match.
#### Status
⚠️ **Partially documented** — exists in `08_identity_agent.md` but missing full response schema and error cases.
---
### `POST /api/v1/agents/identity/match-from-trace`
**Auth**: Required
**Scope**: file-level
Match an identity using a trace. Multi-angle embedding comparison with propagation.
#### Status
⚠️ **Partially documented** — exists in `08_identity_agent.md` but missing full response schema and error cases.
---
## Stubs / Not Implemented
### Visual Search Endpoints
| Method | Endpoint | Status |
|--------|----------|--------|
| POST | `/api/v1/search/visual` | Stub — defined but not functional |
| POST | `/api/v1/search/visual/class` | Stub — defined but not functional |
| POST | `/api/v1/search/visual/density` | Stub — defined but not functional |
| POST | `/api/v1/search/visual/combination` | Stub — defined but not functional |
| POST | `/api/v1/search/visual/stats` | Stub — defined but not functional |
### Unmounted Routes
These endpoints are defined in source code but not mounted in the router:
| Endpoint | Notes |
|----------|-------|
| `/api/v1/search/people` | ✅ Mounted |
| `/api/v1/who` | Defined but not mounted |
| `/api/v1/who/candidates` | Defined but not mounted |
---
## Tracking
| Count | Status |
|-------|--------|
| Undocumented | 3 (resource management) |
| Partially documented | 5 (5W1H ×3, identity agent ×2) |
| Stub/not functional | 5 (visual search) |
| Defined but unmounted | 2 (who, who/candidates) |
| **Total** | **16** |
---
*Created: 2026-06-20 — Gap analysis from core API vs doc_wasm sync*
*Updated: 2026-06-20 — Initial tracking list*
docs_v1.0/API_WORKSPACE/narratives/marcom_intro.md
+36
View File
@@ -0,0 +1,36 @@
<!-- narrative: marcom_intro -->
<!-- description: Intro section for Marcom training manual -->
<!-- depends: -->
## About This Manual
This training manual is designed for the Marcom team to understand and use the Momentry Core API.
### Demo Credentials
**API Key**: `muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69`
**SFTPGo** (for video upload):
| Item | Value |
|------|-------|
| SFTP Host | `sftpgo.momentry.ddns.net` |
| SFTP Port | `2022` |
| Username | `demo` |
| Password | `demopassword123` |
| Web UI | `https://sftpgo.momentry.ddns.net` |
### Quick Examples
**List all videos:**
```bash
curl -s -H "X-API-Key: $KEY" "$API/api/v1/files/scan"
```
**Search:**
```bash
curl -s -X POST "$API/api/v1/search" \
-H "Content-Type: application/json" \
-H "X-API-Key: $KEY" \
-d '{"query": "example", "limit": 5}'
```
docs_v1.0/DESIGN/ASRX_HYBRID_PIPELINE_V1.0.md
+588
View File
@@ -0,0 +1,588 @@
# ASRX Hybrid Pipeline v1.0 — 聲紋分離混合架構
| 項目 | 內容 |
|------|------|
| **範圍** | ASRX 處理器重構：whisperx → VAD-first hybrid pipeline |
| **狀態** | Draft |
| **適用版本** | Momentry Core V4.0+ |
| **作者** | OpenCode / Warren |
| **建立日期** | 2026-06-01 |
---
## 1. 問題
### 1.1 現有問題
| 問題 | 說明 | 影響 |
|------|------|------|
| **Whisper 合併短句** | `whisper small` 會將兩個人的對話錯認成一個連續段 (A+B → 一句) | ASR segment 內混兩人話語，speaker 無法分離 |
| **ASRX v2 speaker_id = null** | `asrx_processor_v2.py` 使用 `whisperx.DiarizationPipeline()` 但該 API 未在 whisperx `__init__.py` 暴露 | 所有 segment speaker 均為 null |
| **文字丟失** | `asrx_processor_custom.py` 的 `SelfASRXFixed.process_with_segments()` 只輸出 `text: ""` | Rule 1 合併時無文字可用 |
| **錯誤的聲紋後端** | `asrx_processor_v2.py` 依賴 whisperx 內建 diarization，但該功能不穩定 | 準確度 ~85%，需 HF token |
| **多版本混亂** | 7 個 root-level 變體、14 個 asrx_self 檔案，生產環境使用錯誤版本 | 維護困難，不知哪個是對的 |
### 1.2 痛點場景
**兩個說話人短句來回切換**（訪談、對話）：
```
Audio: A(2s) → B(1.5s) → A(3s)
Whisper: ───────[0-7s, "A+B+A 全部混在一起"]───────
```
Whisper 在句間停頓處不切段，導致 ASR 時間邊界無法反映 speaker 切換。
---
## 2. 架構
### 2.1 核心原則
1. **VAD 先定邊界** — 用 VAD 在句間停頓處切段，取代 whisper 的邊界
2. **ASR 後做** — 每段各自轉錄，保有獨立文字
3. **聲紋聚類定 speaker** — ECAPA-TDNN + AgglomerativeClustering
### 2.2 5 步 Pipeline
```
Audio
│
① whisper (一次, 粗略定位)
│ 找到說話段 + 初步文字 + 語種
│ [0-7s, "今天天氣很好我覺得也不錯對啊", zh]
│
② VAD scan (在每段內細切)
│ 利用句間停頓切開
│ 段1 [0-2s] 段2 [2-3.5s] 段3 [3.5-7s]
│
③ whisper per refined segment (各段轉錄)
│ 段1 → "今天天氣很好" (zh, 0.98)
│ 段2 → "我覺得也不錯" (zh, 0.97)
│ 段3 → "對啊" (zh, 0.96)
│
④ ECAPA-TDNN per refined segment (聲紋提取)
│ 段1 → emb[0] (192-dim)
│ 段2 → emb[1] (192-dim)
│ 段3 → emb[2] (192-dim)
│
⑤ AgglomerativeClustering (聚類定 speaker)
│ emb[0]=SPEAKER_0, emb[1]=SPEAKER_1, emb[2]=SPEAKER_0
│
輸出:
start end text language speaker_id
0.0 2.0 今天天氣很好 zh SPEAKER_0
2.0 3.5 我覺得也不錯 zh SPEAKER_1
3.5 7.0 對啊 zh SPEAKER_0
```
### 2.3 流程圖
```
┌─────────────────────────────────────────────────────────────────────┐
│ asrx_processor.py │
│ (wrapper) │
│ │
│ ① ffprobe → select best track → ffmpeg → 16kHz WAV │
│ │
│ ② SelfASRXFixed.process(audio_wav, file_uuid) │
│ │ │
│ ├─ Step 1: whisper.transcribe() → rough segments │
│ ├─ Step 2: VAD scan each rough segment │
│ ├─ Step 3: whisper per refined segment → text+language │
│ ├─ Step 4: ECAPA-TDNN per segment → 192-dim embedding │
│ ├─ Step 5: AgglomerativeClustering → speaker_labels │
│ │ │
│ ├─ Step 6: Store embeddings in Qdrant │
│ │ └─ {file_uuid, speaker_id, text, language, start, end} │
│ │ │
│ └─ Step 7: Classify high-quality embeddings │
│ ├─ quality > threshold → reference profile │
│ ├─ 送入聲音分類模型推論性別/屬性 │
│ └─ 寫入 Qdrant (type: speaker_reference) │
│ │
│ ③ 輸出 JSON 格式 (不含 embedding) │
│ │
│ Rust: rule1_ingest.rs │
│ └─ pre_chunks(processor_type='asrx') → chunks │
└─────────────────────────────────────────────────────────────────────┘
```
---
## 3. 檔案組織
### 3.1 最終檔案結構
```
scripts/
├── asrx_processor.py ← production (cleaned custom.py)
│
└── asrx_self/ ← 核心庫
├── __init__.py ← package marker
├── vad.py ← Silero VAD (新增 scan_within_segment)
├── whisper_local.py ← 🆕 封裝 whisper 載入+轉錄
├── speaker_encoder.py ← ECAPA-TDNN 192-dim
├── speaker_cluster_fixed.py ← AgglomerativeClustering
└── main_fixed.py ← 🔧 重寫為 5 步 pipeline
```
### 3.2 刪除清單
**Root-level 變體**（全部刪除）：
| 檔案 | 原因 |
|------|------|
| `asrx_processor.py` | 原始 whisperx 版，diarization 壞的 |
| `asrx_processor_v2.py` | 同上，Rust 目前錯誤呼叫此檔 |
| `asrx_processor_v2_noalign.py` | 跳過對齊但 diarization 仍壞 |
| `asrx_processor_v2_transcribe.py` | 只轉錄不做 speaker |
| `asrx_processor_simplified.py` | 變體 |
| `asrx_processor_contract_v1.py` | 18KB，pyannote，需 HF token |
**asrx_self 內被取代的舊版**：
| 檔案 | 原因 | 取代者 |
|------|------|--------|
| `main.py` | 用 SpectralClustering，有 NaN 問題 | `main_fixed.py` |
| `speaker_cluster.py` | 用 SpectralClustering，不穩定 | `speaker_cluster_fixed.py` |
### 3.3 搬離清單
非生產工具搬至 `tools/asrx/`：
```
tools/asrx/
├── integrate_face_asrx_speaker.py
├── speaker_player_gui.py
├── speaker_player_gui_face.py
├── speaker_player_interactive.py
├── speaker_audio_player.py
├── test_long_movie.py
├── test_gui_face_player.py
└── docs/
├── FINAL_TEST_REPORT.md
├── GUI_FACE_PLAYER_USAGE.md
├── LONG_MOVIE_TEST_SUMMARY.md
└── SPEAKER_PLAYER_GUIDE.md
```
---
---
## 4. Qdrant 聲紋向量儲存
### 4.1 儲存流程
```
Step 4 輸出: 每個 refined segment 有 {embedding: [192-dim], text, language, start, end}
Step 5 輸出: 每個 segment 被標上 speaker_id {SPEAKER_0, SPEAKER_1, ...}
Step 6: Qdrant 儲存
┌─ 每個 segment → Qdrant point
│ point_id = hash(file_uuid + segment_index) ← 可重複查詢
│ vector = embedding (192-dim)
│ payload = {
│ "file_uuid": str, ← 聚類後填入
│ "speaker_id": str, ← 聚類後填入
│ "text": str, ← ASR 轉錄結果
│ "language": str, ← 語種 (zh/en/...)
│ "start_time": f64, ← 秒
│ "end_time": f64, ← 秒
│ "type": "speaker_embedding" ← 便於區分
│ }
└─
```
### 4.2 Qdrant Collection
| 項目 | 內容 |
|------|------|
| Collection Name | `momentry_speaker` (或共用現有 collection) |
| Vector Dimension | 192 (ECAPA-TDNN 輸出) |
| Distance Metric | Cosine |
| Point ID | `hash(file_uuid + "_" + segment_index)` |
### 4.3 Rust `upsert_speaker_embedding`
```rust
impl QdrantDb {
pub async fn upsert_speaker_embedding(
&self,
point_id: u64,
vector: &[f32],
file_uuid: &str,
speaker_id: &str,
text: &str,
language: &str,
start_time: f64,
end_time: f64,
) -> Result<()> {
// Qdrant PUT /collections/{collection}/points?wait=true
// payload: {file_uuid, speaker_id, text, language, start_time, end_time, type: "speaker_embedding"}
}
}
```
### 4.4 與現有 Face Embedding 的關係
| 類別 | Qdrant Collection | Dim | Payload |
|------|-------------------|-----|---------|
| Face | `momentry` (self.collection_name) | 512 (FaceNet) | `file_uuid, trace_id, frame_number` |
| **Speaker** | `momentry` 或獨立 collection | **192** (ECAPA-TDNN) | `file_uuid, speaker_id, text, language, start, end` |
---
## 5. 模組詳細設計
### 5.1 `vad.py` — 語音活動檢測
| 項目 | 內容 |
|------|------|
| 模型 | Silero VAD (torch.hub, snakers4/silero-vad) |
| 現有函數 | `load_vad_model()`, `extract_speech_segments()` |
| **新增函數** | **`scan_within_segment(wav, start_sec, end_sec, model, utils, min_speech_duration_ms=500)`** |
`scan_within_segment` 作用：
- 在一個時間範圍 `[start_sec, end_sec]` 內執行 VAD 掃描
- 只回傳該範圍內的語音子片段 `[(s1, e1), (s2, e2), ...]`
- 利用句間停頓切分，解決 whisper 合併問題
### 5.2 `whisper_local.py` 🆕 — Whisper 封裝
| 項目 | 內容 |
|------|------|
| 模型 | `whisper.load_model("base")` (可設定) |
| 函數 | `load_model()`, `transcribe_segment(audio, start, end)` |
```python
def transcribe_segment(wav, sample_rate, start_sec, end_sec, model) -> dict:
"""轉錄單一段落，回傳 {text, language, lang_prob, segments}"""
```
每段獨立轉錄，保留語言與信心度。
### 5.3 `speaker_encoder.py` — 聲紋編碼器
| 項目 | 內容 |
|------|------|
| 模型 | SpeechBrain ECAPA-TDNN (`spkrec-ecapa-voxceleb`) |
| 輸出維度 | 192-dim |
| EER | 0.80% (VoxCeleb1) |
| 授權 | MIT (不需要 HuggingFace token) |
| 函數 | `load_speaker_encoder()`, `extract_speaker_embedding()`, `extract_speaker_embeddings_batch()` |
### 5.4 `speaker_cluster_fixed.py` — 說話人聚類
| 項目 | 內容 |
|------|------|
| 演算法 | AgglomerativeClustering (cosine + average linkage) |
| 取代 | `speaker_cluster.py` (SpectralClustering, NaN 問題) |
| 函數 | `robust_speaker_clustering(embeddings, n_speakers=None, max_speakers=10)` |
### 5.5 `main_fixed.py` 🔧 — 核心調度器（7 步 Pipeline）
```python
class SelfASRXFixed:
def process(self, audio_path, output_path=None, file_uuid=None):
"""
7 步 speaker diarization pipeline
Steps:
1. whisper.transcribe(audio) → rough segments + text + language
2. VAD scan each rough segment → refined segments
3. whisper per refined segment → {text, language, lang_prob}
4. ECAPA-TDNN per refined segment → 192-dim embeddings
5. AgglomerativeClustering → speaker_labels
6. Store all embeddings in Qdrant (if file_uuid provided)
payload: {file_uuid, speaker_id, text, language, start_time, end_time, type: "speaker_embedding"}
7. High-quality embeddings (quality > threshold) → classify + store reference
payload: {type: "speaker_reference", file_uuid, speaker_id, n_segments, avg_quality, ...}
Returns:
{
"segments": [
{
"start": float, "end": float,
"text": str, "language": str,
"lang_prob": float, "speaker": str,
"speaker_id": str, "quality": float
},
...
],
"speaker_stats": {...},
"n_speakers": int,
"total_duration": float,
"references": [
{
"speaker_id": str,
"n_segments": int,
"avg_quality": float,
"gender": str
}
]
}
"""
def _store_speaker_embeddings(self, segments, file_uuid):
"""Step 6: 每個 segment 的 192-dim embedding 存入 Qdrant"""
def _classify_high_quality_speakers(self, segments, embeddings, labels, file_uuid):
"""Step 7: 高品質聲紋分級 + 分類 → Qdrant reference profile"""
**移除**：
| 舊方法 | 原因 |
|--------|------|
| `process_with_segments(audio, asr_segments)` | 外部 ASR 邊界來源不可靠，被 VAD 取代 |
| `process()` VAD-only fallback | 無文字輸出，被完整 pipeline 取代 |
### 5.6 `speaker_classifier.py` 🆕 — 高品質聲紋分級與分類
#### 目的
聚類後，對每個 cluster 的 embedding 進行品質評估，高於閾值的獨立建檔，並用外部模型做自動分類。
#### 流程
```
Step ⑤ 聚類後，每個 segment 有 {embedding, speaker_id}
│
└─ Compute quality score per embedding
│
├─ 低於閾值 → 寫入 Qdrant (一般 speaker_embedding)
│
└─ 高於閾值 (quality > 0.85)
├─ 獨立建 reference profile
└─ 送入「支持聲音的模型」做分類
├─ 語者性別 (male/female)
├─ 語種口音 (zh-CN / zh-TW / en-US)
└─ 或跨影片 speaker 匹配用
```
#### Quality Score 計算
```python
def compute_embedding_quality(embeddings, labels, threshold=0.85):
"""
每個 embedding 到所屬 cluster centroid 的餘弦相似度
Args:
embeddings: [n_segments, 192]
labels: [n_segments] 聚類標籤
threshold: 高品質門檻
Returns:
qualities: [n_segments] 每個 embedding 的品質分數
high_quality_mask: [n_segments] bool 陣列
"""
from sklearn.metrics.pairwise import cosine_similarity
unique_labels = set(labels)
centroids = {}
for label in unique_labels:
mask = labels == label
centroid = np.mean(embeddings[mask], axis=0)
centroid = centroid / np.linalg.norm(centroid)
centroids[label] = centroid
qualities = []
for i, (emb, label) in enumerate(zip(embeddings, labels)):
sim = cosine_similarity([emb], [centroids[label]])[0][0]
qualities.append(sim)
return np.array(qualities), np.array(qualities) >= threshold
```
#### Reference Profile 格式
```json
{
"point_id": "hash(speaker_reference_" + file_uuid + "_" + speaker_id + "_" + cluster_index)",
"vector": "[192-dim centroid embedding]",
"payload": {
"type": "speaker_reference",
"file_uuid": "來源影片",
"speaker_id": "SPEAKER_0",
"n_segments": 25,
"avg_quality": 0.92,
"total_duration": 45.3,
"language": "zh",
"gender": "male",
"text_samples": ["今天天氣很好", "我覺得也不錯", "..."]
}
}
```
#### 支援的聲音分類模型（選項）
| 模型 | 用途 | 優點 | 缺點 |
|------|------|------|------|
| **SpeechBrain gender classifier** | 性別分類 | 已整合 ECAPA-TDNN | 只分 male/female |
| **CLAP** (LAION) | 零樣本音頻分類 | 可自訂 label text | 需額外安裝 |
| **YAMNet** | 聲音事件分類 | Google 出品，521 classes | 不擅長語者屬性 |
| **Wav2Vec2-BERT** (speechbrain) | 情感/屬性 | 多維度分類 | 模型較大 |
| **自建 identity classifier** | 跨影片 speaker 匹配 | 與現有 identity 系統對接 | 需累積 reference data |
> **待決定**: 選擇哪個分類模型，由後續 POC 決定。
#### `main_fixed.py` 新增方法
```python
class SelfASRXFixed:
# ... 既有 6 個步驟 ...
def _classify_high_quality_speakers(self, segments, embeddings, labels, file_uuid):
"""
步驟 7: 高品質聲紋分級與分類
1. 計算 quality score
2. 高於閾值者建立 reference profile
3. 用分類模型推論性別/屬性
4. 寫入 Qdrant (type: speaker_reference)
"""
qualities, mask = compute_embedding_quality(embeddings, labels)
for i, (seg, emb, label, quality, is_high) in enumerate(
zip(segments, embeddings, labels, qualities, mask)
):
seg["quality"] = float(quality)
if is_high:
profile = self._build_reference_profile(
emb, seg, file_uuid
)
# 分類 (placeholder)
# gender = classify_gender(embedding)
self._store_speaker_reference(profile)
```
### 5.7 `asrx_processor.py` — 清理後的 wrapper
清理項目：
| 問題 | 位置 | 修法 |
|------|------|------|
| 硬編碼 UUID `dd61fda8...` | line 155 | 移除該 fallback path |
| `os.chdir(script_dir)` | line 112 | 改區域性 Path 操作 |
| ASR 文字丟棄 | line 258 | `text` 來自新 pipeline |
| `_debug` dict | line 222 | 移除 |
| `max_speakers=10` 寫死 | line 201 | 改 CLI 參數 `--max-speakers` |
| 載入外部 ASR segments | line 148-174 | 移除（不再需要） |
---
## 6. 輸出格式
### 6.1 ASRX JSON Output (由 `asrx_processor.py` 寫入)
> **注意**: 192-dim embedding 不在此 JSON 中。embedding 在 Python 端直接送入 Qdrant，JSON 只保留中繼資料。
```json
{
"language": "zh",
"segments": [
{
"start_time": 0.0,
"end_time": 2.0,
"start_frame": 0,
"end_frame": 60,
"text": "今天天氣很好",
"speaker_id": "SPEAKER_0",
"language": "zh",
"lang_prob": 0.98
},
{
"start_time": 2.0,
"end_time": 3.5,
"start_frame": 60,
"end_frame": 105,
"text": "我覺得也不錯",
"speaker_id": "SPEAKER_1",
"language": "zh",
"lang_prob": 0.97
}
],
"n_speakers": 2,
"speaker_stats": {
"SPEAKER_0": {"count": 1, "duration": 2.0},
"SPEAKER_1": {"count": 1, "duration": 1.5}
}
}
```
### 6.2 Qdrant Point 格式 (由 Python `_store_speaker_embeddings` 寫入)
> Embedding 不經過 Rust，直接在 Python 端完成 Qdrant HTTP PUT。
| Qdrant 欄位 | 值 | 說明 |
|-------------|-----|------|
| `id` | `hash(file_uuid + "_" + segment_index)` | 可重複查詢的 point ID |
| `vector` | `[f32; 192]` | ECAPA-TDNN 聲紋向量 |
| `payload.file_uuid` | `str` | 影片識別碼 |
| `payload.speaker_id` | `str` | 聚類後的 speaker 標籤 |
| `payload.text` | `str` | 該段的轉錄文字 |
| `payload.language` | `str` | 語種 (`zh`/`en`) |
| `payload.start_time` | `f64` | 開始時間(秒) |
| `payload.end_time` | `f64` | 結束時間(秒) |
| `payload.type` | `"speaker_embedding"` | 便於與 face_embedding 區分 |
### 6.3 Rust `AsrxResult` 對應
```rust
pub struct AsrxSegment {
pub start_time: f64, // serde(alias = "start")
pub end_time: f64, // serde(alias = "end")
pub start_frame: u64, // default 0
pub end_frame: u64, // default 0
pub text: String,
pub speaker_id: Option<String>,
pub language: Option<String>, // 🆕 新增
pub lang_prob: Option<f64>, // 🆕 新增
}
```
---
## 7. Rust 端變動
| 檔案 | 變動 |
|------|------|
| `src/core/processor/asrx.rs` | `asrx_processor_v2.py` → `asrx_processor.py` |
| `src/core/processor/asrx.rs` | `AsrxSegment` 新增 `language`, `lang_prob` 欄位 |
| `src/core/processor/asrx.rs` | 傳遞 `--file-uuid` 給 Python 腳本，讓 Python 端可直接寫入 Qdrant |
| `src/core/chunk/rule1_ingest.rs` | 若 `pre_chunks` data 含 `language` 則帶入 chunk metadata |
| `src/core/db/qdrant_db.rs` | 🆕 新增 `upsert_speaker_embedding()` 方法 (可選，若 Python 端直接寫 Qdrant 則不需) |
---
## 8. 遷移計畫
### 實作順序 (依賴關係排序)
| 步驟 | 內容 | 檔案 | 風險 |
|------|------|------|------|
| **S1** | `vad.py`: 新增 `scan_within_segment()` | `asrx_self/vad.py` | 低 |
| **S2** | 🆕 `whisper_local.py`: 封裝 whisper 載入 + 轉錄 | `asrx_self/whisper_local.py` | 低 |
| **S3** | 🔧 `main_fixed.py`: 重寫為 7 步 pipeline | `asrx_self/main_fixed.py` | 中 |
| **S4** | 🆕 `speaker_classifier.py`: 性別分類器 | `asrx_self/speaker_classifier.py` | 低 |
| **S5** | 🔧 `custom.py` cleanup + rename → `asrx_processor.py` | `asrx_processor_custom.py` | 低 |
| **S6** | 🔧 Rust `asrx.rs`: 改指向 + 傳 `--file-uuid` | `src/core/processor/asrx.rs` | 低 |
| **S7** | ✅ 驗證：build + playground 測試 | — | 中 |
| **S8** | 🧹 刪除變體 + 搬離工具 | — | 低 |
### 驗證標準
1. `cargo build` 通過
2. Playground 3003: 註冊影片 → ASRX processor 完成
3. 輸出 JSON 中 `speaker_id` 非 `null`
4. Qdrant collection 有 `speaker_embedding` 點
5. 性別正確標記 (male/female)
---
## 9. 版本歷史
| 版本 | 日期 | 修改者 | 說明 |
|------|------|--------|------|
| V1.0 | 2026-06-01 | OpenCode | 初始版本：7 步 hybrid pipeline + Qdrant 聲紋儲存 + 高品質分類 |
docs_v1.0/DESIGN/AlwaysProduce_Processing_Contract.md
+148
View File
@@ -0,0 +1,148 @@
---
title: Always-Produce Processing Contract
version: 1.0
date: 2026-07-24
author: OpenCode
status: approved
---
# Always-Produce Processing Contract
## Scope
| Field | Value |
|-------|-------|
| Scope | All frame-based processors (face, pose, appearance, face_cluster, face_trace, etc.) |
| Status | Approved |
| Applies to | Python processors + Rust Worker |
| Related docs | `DESIGN/Processor_Module_V1.0.md`, `DESIGN/Redis_Progress_Reporting_V1.0.md`, `DESIGN/Worker_Health_Check_Mechanism.md` |
## 1. Frame-Scan Model
Video processing is fundamentally frame-based: a processor scans from frame 0 to the last frame.
```
Scan start → frame 0 → frame 1 → ... → frame N → scan complete
↓ ↓ ↓ ↓
Redis Redis Redis {uuid}.{p}.json
progress progress progress (final record)
```
### Key Rules
1. **Progress** = which frame has been scanned so far (`current_frame / total_frames`)
2. **Complete** = scanned to the last frame (proved by `.json` existing)
3. **Result** = always written, even if 0 detections found
## 2. Always-Produce Rule
### Principle
> Every processor MUST write its `{uuid}.{processor}.json` output file after completing its scan, **regardless of whether any results were found**.
### Rationale
The `.json` file serves dual purpose:
- **Proof of completion**: Worker uses `output_path.exists()` (line 580 of `job_worker.rs`) to skip already-finished processors
- **Downstream dependency**: Subsequent processors check this file for input
Without the Always-Produce rule:
- Zero-result processors leave no `.json` → Worker retries infinitely → deadlock
- Stuck jobs block downstream stages (Rule 1/2/3 ingestion, TKG build)
### Format
All processor JSON outputs MUST include:
```json
{
"status": "has_faces" | "no_faces" | "no_face_json" | "no_embeddings" | "success" | "error_*",
"file_uuid": "<uuid>",
...processor-specific fields (empty arrays when zero results)
}
```
Example — face cluster with no faces:
```json
{
"status": "no_faces",
"file_uuid": "9781de6d...",
"clusters": [],
"frames": []
}
```
### Processor Checklist
| Processor | Always-Produce? | Status field on 0 result |
|-----------|----------------|--------------------------|
| `face.py` | ✅ Yes | `"no_faces"` |
| `store_traced_faces.py` | ✅ Yes (already writes) | `"no_faces"` |
| `fast_face_clustering_processor.py` | ❌ **FIX NEEDED** | Early returns, no file written |
| `pose_processor*.py` | ✅ Yes | `"no_faces"` |
| `appearance_processor*.py` | ✅ Yes | `"no_faces"` |
## 3. Redis Progress During Scan
### Purpose
Live frame progress is published to Redis so the QC modal can display real-time status ("scanning frame 1234/5678").
### Mechanism
Use `redis_publisher.py` (`RedisPublisher` class) which publishes to Redis channel `{prefix}progress:{uuid}`:
```python
from redis_publisher import RedisPublisher
pub = RedisPublisher(file_uuid)
# During scan, per batch:
pub.progress("face_cluster", current_frame, total_frames, f"Scanning frame {current_frame}")
# On completion:
pub.complete("face_cluster", f"Done: {cluster_count} clusters")
```
### Frequency
- **Frame-based processors**: publish every N frames (batch/buffer flush)
- **Non-frame processors** (e.g., clustering): publish at meaningful milestones
## 4. Worker Heartbeat
### Problem
`health.rs` currently uses `check_process_running("worker")` which relies on `ps aux | grep momentry.*worker`. This is unreliable:
- Zombie processes show as "running"
- Stale matches from unrelated processes
### Fix
Worker writes a Redis HMSET `{prefix}health` with EXPIRE = `3 × poll_interval_secs` (default: 15s) in every `poll_and_process()` cycle.
Health endpoint checks:
1. Redis key `{prefix}health` exists
2. Key has remaining TTL > 0
3. Key's `status` field is `"healthy"` or `"throttled"`
If Redis key missing or expired → `worker_alive: false`.
## 5. Implementation Plan
| Step | File | Change |
|------|------|--------|
| 1 | `fast_face_clustering_processor.py` | Always-Produce for 3 early returns + Redis progress |
| 2 | `store_traced_faces.py` | Add Redis progress (optional) |
| 3 | `job_worker.rs` | Add EXPIRE after health HMSET |
| 4 | `health.rs` | Replace `check_process_running("worker")` with Redis TTL check |
| 5 | `processing.rs` | (Optional) Reject trigger if Worker not alive |
---
## Version History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0 | 2026-07-24 | OpenCode | Initial specification |
docs_v1.0/DESIGN/Appearance_Feature_System_V1.0.md
+766
View File
@@ -0,0 +1,766 @@
---
title: Appearance Feature System V1.0
version: 1.0.0
date: 2025-06-22
author: OpenCode
status: Draft
---
# Appearance Feature System V1.0
## Overview
### Purpose
Lock onto a target and continuously track across frames using appearance features.
### Architecture
```
Face (identification) → Pose (tracking) → Appearance (tracking)
↓ ↓ ↓
identity_uuid bbox features + proportions
```
### Data Sources
| Source | Provides | Output |
|--------|----------|--------|
| Face | identity, landmarks | face.json |
| Pose | bbox, keypoints | pose.json |
| MediaPipe | detailed landmarks, hands | mediapipe.json |
---
## Keypoint Systems
### Swift Pose (Apple Vision) - 19 Keypoints
| Index | Keypoint | Vision Framework Joint |
|-------|----------|------------------------|
| 0 | nose | .nose (head_joint) |
| 1 | left_eye | .leftEye (left_eye_joint) |
| 2 | right_eye | .rightEye (right_eye_joint) |
| 3 | left_ear | .leftEar (left_ear_joint) |
| 4 | right_ear | .rightEar (right_ear_joint) |
| 5 | neck | .neck (neck_1_joint) |
| 6 | root | .root (center_hip_joint) |
| 7 | left_shoulder | .leftShoulder |
| 8 | right_shoulder | .rightShoulder |
| 9 | left_elbow | .leftElbow |
| 10 | right_elbow | .rightElbow |
| 11 | left_wrist | .leftWrist (left_hand_joint) |
| 12 | right_wrist | .rightWrist (right_hand_joint) |
| 13 | left_hip | .leftHip |
| 14 | right_hip | .rightHip |
| 15 | left_knee | .leftKnee |
| 16 | right_knee | .rightKnee |
| 17 | left_ankle | .leftAnkle |
| 18 | right_ankle | .rightAnkle |
### MediaPipe Pose - 33 Landmarks
| Index | Name | Index | Name |
|-------|------|-------|------|
| 0 | nose | 17 | left_pinky |
| 1 | left_eye_inner | 18 | right_pinky |
| 2 | left_eye | 19 | left_index |
| 3 | left_eye_outer | 20 | right_index |
| 4 | right_eye_inner | 21 | left_thumb |
| 5 | right_eye | 22 | right_thumb |
| 6 | right_eye_outer | 23 | left_hip |
| 7 | left_ear | 24 | right_hip |
| 8 | right_ear | 25 | left_knee |
| 9 | mouth_left | 26 | right_knee |
| 10 | mouth_right | 27 | left_ankle |
| 11 | left_shoulder | 28 | right_ankle |
| 12 | right_shoulder | 29 | left_heel |
| 13 | left_elbow | 30 | right_heel |
| 14 | right_elbow | 31 | left_foot_index |
| 15 | left_wrist | 32 | right_foot_index |
| 16 | right_wrist | | |
### MediaPipe Hand - 21 Landmarks
| Index | Name | Finger |
|-------|------|--------|
| 0 | wrist | - |
| 1-4 | thumb_cmc/mcp/ip/tip | thumb |
| 5-8 | index_mcp/pip/dip/tip | index |
| 9-12 | middle_mcp/pip/dip/tip | middle |
| 13-16 | ring_mcp/pip/dip/tip | ring |
| 17-20 | pinky_mcp/pip/dip/tip | pinky |
### YOLOv8 Pose (Fallback) - 17 Keypoints
| Index | Name |
|-------|------|
| 0 | nose |
| 1 | left_eye |
| 2 | right_eye |
| 3 | left_ear |
| 4 | right_ear |
| 5 | left_shoulder |
| 6 | right_shoulder |
| 7 | left_elbow |
| 8 | right_elbow |
| 9 | left_wrist |
| 10 | right_wrist |
| 11 | left_hip |
| 12 | right_hip |
| 13 | left_knee |
| 14 | right_knee |
| 15 | left_ankle |
| 16 | right_ankle |
---
## Body Proportions Calculation
### Reference Units
Multiple reference units for different shot types:
| Unit | Real Size | Available In | Notes |
|------|-----------|--------------|-------|
| eye_width | ~6cm | Close-up | Most accurate in close-up |
| head_width | ~16cm | Close-up to Medium | Ear-to-ear distance |
| shoulder_width | ~45cm | Medium to Wide | Most stable reference |
```python
# Priority: shoulder_width > head_width > eye_width
# Larger units more stable and available in wider shots
```
### Body Proportions Constants
Standard adult body proportion ratios (used for validation and estimation):
| Ratio | Value | Description |
|-------|-------|-------------|
| head_to_eye | 2.67 | head_width ≈ 2.67 × eye_width |
| eye_to_shoulder | 7.5 | shoulder_width ≈ 7.5 × eye_width |
| head_to_shoulder | 2.8 | shoulder_width ≈ 2.8 × head_width |
| head_to_height | 7.5 | body_height ≈ 7.5 × head_width |
| shoulder_to_height | 3.8 | body_height ≈ 3.8 × shoulder_width |
### Shot Type Detection
Detect shot type based on head position relative to bbox:
| Shot Type | Head Position | Aspect Ratio | Description |
|-----------|---------------|--------------|-------------|
| full_body | < 15% from top | > 2.0 | Full person visible |
| medium_shot | < 30% from top | > 1.5 | Upper body visible |
| close_up | > 30% or middle | < 1.5 | Head/face dominant |
```python
# head_position_ratio = (head_y - bbox_top) / bbox_height
# aspect_ratio = bbox_height / bbox_width
if head_position_ratio < 0.15 and aspect_ratio > 2.0:
shot_type = "full_body"
elif head_position_ratio < 0.30 and aspect_ratio > 1.5:
shot_type = "medium_shot"
else:
shot_type = "close_up"
```
**Usage**: Filter frames by shot type (e.g., find all full-body shots in video).
### Height Estimation
Height estimation strategy based on shot type:
| Shot Type | Method | Formula | Result |
|-----------|--------|---------|--------|
| full_body | Direct measurement | body_height / ref_unit × ref_cm | Accurate |
| medium_shot | Torso extrapolate | torso × (1/0.45) | ~170cm |
| close_up | Proportion estimate | shoulder × 3.8 | ~171cm |
```python
# Close-up: use shoulder_width × 3.8
estimated_height_cm = 45.0 * 3.8 # ≈ 171cm
# Or use head_width × 7.5
estimated_height_cm = 16.0 * 7.5 # ≈ 120cm (lower confidence)
```
### Body Measurements
```python
# Full body height (nose to ankle)
nose_y = keypoints['nose']['y']
ankle_y = max(keypoints['left_ankle']['y'], keypoints['right_ankle']['y'])
body_height = ankle_y - nose_y
# Upper body (neck to hip)
neck_y = keypoints['neck']['y']
hip_y = (keypoints['left_hip']['y'] + keypoints['right_hip']['y']) / 2
torso_height = hip_y - neck_y
# Lower body (hip to ankle)
leg_height = ankle_y - hip_y
# Shoulder width
shoulder_width = distance(left_shoulder, right_shoulder)
# Head width (ear to ear)
head_width = distance(left_ear, right_ear)
```
### Proportion Ratios
```python
proportions = {
'shot_type': detect_shot_type(keypoints, bbox),
'eye_width': eye_width,
'head_width': head_width,
'body_height': body_height,
'torso_height': torso_height,
'leg_height': leg_height,
'shoulder_width': shoulder_width,
'head_ratio': eye_width / body_height if body_height > 0 else 0,
'torso_ratio': torso_height / body_height if body_height > 0 else 0,
'leg_ratio': leg_height / body_height if body_height > 0 else 0,
}
# Validation ratios (should match BODY_PROPORTIONS constants)
proportion_ratios = {
'head_to_eye': head_width / eye_width if eye_width > 0 else 0, # ~2.67
'shoulder_to_head': shoulder_width / head_width if head_width > 0 else 0, # ~2.8
'shoulder_to_eye': shoulder_width / eye_width if eye_width > 0 else 0, # ~7.5
}
```
### Body Shape Classification
Classification based on chest/waist/hip ratios:
| Shape | Criteria | Description |
|-------|----------|-------------|
| hourglass | chest_waist < 1.0, waist_hip < 0.9 | Balanced proportions |
| triangle | chest_waist > 1.2 | Upper body dominant |
| inverted_triangle | waist_hip > 1.1 | Lower body dominant |
| rectangle | chest ≈ hip | Uniform width |
| oval | Other | General classification |
```python
# Measurements
chest_width = distance(left_shoulder, right_shoulder)
waist_width = distance(left_hip, right_hip)
hip_width = distance(left_hip, right_hip)
# Ratios
chest_waist_ratio = chest_width / waist_width
waist_hip_ratio = waist_width / hip_width
```
else:
height_category = "very_tall"
```
---
## Usage
### CLI Commands
#### TKG Level 1 Builder
Build person_trace nodes with Level 1 features:
```bash
# Basic usage (auto-detect video and pose.json paths)
python scripts/tkg_level1_builder.py --file-uuid <uuid> --schema dev
# With explicit paths
python scripts/tkg_level1_builder.py \
--file-uuid <uuid> \
--schema dev \
--video /path/to/video.mp4 \
--pose-json /path/to/pose.json
```
Output: Creates `person_trace` nodes in `tkg_nodes` table with:
- frame_count
- height_estimate (from shoulder_width or head_width)
- level1_features (body, head_top, upper_body, lower_body colors)
#### Query TKG Nodes
```python
import psycopg2
conn = psycopg2.connect('postgresql://accusys@localhost:5432/momentry')
cur = conn.cursor()
cur.execute("SELECT external_id, properties FROM dev.tkg_nodes WHERE node_type='person_trace'")
for row in cur.fetchall():
external_id, props = row
print(f'{external_id}: height={props["height_estimate"]["estimated_height_cm"]}cm')
```
---
## Appearance Feature Location Mapping
### Environment Factors
| Feature | Location | Detection Method |
|---------|----------|------------------|
| Light type | Frame background | HSV H distribution |
| Light direction | Shadow analysis | Shadow orientation |
| Light intensity | Overall brightness | HSV V mean |
### Head Features
#### Hair Style
| Feature | Keypoints Range |
|---------|-----------------|
| Short hair | head_top → ear/neck |
| Long hair | head_top → shoulder/back |
| Ponytail | head_top → neck (tied) |
| Braids | head_top → shoulder (braided) |
| Curly hair | hair region texture |
| Straight hair | hair region texture |
#### Hair Accessories
| Feature | Keypoints |
|---------|-----------|
| Hair band | eye_distance (head top) |
| Hair clip | ear/head |
| Hair wrap | ear_distance |
| Hair tie | neck (ponytail position) |
| Hair pin | head |
#### Head Accessories
| Feature | Keypoints |
|---------|-----------|
| Hat | head_top → eye |
| Headscarf | ear_distance (wrapped) |
| Hood | head_top → neck (full head) |
#### Hair Color
| Feature | Detection |
|---------|-----------|
| Hair color HSV | hair region HSV histogram |
### Face Features
#### Eye Accessories
| Feature | Keypoints |
|---------|-----------|
| Glasses | eye_distance |
| Sunglasses | eye_distance (larger) |
#### Ear Accessories
| Feature | Keypoints |
|---------|-----------|
| Earrings | ear_position |
| Headphones (over-ear) | ear_distance (wrapped) |
| Earphones (in-ear) | ear_position |
| Earphones (ear-hook) | ear_position |
#### Face Accessories
| Feature | Keypoints |
|---------|-----------|
| Blush | cheeks (below eye) |
| Lipstick | lips (nose + eye_width * 0.5) |
| Mask | ear_distance, eye → neck |
#### Skin Tone
| Feature | Detection |
|---------|-----------|
| Skin color HSV | face region HSV histogram |
### Neck Features
#### Neck Accessories
| Feature | Keypoints |
|---------|-----------|
| Collar | neck |
| Bow tie | neck → chest |
| Tie | neck → hip |
| Scarf | neck → shoulder |
| Necklace | neck |
#### Hanging Accessories
| Feature | Keypoints |
|---------|-----------|
| Pendant (necklace) | neck → chest |
| Charm (bag) | bag_position |
| Charm (phone) | phone_position |
### Upper Body Features
#### Clothing
| Feature | Keypoints |
|---------|-----------|
| Shirt color | neck → hip |
| Shirt material | clothing texture (LBP) |
| Clothing pattern | pattern detection |
#### Sleeves
| Feature | Keypoints |
|---------|-----------|
| Long sleeve | shoulder → wrist |
| Short sleeve | shoulder → elbow |
| Arm sleeve | elbow → wrist |
#### Back Features
| Feature | Keypoints |
|---------|-----------|
| Back exposed | shoulder → hip (view angle) |
| Back tattoo | back exposed skin |
### Bags
| Feature | Keypoints |
|---------|-----------|
| Handbag | hand_position |
| Shoulder bag | shoulder_position |
| Backpack | shoulder → hip (back) |
| Waist bag | hip_position |
### Hand Features
#### Hand Accessories
| Feature | Keypoints |
|---------|-----------|
| Watch | wrist |
| Bracelet | wrist → hand |
| Ring | finger (MediaPipe hand landmarks 13-16) |
| Gloves | wrist → hand |
| Nail polish | finger tips |
#### Handheld Objects
| Feature | Keypoints |
|---------|-----------|
| Phone | hand + object detection |
| Handbag | hand + object detection |
### Lower Body Features
#### Pants
| Feature | Keypoints |
|---------|-----------|
| Long pants | hip → ankle |
| Shorts | hip → knee |
#### Waist Accessories
| Feature | Keypoints |
|---------|-----------|
| Belt | hip |
### Foot Features
#### Foot Accessories
| Feature | Keypoints |
|---------|-----------|
| Anklet | ankle |
| Socks | ankle → foot |
| Shoes | ankle |
### Skin Features
| Feature | Detection |
|---------|-----------|
| Tattoo | exposed skin anomaly color block |
### Exposed Skin Detection
| Location | Coverage Detection |
|----------|-------------------|
| Face | always exposed |
| Arms | exposed if short sleeve |
| Legs | exposed if shorts |
| Hands | exposed if no gloves |
| Feet | exposed if no socks |
---
## Mobility Aids / Vehicles
### Walking Aids (Object Detection)
| Feature | Keypoints |
|---------|-----------|
| Cane | hand + object |
| Wheelchair | hip + object |
| Walker | both hands + object |
### Mobility Tools (Object Detection)
| Feature | Keypoints |
|---------|-----------|
| Roller skates | ankle + object |
| Skateboard | ankle + object |
| Scooter | hand + ankle + object |
### Vehicles (Object Detection)
| Feature | Keypoints |
|---------|-----------|
| Motorcycle | hip + ankle + object |
| Bicycle | hip + ankle + object |
| Tricycle | hip + ankle + object |
| Car | hip + object |
---
## Feature Extraction Techniques
### Color Extraction (HSV Histogram)
```python
def extract_color(roi):
hsv = cv2.cvtColor(roi, cv2.COLOR_BGR2HSV)
h_hist = cv2.calcHist([hsv], [0], None, [30], [0, 180])
s_hist = cv2.calcHist([hsv], [1], None, [32], [0, 256])
v_hist = cv2.calcHist([hsv], [2], None, [32], [0, 256])
return {
'h_histogram': normalize(h_hist),
's_histogram': normalize(s_hist),
'v_histogram': normalize(v_hist),
}
```
### Dominant Color (K-means)
```python
def extract_dominant_colors(roi, k=5):
hsv = cv2.cvtColor(roi, cv2.COLOR_BGR2HSV)
pixels = hsv.reshape(-1, 3).astype(np.float32)
_, labels, centers = cv2.kmeans(pixels, k, None, criteria, 10, cv2.KMEANS_RANDOM_CENTERS)
counts = np.bincount(labels.flatten())
return centers[np.argsort(-counts)[:k]]
```
### Texture Extraction (LBP)
```python
def extract_texture(roi):
gray = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
lbp = local_binary_pattern(gray, P=8, R=1)
return {
'lbp_variance': np.var(lbp),
'lbp_histogram': np.histogram(lbp, bins=256)[0],
}
```
### Shininess Detection
```python
def detect_shininess(roi):
hsv = cv2.cvtColor(roi, cv2.COLOR_BGR2HSV)
v_mean = np.mean(hsv[:,:,2])
v_std = np.std(hsv[:,:,2])
return {
'brightness': v_mean,
'brightness_variance': v_std,
}
```
---
## Tracking Flow
### Feature Storage Strategy
| Level | Storage | Reason |
|-------|---------|--------|
| **Level 1** | TKG nodes | Stable features for tracking |
| **Level 2** | Dynamic | On-demand calculation |
| **Level 3** | Dynamic | On-demand calculation |
### Level 1 in TKG
```sql
-- New node_type: person_trace
INSERT INTO tkg_nodes (
node_type = 'person_trace',
external_id = 'person_{frame}_{index}',
file_uuid = 'xxx',
properties = {
'frame_count': 100,
'frames': [1, 30, 60, ...],
'avg_bbox': {...},
'height_estimate': {
'estimated_height_cm': 170.5,
'height_ratio': 28.4,
'height_category': 'tall'
},
'body_shape': {
'chest_width': 150.2,
'waist_width': 100.5,
'hip_width': 120.3,
'chest_waist_ratio': 1.49,
'waist_hip_ratio': 0.84,
'body_shape': 'hourglass'
},
'level1_features': {
'body': {...},
'head_top': {...},
'upper_body': {...},
'lower_body': {...}
}
}
)
```
### Level 2/3 Dynamic Calculation
```python
# Level 2: computed on query
face_features = extractor.extract_level2(frame, regions)
# Level 3: computed on query
accessory_features = extractor.extract_level3(frame, keypoints, eye_width)
```
### Matching Strategy
```
Frame N → Frame N+1:
1. Pose bbox IoU → same person position
2. Level 1 similarity (TKG) → same feature combination
3. Level 2/3 dynamic → detailed verification
4. Face identity → final confirmation (if face detected)
Result: Continuous tracking of same identity
```
### IoU Calculation
```python
def calculate_iou(bbox1, bbox2):
x1, y1, w1, h1 = bbox1
x2, y2, w2, h2 = bbox2
xi1 = max(x1, x2)
yi1 = max(y1, y2)
xi2 = min(x1 + w1, x2 + w2)
yi2 = min(y1 + h1, y2 + h2)
inter_area = max(0, xi2 - xi1) * max(0, yi2 - yi1)
union_area = w1 * h1 + w2 * h2 - inter_area
return inter_area / union_area if union_area > 0 else 0
```
### Feature Similarity
```python
def calculate_similarity(features1, features2):
# HSV histogram similarity
h_sim = cv2.compareHist(features1['h_histogram'], features2['h_histogram'], cv2.HISTCMP_CORREL)
# Dominant color similarity
color_dist = np.linalg.norm(features1['dominant_colors'] - features2['dominant_colors'])
# Combined score
return {
'color_similarity': h_sim,
'color_distance': color_dist,
'overall_score': h_sim * 0.7 + (1 - color_dist/255) * 0.3,
}
```
---
## Output Format
### appearance.json Structure
```json
{
"frame_count": 100,
"fps": 30.0,
"frames": [
{
"frame": 1,
"timestamp": 0.033,
"persons": [
{
"person_index": 0,
"bbox": {"x": 100, "y": 200, "width": 400, "height": 600},
"identity_uuid": "xxx-xxx-xxx",
"proportions": {
"eye_width": 50.0,
"body_height": 600.0,
"torso_height": 200.0,
"leg_height": 300.0,
"shoulder_width": 150.0,
"head_ratio": 0.08,
"torso_ratio": 0.33,
"leg_ratio": 0.50
},
"features": {
"hair": {
"color": {"h_histogram": [...], "dominant_colors": [...]},
"length": "long",
"style": "straight"
},
"skin": {
"color": {"h_histogram": [...], "dominant_colors": [...]}
},
"clothing": {
"upper": {
"color": {...},
"material": "cotton",
"pattern": "solid",
"sleeve": "short"
},
"lower": {
"color": {...},
"length": "long"
}
},
"accessories": {
"earring": true,
"watch": true,
"shoes_color": {...}
}
}
}
]
}
]
}
```
---
## Dependencies
### Processor Dependencies
| Processor | Depends On | Reason |
|-----------|------------|--------|
| Appearance | Pose | bbox for region extraction |
| Appearance | Face | identity matching + face landmarks |
| Appearance | MediaPipe | hand landmarks + detailed pose |
### Data Flow
```
pose.json → bbox + keypoints
face.json → identity + face landmarks
mediapipe.json → hand landmarks + pose landmarks
↓
appearance.json → features + proportions + tracking
```
---
## Implementation Phases
### Phase 1: Design Document
- Create this design document
- Define all feature mappings
- Define output format
### Phase 2: Appearance Processor Refactor
- Add proportion calculation module
- Add feature extraction module
- Integrate Pose + MediaPipe + Face data
- Add IoU matching for pose-face
### Phase 3: Output Format Update
- Update appearance.json structure
- Update Rust structs
- Update DB schema
### Phase 4: Testing
- Unit tests for proportion calculation
- Integration tests for full pipeline
- Real video tracking validation
---
## Version History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0.0 | 2025-06-22 | OpenCode | Initial design document |
docs_v1.0/DESIGN/Audio_Scene_Detection_POC.md
+162
View File
@@ -0,0 +1,162 @@
---
title: Audio Scene & Instrument Detection POC Plan
version: 0.1
date: 2026-07-02
author: OpenCode
status: planned
---
| scope | status | applicable to |
|-------|--------|---------------|
| Audio processing pipeline | planned | Video files with non-speech audio |
## Goal
Detect non-speech audio events (instruments, music, environmental sounds) in video files alongside existing ASRX speech recognition.
## Why
Current pipeline only detects speech (ASRX → 64 segments + 1554 speaker embeddings). Instrument sounds, background music, and environmental audio are completely ignored.
## Technical Options
### Option A: PANNs (Pre-trained Audio Neural Networks)
- **Model**: Cnn14 (313M params, 700MB weights)
- **Classes**: 527 AudioSet classes (piano, guitar, drums, speech, etc.)
- **Pros**: Production-ready, accurate, PyTorch-based
- **Cons**: Large download, ~200MB RAM per inference
- **Install**: `pip install panns-inference`
### Option B: YAMNet (Google)
- **Model**: MobileNet-based, 4MB weights
- **Classes**: 521 AudioSet classes
- **Pros**: Lightweight, fast
- **Cons**: Requires TensorFlow (not currently installed)
- **Install**: `pip install yamnet` + TensorFlow
### Option C: torchaudio + heuristics (lightweight fallback)
- Use existing PyTorch + torchaudio
- Extract spectral features (MFCC, centroid, energy)
- Simple classification: speech vs music vs silence
- **Pros**: No extra dependencies
- **Cons**: Less accurate, limited classes
## Recommended: Option A (PANNs)
## Pipeline Integration
```
Video → Audio Extract → ASRX (speech) → Speaker Embeddings (3.4/s)
→ Audio Scene (new) → Scene Labels (1/s)
```
### New Processor: `audio_scene`
| Field | Value |
|-------|-------|
| Processor type | `audio_scene` |
| Input | Video file (audio track) |
| Output | `file_uuid.audio_scene.json` |
| Sampling | 1-second segments |
| Qdrant collection | `momentry_{schema}_audio_scene` |
### Output Format
```json
{
"file_uuid": "...",
"segments": [
{
"start_time": 0.0,
"end_time": 1.0,
"primary_class": "speech",
"confidence": 0.95,
"top_classes": [
{"class": "speech", "score": 0.95},
{"class": "music", "score": 0.03},
{"class": "piano", "score": 0.01}
]
}
],
"summary": {
"speech_ratio": 0.72,
"music_ratio": 0.15,
"silence_ratio": 0.08,
"instrument_ratio": 0.05,
"instruments_detected": ["piano", "guitar"]
}
}
```
### Qdrant Storage
| Field | Type | Purpose |
|-------|------|---------|
| `file_uuid` | string | Filter by file |
| `start_time` | float | Segment start |
| `end_time` | float | Segment end |
| `primary_class` | keyword | Filter by class |
| `confidence` | float | Filter by confidence |
| `instrument_name` | keyword | Search by instrument |
| `vector` | f32[2048] | Audio embedding for similarity search |
### Processor Dependencies
```
audio_scene → (no dependencies, runs parallel with ASRX)
```
## Key AudioSet Instrument Classes
| Category | Classes |
|----------|---------|
| Piano | Piano, Electric piano, Keyboard |
| Guitar | Guitar, Electric guitar, Acoustic guitar |
| Drums | Drum kit, Snare drum, Cymbal, Hi-hat |
| Strings | Violin, Cello, Harp, Double bass |
| Wind | Flute, Saxophone, Trumpet, Clarinet |
| Voice | Speech, Singing, Chant, Choir |
| Other | Music, Percussion, Organ, Synthesizer |
## POC Steps
1. **Install panns-inference**
```bash
pip install panns-inference
```
2. **Create `scripts/audio_scene_processor.py`**
- Load audio via ffmpeg → numpy array
- Process 1-second segments through Cnn14
- Save results to JSON + Qdrant
3. **Add processor type to pipeline**
- Add `AudioScene` to `ProcessorType` enum
- Add to worker's processor dispatch
- Add `AUDIO_SCENE_TIMEOUT` config
4. **Test with existing video**
- Run on KOBA interview video
- Verify instrument detection accuracy
- Check performance (time, memory)
5. **Integrate with search**
- Add audio_scene to universal_search
- Add filter by audio class (speech/music/instrument)
## Estimated Effort
| Step | Time |
|------|------|
| Install + prototype script | 2-3 hours |
| Pipeline integration | 1-2 hours |
| Qdrant + search integration | 1 hour |
| Testing + tuning | 1-2 hours |
| **Total** | **5-8 hours** |
## Future Enhancements
- Real-time audio classification during processing
- Audio event timeline visualization
- Combine with TKG for audio-visual relationships
- Background music detection for copyright checks
docs_v1.0/DESIGN/FACE_DETECTIONS_DEPRECATION_PLAN.md
+189
View File
@@ -0,0 +1,189 @@
---
title: face_detections Table Deprecation Plan
version: 1.0
date: 2026-06-21
author: OpenCode
status: Draft
---
## Overview
`face_detections` 表在 TKG Phase 0-2.7 迁移后，大部分功能已迁移到 Qdrant。本文档规划后续 deprecation 策略。
## Current Usage Analysis
### TKG Builders (PostgreSQL Fallback)
**状态**: 可保留作为 fallback
| Function | 用途 | 状态 |
|----------|------|------|
| `build_face_trace_nodes_from_pg()` | Fallback | ⚠️ 保留 |
| `build_gaze_trace_nodes_from_pg()` | Fallback | ⚠️ 保留 |
| `build_lip_trace_nodes_from_pg()` | Fallback | ⚠️ 保留 |
| `build_co_occurrence_edges_from_pg()` | Fallback | ⚠️ 保留 |
| `build_face_face_edges_from_pg()` | Fallback | ⚠️ 保留 |
| `build_speaker_face_edges_from_pg()` | Fallback | ⚠️ 保留 |
**总计**: 12 fallback functions
**建议**: 保留 PostgreSQL fallback，作为 Qdrant 失败时的备用方案。
### API Endpoints (Direct Queries)
**状态**: 需要迁移或保留
| Module | 功能 | 依赖程度 | 迁移难度 |
|--------|------|---------|----------|
| `files.rs` | 文件处理 | 高 | 中等 |
| `five_w1h_agent_api.rs` | Five W1H agent | 中 | 低 |
| `identities.rs` | Identity 管理 | 高 | 高 |
| `identity_agent_api.rs` | Identity Agent | 高 | 高 |
| `identity_api.rs` | Identity API | 高 | 高 |
| `identity_binding.rs` | Face binding | **非常高** | **非常高** |
| `media_api.rs` | Media API | 中 | 中 |
| `scan.rs` | Scan 功能 | 低 | 低 |
| `tmdb_api.rs` | TMDb API | 中 | 中 |
| `trace_agent_api.rs` | Trace Agent | 高 | 中 |
**总计**: 11 modules with direct queries
**关键依赖**:
- **Identity binding**: 使用 `face_detections.trace_id` 进行 face binding
- **Identity Agent**: 使用 `face_detections.trace_id` 进行 identity matching
### Identity Binding Dependencies
**最关键依赖**: `src/api/identity_binding.rs`
**用途**:
- `bind_identity_trace()`: 绑定 identity 到 trace_id
- `unbind_identity()`: 解绑 identity
- Face ↔ Identity mapping
**现状**:
- Phase 2.3 已迁移到 TKG nodes properties
- 但 identity binding API 仍使用 face_detections 查询
**迁移方案**:
1. 查询 TKG nodes by identity_id
2. 更新 TKG nodes properties
3. 移除 face_detections 查询
## Deprecation Strategy
### Phase A: Documentation (Immediate)
- [x] 标记 `face_detections` 为 deprecated (in docs)
- [x] 文档说明迁移路径
- [x] 保留 PostgreSQL fallback
### Phase B: Gradual Migration (Future)
**优先级**:
| Priority | Module | Migration | Timeline |
|----------|--------|-----------|----------|
| P1 | identity_binding.rs | TKG-based binding | TBD |
| P2 | identity_agent_api.rs | TKG-based matching | TBD |
| P3 | identity_api.rs | TKG queries | TBD |
| P4 | Other APIs | Case-by-case | TBD |
### Phase C: Removal (Long-term)
**条件**:
- 所有 API endpoints 迁移完成
- TKG-only architecture 完全稳定
- 经过充分测试验证
**时间**: TBD (至少 6 个月后)
## Current Status
### What We Can Deprecate Now
**Nothing**: 所有功能仍有 PostgreSQL fallback 或 API dependencies
**原因**:
1. Production Qdrant collection 为空 (0 points)
2. PostgreSQL fallback 是必要的安全机制
3. Identity binding APIs 依赖 face_detections
### What We Keep
- ✅ PostgreSQL fallback functions
- ✅ face_detections table
- ✅ populate_face_detections_from_face_json (Phase 0)
### What We Document
- ⚠️ face_detections deprecated (but still used)
- ⚠️ New features should use Qdrant/TKG
- ⚠️ Migration path documented
## Recommendations
### Immediate Actions
1. **标记为 deprecated**: 在 AGENTS.md 中说明
2. **文档迁移路径**: 记录 TKG-based alternatives
3. **保留 fallback**: 确保 Production 稳定性
### Short-term Actions
1. **测试新视频**: 注册新视频验证 Qdrant-based
2. **监控 Production**: 观察 PostgreSQL fallback 使用率
3. **性能对比**: Qdrant vs PostgreSQL
### Long-term Actions
1. **API migration**: 逐步迁移 identity binding APIs
2. **数据迁移**: 批量迁移现有数据到 Qdrant
3. **最终移除**: 在验证完成后移除 face_detections
## Migration Path for Identity Binding
### Current Implementation
```rust
// identity_binding.rs
let trace_id = sqlx::query_scalar(
"SELECT trace_id FROM face_detections WHERE ..."
)
```
### Future Implementation (TKG-based)
```rust
// Query TKG nodes with identity_id
let nodes = sqlx::query_as(
"SELECT id, external_id FROM tkg_nodes
WHERE file_uuid=$1 AND node_type='face_trace'
AND properties->>'identity_id' IS NOT NULL"
)
```
**优势**:
- 无需 face_detections
- TKG-only architecture
- 性能更好 (TKG nodes 缓存)
## Conclusion
**当前**: face_detections **不能** deprecated
- PostgreSQL fallback 必要
- API endpoints 仍有依赖
- Production 稳定性优先
**未来**: 逐步迁移到 TKG-only
- 按优先级迁移 API endpoints
- 验证后考虑移除 face_detections
- 至少 6 个月后评估
**建议**: 保持现状，文档化迁移路径，新功能使用 Qdrant/TKG。
---
**状态**: Draft (不执行 deprecation)
**原因**: Production 稳定性 + API dependencies
**下一步**: 文档化 + 测试新视频
docs_v1.0/DESIGN/FACE_TRACKING_PIPELINE_V1.0.md
+341
View File
@@ -0,0 +1,341 @@
---
title: Face Tracking Pipeline Structure
version: 1.0
date: 2026-07-22
author: OpenCode
status: Active
---
# Face Tracking Pipeline — Structure Design
## Overview
```
Video
│
▼
┌──────────────────────────────┐
│ Stage 1: Face Detection │ face_processor.py
│ swift_face (Apple Vision) │ → {uuid}.face.json
│ CoreML FaceNet embedding │ → Qdrant _faces (initial)
└──────────────────────────────┘
│
▼
┌──────────────────────────────┐
│ Stage 2: Face Tracking │ store_traced_faces.py
│ face_tracker.py (IoU) │ → {uuid}.face_traced.json
│ trace_id assignment │ → Qdrant _faces (trace_id update)
└──────────────────────────────┘
│
▼
┌──────────────────────────────┐
│ Stage 3: Trace Profile │ backfill_trace_profiles.py
│ Qdrant _faces 分組 │ → output/{uuid}/trace_{N}/
│ key_frame + key_face │ trace_profile.json
└──────────────────────────────┘
│
▼
┌──────────────────────────────┐
│ Stage 4: TKG Nodes │ tkg.rs
│ Qdrant _faces → trace_id │ → tkg_nodes (face_track, etc.)
└──────────────────────────────┘
```
---
## Stage 1: Face Detection
**Script**: `scripts/face_processor.py`
### Flow
1. `swift_face` (Swift/Apple Vision/ANE) → bbox detection per sampled frame
2. `cv2` opens video, crops face from bbox
3. CoreML FaceNet → 512D embedding per face
4. Output: `{uuid}.face.json`
5. Push embeddings to Qdrant `_faces` collection
### Output Format: `{uuid}.face.json`
```json
{
"status": "has_faces",
"frame_count": 563,
"fps": 29.97,
"total_faces": 1200,
"frames": [
{
"frame": 743,
"timestamp": 24.78,
"faces": [
{
"x": 892, // int, pixel
"y": 313, // int, pixel
"width": 78, // int, pixel
"height": 78, // int, pixel
"confidence": 0.733,
"pose_angle": { "angle": "frontal", "roll": 0.77, "yaw": -1.24, "pitch": 0.23 },
"landmarks": { "right_eye": [...], "nose": [...], "left_eye": [...] },
"lips": { "inner_lips": [...], "outer_lips": [...] }
}
]
}
]
}
```
**Key points**:
- bbox is **pixel integer** from Apple Vision, never modified
- face.json uses **list format** (not dict)
- Sampling at ~8Hz (`sample_interval = round(fps / 8)`)
### Qdrant Initial Push
`push_face_embeddings_batch()` in `qdrant_faces.py`:
```python
payload = {
"file_uuid": file_uuid,
"frame": frame_num,
"trace_id": face_idx, # ⚠️ frame-internal index (0, 1, 2...), NOT tracking trace_id
"bbox": {"x": x, "y": y, "width": w, "height": h}, # int pixel
"confidence": 0.5,
"identity_id": None,
"identity_uuid": None,
"stranger_id": None,
}
```
**Important**: `trace_id` at this stage is `face_idx` (index within the frame), used only as a temporary placeholder. It gets overwritten in Stage 2.
---
## Stage 2: Face Tracking
**Scripts**: `scripts/store_traced_faces.py` → `scripts/utils/face_tracker.py`
### Trigger
`job_worker.rs` P2 trigger (line ~1877): after face + asrx processors complete.
```rust
tokio::spawn(async move {
executor.run("store_traced_faces.py", &["--file-uuid", &uuid], ...)
});
```
Skip if `{uuid}.face_traced.json` already exists.
### Flow
1. `store_traced_faces.py` reads `{uuid}.face.json`
2. Converts face.json from list to dict format (frame_num_str → {frame_number, time_seconds, faces})
3. Loads cut boundaries from `{uuid}.cut.json` (if exists)
4. Calls `face_tracker.track_faces(face_data, use_embedding=False, cut_boundaries=...)`
5. Writes `{uuid}.face_traced.json`
6. Calls `update_trace_ids(file_uuid, trace_mapping)` to update Qdrant
### `face_tracker.py:track_faces()`
**Algorithm** (IoU-only, no embedding):
```
For each frame (sorted):
For each face in current frame:
Match against previous frame faces:
- Calculate IoU
- Calculate bbox center distance
- Reject if area ratio > 5x (different zoom level)
- Reject if at-edge → not-at-edge transition (person exited)
If match found → same trace_id as matched face
If no match → new trace_id (next_trace_id++)
Scene cut boundary between frames → force all new traces
```
**Matching conditions** (IoU-only mode):
- IoU > 0.5 AND IoU > 0.35 + distance < 100px → match
- IoU > 0.5 + similarity > 0.65 → match (similarity not used but condition exists)
- similarity > 0.85 → match (not used in IoU-only mode)
- Scene cut boundary → all new traces
### Output Format: `{uuid}.face_traced.json`
Same structure as face.json, but:
- Format converted to **dict** (`frames[str(frame_num)]` → face data)
- Each face gains `trace_id` field (integer)
- Top-level `traces` dict with per-trace statistics
- `metadata.tracking_method = "iou_only"`
- `metadata.traced_at = ISO timestamp`
```json
{
"metadata": {
"fps": 29.97,
"total_frames": 43977,
"tracking_method": "iou_only",
"trace_stats": {
"total_traces": 107,
"active_traces": 107,
"long_traces": 95
}
},
"frames": {
"743": {
"frame_number": 743,
"faces": [
{ "x": 892, "y": 313, "width": 78, "height": 78, "trace_id": 0, ... }
]
}
},
"traces": {
"0": {
"trace_id": 0,
"start_frame": 743,
"end_frame": 783,
"duration_frames": 41,
"total_appearances": 11,
"avg_confidence": 0.72
}
}
}
```
### Qdrant Trace Update
`update_trace_ids()` in `qdrant_faces.py`:
1. Scroll all Qdrant `_faces` points for `file_uuid` (with vector + payload)
2. For each point, build `bbox_key = f"{bbox.x}_{bbox.y}_{bbox.width}_{bbox.height}"`
3. Look up `trace_mapping[frame][bbox_key]` from face_traced.json
4. If match found → set `payload["trace_id"] = real_trace_id`
5. PUT updated points back to Qdrant
**Matching key**: `frame` + `bbox_key` (pixel integer string)
---
## Stage 3: Trace Profile
**Script**: `scripts/backfill_trace_profiles.py`
### Data Source
Qdrant `_faces` collection (source of truth for trace_id assignments).
### Flow
1. Scroll all `_faces` points for each `file_uuid` with `trace_id >= 0`
2. Group by `(file_uuid, trace_id)`
3. For each group:
- `frame_count` = count of points
- `start_frame` = min(frame)
- `end_frame` = max(frame)
- `representative_frame` = frame with max(confidence)
- `representative_bbox` = bbox at representative frame
4. Extract `key_frame.jpg` via ffmpeg at representative frame
5. Crop `key_face.jpg` from key_frame using representative bbox
6. Write `output/{uuid}/trace_{N}/trace_profile.json`
### Output: `output/{uuid}/trace_{N}/trace_profile.json`
```json
{
"version": "1.0",
"file_uuid": "d8acb03870f0cc9b14e01f14a7bf24d6",
"trace_id": 37,
"label": "",
"frame_count": 38,
"start_frame": 1859,
"end_frame": 2100,
"avg_confidence": 0.754,
"key_frame": "key_frame.jpg",
"key_face": "key_face.jpg",
"status": "pending"
}
```
### File Layout
```
output/{uuid}/
trace_0/
trace_profile.json
key_frame.jpg
key_face.jpg
trace_1/
trace_profile.json
key_frame.jpg
key_face.jpg
...
```
---
## Stage 4: TKG Node Construction
**File**: `src/core/processor/tkg.rs`
Reads trace_id from Qdrant `_faces` payload to build knowledge graph nodes:
- `face_track` nodes: one per trace
- `gaze_track`, `lip_track`: linked to face_track via frame alignment
- `co_occurrence` edges: traces that appear in same frame
---
## Qdrant `_faces` Collection Schema
| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | string | Video file identifier |
| `frame` | int | Video frame number (absolute, not sampled) |
| `trace_id` | int | Face tracking ID (set by Stage 2) |
| `bbox` | `{x, y, width, height}` | Pixel integer coordinates |
| `confidence` | float | Detection confidence |
| `identity_id` | int? | Identity binding (set by identity agent) |
| `identity_uuid` | string? | Identity UUID |
| `stranger_id` | int? | Stranger classification |
**Point ID**: `generate_point_id(file_uuid, frame, face_idx)` — deterministic hash.
---
## Known Issues
### bfba056f5021e2404b0870cc0b1fa851
- **Qdrant**: trace_id = 0,1,2 (face_idx, never updated)
- **face_traced.json**: trace_id = 0-8209 (8210 traces, iou_only)
- **Root cause**: `face_processor.py` re-ran after `store_traced_faces.py`, pushing fresh embeddings with `trace_id=face_idx`, overwriting the updated trace_ids
- **Other 12 files**: all correct
### `update_trace_ids` bbox matching
Matching is by exact `frame` + `bbox_key` string (`x_y_width_height`). Since bbox is pixel integer from the same source, values are identical across face_traced.json and Qdrant. Mismatch only occurs when face_processor.py re-runs and generates different detection results.
---
## File Inventory (2026-07-22)
| file_uuid | traces (Qdrant) | traces (face_traced) | status |
|-----------|-----------------|----------------------|--------|
| 30affad3... | 52 | 53 | ✅ |
| 31a6b821... | 31 | 36 | ⚠️ minor mismatch |
| 352cf73a... | 16 | 25 | ⚠️ minor mismatch |
| 57bd7e43... | 3 | 4 | ✅ |
| 5e5f3de8... | 21 | 22 | ✅ |
| 84d838f2... | 88 | 89 | ✅ |
| 88e72467... | 18 | 19 | ✅ |
| 9cbeb112... | 9 | 17 | ⚠️ minor mismatch |
| bfba056f... | 15 | 8210 | ❌ face_idx not updated |
| c0a9dc37... | 77 | 78 | ✅ |
| c36f3568... | 5601 | 5616 | ⚠️ minor mismatch |
| d8acb038... | 106 | 107 | ✅ |
| fbd82072... | 12 | 13 | ✅ |
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2026-07-22 | Initial document: face detection → tracking → Qdrant → TKG pipeline structure |
docs_v1.0/DESIGN/FILE_LIFECYCLE_V1.0.md
+476 -161
View File
@@ -1,198 +1,513 @@
---
document_type: "design_doc"
service: "MOMENTRY_CORE"
title: "File Lifecycle — Pre-Processing & Registration"
version: "V1.2"
date: "2026-05-15"
author: "M5"
status: "draft"
title: File Lifecycle Architecture
version: 1.0
date: 2026-07-22
author: OpenCode
status: Active
scope: File processing pipeline — stages, verification, rebuild
---
# File Lifecycle — Pre-Processing & Registration
# File Lifecycle Architecture V1.0
| Item | Value |
|------|-------|
| Scope | All managed file types (video, image, document, spreadsheet, presentation) |
| Status | Draft |
| Applies to | Pre-process API (explicit) + Register API |
| Key concept | Two-phase flow: birth certificate (`.pre.json`) → civil registration (DB INSERT) |
| Field | Value |
|-------|-------|
| Scope | Complete file processing lifecycle |
| Status | Active |
| Applies to | Pipeline stages, progress tracking, verification, rebuild |
| Related | `FILE_PROFILE_V1.0.md`, `FACE_TRACKING_PIPELINE_V1.0.md` |
> **Applicable to all managed file types**: video, image, document (pdf, docx, pages, key, numbers), spreadsheet, presentation, and any other file registered in the system. The pre-processor registers any file type found by the watcher. ffprobe is used when applicable; files that ffprobe cannot parse receive minimal filesystem metadata as a fallback.
---
## Metaphor
## 1. Overview
Every registered video file passes through a deterministic pipeline of stages.
Each stage must produce a `.json` (or `.jpg`) artifact on disk.
This enables:
- **Verification**: Check pipeline completeness by inspecting artifact existence
- **Rebuild**: Re-run any stage from its input artifacts without re-running the entire pipeline
- **Progress tracking**: Two-layer display (high-level summary + expandable sub-stages)
### Design Principles
1. **Every stage has a `.json` output** — no silent DB-only writes
2. **Any stage can be rebuilt** from its input artifacts
3. **Frontend reads stages from API** — not hardcoded
4. **Verification is disk-first** — check `.json` exists, then validate content, then check DB/Qdrant consistency
5. **Processors are not modified** — this document defines tracking/verification/rebuild only
---
## 2. Stage Architecture
### 2.1 High-Level Stages (6)
| # | Stage | Weight | Sub-Stages | Description |
|---|-------|--------|------------|-------------|
| S0 | Register | 5% | 4 | File metadata + audio track + key frame extraction |
| S1 | Processors | 40% | 8 | Individual processor execution |
| S2 | Post-Process | 20% | 4 | Face trace, Rule1, Vectorize, Identity Agent |
| S3 | TKG Build | 20% | 2 | Temporal Knowledge Graph nodes + edges |
| S4 | Rule2 | 10% | 1 | Relationship chunk ingestion |
| S5 | Complete | 5% | 1 | Final status update |
### 2.2 Sub-Stages (15)
```
SHA256 = DNA or fingerprint (immutable biometric identity)
file mtime = birth moment (preserved by rsync across systems)
birthday (file_uuid anchor) = mtime timestamp
.pre.json = birth certificate
POST /api/v1/files/register = civil registration
status = registered = citizenship completed
S0: Register (5%)
├─ 0a: probe → probe.json
├─ 0b: audio_track → DB: audio_track column (no disk artifact)
├─ 0c: profile → profile.json
└─ 0d: key_frame → key_frame.jpg
S1: Processors (40%)
├─ 1a: cut → cut.json
├─ 1b: asr → asr.json
├─ 1c: asrx → asrx.json (depends: 1a + 1b)
├─ 1d: ocr → ocr.json
├─ 1e: face → face.json (+ Qdrant _faces initial)
├─ 1f: pose → pose.json (depends: 1e)
├─ 1g: appearance → appearance.json (depends: 1f)
└─ 1h: face_dedup → face_cluster.json (depends: 1e) [OPTIONAL + MANUAL]
S2: Post-Process (20%)
├─ 2a: face_trace → face_traced.json (+ Qdrant trace_id update)
├─ 2b: rule1 → rule1.json (ASRX → sentence chunks)
├─ 2c: vectorize → vectorize.json (embeddings → PG + Qdrant)
└─ 2d: identity_agent → identity_agent.json (optional)
S3: TKG Build (20%)
├─ 3a: tkg_nodes → tkg_nodes.json
└─ 3b: tkg_edges → tkg_edges.json
S4: Rule2 (10%)
└─ 4a: rule2 → rule2.json (relationship chunks)
S5: Complete (5%)
└─ 5a: complete → status = "completed"
```
## Two-Phase Flow
A file enters the system in two distinct phases:
| Phase | Action | Analogy | Automatic? | Status |
|-------|--------|---------|:----------:|:------:|
| **Birth** | Pre-process: SHA256 + probe + file_uuid | 出生 + 醫院開出生證明 | ✅ Watcher | `unregistered` |
| **Citizenship** | Register: INSERT into DB | 戶政事務所登記 | ❌ User API | `registered` |
## Phase 1: Pre-Processing (Birth)
### Trigger
Pre-processing is triggered explicitly via the register API or a dedicated pre-process endpoint. It is NOT automatic — the watcher only detects new files without modifying them.
### Computation Steps
### 2.3 Dependency Graph
```
1. fs::metadata(path).modified()
→ birthday = file modification time (mtime, RFC 3339; preserved by rsync -a across systems)
2. SHA256(full file, streaming 64KB chunks)
→ content_hash = 512-bit hex string (file DNA / fingerprint)
3. ffprobe (or minimal fs metadata fallback for non-video)
→ probe_json
4. compute_birth_uuid(mac, birthday, canonical_path, filename)
→ file_uuid = SHA256(mac | birthday | path | filename)[0:32]
5. Write {OUTPUT_DIR}/{file_uuid}.pre.json
S0 (Register)
└─→ S1 (Processors)
├─ 1a (CUT) ─────┐
├─ 1b (ASR) ─────┤
│ └─→ 1c (ASRX) ──→ 2b (Rule1)
├─ 1d (OCR) ──────────────────────→ 3a (TKG Nodes)
├─ 1e (Face) ──┬─→ 1f (Pose) ──→ 1g (Appearance) ──→ 3a
│ ├─→ 1h (FaceDedup) [manual]
│ └─→ 2a (Face Trace) ──→ 3a
└─────────────────────────────────────→ 3a
│
S2: 2c (Vectorize) ←── DB chunks │
S2: 2d (IdentityAgent) ←── face_clusters │
↓
3b (TKG Edges)
│
↓
4a (Rule2)
│
↓
5a (Complete)
```
### Output: `.pre.json` Schema
---
Stored alongside other processor outputs:
## 3. I/O Specification
### 3.1 Register (S0)
| Sub-Stage | Input | Output Artifact | DB Tables | Qdrant |
|-----------|-------|----------------|-----------|--------|
| 0a: probe | video file on disk | `{uuid}.probe.json` | — | — |
| 0b: audio_track | probe.json, video file | DB column only | videos.audio_track | — |
| 0c: profile | probe.json | `{uuid}.profile.json` | videos (INSERT/UPDATE) | — |
| 0d: key_frame | probe.json | `{uuid}.key_frame.jpg` | — | — |
**Audio Track Classification** (S0b):
| Classification | Condition | ASR Behavior |
|----------------|-----------|--------------|
| `no_audio` | No audio track in video | Skip ASR, output `{"status": "no_audio"}` |
| `silent_audio` | Audio track exists but no speech detected | Skip ASR, output `{"status": "silent_audio"}` |
| `music_only` | Audio with no speech (music/sound effects) | Skip ASR, output `{"status": "music_only"}` |
| `speech_only` | Audio with speech only (≥30% speech ratio) | Run ASR normally |
| `speech_with_music` | Speech with background music (<30% speech ratio) | Run ASR normally |
### 3.2 Processors (S1)
| Sub-Stage | Input Artifacts | Output Artifact | DB Tables | Qdrant |
|-----------|----------------|----------------|-----------|--------|
| 1a: cut | probe.json | `{uuid}.cut.json` + `{uuid}_scene_{n}.jpg` | processor_results | — |
| 1b: asr | video file | `{uuid}.asr.json` | processor_results | — |
| 1c: asrx | cut.json, asr.json | `{uuid}.asrx.json` | speaker_detections | — |
| 1d: ocr | video file | `{uuid}.ocr.json` | processor_results | — |
| 1e: face | video file | `{uuid}.face.json` | processor_results | `_faces` (initial push) |
| 1f: pose | face.json, video file | `{uuid}.pose.json` | processor_results | — |
| 1g: appearance | pose.json, video file | `{uuid}.appearance.json` | processor_results | — |
| 1h: face_dedup | face.json | `{uuid}.face_cluster.json` | face_clusters | — |
**Note**: 1h (Face Deduplication) is currently `optional + manual`. It will be integrated into the automated pipeline after testing is complete.
**Scene Key Frames** (1a post-process):
After CUT completes, extracts the middle frame from each scene as `{uuid}_scene_{n}.jpg` for VLM analysis:
| Output | Purpose |
|---------|---------|
| `{uuid}_scene_1.jpg` | Representative frame from scene 1 |
| `{uuid}_scene_2.jpg` | Representative frame from scene 2 |
| ... | ... |
These key frames enable:
- VLM scene understanding (caption, objects, actions)
- Scene-level search and filtering
- Thumbnail generation for scene navigation
### 3.3 Post-Process (S2)
| Sub-Stage | Input Artifacts | Output Artifact | DB Tables | Qdrant |
|-----------|----------------|----------------|-----------|--------|
| 2a: face_trace | face.json | `{uuid}.face_traced.json` | — | `_faces` (trace_id update) |
| 2b: rule1 | asrx.json | `{uuid}.rule1.json` | chunk, pre_chunks | — |
| 2c: vectorize | chunk (DB) | `{uuid}.vectorize.json` | chunk_vectors | main collection |
| 2d: identity_agent | face_cluster.json | `{uuid}.identity_agent.json` | file_identities | — |
### 3.4 TKG Build (S3)
| Sub-Stage | Input Artifacts | Output Artifact | DB Tables | Qdrant |
|-----------|----------------|----------------|-----------|--------|
| 3a: tkg_nodes | All processor JSONs, trace profiles | `{uuid}.tkg_nodes.json` | tkg_nodes | — |
| 3b: tkg_edges | tkg_nodes.json, asrx.json | `{uuid}.tkg_edges.json` | tkg_edges | — |
### 3.5 Rule2 (S4)
| Sub-Stage | Input Artifacts | Output Artifact | DB Tables | Qdrant |
|-----------|----------------|----------------|-----------|--------|
| 4a: rule2 | tkg_edges.json, chunk (DB) | `{uuid}.rule2.json` | chunk (relationship type) | main collection |
### 3.6 Complete (S5)
| Sub-Stage | Input | Output | DB Tables |
|-----------|-------|--------|-----------|
| 5a: complete | All above stages verified | status = "completed" | videos.status |
---
## 4. Verification
### 4.1 Verification Levels
Each sub-stage has three verification levels:
| Level | Check | Description |
|-------|-------|-------------|
| L1: Artifact exists | `{uuid}.{stage}.json` on disk | Required for all stages |
| L2: Content valid | JSON parseable + non-empty array/object | Ensures output is usable |
| L3: DB/Qdrant consistent | Row count > 0 or point count > 0 | Ensures data was written |
### 4.2 Verification Matrix
| Sub-Stage | L1 (exists) | L2 (valid) | L3 (DB/Qdrant) |
|-----------|:-----------:|:----------:|:--------------:|
| 0a: probe | `.probe.json` | non-empty | — |
| 0b: profile | `.profile.json` | has file_uuid | videos row exists |
| 0c: key_frame | `.key_frame.jpg` | file size > 0 | — |
| 1a: cut | `.cut.json` | non-empty | processor_results > 0 |
| 1b: asr | `.asr.json` | non-empty | processor_results > 0 |
| 1c: asrx | `.asrx.json` | non-empty | speaker_detections > 0 |
| 1d: ocr | `.ocr.json` | non-empty | processor_results > 0 |
| 1e: face | `.face.json` | non-empty | Qdrant `_faces` > 0 |
| 1f: pose | `.pose.json` | non-empty | processor_results > 0 |
| 1g: appearance | `.appearance.json` | non-empty | processor_results > 0 |
| 1h: face_dedup | `.face_cluster.json` | non-empty | face_clusters > 0 |
| 2a: face_trace | `.face_traced.json` | non-empty | Qdrant `_faces` trace_id set |
| 2b: rule1 | `.rule1.json` | non-empty | chunk (sentence) > 0 |
| 2c: vectorize | `.vectorize.json` | non-empty | chunk_vectors > 0 |
| 2d: identity_agent | `.identity_agent.json` | non-empty | file_identities > 0 |
| 3a: tkg_nodes | `.tkg_nodes.json` | non-empty | tkg_nodes > 0 |
| 3b: tkg_edges | `.tkg_edges.json` | non-empty | tkg_edges > 0 |
| 4a: rule2 | `.rule2.json` | non-empty | chunk (relationship) > 0 |
### 4.3 Status Values
| Status | Meaning |
|--------|---------|
| `pending` | Not yet started |
| `running` | Currently executing |
| `completed` | L1 + L2 + L3 all pass |
| `failed` | L1 passes but L2 or L3 fails |
| `missing` | L1 fails (artifact not on disk) |
| `skipped` | Optional stage not run |
---
## 5. Rebuild
### 5.1 Rebuild Principle
Any sub-stage can be rebuilt independently:
1. Read input artifacts (from disk or DB)
2. Re-run the stage logic (processor or post-processor)
3. Write output artifact + update DB/Qdrant
### 5.2 Rebuild Dependency
To rebuild stage N, all its dependency stages must be `completed`:
| Stage | Required Dependencies |
|-------|----------------------|
| 0a-0c | video file on disk |
| 1a: cut | 0a (probe) |
| 1b: asr | video file |
| 1c: asrx | 1a (cut) + 1b (asr) |
| 1d: ocr | video file |
| 1e: face | video file |
| 1f: pose | 1e (face) |
| 1g: appearance | 1f (pose) |
| 1h: face_dedup | 1e (face) |
| 2a: face_trace | 1e (face) |
| 2b: rule1 | 1c (asrx) |
| 2c: vectorize | 2b (rule1) — chunks in DB |
| 2d: identity_agent | 1h (face_dedup) — optional |
| 3a: tkg_nodes | 1e (face), 2a (face_trace), 1c (asrx), 1d (ocr), 1g (appearance) |
| 3b: tkg_edges | 3a (tkg_nodes) + 1c (asrx) |
| 4a: rule2 | 3b (tkg_edges) + 2b (rule1) — chunks in DB |
| 5a: complete | All required stages completed |
### 5.3 Rebuild API
```
{OUTPUT_DIR}/
{file_uuid}.probe.json ← ffprobe
{file_uuid}.face.json ← face detection
{file_uuid}.pre.json ← pre-processor (NEW)
POST /api/v1/file/:file_uuid/rebuild/:stage
```
- Validates dependencies are met
- Re-runs the stage
- Returns updated verification status
### 5.4 Rebuild via CLI
```bash
# Check all stages
python3 scripts/lifecycle_check.py --file-uuid <UUID>
# Rebuild specific stage
python3 scripts/lifecycle_check.py --file-uuid <UUID> --rebuild 1c
# Rebuild from first missing stage
python3 scripts/lifecycle_check.py --file-uuid <UUID> --rebuild auto
```
---
## 6. Frontend Display
### 6.1 Two-Layer Architecture
**Layer 1: High-Level Summary** (default view)
```
┌─────────────────────────────────────────────────────────┐
│ ▶ S0: Register ████████████ 3/3 completed │
│ ▶ S1: Processors ████████░░░░ 6/8 partial │
│ ▶ S2: Post-Process ██░░░░░░░░░░ 1/4 running │
│ ▶ S3: TKG Build ░░░░░░░░░░░░ 0/2 pending │
│ ▶ S4: Rule2 ░░░░░░░░░░░░ 0/1 pending │
│ ▶ S5: Complete ░░░░░░░░░░░░ 0/1 pending │
└─────────────────────────────────────────────────────────┘
```
**Layer 2: Expandable Sub-Stages** (click to expand)
```
┌─────────────────────────────────────────────────────────┐
│ ▼ S1: Processors ████████░░░░ 6/8 partial │
│ ├─ 1a: CUT ✅ completed │
│ ├─ 1b: ASR ✅ completed │
│ ├─ 1c: ASRX ✅ completed │
│ ├─ 1d: OCR ✅ completed │
│ ├─ 1e: Face ✅ completed │
│ ├─ 1f: Pose ✅ completed │
│ ├─ 1g: Appearance ❌ missing │
│ └─ 1h: Face Dedup ⏭ skipped (manual) │
└─────────────────────────────────────────────────────────┘
```
### 6.2 Sub-Stage Display Names
| Code Name | Display Name |
|-----------|-------------|
| probe | Probe (ffprobe) |
| profile | File Profile |
| key_frame | Key Frame |
| cut | Scene Detection (CUT) |
| asr | Speech Recognition (ASR) |
| asrx | Speaker Diarization (ASRX) |
| ocr | Text Recognition (OCR) |
| face | Face Detection |
| pose | Pose Estimation |
| appearance | Appearance Features |
| face_dedup | Face Deduplication |
| face_trace | Face Tracking |
| rule1 | Rule1 Ingestion |
| vectorize | Vector Embedding |
| identity_agent | Identity Agent |
| tkg_nodes | TKG Nodes |
| tkg_edges | TKG Edges |
| rule2 | Rule2 Ingestion |
| complete | Complete |
### 6.3 API Contract
The frontend fetches stage data from:
```
GET /api/v1/stats/pipeline/:file_uuid
```
Response:
```json
{
"file_name": "charade.mp4",
"file_path": "/data/demo/charade.mp4",
"canonical_path": "/private/data/demo/charade.mp4",
"content_hash": "a1b2c3d4e5f6...",
"probe_json": {
"format": { "duration": "6879.3", "size": "2147483648" },
"streams": [...]
},
"birthday": "2026-05-15T02:15:00Z",
"file_uuid": "aeed71342a899fe4b4c57b7d41bcb692",
"file_size": 2147483648,
"file_type": "video | image | document | audio",
"pre_processed_at": "2026-05-15T02:15:05Z"
"file_uuid": "abc123",
"overall_progress": 0.45,
"stages": [
{
"name": "register",
"weight": 0.05,
"progress": 1.0,
"status": "completed",
"detail": "3/3 sub-stages",
"sub_stages": [
{"name": "probe", "status": "completed", "artifact": "probe.json"},
{"name": "profile", "status": "completed", "artifact": "profile.json"},
{"name": "key_frame", "status": "completed", "artifact": "key_frame.jpg"}
]
},
{
"name": "processors",
"weight": 0.40,
"progress": 0.75,
"status": "partial",
"detail": "6/8 sub-stages",
"sub_stages": [
{"name": "cut", "status": "completed", "artifact": "cut.json"},
{"name": "asr", "status": "completed", "artifact": "asr.json"},
{"name": "asrx", "status": "completed", "artifact": "asrx.json"},
{"name": "ocr", "status": "completed", "artifact": "ocr.json"},
{"name": "face", "status": "completed", "artifact": "face.json"},
{"name": "pose", "status": "completed", "artifact": "pose.json"},
{"name": "appearance", "status": "missing", "artifact": "appearance.json"},
{"name": "face_dedup", "status": "skipped", "artifact": "face_cluster.json"}
]
}
],
"updated_at": "2026-07-22T18:00:00Z"
}
```
### Key Design: file_uuid = f(mac, birthday, path, filename)
---
The `birthday` is `file modification time` (mtime) — obtained from `fs::metadata().modified()`. Using mtime instead of birthtime ensures file_uuid stability when files are transferred between systems via rsync (which preserves mtime but not birthtime on macOS).
## 7. Weight Distribution
### 7.1 High-Level Stage Weights
| Stage | Weight | Rationale |
|-------|--------|-----------|
| S0: Register | 5% | Fast, prerequisite for everything |
| S1: Processors | 40% | Most time-consuming, GPU-bound |
| S2: Post-Process | 20% | Face trace + Rule1 + Vectorize |
| S3: TKG Build | 20% | Node + edge construction |
| S4: Rule2 | 10% | Relationship chunk creation |
| S5: Complete | 5% | Final status update |
### 7.2 Processor Sub-Weights (within S1 = 40%)
| Processor | Sub-Weight | Rationale |
|-----------|-----------|-----------|
| CUT | 5% | Scene detection, ~10s |
| ASR | 15% | whisper-small, ~2min/10min video |
| ASRX | 20% | Speaker diarization, ~3min |
| OCR | 10% | PaddleOCR, ~1min |
| Face | 15% | CoreML FaceNet, ~1min |
| Pose | 10% | mediapipe, ~1min |
| Appearance | 5% | Feature extraction, ~30s |
| Face Dedup | 0% | Manual (not in automated pipeline) |
---
## 8. Artifact Naming Convention
All artifacts live in the output directory (`MOMENTRY_OUTPUT_DIR`):
```
birthday = 2026-05-15T02:15:00Z ← file birth time, never changes
↓
file_uuid = SHA256(mac | birthday | path | filename)
↓
Same file: same path + filename → same file_uuid, regardless of registration count
Different files: different content_hash → different file_uuid (even if same name)
{output_dir}/
├─ {uuid}.probe.json # S0: ffprobe metadata
├─ {uuid}.profile.json # S0: FileProfile
├─ {uuid}.key_frame.jpg # S0: extracted key frame
├─ {uuid}.cut.json # S1: scene boundaries
├─ {uuid}.asr.json # S1: speech transcription
├─ {uuid}.asrx.json # S1: speaker diarization
├─ {uuid}.ocr.json # S1: text detections
├─ {uuid}.face.json # S1: face detections + embeddings
├─ {uuid}.face_cluster.json # S1: face clustering (optional)
├─ {uuid}.pose.json # S1: pose estimations
├─ {uuid}.appearance.json # S1: appearance features
├─ {uuid}.face_traced.json # S2: face tracking with trace_id
├─ {uuid}.rule1.json # S2: sentence chunks
├─ {uuid}.vectorize.json # S2: embedding stats
├─ {uuid}.identity_agent.json # S2: identity matching (optional)
├─ {uuid}.tkg_nodes.json # S3: TKG node dump
├─ {uuid}.tkg_edges.json # S3: TKG edge dump
├─ {uuid}.rule2.json # S4: relationship chunks
└─ {uuid}/ # Trace profiles directory
├─ trace_0/
│ ├─ trace_profile.json
│ ├─ key_frame.jpg
│ └─ key_face.jpg
├─ trace_1/
│ └─ ...
└─ trace_N/
```
## Phase 2: Registration (Citizenship)
---
### POST /api/v1/files/register
## 9. Current State Audit (Gamma 8)
```bash
curl -X POST http://localhost:3002/api/v1/files/register \
-H "X-API-Key: ..." \
-H "Content-Type: application/json" \
-d '{"file_path":"/data/demo/charade.mp4"}'
```
File: `d3f9ae8e471a1fc4d47022c66091b920` (Gamma 8-Director Chih-Lin Yang)
### Flow
| Sub-Stage | Artifact | Status |
|-----------|----------|--------|
| 0a: probe | probe.json | ✅ exists |
| 0b: profile | profile.json | ❌ missing |
| 0c: key_frame | key_frame.jpg | ❌ missing |
| 1a: cut | cut.json | ✅ exists |
| 1b: asr | asr.json | ✅ exists |
| 1c: asrx | asrx.json | ✅ exists |
| 1d: ocr | ocr.json | ✅ exists |
| 1e: face | face.json | ✅ exists |
| 1f: pose | pose.json | ✅ exists |
| 1g: appearance | appearance.json | ❌ missing |
| 1h: face_dedup | face_cluster.json | ⏭ skipped (manual) |
| 2a: face_trace | face_traced.json | ✅ exists |
| 2b: rule1 | rule1.json | ❌ missing |
| 2c: vectorize | vectorize.json | ❌ missing |
| 2d: identity_agent | identity_agent.json | ❌ missing |
| 3a: tkg_nodes | tkg_nodes.json | ❌ missing |
| 3b: tkg_edges | tkg_edges.json | ❌ missing |
| 4a: rule2 | rule2.json | ❌ missing |
```
1. Check {OUTPUT_DIR}/{file_uuid}.pre.json
├─ Exists AND content_hash matches → use cached (skip SHA256 + probe)
└─ Not exists OR hash mismatch → compute fresh (existing logic)
2. Dedup check: SELECT file_uuid FROM videos WHERE content_hash = $1
├─ Found → already_exists: true (identical DNA = same file)
└─ Not found → continue
3. Name conflict check + auto-rename if needed
└─ charade.mp4 → charade (1).mp4 (same name, different content)
4. INSERT INTO videos (
file_uuid, file_path, file_name, file_type,
duration, width, height, fps,
probe_json, content_hash, status, registration_time
) VALUES (
$1, $2, $3, $4, $5, $6, $7, $8, $9, $10,
'registered', NOW() ← status=registered, registration_time=NOW()
)
```
## Data Separation
| Field | Source | Computed When | Mutable |
|-------|--------|---------------|:------:|
| `birthday` | `fs::metadata().modified()` (mtime) | Pre-process (once) | ❌ Never (stable across rsync) |
| `content_hash` (SHA256) | Full file | Pre-process (once) | ❌ Never (unless file modified) |
| `file_uuid` | SHA256(mac\|birthday\|path\|filename) | Pre-process (once) | ❌ Never |
| `registration_time` | `NOW()` at register | Register API | ✅ Per registration |
| `status` | — | Register API | `unregistered` → `registered` |
## File Lifecycle State Diagram
```
File detected by watcher (detection only, no modification)
│
│ Pre-processing triggered explicitly (API or register)
▼
[Pre-Processor]
├─ SHA256 (DNA / fingerprint)
├─ ffprobe (metadata extraction)
└─ file_uuid (birth certificate ID)
│
▼
{file_uuid}.pre.json
status = unregistered (no DB record)
│
│ (user calls POST /api/v1/files/register)
▼
[Register Handler]
├─ Read .pre.json → skip recomputation
├─ Dedup check (content_hash collision?)
├─ Name check + rename?
└─ INSERT INTO videos
│
▼
status = registered
registration_time = NOW()
```
## Implementation Checklist
| # | Task | File |
|---|------|------|
| 1 | Expose `pre_process_file()` as public function (SHA256 + probe + file_uuid → `.pre.json`) | `src/watcher/watcher.rs` |
| 2 | Register: read `.pre.json`, skip SHA256/probe if cached | `src/api/server.rs` → `register_single_file` |
| 3 | file_uuid: use `birthday` from `.pre.json` (or `fs::metadata().modified()` fallback) | `src/api/server.rs` |
| 4 | INSERT status: `registered`, registration_time: `NOW()` | `src/api/server.rs` |
**Observations**:
- S1 processors mostly complete, but Appearance missing (1g)
- S0 profile/key_frame missing (registration may not have created them)
- S2-S4 all have DB data but no flat `.json` dumps
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| V1.0 | 2026-05-15 | Initial design — birth certificate (pre-process) + civil registration two-phase flow |
| V1.1 | 2026-05-15 | Reclassified from DESIGN to STANDARDS as design standard |
| V1.2 | 2026-05-15 | mtime replaces birthtime for file_uuid stability across rsync; watcher is detection-only |
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.2 | 2026-07-22 | OpenCode | Added CUT scene key frames extraction for VLM analysis |
| 1.1 | 2026-07-22 | OpenCode | Added S0b: audio_track classification (VAD) — 6 stages, 15 sub-stages |
| 1.0 | 2026-07-22 | OpenCode | Initial design — 6 stages, 14 sub-stages, I/O specs, verification, rebuild |
docs_v1.0/DESIGN/FILE_PROFILE_V1.0.md
+153
View File
@@ -0,0 +1,153 @@
# File Profile V1.0
**Status:** Active
**Version:** 1.0
**Date:** 2026-07-22
**Scope:** File identity artifact — persistent on-disk profile per registered file
---
## Problem
- 5 zombie files in DB: `file_uuid` exists but `file_name` and `file_path` are empty — no way to recover
- File identity lives only in PostgreSQL; no on-disk fallback
- `birth_registration` written by `ingestion.rs` but **not** by `files.rs` API path
- No file history — if a file moves or is renamed, no record of where it was
## Design
A JSON file created **first** during registration, stored flat in `MOMENTRY_OUTPUT_DIR`:
```
{MOMENTRY_OUTPUT_DIR}/{file_uuid}.profile.json
```
### JSON Structure
```json
{
"version": "1.0",
"file_uuid": "84d838f260e1881a0daa55fabbc8e434",
"file_name": "view28.mp4",
"file_type": "video",
"birth": {
"mac_address": "a1:b2:c3:d4:e5:f6",
"birthday": "2026-04-13T23:00:49+08:00",
"original_path": "/Users/accusys/momentry/var/sftpgo/data/demo",
"original_filename": "view28.mp4",
"canonical_path": "/Users/accusys/momentry/var/sftpgo/data/demo/view28.mp4",
"content_hash": "abc123..."
},
"current": {
"path": "/Users/accusys/momentry/var/sftpgo/data/demo/view28.mp4",
"file_name": "view28.mp4",
"file_type": "video"
},
"history": [
{
"action": "registered",
"timestamp": "2026-07-22T14:30:00+08:00",
"path": "/Users/accusys/momentry/var/sftpgo/data/demo/view28.mp4",
"file_name": "view28.mp4"
}
],
"metadata": {
"duration": 243.24,
"width": 720,
"height": 890,
"fps": 60.0,
"total_frames": 7297
},
"key_frame": null
}
```
### Fields
| Field | Purpose |
|-------|---------|
| `version` | Profile schema version (for future migration) |
| `file_uuid` | The deterministic UUID |
| `file_name` | Original filename at registration |
| `file_type` | video/audio/image/document/... |
| `birth.mac_address` | MAC address used to compute UUID |
| `birth.birthday` | File mtime (RFC3339) used to compute UUID |
| `birth.original_path` | Parent directory at registration |
| `birth.original_filename` | Filename at registration |
| `birth.canonical_path` | Canonical (resolved symlinks) path at registration |
| `birth.content_hash` | SHA256 of file content |
| `current.path` | Latest known path (updated when file moves) |
| `current.file_name` | Latest known filename (updated on rename) |
| `current.file_type` | Latest file type |
| `history` | Array of all path/name changes with timestamps |
| `metadata` | Media info (duration, resolution, etc.) |
| `key_frame` | Base64-encoded JPEG of representative frame (video only), or null |
### key_frame
For video files, `key_frame` stores a **base64-encoded JPEG** of a representative frame extracted at registration time (typically at 10% of duration or the first non-black frame). For non-video files, this field is `null`.
Purpose:
- Instant visual identification without needing to open the video
- Fallback if thumbnails or `.faces/` crops are deleted
- Portable — the profile file is self-contained
Extraction:
- Uses ffmpeg to grab a frame at `duration * 0.1` (or first frame if duration unknown)
- JPEG quality 85, max width 640px
- Stored inline as base64 string in the JSON
## Implementation
### New Module
`src/core/file_profile.rs` — `FileProfile` struct with:
- `from_registration_params(...)` — build at registration time
- `load_from_disk(uuid, output_dir)` — read `{uuid}.profile.json`
- `save_to_disk(&self, output_dir)` — write `{uuid}.profile.json`
- `update_current_path(&mut self, new_path, new_name)` — append to history, update current
- `extract_key_frame(video_path, duration)` — ffmpeg frame extraction + base64
### Registration Flow
1. DB INSERT (existing)
2. **Build FileProfile** from all params (mac, birthday, path, name, content_hash, probe metadata)
3. **Extract key_frame** if video (ffmpeg)
4. **Save `{uuid}.profile.json`** — first artifact on disk
5. CUT processing (existing)
### Update Flow
When `file_path` or `file_name` changes via API:
1. Load profile from disk
2. `profile.update_current_path(new_path, new_name)`
3. Save updated profile
### Fallback Flow
If DB data is missing (zombie files):
1. Load profile from disk
2. Profile's `current.path` and `current.file_name` provide recovery data
### Files Changed
| File | Change |
|------|--------|
| `src/core/file_profile.rs` | **NEW** — FileProfile struct |
| `src/core/mod.rs` | Add `pub mod file_profile` |
| `src/api/files.rs` | Write profile after registration; fallback; enrich GET; cleanup |
| `src/core/ingestion.rs` | Write profile after registration |
| `src/api/profile.rs` | Enrich GET file-profile with profile data + history |
### Backfill
Existing 16 files get profiles generated from DB fields + probe.json data.
5 zombie files get minimal profiles (UUID + file_type + content_hash from DB).
---
## Version History
| Version | Date | Change |
|---------|------|--------|
| 1.0 | 2026-07-22 | Initial design — file profile with key_frame |
docs_v1.0/DESIGN/Face_Pose_Appearance_Design.md
+339
View File
@@ -0,0 +1,339 @@
# Face-Pose-Appearance Tracking Design
**Version**: 1.0
**Date**: 2026-07-19
**Status**: Ready for Implementation
---
## 1. Overview
本文件定義 Face、Pose、Appearance 的追蹤系統設計，包含：
- Trace ID 繼承規則
- 擴張邏輯
- Appearance 色塊提取
- Agent Search 整合
---
## 2. Core Concepts
### 2.1 Identity vs Tracking
| Processor | Purpose | Description |
|-----------|---------|-------------|
| **Face** | Identity | Who is this person? 需要高品質 embedding |
| **Pose** | Tracking | Where is this person? 當 face occluded 時維持追蹤 |
| **Appearance** | Tracking | What do they look like? 當 pose occluded 時維持追蹤 |
### 2.2 Offline Processing Advantage
Offline 處理可以先做 face detection，再從 face traces 擴張 pose/appearance：
```
Face Detection → 知道身份錨點
↓
Face Tracking → 給予 trace_id
↓
Pose Expansion → 從 face traces 向外擴張
↓
Appearance Expansion → 從 pose traces 向外擴張
```
---
## 3. Processing Pipeline
### 3.1 Pipeline Order
| Order | Processor | Dependencies | Output | Description |
|-------|-----------|-------------|--------|-------------|
| 1 | `cut` | — | cut.json | Scene detection |
| 2 | `face` | — | face.json | Face detection (8Hz) + embedding |
| 3 | `face_trace` | face | face_traced.json | Face tracking (IoU + embedding) |
| 4 | `pose` | face_trace | pose.json | Pose expansion from traces |
| 5 | `appearance` | pose | appearance.json | Appearance extraction |
| 6 | `asr` | cut | asr.json | Speech-to-text |
| 7 | `asrx` | asr | asrx.json | Speaker diarization |
### 3.2 Sampling Rate
- **公式**: `sample_interval = floor(fps / 8)`
- **確保**: ≥ 8Hz 取樣率
- **範例**:
- 24fps → interval = 3 → 8Hz
- 30fps → interval = 3 → 10Hz
- 60fps → interval = 7 → 8.6Hz
---
## 4. Trace ID Inheritance
### 4.1 Inheritance Chain
```
Face Trace (identity anchor)
│ trace_id = 1, 2, 3, ...
│
▼ inherits trace_id
Pose Expansion
│ same trace_id per person
│
▼ inherits trace_id
Appearance Expansion
│ same trace_id per person
```
### 4.2 Frame Count Relationship
```
face_frames ≤ pose_frames ≤ appearance_frames
```
**原因**:
- Face: 只有臉部可見的 frames
- Pose: Face frames + 擴張 frames（臉被遮但身體可見）
- Appearance: Pose frames + 擴張 frames
### 4.3 Trace Connection
```
Face trace A (frames 1-10) Face trace B (frames 20-30)
↘ ↙
Pose 連接 (frames 15-18)
（同一人，中間臉被遮住）
```
**意義**: Pose 可以連接斷開的 face traces，屬於同一人。
---
## 5. Expansion Rules
### 5.1 Pose Expansion
**Algorithm**:
1. 讀取 face_traced.json，取得每個 trace_id 的 frames
2. 對每個 trace 的 frames 向外擴張（逐幀檢查）
3. 連續 3 幀無 pose detection → 停止擴張
4. 繼承 trace_id
5. 輸出 8Hz 取樣
**Parameters**:
| Parameter | Value | Description |
|-----------|-------|-------------|
| `miss_threshold` | 3 | 連續無檢測幀數 |
| `output_rate` | 8Hz | 輸出取樣率 |
### 5.2 Appearance Expansion
**Algorithm**:
1. 讀取 pose.json，取得每個 trace_id 的 frames
2. 對每個 pose frame，在 keypoint 位置提取顏色
3. 記錄整體亮度
4. 輸出 8Hz 取樣
**Parameters**:
| Parameter | Value | Description |
|-----------|-------|-------------|
| `color_radius` | 15 | 顏色取樣半徑（pixels） |
| `output_rate` | 8Hz | 輸出取樣率 |
---
## 6. Pose Output
### 6.1 Data Structure
```json
{
"frame_count": 1000,
"fps": 24.0,
"frames": [
{
"frame": 100,
"timestamp": 4.16,
"trace_id": 1,
"persons": [
{
"keypoints": [
{"name": "nose", "x": 315.9, "y": 364.2, "confidence": 0.85},
{"name": "left_shoulder", "x": 290.0, "y": 400.0, "confidence": 0.92}
],
"bbox": {"x": 280, "y": 350, "width": 100, "height": 200}
}
]
}
]
}
```
### 6.2 Bbox Validation
**原則**: Face bbox 應在 Pose bbox 內，或 IoU > 0.5
```
┌─────────────────────────┐
│ Pose bbox │
│ ┌─────────┐ │
│ │ Face │ │
│ │ bbox │ │
│ └─────────┘ │
└─────────────────────────┘
```
**用途**:
- 品質驗證：確保 pose/face 屬於同一人
- 匹配追蹤：用 bbox overlap 匹配 face/pose
---
## 7. Appearance Output
### 7.1 Keypoint-based Color Extraction
**原理**: 在 pose keypoint 位置取周圍平均色
```
Pose Keypoints 座標
↓
在每個 keypoint 位置取色
↓
記錄為 appearance
```
### 7.2 Body Part Mapping
| Keypoints | Body Part | Description |
|-----------|-----------|-------------|
| nose, eyes, ears | head | 帽子、頭髮顏色 |
| shoulders | torso | 上衣顏色 |
| hips, knees | legs | 褲子顏色 |
| ankles | feet | 鞋子顏色 |
### 7.3 Data Structure
```json
{
"frame_count": 1000,
"fps": 24.0,
"frames": [
{
"frame": 100,
"timestamp": 4.16,
"trace_id": 1,
"brightness": 0.75,
"colors": {
"head": [180, 150, 120],
"torso": [255, 50, 50],
"legs": [50, 50, 200],
"feet": [50, 200, 50]
}
}
]
}
```
### 7.4 Lighting Record
```json
{
"brightness": 0.75
}
```
**用途**: 不同光源下的顏色校正
---
## 8. VLM Complementary Strategy
### 8.1 Two-Level Approach
| Level | Method | Purpose |
|-------|--------|---------|
| **L1** | Keypoint 快取色 | 快速搜尋、初步候選 |
| **L2** | VLM 驗證 | 複雜情況、細節補充（可選） |
### 8.2 Workflow
```
搜尋「穿紅上衣的人」
↓
L1: Keypoint 取色搜尋 → Top 20 候選
↓
L2: VLM 驗證（需要時）→ 確認顏色、補充細節
↓
最終結果 → Top 10 + 置信度
```
---
## 9. Agent Search Integration
### 9.1 Agent Tool Design
```python
def search_by_appearance(
color: str, # "red", "blue", "green"
body_part: str, # "torso", "legs", "feet"
top_k: int = 10
) -> List[SearchResult]:
"""
搜尋穿特定顏色衣物的人
Returns:
[
{"trace_id": 1, "identity": "John", "confidence": 0.85},
{"trace_id": 2, "identity": "Mary", "confidence": 0.72},
]
"""
```
### 9.2 Query Examples
| User Query | Agent Action |
|------------|--------------|
| 「穿紅上衣的人是誰？」 | search_by_appearance("red", "torso") → match identity |
| 「穿綠鞋子的人」 | search_by_appearance("green", "feet") |
| 「戴黑帽子的人」 | search_by_appearance("black", "head") |
### 9.3 Top-K Strategy
- **原則**: 找 top 10-20 最相似的
- **容許誤差**: 光源、角度差異可接受
- **近似即可**: 不需精確匹配
---
## 10. Implementation Files
| Component | File | Status |
|-----------|------|--------|
| Face Detection | `swift_face.swift` | ✅ Complete |
| Face Tracking | `store_traced_faces.py` | ✅ Complete |
| Pose Expansion | `swift_pose_expansion.swift` | ✅ Complete |
| Appearance Expansion | `swift_appearance_expansion.swift` | ✅ Complete |
| Pose Processor | `pose_processor_v2.py` | ✅ Complete |
| Appearance Processor | `appearance_processor_v2.py` | ✅ Complete |
---
## 11. Testing Checklist
- [ ] 清除測試檔案重新註冊
- [ ] 執行完整流程：face → trace → pose → appearance
- [ ] 驗證 trace_id 繼承正確性
- [ ] 驗證 frame count 關係 (face ≤ pose ≤ appearance)
- [ ] 驗證 bbox 包含關係 (face bbox ⊂ pose bbox)
- [ ] 測試 Agent search_by_appearance
---
## 12. Version History
| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2026-07-19 | Initial design |
docs_v1.0/DESIGN/IDENTITY_AGENT_REDESIGN_V2.0.md
+538
View File
@@ -0,0 +1,538 @@
---
title: Identity Agent V2.0 Redesign
version: 2.0
date: 2026-07-15
author: OpenCode
status: Draft
---
# Identity Agent V2.0 重新設計
## 1. 概述
### 目的
將 face trace 與 seed photos 透過 Qdrant 向量比對，建立 pending people 綁定建議，由用戶確認後升級為 known people。
### 核心原則
| 原則 | 說明 |
|------|------|
| **只綁定，不確認** | Identity Agent 只建立 suggested 綁定，不升級為 confirmed |
| **所有綁定都需確認** | 即使 matching 的是已知人物（known people），也要建立 pending 綁定 |
| **用戶決定** | 每個新綁定都需要用戶在 UI 上點擊確認 |
| **多重綁定** | 一個 trace 可同時匹配多個 seeds，都記錄為建議 |
### 流程圖
```
┌─────────────────────────────────────────────────────────────┐
│ Identity Agent V2.0 │
├─────────────────────────────────────────────────────────────┤
│ │
│ Data Stores: │
│ ├─ Qdrant _faces (512D, trace-level faces) │
│ ├─ Qdrant _seeds (512D, seed photos per file_uuid) │
│ ├─ PG identities (metadata: name, tmdb_id, status) │
│ └─ PG tkg_nodes (face_track, status + suggestions) │
│ │
│ Flow: │
│ 1. TMDb/Upload/Name → Seeds (_seeds collection) │
│ 2. Identity Agent → Match _seeds vs _faces │
│ 3. Write suggestions → TKG + Qdrant (suggested) │
│ 4. User Confirm → status = confirmed │
│ 5. User Reject → status = stranger │
│ │
│ Rules: │
│ - All bindings start as "suggested" │
│ - Even known people get new pending bindings │
│ - Multiple suggestions per trace allowed │
│ - User confirms each binding individually │
│ │
└─────────────────────────────────────────────────────────────┘
```
---
## 2. 種子管理
### 種子來源
| 來源 | 觸發方式 | file_uuid 綁定 |
|------|----------|---------------|
| TMDb probe | 檔案註冊時自動探測 | 探測檔案的 uuid |
| 用戶上傳照片 | UI 上傳 face photo | 當前查看檔案的 uuid |
| Pending Face 命名 | 選擇 trace 並命名 | 該 face 所在檔案的 uuid |
### Qdrant _seeds 結構
```json
{
"vector": [512 floats],
"payload": {
"identity_id": 3,
"identity_uuid": "69e50666-0c2d-4215-968b-d1fdd820fb7d",
"name": "Walter Matthau",
"source": "tmdb",
"file_uuid": "4bb70f2b0d6d9c1a2900298666189a73",
"tmdb_id": 3490,
"status": "confirmed",
"angles": [
{"frame": 100, "embedding": [...]},
{"frame": 150, "embedding": [...]}
]
}
}
```
### 種子產生流程
```
TMDb Probe
→ 下載演員照片
→ 提取 face embedding
→ Push to _seeds (file_uuid 綁定)
用戶上傳
→ 上傳照片
→ 檢測 face + 提取 embedding
→ Push to _seeds (file_uuid 綁定)
Pending Face 命名
→ 選擇 trace
→ 輸入名稱
→ 從 trace 提取 representative embedding
→ Push to _seeds (file_uuid 綁定)
```
---
## 3. 匹配流程
### 匹配閾值
| Round | Threshold | Seed Source |
|-------|-----------|-------------|
| Round 1 | 0.55 | TMDb seeds |
| Round 2 | 0.55 | Confirmed traces (propagation) |
| Round 3+ | 0.50 | More confirmed traces |
| Stranger | 0.40 | Unmatched traces (clustering) |
### 匹配邏輯
```python
def match_faces_round_1(file_uuid: str) -> dict:
"""
Returns: {trace_id: [suggestions]}
每個 trace 可有多個 suggestions（多重綁定）
"""
traces = get_trace_representatives(file_uuid)
seeds = get_seeds(file_uuid=file_uuid) # 只查該檔案的 seeds
suggestions = {}
for trace_id, reps in traces.items():
trace_suggestions = []
for seed in seeds:
score = multi_angle_match(seed["embedding"], reps)
if score >= TH_ROUND_1:
trace_suggestions.append({
"identity_id": seed["identity_id"],
"identity_uuid": seed["identity_uuid"],
"name": seed["name"],
"score": round(score, 4),
"source": seed["source"],
"is_known": seed.get("status") == "confirmed"
})
if trace_suggestions:
trace_suggestions.sort(key=lambda x: x["score"], reverse=True)
suggestions[trace_id] = trace_suggestions
return suggestions
```
---
## 4. 多重綁定
### 設計原則
一個 face trace 可以同時匹配多個 seeds（分數都很高），所有匹配都記錄為建議，由用戶決定選擇哪一個。
### 範例
```
Trace #432 (7 frames, confidence: 0.83)
├─ Walter Matthau (58.6%) ← is_known: true
├─ George Kennedy (57.2%) ← is_known: true
└─ [Not yet assigned]
```
---
## 5. 資料儲存
### Qdrant _faces payload（多重綁定）
```json
{
"file_uuid": "9f880a88e9f027255714906d84865fd4",
"trace_id": 432,
"frame": 13494,
"confidence": 0.829,
"bbox": {"x": 272, "y": 38, "width": 87, "height": 87},
"identity_id": null,
"identity_uuid": null,
"suggestions": [
{"identity_id": 3, "identity_uuid": "69e50666...", "name": "Walter Matthau", "score": 0.586, "source": "tmdb", "is_known": true},
{"identity_id": 5, "identity_uuid": "102e8ed0...", "name": "George Kennedy", "score": 0.572, "source": "tmdb", "is_known": true}
],
"suggested_by": "tmdb"
}
```
### TKG face_track properties（多重綁定）
```json
{
"trace_id": 432,
"frame_count": 7,
"start_frame": 100,
"end_frame": 300,
"status": "suggested",
"suggestions": [
{"identity_id": 3, "identity_uuid": "69e50666...", "name": "Walter Matthau", "score": 0.586, "source": "tmdb", "is_known": true},
{"identity_id": 5, "identity_uuid": "102e8ed0...", "name": "George Kennedy", "score": 0.572, "source": "tmdb", "is_known": true}
],
"suggested_by": "tmdb"
}
```
### 雙寫入邏輯
```python
def write_suggestions(file_uuid: str, suggestions: Dict):
"""同時寫入 TKG 和 Qdrant"""
# 1. 寫入 TKG
tkg_updated = batch_mark_suggestions(file_uuid, suggestions)
# 2. 寫入 Qdrant _faces
qdrant_updated = update_faces_suggestions(file_uuid, suggestions)
return {"tkg": tkg_updated, "qdrant": qdrant_updated}
```
---
## 6. Pending People
### 查詢邏輯
**API:** `GET /api/v1/file/:file_uuid/pending-persons`
**查詢：** PG tkg_nodes WHERE status = 'suggested'
```sql
SELECT
jsonb_array_elements(properties->'suggestions') as suggestion,
COUNT(DISTINCT (properties->>'trace_id')) as trace_count,
MIN(created_at) as created_at
FROM tkg_nodes
WHERE file_uuid = $1
AND node_type = 'face_track'
AND properties->>'status' = 'suggested'
AND jsonb_array_length(properties->'suggestions') > 0
GROUP BY suggestion
ORDER BY trace_count DESC
```
### 回傳格式
```json
{
"success": true,
"message": "Found 5 pending persons",
"data": [
{
"identity_uuid": "69e50666-0c2d-...",
"identity_id": 3,
"name": "Walter Matthau",
"trace_count": 4,
"is_known": true,
"created_at": "2026-07-15 12:00:00"
}
]
}
```
---
## 7. Known Person 複製邏輯
### 規則
當 Identity Agent 匹配到已知人物（status = 'confirmed'）時：
1. **不自動確認** — 仍建立 suggested 綁定
2. **標記 is_known** — 讓 UI 顯示為已知人物
3. **用戶仍需確認** — 每個新綁定都需要用戶操作
### 匹配結果結構
```json
{
"trace_id": 432,
"suggestions": [
{
"identity_id": 3,
"name": "Walter Matthau",
"score": 0.586,
"is_known": true // ← 已知人物標記
}
]
}
```
### UI 顯示
```
Pending People (Charade)
Trace #432
├─ Walter Matthau (58.6%) ✓ Known
│ [Confirm] [Reject]
├─ George Kennedy (57.2%) ✓ Known
│ [Confirm] [Reject]
```
---
## 8. Speaker Binding
### 改查 TKG suggested
**舊邏輯：** 查 Qdrant _faces WHERE identity_id EXISTS（永遠 0 筆）
**新邏輯：** 查 PG tkg_nodes WHERE status = 'suggested'
### 具體實現
```rust
pub async fn bind_speakers(pool: &PgPool, file_uuid: &str) -> Result<usize> {
// 1. 查 TKG suggested 節點
let rows: Vec<(i32, i32, i64, i64, f64)> = sqlx::query_as(
"SELECT (properties->>'trace_id')::int,
(properties->'suggestions'->0->>'identity_id')::int,
(properties->>'start_frame')::bigint,
(properties->>'end_frame')::bigint,
COALESCE((properties->>'confidence')::float, 0.0)
FROM tkg_nodes
WHERE file_uuid = $1 AND node_type = 'face_track'
AND properties->>'status' = 'suggested'
AND jsonb_array_length(properties->'suggestions') > 0"
).bind(file_uuid).fetch_all(pool).await?;
// 2. 讀取 ASRX segments
let asrx_data = load_asrx(file_uuid).await?;
let speakers = extract_speakers(&asrx_data);
// 3. 計算重疊 + 寫入 identity_bindings
let mut bindings = 0;
for (trace_id, identity_id, start_frame, end_frame, confidence) in rows {
let best_speaker = compute_overlap(start_frame, end_frame, &speakers, fps);
if best_speaker.overlap_ratio > 0.3 {
insert_identity_binding(identity_id, &best_speaker.speaker_id, ...);
bindings += 1;
}
}
Ok(bindings)
}
```
---
## 9. API 設計
### 完整 API 清單
| Method | Path | 說明 |
|--------|------|------|
| POST | `/api/v1/agents/identity/run` | 執行 Identity Agent（全部 seeds） |
| POST | `/api/v1/agents/identity/run-for-seed` | 執行單一 seed 匹配 |
| POST | `/api/v1/agents/identity/cluster` | 執行 trace clustering（無 seed） |
| POST | `/api/v1/file/:file_uuid/cluster-agent` | 執行 trace clustering（檔案專用） |
| POST | `/api/v1/agents/identity/generate-seeds` | 產生種子 embeddings |
| GET | `/api/v1/file/:file_uuid/pending-persons` | 列出 pending people |
| POST | `/api/v1/file/:file_uuid/pending-person` | 建立 pending person |
| POST | `/api/v1/identity/:uuid/bind` | 綁定 identity |
| POST | `/api/v1/identity/:uuid/unbind` | 解綁 identity |
| POST | `/api/v1/identity/:uuid/confirm` | 確認 pending → confirmed |
| POST | `/api/v1/identity/:uuid/reject` | 拒絕 pending → stranger |
### run-for-seed API
**Request:**
```json
{
"file_uuid": "9f880a88e9f027255714906d84865fd4",
"identity_uuid": "c3545906-c82d-4b66-aa1d-150bc02decce"
}
```
**Response:**
```json
{
"success": true,
"message": "Found 19 new matches for seed",
"matches": 19
}
```
---
## 10. 前端 UI
### Pending People 顯示
```
┌─────────────────────────────────────────────────────────┐
│ Pending People (Charade) │
├─────────────────────────────────────────────────────────┤
│ │
│ Audrey Hepburn (6 traces) ✓ Known Person │
│ [Confirm All] [Review] [Skip] │
│ │
│ George Kennedy (4 traces) ✓ Known Person │
│ [Confirm All] [Review] [Skip] │
│ │
│ Walter Matthau (4 traces) ✓ Known Person │
│ [Confirm All] [Review] [Skip] │
│ │
│ Bernard Musson (1 trace) │
│ [Confirm] [Skip] │
│ │
│ James Coburn (1 trace) │
│ [Confirm] [Skip] │
│ │
└─────────────────────────────────────────────────────────┘
```
### Trace 層級多重綁定
```
┌─────────────────────────────────────────────────────────┐
│ Trace #432 (7 frames) │
├─────────────────────────────────────────────────────────┤
│ │
│ Suggestions: │
│ ○ Walter Matthau (58.6%) ✓ Known │
│ ○ George Kennedy (57.2%) ✓ Known │
│ │
│ [Select Walter Matthau] [Select George Kennedy] │
│ [Skip - Not Recognized] │
│ │
└─────────────────────────────────────────────────────────┘
```
---
## 11. Cluster Agent（無 Seed 分群）
### 目的
不需要 seed photos，直接對現有 face traces 進行相似度分群，找出相似臉並分組。
### 與 Identity Agent 的區別
| | Identity Agent | Cluster Agent |
|---|---|---|
| Seed | 需要 seed person | 不需要 seed |
| 目標 | TMDB 匹配已知演員 | 用現有 face traces 找出相似臉分群 |
| 輸出 | identity_match_round1.json | cluster_result.json |
| 算法 | Cosine similarity vs seeds | Greedy clustering with centroids |
### API
**Request:**
```json
POST /api/v1/file/:file_uuid/cluster-agent
{}
```
**Response:**
```json
{
"success": true,
"message": "Found 30 clusters from 412 traces",
"clusters": 30,
"total_traces": 412,
"output_path": "/Users/accusys/momentry/output/{uuid}/{uuid}.cluster_result.json",
"cluster_details": [
{"cluster_id": 1, "trace_count": 49, "trace_ids": [...], "representative_trace": 2136},
{"cluster_id": 2, "trace_count": 36, "trace_ids": [...], "representative_trace": 2}
]
}
```
### 輸出檔案
`{output}/{uuid}/{uuid}.cluster_result.json`
```json
{
"file_uuid": "9f880a88e9f027255714906d84865fd4",
"threshold": 0.40,
"total_traces": 412,
"clusters": [
{"cluster_id": 1, "trace_count": 49, "trace_ids": [2136, ...], "representative_trace": 2136}
]
}
```
---
## 12. 修改計畫
### Python Scripts
| 檔案 | 改動 |
|------|------|
| `identity_matcher.py` | 輸出 suggestions 陣列，支援多重綁定 |
| `qdrant_faces.py` | 新增 `update_faces_suggestions()` |
| `tkg_helper.py` | 修改 `batch_mark_suggestions()` 寫入 suggestions 陣列 |
### Rust 後端
| 檔案 | 改動 |
|------|------|
| `identity_agent_api.rs` | `bind_speakers()` 改查 TKG suggested |
| `identity_binding.rs` | `list_pending_persons()` 已改為查 TKG ✓ |
### 前端
| 檔案 | 改動 |
|------|------|
| `PeopleView.vue` | 顯示多重綁定、is_known 標記、觸發按鈕 |
| `store.ts` | `ensureFaceCandidates` 傳 fileUuid ✓ |
| `api/index.ts` | 新增 `run_identity_for_seed` API 映射 |
### Pending People Card UI
每個 pending person 卡片新增觸發按鈕（右上角 refresh icon）：
- 點擊後觸發單一 seed 匹配
- 顯示匹配結果數量
- 自動重新載入 pending people 列表
---
## 12. 版本歷史
| 版本 | 日期 | 說明 |
|------|------|------|
| V1.0 | 2026-05-07 | 初始設計 |
| V2.0 | 2026-07-15 | 重新設計：多重綁定、只綁定不確認、TKG 雙寫入 |
| V2.1 | 2026-07-16 | 新增 run-for-seed API、前端觸發按鈕 |
| V2.2 | 2026-07-16 | 新增 Cluster Agent（無 seed 分群）、cluster-agent API |
docs_v1.0/DESIGN/Modular_Doc_System_V1.0.md
+385
View File
@@ -0,0 +1,385 @@
---
document_type: "design"
service: "MOMENTRY_CORE"
title: "模組生成式文件產出系統"
date: "2026-05-17"
version: "V1.0"
status: "active"
owner: "M5"
created_by: "OpenCode"
tags:
- "documentation"
- "modular"
- "generated-docs"
- "workspace"
ai_query_hints:
- "查詢模組生成式文件產出系統的設計理念"
- "如何使用 API_WORKSPACE"
- "如何新增 API endpoint 文檔"
- "make deploy 流程"
- "自定義交付文件"
related_documents:
- "STANDARDS/USER_DOCS_STANDARD.md"
- "STANDARDS/DOCS_STANDARD.md"
- "API_WORKSPACE/README.md"
- "API_WORKSPACE/modules/_template.md"
---
# 模組生成式文件產出系統
| 項目 | 內容 |
|------|------|
| 建立者 | OpenCode |
| 建立時間 | 2026-05-17 |
| 文件版本 | V1.0 |
| 目標讀者 | developer, documentation maintainer |
---
## 版本歷史
| 版本 | 日期 | 目的 | 操作人 |
|------|------|------|--------|
| V1.0 | 2026-05-17 | 建立設計文件 | OpenCode |
---
## 1. 設計理念
### 1.1 痛點
傳統 API 文件維護有常見問題：
| 問題 | 具體表現 |
|------|----------|
| **內容重複** | 同一個 endpoint 在快速參考、完整手冊、教育訓練文件中寫三次 |
| **更新遺漏** | 修改 curl 範例後，忘記同步到另一份文件 |
| **交付僵化** | 無法按對象產出不同版本的 API 文件 |
| **版本失靈** | YAML frontmatter 版本號與實際內容脫節 |
### 1.2 核心原則
```
單一真理源（modules/）→ 組裝引擎（assemble_docs.sh）→ 多種交付產品（GUIDES/）
編輯 ──→ 生成 ──→ 部署
1 處修改模組 make all make deploy
```
| 原則 | 說明 |
|------|------|
| **單一真理源** | 每個 endpoint 只在 `modules/` 中定義一次 |
| **組裝而非撰寫** | 交付文件是 modules 的組合，不是手寫 |
| **開發與交付分離** | `API_WORKSPACE/` 開發，`GUIDES/` 交付 |
| **模組為最小可測試單位** | 每個 module 可獨立驗證正確性 |
| **配置驅動** | `.toml` 配置定義哪些 module 以何種模式組裝成何種輸出 |
### 1.3 檔案類型對照
| 類型 | 角色 | 可編輯 | 位置 |
|------|------|--------|------|
| Module (模組) | 不可再拆的內容最小單位 | ✅ 是 | `API_WORKSPACE/modules/` |
| Config (配方) | 定義組裝規則 | ✅ 是 | `API_WORKSPACE/configs/` |
| Narrative (敘事) | 非結構化的前言/背景 | ✅ 是 | `API_WORKSPACE/narratives/` |
| Assembled (產出) | 從模組組裝的交付文件 | ❌ 否（generated） | `API_WORKSPACE/_build/` → `GUIDES/` |
---
## 2. 目錄結構
```
docs_v1.0/
├── API_WORKSPACE/ ← 開發區
│ ├── modules/ ← 端點模組（單一真理源）
│ │ ├── _template.md ← 模組撰寫規範
│ │ ├── 01_auth.md ← 認證、Base URL
│ │ ├── 02_health.md ← 健康檢查
│ │ ├── 03_register.md ← 註冊、掃描
│ │ ├── 04_lookup.md ← 查詢、刪除
│ │ ├── 05_process.md ← 處理、進度、任務
│ │ ├── 06_search.md ← 搜尋（向量、n8n、視覺）
│ │ ├── 07_identity.md ← 身份 CRUD、bind/unbind
│ │ ├── 08_identity_agent.md ← Identity Agent
│ │ ├── 09_tmdb.md ← TMDb Enrichment
│ │ ├── 10_pipeline.md ← Stats、配置、未掛載端點
│ │ └── 11_error_codes.md ← 錯誤碼對照表
│ │
│ ├── configs/ ← 組裝配方（每個輸出一份）
│ │ ├── reference.toml → API_REFERENCE.md
│ │ ├── endpoints.toml → API_ENDPOINTS.md
│ │ ├── quickref.toml → API_QUICK_REFERENCE.md
│ │ ├── errors.toml → API_ERROR_CODES.md
│ │ ├── index.toml → API_INDEX.md
│ │ ├── marcom.toml → API_TRAINING_MARCOM.md
│ │ └── tmdb.toml → TMDb_User_Guide.md
│ │
│ ├── narratives/ ← 非端點敘事前言
│ │ └── marcom_intro.md
│ │
│ ├── _build/ ← 生成暫存區（gitignored）
│ ├── Makefile ← 組裝自動化入口
│ ├── assemble_docs.sh ← 組裝引擎
│ └── README.md ← 開發者速查
│
├── GUIDES/ ← 交付區
│ ├── API_REFERENCE.md (generated)
│ ├── API_ENDPOINTS.md (generated)
│ ├── API_QUICK_REFERENCE.md (generated)
│ ├── API_ERROR_CODES.md (generated)
│ ├── API_INDEX.md (generated)
│ ├── API_TRAINING_MARCOM.md (generated)
│ ├── TMDb_User_Guide.md (generated)
│ ├── Demo_EndToEnd.md (手寫保留)
│ ├── Pipeline_API_Demo.md (手寫保留)
│ └── ... (其他手寫文件)
│
├── DESIGN/
├── REFERENCE/
├── OPERATIONS/
├── INTEGRATIONS/
└── STANDARDS/
```
---
## 3. 模組規範
### 3.1 檔名規則
- 格式：`NN_<name>.md`（NN = 兩位數排序 01-99）
- 範例：`03_register.md`, `09_tmdb.md`
- 依賴序號決定組裝時的 endpoint 順序
### 3.2 Module Metadata 註解
每個 module 開頭必須有 metadata 註解：
```markdown
<!-- module: auth -->
<!-- description: Authentication, API Key, Base URL configuration -->
<!-- depends: -->
```
| 欄位 | 必填 | 說明 |
|------|------|------|
| `module` | Yes | 唯一名稱，無空格無數字開頭 |
| `description` | Yes | 一句話說明 |
| `depends` | No | 依賴的其他 module 名稱（逗號分隔） |
### 3.3 Endpoint 結構
每個 endpoint 必須使用一致結構：
```markdown
### `METHOD /path/to/endpoint`
**Auth**: Required / Optional / Public
**Scope**: file-level / identity-level / system-level
#### Request Parameters
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
#### Example
```bash
curl -s -X METHOD "$API/path" \
-H "X-API-Key: $KEY" \
-d '{"field": "value"}'
```
#### Response (200)
```json
{ ... }
```
#### Error Codes
| Code | HTTP | When |
|------|------|------|
```
```
### 3.4 變數規則
| 變數 | 用途 | 範例值 |
|------|------|--------|
| `$API` | Base URL | `http://localhost:3003` |
| `$KEY` | API Key | `your-api-key-here` |
| `$FILE_UUID` | File UUID | `3a6c1865...` |
| `$IDENTITY_UUID` | Identity UUID | `a9a90105...` |
---
## 4. 組裝引擎
### 4.1 `assemble_docs.sh`
Shell 腳本，接收三個參數：
| 參數 | 說明 | 範例 |
|------|------|------|
| `--config` | TOML 配方路徑 | `configs/reference.toml` |
| `--modules` | Module 目錄 | `modules/` |
| `--build` | 輸出目錄 | `_build/` |
### 4.2 三種組裝模式
| mode | 行為 | 適用 |
|------|------|------|
| `full` | 完整包含 module 全部內容（除 metadata） | API_REFERENCE, API_ENDPOINTS |
| `summary` | 僅擷取 endpoint 表格 + curl 範例 | API_QUICK_REFERENCE |
| `index` | 生成文件總覽（掃描 modules 目錄自動產生索引） | API_INDEX |
### 4.3 組裝流程
```
1. 讀取 config.toml → 解析 title, modules, mode, narrative
2. 生成 YAML frontmatter（含 document_type, date, version）
3. 生成 title heading + info block
4. （可選）摘自 TOC：從 modules ## headings 生成目錄
5. （可選）插入 narrative intro
6. 遍歷 modules：
- full mode: 複製整份內容（跳過 <!-- --> 註解）
- summary mode: 只提取 | table | + ```bash code block
- index mode: 自動掃描 modules 目錄生成清單
7. 寫入 _build/ 輸出檔案
```
---
## 5. 配方格式（config.toml）
```toml
title = "輸出文件標題"
output = "_build/FILENAME.md" # 輸出路徑（相對於 API_WORKSPACE）
mode = "full" # full | summary | index
modules = ["01_auth", "03_register"] # 要包含的 module 名稱
narrative = "narratives/xxx.md" # （可選）包含的敘事前言
toc = true # （可選）是否生成目錄
[frontmatter]
document_type = "api_reference" # 用於 YAML frontmatter
service = "MOMENTRY_CORE"
version = "V1.0"
owner = "M5"
created_by = "OpenCode"
```
### 內建配方一覽
| 檔案 | 輸出 | Modules | Mode |
|------|------|---------|------|
| `reference.toml` | API_REFERENCE.md | 01-11 | full |
| `endpoints.toml` | API_ENDPOINTS.md | 01-10 | full |
| `quickref.toml` | API_QUICK_REFERENCE.md | 01-06,09 | summary |
| `errors.toml` | API_ERROR_CODES.md | 11 | full |
| `index.toml` | API_INDEX.md | (auto) | index |
| `marcom.toml` | API_TRAINING_MARCOM.md | 01,03,06 + narrative | full |
| `tmdb.toml` | TMDb_User_Guide.md | 01,03,09 | full |
---
## 6. 工作流程
### 6.1 日常修改
```bash
# 1. 編輯模組
cd API_WORKSPACE
vim modules/09_tmdb.md
# 2. 重新生成單一文件
make tmdb
# 3. 預覽結果
less _build/TMDb_User_Guide.md
# 4. 部署
make deploy
```
### 6.2 新增端點
```bash
# 1. 找到所屬模組
ls modules/
# 決定該 endpoint 屬於哪個模組（如 tmdb, identity, search）
# 2. 在對應模組加入 endpoint 文檔
vim modules/09_tmdb.md
# 3. 重新生成所有文件
make all
# 4. 確認所有引用此端點的文件都有正確更新
make check
# 5. 部署
make deploy
```
### 6.3 客製化交付
```bash
# 新增一個客製化配方
cat > configs/integration_partner.toml << TOML
title = "Integration Partner API Guide"
output = "_build/PARTNER_GUIDE.md"
mode = "full"
modules = ["01_auth", "06_search", "09_tmdb", "11_error_codes"]
toc = true
[frontmatter]
document_type = "user_manual"
service = "MOMENTRY_CORE"
version = "V1.0"
owner = "M5"
created_by = "OpenCode"
TOML
# 在 Makefile 中加入對應 target
echo "partner:" >> Makefile
echo ' @$$(SCRIPT) --config configs/integration_partner.toml --modules $$(MODULES) --build $$(BUILD)' >> Makefile
# 生成
make partner
# 部署
make deploy
```
---
## 7. 交付客製化對照表
| 對象 | 需要 modules | make target | 輸出 |
|------|-------------|-------------|------|
| API Developer | 01-11 (all) | `make reference` | API_REFERENCE.md |
| Quick Start User | 01-06,09 | `make quickref` | API_QUICK_REFERENCE.md |
| Marcom Team | 01,03,06 + narrative | `make marcom` | API_TRAINING_MARCOM.md |
| TMDb User | 01,03,09 | `make tmdb` | TMDb_User_Guide.md |
| Integration Partner | 01,06,09,11 | Custom config | PARTNER_GUIDE.md |
---
## 8. GUIDES/ 文件類型說明
| 類型 | 來源 | 說明 |
|------|------|------|
| `API_*.md` (7 files) | Generated from API_WORKSPACE | API 功能文件，endpoint 列表 + curl 範例 |
| `Demo_*.md`, `M5API_*.md` | 手寫 | 敘事性指引，含完整 step-by-step 流程 |
| `PORTAL_*.md` | 手寫 | Portal 開發計畫與 Demo 指引 |
| `USER_MANUAL.md` | 手寫 | 系統操作使用手冊 |
> **提醒**：不要直接修改 GUIDES/ 中的 generated files。修改應在 API_WORKSPACE/modules/ 中進行，然後執行 `make deploy`。
---
## 相關文件
- `API_WORKSPACE/README.md` — 開發者快速上手指南
- `API_WORKSPACE/modules/_template.md` — 模組撰寫範本
- `STANDARDS/DOCS_STANDARD.md` — 文件創建規範
- `STANDARDS/USER_DOCS_STANDARD.md` — 使用者文件規範
docs_v1.0/DESIGN/PER_FILE_VOICE_COLLECTION_V1.0.md
+143
View File
@@ -0,0 +1,143 @@
---
title: Per-File Voice Collection V1.0
version: 1.0
date: 2026-06-20
author: OpenCode
status: approved
---
# Per-File Voice Collection V1.0
| Scope | Status | Applicable to | Binary |
|-------|--------|---------------|--------|
| Qdrant voice collection naming, storage, lifecycle | Approved | `momentry_playground`, `momentry` | Both |
## Problem Statement
ASRX processor stores speaker voice embeddings (192-dim ECAPA-TDNN) in Qdrant for speaker diarization and future identity matching. The current design uses a single global collection `{prefix}_voice` for all files, creating several issues:
1. **No isolation**: All files' voice embeddings share one collection, making per-file cleanup error-prone
2. **Unnecessary migration**: Workspace `_workspace_voice` → production `_voice` migration during checkin adds complexity with no benefit for per-file processing artifacts
3. **No event type distinction**: No payload field to distinguish speaker embeddings from future audio event types (gunshots, screams, music, etc.)
4. **Cross-file matching is impractical**: Current point ID includes file_uuid, but querying across files requires filtering rather than direct collection access
## Design
### Collection Naming: Per-File
```
{file_uuid}_voice
```
Examples:
- `d3f9ae8e471a1fc4d47022c66091b920_voice`
- `92ed12dbb7fbea5e6ddfe668e1f31444_voice`
### Collection Schema
| Property | Value |
|----------|-------|
| Name | `{file_uuid}_voice` |
| Vector dimension | 192 |
| Distance metric | Cosine |
| On-disk | false (default, in-memory for fast search during processing) |
### Point Schema
**Point ID**: `SHA256(speaker_id + "_" + segment_index)` → first 8 bytes as u64
- No file_uuid in hash (redundant, collection is per-file)
**Payload**:
| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `speaker_id` | String | Speaker label from ASRX | `"SPEAKER_00"` |
| `segment_index` | Integer | Segment index within ASRX result | `5` |
| `start_frame` | Integer | Start frame number | `120` |
| `end_frame` | Integer | End frame number | `240` |
| `start_time` | Float | Start time in seconds | `4.0` |
| `end_time` | Float | End time in seconds | `8.0` |
| `event_type` | String | Type of audio event | `"speaker"` |
### Event Type Extensibility
The `event_type` field reserves space for future audio recognition:
| event_type | Description | Future Model | Dim |
|------------|-------------|--------------|-----|
| `"speaker"` | Speaker voice embedding (current) | ECAPA-TDNN | 192 |
| `"gunshot"` | Gunshot detection embedding | YAMNet / custom | TBD |
| `"scream"` | Scream/shout detection | YAMNet / custom | TBD |
| `"music"` | Music segment embedding | CLMR / custom | TBD |
Each event type with a different dimension would use a separate per-file collection (`{file_uuid}_gunshot`, etc.).
### Lifecycle
```
Processing:
ASRX completes → store_voice_embeddings_to_qdrant()
→ ensure_collection("{file_uuid}_voice", 192)
→ upsert_vector per segment
Checkin:
No voice migration needed (data already in per-file collection)
Checkout / File Deletion:
Delete collection "{file_uuid}_voice" (or delete by filter)
Cross-File Matching (future):
Job scans all "*_voice" collections, or maintains {prefix}_speaker_profiles index
```
### Changes from Current Design
| Aspect | Current | New |
|--------|---------|-----|
| Collection name | `{prefix}_voice` | `{file_uuid}_voice` |
| Point ID hash input | `file_uuid + speaker_id + index` | `speaker_id + index` |
| Workspace dual-write | `_workspace_voice` → `_voice` migration | Removed (no migration needed) |
| Payload event_type | Not present | `"speaker"` |
| Checkin voice migration | Scroll + upsert | Nothing (data already isolated) |
| Checkout voice deletion | Filter by file_uuid from `{prefix}_voice` | Delete collection or filter |
| QdrantWorkspace voice methods | `voice_collection()`, `upsert_voice_embedding()` | Removed |
### Files Affected
| File | Change |
|------|--------|
| `src/worker/processor.rs:1291-1360` | `store_voice_embeddings_to_qdrant()` — per-file collection, event_type payload |
| `src/worker/processor.rs:919-942` | Remove workspace voice dual-write |
| `src/core/checkin.rs:208-242` | Remove voice migration block |
| `src/core/checkin.rs:358-379` | Update checkout voice deletion to target `{file_uuid}_voice` |
| `src/core/db/qdrant_workspace.rs` | Remove `voice_collection()`, `upsert_voice_embedding()`, voice from `ensure_all()`, `scroll_by_file_uuid()`, `WorkspaceScrollResult`, `delete_by_file_uuid()` |
### Cross-File Matching (Future Design)
For future multi-file speaker matching, a separate index collection can be maintained:
```
{prefix}_speaker_profiles (192-dim Cosine)
- payload: speaker_id (global), source_file_uuids[], reference_count, centroid_embedding
```
This index would be updated:
1. During a periodic batch job that scans all `*_voice` collections
2. Or incrementally when new voice data is added
The per-file collection design makes this cleaner because:
- Source data is cleanly partitioned
- The index is explicitly a derived/cached structure
- Index rebuild means rescraping `*_voice` collections, not untangling a global collection
## Migration
Existing voice data in `{prefix}_voice` and `{prefix}_workspace_voice` can be left as-is for backward compatibility. New processing will write to `{file_uuid}_voice`. Old data in `{prefix}_voice` will remain queryable if needed.
No data migration script is required — old data is read-only legacy.
## Version History
| Version | Date | Author | Change |
|---------|------|--------|--------|
| 1.0 | 2026-06-20 | OpenCode | Initial design |
docs_v1.0/DESIGN/Processor_Module_V1.0.md
+758
View File
@@ -0,0 +1,758 @@
# Processor Module V1.0
**Date**: 2026-06-19
**Version**: 1.0.0
**Status**: Draft
---
## 1. 架構總覽
### 1.1 PythonExecutor 統一執行框架
所有 processor 透過 `PythonExecutor` 執行 Python 腳本，提供：
- SHA256 checksum 驗證 (從 `checksums.sha256` 讀取)
- Retry 機制 (exponential backoff: 1s → 2s → 4s → ...)
- Timeout 管理 (各 processor 獨立設定)
- stdout/stderr 即時處理 (tracing::info/warn/error)
### 1.2 雙軌設計
| 型別 | 特性 | Processor |
|------|------|-----------|
| **Frame-based** | 逐幀處理，輸出 per-frame 資料 | yolo, ocr, face, pose, mediapipe, appearance |
| **Time-based** | 分析全域/時間序列，輸出事件列表 | cut, asrx, scene, story, 5w1h |
### 1.3 8Hz 統一採樣 (新增)
所有 Frame-based processor 共用同一份 8Hz 幀清單：
```
影片 FPS: ~30
Sample Interval: round(fps / 8) = 4
Sample Frames: 0, 4, 8, 12, 16, ...
```
---
## 2. Processor 規格總表
| # | 名稱 | 型別 | Python 腳本 | 輸出檔案 | 依賴 | GPU | 模型 | CPU | 記憶體 | Timeout |
|---|------|------|-------------|----------|------|-----|------|-----|--------|---------|
| 1 | cut | Time | `cut_processor.py` | `.cut.json` | — | ❌ | PySceneDetect | 0.5 | 512MB | 3600s |
| 2 | asrx | Time | `asrx_processor.py` | `.asrx.json` | cut | ❌ | speechbrain | 0.8 | 2048MB | 7200s |
| 3 | yolo | Frame | `yolo_processor.py` | `.yolo.json` | — | ✅ | yolov8n | 0.3 | 1024MB | 7200s |
| 4 | ocr | Frame | `ocr_processor.py` | `.ocr.json` | — | ❌ | paddleocr | 0.8 | 1024MB | 7200s |
| 5 | face | Frame | `face_processor.py` | `.face.json` | — | ✅ | insightface/buffalo_l | 0.6 | 1536MB | 7200s |
| 6 | pose | Frame | `pose_processor.py` | `.pose.json` | — | ✅ | mediapipe/pose | 0.4 | 1024MB | 7200s |
| 7 | mediapipe | Frame | `mediapipe_holistic_processor.py` | `.mediapipe.json` | — | ❌ | mediapipe/holistic | 0.3 | 1024MB | 7200s |
| 8 | appearance | Frame | `appearance_processor.py` | `.appearance.json` | pose | ❌ | HSV | 0.3 | 512MB | 7200s |
| 9 | scene | Time | `scene_classifier.py` | `.scene.json` | cut | ❌ | places365 | 0.3 | 512MB | 7200s |
| 10 | story | Time | `story_processor.py` | `.story.json` | asrx+cut+yolo+face | ❌ | gemma4 | 0.1 | 256MB | 7200s |
| 11 | 5w1h | Time | `parent_chunk_5w1h.py` | — | story | ❌ | gemma4 | 0.1 | 256MB | 7200s |
---
## 3. 各 Processor 詳細規格
### 3.1 Cut — 場景切換偵測
**型別**: Time-based
**腳本**: `cut_processor.py`
**模型**: PySceneDetect
```rust
pub struct CutResult {
pub frame_count: u64,
pub fps: f64,
pub scenes: Vec<CutScene>,
}
pub struct CutScene {
pub scene_number: u32,
pub start_frame: u64,
pub end_frame: u64,
pub start_time: f64,
pub end_time: f64,
}
```
**輸出 JSON**:
```json
{
"frame_count": 8951,
"fps": 29.97,
"scenes": [
{"scene_number": 1, "start_frame": 0, "end_frame": 150, "start_time": 0.0, "end_time": 5.0},
...
]
}
```
---
### 3.2 ASRX — 語音辨識 + Speaker Diarization
**型別**: Time-based
**腳本**: `asrx_processor.py`
**模型**: speechbrain/ecapa-tdnn
**依賴**: cut (需要場景邊界)
```rust
pub struct AsrxResult {
pub language: Option<String>,
pub segments: Vec<AsrxSegment>,
pub embeddings: Option<Vec<Vec<f32>>>,
}
pub struct AsrxSegment {
pub start_time: f64,
pub end_time: f64,
pub start_frame: u64,
pub end_frame: u64,
pub text: String,
pub speaker_id: Option<String>,
}
```
**輸出 JSON**:
```json
{
"language": "zh",
"segments": [
{
"start_time": 0.1,
"end_time": 2.0,
"start_frame": 3,
"end_frame": 60,
"text": "大家好",
"speaker_id": "SPEAKER_0"
},
...
]
}
```
---
### 3.3 YOLO — 物件偵測
**型別**: Frame-based
**腳本**: `yolo_processor.py`
**模型**: yolov8n
**GPU**: ✅
**採樣**: 8Hz
```rust
pub struct YoloResult {
pub frame_count: u64,
pub fps: f64,
pub frames: Vec<YoloFrame>,
}
pub struct YoloFrame {
pub frame: u64,
pub timestamp: f64,
pub objects: Vec<YoloObject>,
}
pub struct YoloObject {
pub class_name: String,
pub class_id: u32,
pub x: i32,
pub y: i32,
pub width: i32,
pub height: i32,
pub confidence: f32,
}
```
**輸出 JSON**:
```json
{
"frame_count": 2238,
"fps": 29.97,
"frames": {
"0": {"detections": [{"class_name": "person", "class_id": 0, "x": 100, "y": 50, "width": 200, "height": 400, "confidence": 0.95}]},
"4": {"detections": [...]},
...
}
}
```
**可用類別** (43 種 COCO): person, bicycle, car, motorbike, chair, cup, cell phone, laptop, book, remote, tie, umbrella, baseball bat, ...
---
### 3.4 OCR — 文字辨識
**型別**: Frame-based
**腳本**: `ocr_processor.py`
**模型**: paddleocr
**採樣**: 8Hz
```rust
pub struct OcrResult {
pub frame_count: u64,
pub fps: f64,
pub frames: Vec<OcrFrame>,
}
pub struct OcrFrame {
pub frame: u64,
pub timestamp: f64,
pub texts: Vec<OcrText>,
}
pub struct OcrText {
pub text: String,
pub x: i32,
pub y: i32,
pub width: i32,
pub height: i32,
pub confidence: f32,
}
```
---
### 3.5 Face — 人臉偵測 + Embedding
**型別**: Frame-based
**腳本**: `face_processor.py`
**模型**: insightface/buffalo_l
**GPU**: ✅
**採樣**: 8Hz
```rust
pub struct FaceResult {
pub frame_count: u64,
pub fps: f64,
pub frames: Vec<FaceFrame>,
}
pub struct FaceFrame {
pub frame: u64,
pub timestamp: f64,
pub faces: Vec<Face>,
}
pub struct Face {
pub face_id: Option<String>,
pub x: i32,
pub y: i32,
pub width: i32,
pub height: i32,
pub confidence: f32,
pub embedding: Option<Vec<f32>>,
pub landmarks: Option<serde_json::Value>,
pub attributes: Option<FaceAttributes>,
}
pub struct FaceAttributes {
pub age: Option<i32>,
pub gender: Option<String>,
}
```
**輸出 JSON**:
```json
{
"frame_count": 2238,
"fps": 29.97,
"frames": [
{
"frame": 0,
"timestamp": 0.0,
"faces": [{
"face_id": "face_0",
"x": 500, "y": 300, "width": 200, "height": 250,
"confidence": 0.98,
"embedding": [0.12, -0.34, ...],
"landmarks": {
"nose": [[x,y], ...],
"left_eye": [[x,y], ...],
"right_eye": [[x,y], ...]
},
"attributes": {"age": 35, "gender": "male"}
}]
}
]
}
```
**Landmarks**: nose (8pts) + left_eye (6pts) + right_eye (6pts) = 20 pts
---
### 3.6 Pose — 身體姿勢
**型別**: Frame-based
**腳本**: `pose_processor.py`
**模型**: mediapipe/pose
**GPU**: ✅
**採樣**: 8Hz
```rust
pub struct PoseResult {
pub frame_count: u64,
pub fps: f64,
pub frames: Vec<PoseFrame>,
}
pub struct PoseFrame {
pub frame: u64,
pub timestamp: f64,
pub persons: Vec<PersonPose>,
}
pub struct PersonPose {
pub keypoints: Vec<Keypoint>,
pub bbox: Bbox,
}
pub struct Keypoint {
pub x: f64,
pub y: f64,
pub z: f64,
pub visibility: f64,
}
pub struct Bbox {
pub x: i32,
pub y: i32,
pub width: i32,
pub height: i32,
}
```
**輸出 JSON**:
```json
{
"frame_count": 2238,
"fps": 29.97,
"frames": [
{
"frame": 0,
"timestamp": 0.0,
"persons": [{
"keypoints": [
{"x": 0.5, "y": 0.3, "z": 0.1, "visibility": 0.95},
...
],
"bbox": {"x": 400, "y": 100, "width": 300, "height": 600}
}]
}
]
}
```
**Keypoints**: 33 個身體關節 (nose, shoulders, elbows, wrists, hips, knees, ankles, ...)
**用途**: 提供 appearance_processor 的 bbox 來源，計算上下半身色彩 ROI
---
### 3.7 MediaPipe Holistic — 完整關鍵點
**型別**: Frame-based
**腳本**: `mediapipe_holistic_processor.py`
**模型**: mediapipe/holistic
**GPU**: ❌
**採樣**: 8Hz
```rust
pub struct MediaPipeResult {
pub metadata: MediaPipeMetadata,
pub frames: HashMap<String, MediaPipeDictEntry>,
}
pub struct MediaPipeMetadata {
pub fps: f64,
pub total_frames: i64,
pub processed_frames: i64,
pub sample_interval: i64,
pub width: i64,
pub height: i64,
pub processor: String,
}
pub struct MediaPipeDictEntry {
pub frame: String,
pub timestamp: f64,
pub persons: Vec<MediaPipePerson>,
}
pub struct MediaPipePerson {
pub person_id: u64,
pub bbox: Option<MediaPipeBBox>,
pub face_mesh: Option<MediaPipeFaceMesh>,
pub pose: Option<MediaPipePose>,
pub hands: MediaPipeHands,
}
pub struct MediaPipeHands {
pub left: Option<MediaPipeHand>,
pub right: Option<MediaPipeHand>,
}
```
**輸出 JSON**:
```json
{
"metadata": {
"fps": 29.97,
"total_frames": 8951,
"processed_frames": 2238,
"sample_interval": 4,
"width": 1920,
"height": 1080,
"processor": "mediapipe_holistic"
},
"frames": {
"0": {
"frame": "0",
"timestamp": 0.0,
"persons": [{
"person_id": 0,
"bbox": {"x": 400, "y": 100, "width": 300, "height": 600},
"face_mesh": {
"landmarks": [[x,y,z], ...],
"eye_features": {"left_openness": 0.85, "right_openness": 0.82},
"mouth_features": {"openness": 0.3, "width": 45}
},
"pose": {
"landmarks": [[x,y,z,visibility], ...],
"arm_features": {"left_angle": 45, "right_angle": 30},
"leg_features": {"left_angle": 180, "right_angle": 175}
},
"hands": {
"left": {"landmarks": [[x,y,z], ...], "gesture": "point"},
"right": {"landmarks": [[x,y,z], ...], "gesture": "fist"}
}
}]
}
}
}
```
**關鍵點總計**:
| 部位 | 數量 | 說明 |
|------|------|------|
| Face Mesh | 468 | 臉部完整網格 |
| Pose | 33 | 身體關節 |
| Left Hand | 21 | 左手關鍵點 |
| Right Hand | 21 | 右手關鍵點 |
| **總計** | **543** | |
### Pose vs MediaPipe 對比
| | Pose Processor | MediaPipe Holistic |
|--|----------------|--------------------|
| **Landmarks** | 33 pts (pose only) | 543 pts (face + pose + hands) |
| **速度** | 快 (GPU 加速) | 較慢 (CPU) |
| **GPU** | ✅ | ❌ |
| **輸出檔案** | `.pose.json` | `.mediapipe.json` |
| **Appearance 共用** | 身體 ROI (neck, foot) | 臉部 ROI (hat, glasses)、手部 ROI (watch, phone) |
| **用途** | 身體姿勢、bbox 來源 | 完整關鍵點、手勢辨識、唇型分析 |
---
### 3.8 Appearance — 色彩特徵 + 配件偵測
**型別**: Frame-based
**腳本**: `appearance_processor.py`
**依賴**: pose (bbox 來源)
**採樣**: 8Hz
**ROI 共用**: 緊密貼合 face/pose/mediapipe landmarks
```rust
pub struct AppearanceResult {
pub frame_count: u64,
pub fps: f64,
pub frames: Vec<AppearanceFrame>,
}
pub struct AppearanceFrame {
pub frame: u64,
pub timestamp: f64,
pub persons: Vec<AppearancePerson>,
}
pub struct AppearancePerson {
pub person_id: u64,
pub bbox: BBox,
pub hsv_histogram: Vec<Vec<f64>>,
pub dominant_colors: Vec<Vec<f64>>,
pub upper_body: Option<Vec<Vec<f64>>>,
pub lower_body: Option<Vec<Vec<f64>>>,
}
```
**輸出 JSON**:
```json
{
"frame_count": 2238,
"fps": 29.97,
"frames": [
{
"frame": 0,
"timestamp": 0.0,
"persons": [{
"person_id": 0,
"bbox": {"x": 400, "y": 100, "width": 300, "height": 600},
"hsv_histogram": [
[H0, H1, ...H29],
[S0, S1, ...S31],
[V0, V1, ...V31]
],
"dominant_colors": [[H,S,V], ...],
"upper_body": [[H...], [S...], [V...]],
"lower_body": [[H...], [S...], [V...]]
}]
}
]
}
```
#### ROI 定位方式
```python
def get_accessory_rois(frame, face_data, pose_data, hand_data):
rois = {}
# 臉部區域 — 用 face bbox + landmarks
face_bbox = face_data['bbox']
landmarks = face_data['landmarks'] # nose, left_eye, right_eye
# 帽子 ROI: 臉部 bbox 上方延伸
rois['hat'] = expand_region(face_bbox, direction='up', factor=0.5)
# 眼鏡 ROI: 眼部 landmarks 水平帶
rois['glasses'] = bbox_around_points(landmarks['left_eye'], landmarks['right_eye'], padding=10)
# 口罩 ROI: 鼻子下方到下顎
rois['mask'] = region_below_point(landmarks['nose'], face_bbox.bottom)
# 脖子 ROI — 用 pose neck keypoints
rois['neck'] = region_between(pose_data['keypoints']['nose'], pose_data['keypoints']['neck'], width=80)
# 手腕 ROI — 用 MediaPipe hand landmarks
rois['left_wrist'] = circle_around(hand_data['left']['wrist'], radius=30)
# 腳部 ROI — 用 pose ankle/toe keypoints
rois['left_foot'] = bbox_around_points(pose_data['left_ankle'], pose_data['left_toe'], padding=20)
return rois
```
#### 配件偵測方式
| 方式 | 適用配件 | 說明 |
|------|----------|------|
| **HSV 色塊** | tie, phone, watch, ring, bracelet, glasses, mask, hat, shoes, backpack, handbag | 主要方式 — 異色區塊分析 |
| **CLIP** | hairstyle, beard, face_tattoo, earrings, nose_ring, necklace, gloves | 輔助 — 色塊不易區分時 |
| **MediaPipe** | gesture, arm_pose | 21 hand pts + 33 pose pts |
| **HSV** | upper_body_color, lower_body_color, skin_tone | 色彩特徵提取 |
#### 配件完整清單 (49 種)
| 部位 | 配件 | 偵測 |
|------|------|------|
| 頭部 (12) | hat, hairstyle, hair_accessory, earrings, nose_ring, lip_ring, face_tattoo, eyebrow_tattoo, glasses, mask, beard, headscarf | HSV 色塊 + CLIP |
| 脖子 (5) | tie, scarf, shawl, necklace, neck_tattoo | HSV 色塊 + CLIP |
| 手部/手臂 (16) | ring, bracelet, watch, gloves, phone, pen, laptop, book, cup, remote, tool, knife, gun, baseball_bat, gesture, arm_pose | HSV 色塊 + CLIP + MP |
| 足部/載具 (8) | shoes, socks, barefoot, skateboard, scooter, bicycle, motorbike, roller_skates | HSV 色塊 + CLIP |
| 攜帶/環境 (5) | backpack, handbag, luggage, chair, diningtable | HSV 色塊 + CLIP |
| 色彩 (3) | upper_body_hsv, lower_body_hsv, skin_tone | HSV |
---
### 3.9 Scene — 場景分類
**型別**: Time-based
**腳本**: `scene_classifier.py`
**模型**: places365
**依賴**: cut
---
### 3.10 Story — 故事生成
**型別**: Time-based
**腳本**: `story_processor.py`
**模型**: gemma4
**依賴**: asrx + cut + yolo + face
---
### 3.11 5W1H — 故事摘要
**型別**: Time-based
**腳本**: `parent_chunk_5w1h.py`
**模型**: gemma4
**依賴**: story
---
## 4. PythonExecutor 統一框架
### 4.1 RetryConfig
```rust
pub struct RetryConfig {
pub max_attempts: u32, // 預設 3
pub initial_delay_ms: u64, // 預設 1000 (1s)
pub max_delay_ms: u64, // 預設 30000 (30s)
pub backoff_multiplier: f64, // 預設 2.0
}
```
**退避策略**: 1s → 2s → 4s → 8s → ... → max 30s
### 4.2 SHA256 Checksum 驗證
```
scripts/
├── checksums.sha256 # SHA256 manifest
├── face_processor.py
├── yolo_processor.py
└── ...
```
`checksums.sha256` 內容:
```
a1b2c3d4... face_processor.py
e5f6g7h8... yolo_processor.py
...
```
Executor 啟動前驗證腳本完整性，防止腳本被篡改。
### 4.3 Timeout 管理
| Processor | Timeout |
|-----------|---------|
| cut | 3600s (1h) |
| asrx, yolo, ocr, face, pose, mediapipe, appearance, scene, story, 5w1h | 7200s (2h) |
---
## 5. 8Hz 採樣框架
### 5.1 基本原理
```
影片 FPS: ~30
Sample Interval: round(fps / 8) = 4
Sample Frames: 0, 4, 8, 12, 16, ...
```
| 影片長度 | 總幀數 | 8Hz 樣本數 |
|----------|--------|------------|
| 5 分鐘 | 9,000 | ~2,250 |
| 10 分鐘 | 18,000 | ~4,500 |
| 30 分鐘 | 54,000 | ~13,500 |
### 5.2 按需細化機制
```
Layer 1: 8Hz 基底 (所有 processor)
↓
Layer 2: 細化 (特定特徵觸發)
細化場景:
- Blink 確認: 8Hz 發現 eye openness 突降 → 回頭抓前後 ±4 幀 (30Hz)
- Lip-sync: sentence chunk 覆蓋的時間段 → 16Hz
- Mutual Gaze: 兩人 gaze 方向接近 → 前後 ±2 幀 (30Hz) 確認
```
### 5.3 樣本幀計算
```rust
fn compute_sample_frames(total_frames: i64, fps: f64) -> Vec<i64> {
let interval = (fps / 8.0).round() as i64;
(0..total_frames).step_by(interval.max(1) as usize).collect()
}
```
---
## 6. DAG 依賴圖
```
┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐
│ cut │───►│asrx │───►│story│───►│5w1h │
└──┬──┘ └──┬──┘ └──┬──┘ └─────┘
│ │ │
│ ┌─────┘ │
▼ ▼ │
┌─────┐ ┌─────┐ ┌─────┐ │
│yolo │ │face │ │pose │ │
└──┬──┘ └──┬──┘ └──┬──┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌────────┐ │
│ └─►│appear │ │
│ └────────┘ │
▼ ▼ ▼
┌─────────────────────────┐
│ TKG (build_tkg) │
└─────────────────────────┘
獨立處理器 (無依賴):
┌─────┐ ┌─────┐ ┌───────────┐
│ ocr │ │mediap│ │ scene │
└─────┘ └─────┘ └─────┬─────┘
│ (依賴 cut)
```
---
## 7. Worker 整合
### 7.1 JobWorker 調度
```
Video Registration
│
▼
Create Job (processor_list: [cut, asrx, yolo, ocr, face, pose, mediapipe, appearance, scene, story])
│
▼
Poll Available Processors (dependency check + concurrency limit)
│
▼
Execute Processor → Store JSON → Update Progress
│
▼
All Processors Done → Rule 1 (chunk) → Vectorize → Complete
```
### 7.2 並發控制
- **Dynamic concurrency**: 根據 CPU/Memory/GPU 動態調整 (預設 2)
- **Processor pool**: 同時執行最多 N 個 processor
### 7.3 進度回報 (Redis)
```
Redis Key: momentry_dev:progress:{file_uuid}
Value: {
"phase": "PROCESSING",
"progress": {
"FACE": {"current": 150, "total": 2238, "status": "running"},
"YOLO": {"current": 2238, "total": 2238, "status": "completed"},
...
},
"active_processors": ["FACE", "POSE"]
}
```
---
## Version History
| Version | Date | Author | Description |
|---------|------|--------|-------------|
| 1.0.0 | 2026-06-19 | OpenCode | Initial design document |
docs_v1.0/DESIGN/REPRESENTATIVE_FRAME_API_V1.md
+128
View File
@@ -0,0 +1,128 @@
# Representative Frame API V1.0
Portal 影片代表畫面 API — 沒有指定 frame_number 時自動偵測男女主角找到最佳互動 frame。
---
## 1. Overview
### Purpose
Portal 需要為每個影片顯示一張代表畫面（thumbnail），內容應為該影片最具代表性的 scene — 通常包含男女主角同框且互看的時刻。
### Principle
**沒有指定 frame_number → auto-detect representative frame**
既有端點不需改動，只需在 `frame` 參數為空時自動偵測。
---
## 2. Endpoint
### `GET /api/v1/file/:file_uuid/thumbnail`
**Query Parameters**:
| Param | Type | Required | Description |
|-------|------|----------|-------------|
| `frame` | i64 | ❌ | 指定 frame；不傳則 auto-detect |
| `x` | i32 | ❌ | bbox crop x |
| `y` | i32 | ❌ | bbox crop y |
| `w` | i32 | ❌ | bbox crop width |
| `h` | i32 | ❌ | bbox crop height |
**Response**: Pure JPEG bytes (Content-Type: image/jpeg)
**Examples**:
```
GET /api/v1/file/:uuid/thumbnail → auto-detect
GET /api/v1/file/:uuid/thumbnail?frame=38165 → 指定 frame
GET /api/v1/file/:uuid/thumbnail?frame=38165&x=723&y=205&w=221&h=221 → 指定 crop
```
---
## 3. Internal Algorithm
### Auto-detect Fallback Chain
```
Step 1: Auto-detect 主角 (top 2 by face count)
└─ face_detections JOIN identities
Step 2: TKG Bridge — mutual_gaze?
├── 有 mutual_gaze edge → first_frame ✅
└── 無 → face_detections 第一次同框 frame ✅
Step 3: 只有一個主角?
└─ 該主角 face_quality (w×h×confidence) 最高 frame
Step 4: 完全無 identity?
└─ 任 identity 的 face_quality 最高 frame
Step 5: 完全無 face?
└─ 404 "No faces in this file"
```
### TKG Bridge Query
```sql
-- 找兩主角各自的 main trace
SELECT trace_id FROM face_detections
WHERE file_uuid = $1 AND identity_id = $2 AND trace_id IS NOT NULL
GROUP BY trace_id ORDER BY COUNT(*) DESC LIMIT 1;
-- TKG mutual_gaze 查詢
SELECT (e.properties->>'first_frame')::bigint
FROM tkg_edges e
JOIN tkg_nodes a ON a.id = e.source_node_id
JOIN tkg_nodes b ON b.id = e.target_node_id
WHERE e.file_uuid = $1
AND a.external_id = concat('trace_', $4)
AND b.external_id = concat('trace_', $5)
AND e.properties->>'mutual_gaze' = 'true'
LIMIT 1;
-- Fallback: 第一次同框
SELECT MIN(fd_a.frame_number)::bigint
FROM face_detections fd_a
JOIN face_detections fd_b ON fd_a.frame_number = fd_b.frame_number
WHERE fd_a.file_uuid = $1 AND fd_a.identity_id = $2 AND fd_b.identity_id = $3;
```
---
## 4. Implementation
### Files Changed
| File | Change |
|------|--------|
| `src/api/media_api.rs` | `ThumbQuery.frame` → `Option<i64>`; add auto-detect fallback |
| `src/core/processor/tkg.rs` | Add `query_auto_representative_frame()` + structs (已實作) |
| `src/core/processor/mod.rs` | Export new function + structs (已實作) |
### Existing Trace-level Endpoints (不變)
```
GET /api/v1/file/:uuid/trace/:tid/representative-face → JSON (legacy)
GET /api/v1/file/:uuid/trace/:tid/thumbnail → JPEG (auto via select_rep_face)
```
### No Changes
- ❌ No new DB tables / migrations
- ❌ No changes to `select_rep_face` / blurdetect
- ❌ No chunk / cut / pre_chunks dependency
---
## 5. Version History
| Date | Version | Author | Change |
|------|---------|--------|--------|
| 2026-05-22 | 1.0 | OpenCode | Initial design |
| 2026-05-22 | 1.1 | OpenCode | 簡化為單一 endpoint: frame 為 None 時 auto-detect |
*Updated: 2026-05-22*
docs_v1.0/DESIGN/RULE1_CHUNK_V1.0.md
+187
View File
@@ -0,0 +1,187 @@
---
title: Rule 1 Chunk Ingestion V1.0
version: 1.0
date: 2026-06-20
author: OpenCode
status: approved
---
# Rule 1 Chunk Ingestion V1.0
| Scope | Status | Applicable to | Binary |
|-------|--------|---------------|--------|
| Sentence chunk creation from ASR + OCR | Approved | `momentry_playground`, `momentry` | Both |
## Overview
Rule 1 is the first chunking rule in Momentry's pipeline. It creates **sentence-level chunks** (`ChunkType::Sentence`, `ChunkRule::Rule1`) by taking ASR transcription segments and enriching them with OCR on-screen text from the same time range. Each chunk represents a spoken segment annotated with the visible text in the video frames.
These chunks are vectorized by the downstream `vectorize_chunks` step and become searchable through semantic search (Qdrant), keyword search (BM25 ILIKE), and identity-based search.
## Data Flow
```
┌─────────────────────────────────────────────────────────┐
│ UPSTREAM: pre_chunks table │
│ │
│ Processor outputs stored by store_raw_pre_chunks_batch: │
│ processor_type='asr' → ASR segments (text, timestamps) │
│ processor_type='ocr' → OCR texts per frame │
└─────────────────────────────────────────────────────────┘
│
▼ wait for ASRX completion
│
┌─────────────────────────────────────────────────────────┐
│ RULE 1 PROCESSING │
│ │
│ Triggered by: │
│ 1. Worker auto: job_worker.rs after ASRX completes │
│ 2. HTTP API: POST /api/v1/file/:file_uuid/rule1 │
│ 3. Pipeline: pipeline_core::execute_rule1 │
│ │
│ execute_rule1(file_uuid, fps): │
│ ├─ fetch_asr_segments() → Vec<AsrSegment> │
│ ├─ fetch_ocr_texts() → BTreeMap<frame, [texts]> │
│ │ │
│ └─ for each ASR segment: │
│ ├─ collect_ocr_text(frame_range, ocr_map) │
│ │ → deduplicated OCR texts within range │
│ ├─ build combined_text = "<ASR> <OCR>" │
│ ├─ build content = {text, ocr_text} │
│ ├─ build metadata = {language} │
│ └─ store_chunk_in_tx() → chunk table │
│ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ DOWNSTREAM: vectorize_chunks() │
│ │
│ SELECT ... WHERE chunk_type='sentence' AND embedding │
│ IS NULL │
│ │
│ 1. embedder.embed_document(combined_text) → vector │
│ 2. db.store_vector() → PG chunk.embedding │
│ 3. qdrant.upsert_vector() → momentry_rule1 collection │
│ │
└─────────────────────────────────────────────────────────┘
```
## Chunk Data Structure
### Content JSON (`content` column)
```json
{
"text": "今天的會議我們要討論 ...",
"ocr_text": "Q3 Revenue Slides Agenda"
}
```
| Field | Source | Purpose |
|-------|--------|---------|
| `text` | ASR transcription | Original spoken text, used by UI/reference |
| `ocr_text` | OCR detections in frame range | On-screen text (titles, labels, signs) |
### Text Content (`text_content` column)
```
"今天的會議我們要討論 Q3 Revenue Slides Agenda"
```
Combined ASR + OCR text used for:
- **Embedding generation**: The combined text is embedded to Qdrant, enabling semantic search to find segments based on both spoken and on-screen content
- **Keyword search (BM25 ILIKE)**: Queries match against this field, so searching for "Q3 Revenue" finds the segment even if not spoken aloud
### Metadata JSON (`metadata` column)
```json
{
"language": "zh"
}
```
Only the ASR-detected language is stored. See Design Decisions below.
## Search Contribution Analysis
| Search Path | Mechanism | Rule 1 Contribution |
|-------------|-----------|-------------------|
| **Semantic search** (Qdrant) | `chunk_type='sentence'` → embedding query | ASR + OCR text in embedding captures both spoken and visual content |
| **Keyword search** (BM25 ILIKE) | `text_content ILIKE '%query%'` | Both ASR and OCR text are searchable |
| **Title match** (smart_search) | `chunk_type='sentence' AND embedding IS NOT NULL` | Rule 1 chunks are the primary sentence chunks |
| **Identity search** | `face_detections` time overlap join | Rule 1 chunks match via frame ranges |
### What Was Excluded and Why
| Data Source | Considered For | Decision | Reason |
|-------------|---------------|----------|--------|
| **YOLO detections** | Adding class names to text_content | ❌ **Excluded** | 80 COCO classes are too generic ("person", "chair" appear in almost every segment). High error rate adds noise, dilutes embedding semantic density. Cross-segment distinctiveness is near zero. |
| **ASRX speaker** | Adding speaker_id to metadata | ❌ **Excluded** | At Rule 1 time, identity has not been paired yet. Speaker IDs are temporary labels without identity binding, providing no search value. |
| **Face detections** | Adding face_ids to metadata | ❌ **Excluded** | Same as speaker — identity not yet available. Face detection IDs alone have no search meaning. |
| **OCR text** | Adding to text_content + embedding | ✅ **Included** | OCR provides specific on-screen text (titles, labels, signs) that directly matches user search queries. Highly complementary to ASR. |
## Implementation Details
### `fetch_ocr_texts()`
Reads OCR per-frame data from `pre_chunks`:
```sql
SELECT coordinate_index as frame, data
FROM pre_chunks
WHERE file_uuid = $1 AND processor_type = 'ocr'
ORDER BY coordinate_index
```
Parses the `data.texts` JSON array, extracting `text` fields where `confidence > 0.5`. Returns `BTreeMap<i64, Vec<String>>` mapping frame number to list of recognized text strings.
### `collect_ocr_text()`
For a given frame range `[start_frame, end_frame]`:
1. Iterates frames using `BTreeMap::range(start_frame..=end_frame)`
2. Collects all OCR texts from those frames
3. Deduplicates using a `HashSet` (case-sensitive)
4. Joins with spaces: `"text1 text2 text3"`
Returns empty string if no OCR data exists in the range.
### `text_content` Composition Rules
```
if OCR text exists:
combined = "{asr_text} {ocr_text}"
else:
combined = "{asr_text}"
```
The combined string is used for both embedding and keyword search. The original ASR text is preserved separately in `content.text`.
## Trigger Points
| Trigger | Location | Condition |
|---------|----------|-----------|
| Worker auto | `job_worker.rs:1135` | After ASRX processor completes and no sentence chunks exist yet |
| HTTP API | `POST /api/v1/file/:file_uuid/rule1` | Manual trigger via `pipeline_core::execute_rule1` |
| Programmatic | `pipeline_core::execute_rule1` | Called by other modules needing sentence chunks |
The worker guard checks idempotency:
```sql
SELECT 1 FROM chunk WHERE file_uuid = $1 AND chunk_type = 'sentence' LIMIT 1
```
## Edge Cases
| Scenario | Behavior |
|----------|----------|
| No ASR segments | Returns 0 immediately with info log |
| No OCR data in pre_chunks | `ocr_text` is empty string; `text_content` = ASR only |
| OCR frame with no valid text | Skipped (confidence < 0.5 or empty string) |
| ASR segment end_time = 0.0 | Logs warning; overlap-based matching degrades gracefully |
| Large number of segments | Batches in single transaction; progress logged every 100 segments |
## Version History
| Version | Date | Author | Change |
|---------|------|--------|--------|
| 1.0 | 2026-06-20 | OpenCode | Initial design: ASR + OCR → sentence chunks |
docs_v1.0/DESIGN/RULE2_TKG_RELATIONSHIP_V1.0.md
+249
View File
@@ -0,0 +1,249 @@
---
title: Rule 2 TKG Relationship Chunks V1.0
version: 1.1
date: 2026-06-22
author: OpenCode
status: approved
---
# Rule 2 TKG Relationship Chunks V1.0
| Scope | Status | Applicable to | Binary |
|-------|--------|---------------|--------|
| TKG relationship vectorization | Approved | `momentry_playground`, `momentry` | Both |
## Overview
Rule 2 creates **relationship chunks** by converting TKG edges into searchable, vectorized units. Each TKG edge becomes a chunk with LLM-generated natural language description, enabling semantic search for relationship queries.
**Key Change:** Original Rule 2 (YOLO frame objects) is deprecated due to COCO classes being too generic. New Rule 2 focuses on TKG relationships.
## Node Types (V2.0 - Intuitive Naming)
| Old Name | New Name | Description | external_id Format |
|----------|----------|-------------|-------------------|
| `face_trace` | `face_track` | Face tracking across frames | `face_track_1` |
| `person_trace` | `body_track` | Body appearance tracking | `body_track_0` |
| `gaze_trace` | `gaze_track` | Gaze direction sequence | `gaze_track_1` |
| `lip_trace` | `lip_track` | Lip sync sequence | `lip_track_1` |
| `hand_trace` | `hand_track` | Hand state sequence | `hand_track_0` |
| `speaker` | `speaker_segment` | Speaker segment | `speaker_01` |
| `object` | `detected_object` | YOLO detected object | `car`, `phone` |
| `text_trace` | `text_region` | OCR text region | `text_1` |
## Data Flow
```
┌─────────────────────────────────────────────────────────┐
│ UPSTREAM: TKG Builder │
│ │
│ tkg_nodes: face_track, speaker_segment, detected_object │
│ tkg_edges: speaker_face, mutual_gaze, co_occurs, etc. │
│ │
└─────────────────────────────────────────────────────────┘
│
▼ after TKG complete
│
┌─────────────────────────────────────────────────────────┐
│ RULE 2 PROCESSING │
│ │
│ Triggered by: │
│ 1. Worker auto: job_worker.rs after TKG completes │
│ 2. HTTP API: POST /api/v1/file/:file_uuid/rule2 │
│ │
│ ingest_rule2(file_uuid): │
│ ├─ Query tkg_edges by type (priority order) │
│ ├─ For each edge: │
│ │ ├─ Resolve source_node / target_node │
│ │ ├─ Resolve identity names (if face_track) │
│ │ ├─ Build context JSON │
│ │ ├─ call_llm(context) → text_content │
│ │ └─ INSERT INTO chunk (chunk_type='relationship') │
│ │ │
│ │
└─────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────┐
│ DOWNSTREAM: vectorize_chunks() │
│ │
│ SELECT ... WHERE chunk_type='relationship' │
│ AND embedding IS NULL │
│ │
│ 1. embedder.embed_document(text_content) → vector │
│ 2. db.store_vector() → PG chunk.embedding │
│ 3. qdrant.upsert_vector() → momentry_rule2 collection │
│ │
└─────────────────────────────────────────────────────────┘
```
## Edge Type Priority
| Priority | Edge Type | Description | Example Output |
|----------|-----------|-------------|----------------|
| P0 | `speaker_face` | Speaker ↔ Face track | "SPEAKER_01 以 Cary Grant 的身份說話，從 frame 100 到 350" |
| P0 | `mutual_gaze` | Two face tracks looking at each other | "Cary Grant 和 Grace Kelly 互相看對方 24 幀，起始於 frame 450" |
| P1 | `face_face` | Two face tracks co-occurring | "Cary Grant 和 Grace Kelly 同框 180 幀" |
| P1 | `co_occurs` | Detected object ↔ Detected object co-occurrence | "物件 'car' 和 'person' 在同一畫面出現 60 幀" |
| P2 | `has_appearance` | Face track ↔ Body track | "Cary Grant 穿著藍色上衣，戴眼鏡" |
| P2 | `wears` | Face track ↔ Accessory | "Cary Grant 戴帽子，信心值 0.82" |
## Chunk Data Structure
### Content JSON (`content` column)
```json
{
"edge_type": "speaker_face",
"edge_id": 123,
"source_node": {
"id": 45,
"node_type": "speaker_segment",
"external_id": "speaker_01",
"label": "SPEAKER_01"
},
"target_node": {
"id": 67,
"node_type": "face_track",
"external_id": "face_track_5",
"label": "Face Track 5",
"identity_name": "Cary Grant"
},
"properties": {
"first_frame": 100,
"last_frame": 350,
"frame_count": 250,
"lip_sync_confidence": 0.85
}
}
```
### Text Content (`text_content` column)
LLM-generated natural language description in Traditional Chinese:
```
"SPEAKER_01 以 Cary Grant 的身份說話，從 frame 100 到 frame 350，唇語同步信心值 0.85"
```
### Metadata JSON (`metadata` column)
```json
{
"source_type": "speaker",
"target_type": "face_trace",
"has_identity": true,
"identity_source": "tmdb"
}
```
## LLM Prompt Template
```text
你是影片關係描述專家。請用繁體中文描述以下人物/物件關係：
關係類型: {edge_type}
來源節點: {source_node.node_type} - {source_node.external_id}
身份名稱: {identity_name} (如果有)
目標節點: {target_node.node_type} - {target_node.external_id}
身份名稱: {identity_name} (如果有)
關係屬性:
- 起始幀: {first_frame}
- 結束幀: {last_frame}
- 幀數: {frame_count}
- 信心值: {confidence}
要求：
1. 使用自然語言，不要輸出 JSON
2. 包含時間範圍（幀號）
3. 包含人物名字（如有 identity）
4. 簡潔，20-50 字
5. 用繁體中文
範例輸出：
"SPEAKER_01 以 Cary Grant 的身份說話，從 frame 100 到 frame 350"
"Cary Grant 和 Grace Kelly 互相看對方 24 幀，起始於 frame 450"
```
## Edge → Chunk Conversion Rules
### speaker_face Edge
```rust
// Source: speaker_segment node
// Target: face_track node
// Properties: first_frame, last_frame, lip_sync_confidence
let text_content = call_llm(format!(
"SPEAKER {} 對應 face track {}，身份 {}，frame {}-{}",
speaker_id, track_id, identity_name, first_frame, last_frame
));
```
### mutual_gaze Edge
```rust
// Source: face_track node A
// Target: face_track node B
// Properties: first_frame, gaze_frame_count, yaw_a_avg, yaw_b_avg
let text_content = call_llm(format!(
"人物 {} 和 {} 互相看對方 {} 幀，起始於 frame {}",
identity_a, identity_b, gaze_frame_count, first_frame
));
```
### has_appearance Edge
```rust
// Source: face_track node
// Target: body_track node
// Properties: clothing colors, accessories
let text_content = call_llm(format!(
"人物 {} 穿著 {} 上衣，{} 下衣",
identity_name, upper_color, lower_color
));
```
## Search Contribution
| Search Path | Mechanism | Rule 2 Contribution |
|-------------|-----------|-------------------|
| **Semantic search** (Qdrant) | `chunk_type='relationship'` → embedding query | LLM descriptions enable natural language queries |
| **Keyword search** (BM25 ILIKE) | `text_content ILIKE '%互相看%'` | Relationship keywords searchable |
| **Agent tkg_query** | Direct edge queries | Rule 2 complements with vectorized search |
| **identity_text** | Reverse lookup | "誰戴眼鏡" → has_appearance chunks |
## Trigger Points
| Trigger | Location | Condition |
|---------|----------|-----------|
| Worker auto | `job_worker.rs` | After TKG builder completes |
| HTTP API | `POST /api/v1/file/:file_uuid/rule2` | Manual trigger |
| Pipeline | `pipeline_core::execute_rule2` | Called by other modules |
## Edge Cases
| Scenario | Behavior |
|----------|----------|
| No tkg_edges | Returns 0 immediately with info log |
| Edge without identity | Use node external_id (e.g., "trace_5") in description |
| LLM call fails | Fallback to template-based description |
| Multiple edges same type | Each edge becomes separate chunk |
## Qdrant Collection
| Property | Value |
|----------|-------|
| Collection name | `momentry_rule2` |
| Vector size | 768 (nomic-embed-text-v2-moe) |
| Distance | Cosine |
| Payload | `{chunk_id, file_uuid, edge_type, source_type, target_type}` |
## Version History
| Version | Date | Author | Change |
|---------|------|--------|--------|
| 1.1 | 2026-06-22 | OpenCode | Node type renaming: face_trace→face_track, person_trace→body_track, etc. |
| 1.0 | 2026-06-20 | OpenCode | Initial design: TKG edges → relationship chunks |
docs_v1.0/DESIGN/Redis_Prefix_Configuration.md
+179
View File
@@ -0,0 +1,179 @@
---
title: Redis Prefix Configuration
version: 1.0
date: 2026-06-21
author: momentry_core development
status: active
---
## Overview
Momentry Core uses Redis key prefixes to isolate namespaces between Production and Playground environments. This prevents cross-contamination of job queues, progress data, and cache entries.
## Environment Configuration
| Environment | Port | Redis Prefix | Config File |
|-------------|------|--------------|-------------|
| **Production** | 3002 | `momentry:` | `.env` (default) |
| **Playground** | 3003 | `momentry_dev:` | `.env.development` |
### Configuration
```bash
# Production (.env)
MOMENTRY_REDIS_PREFIX=momentry: # Default if not set
# Playground (.env.development)
MOMENTRY_REDIS_PREFIX=momentry_dev:
```
## Redis Key Structure
All Redis keys follow this pattern:
```
{prefix}{key_type}:{identifier}
```
### Key Types
| Key Type | Pattern | Example |
|----------|---------|---------|
| Job | `{prefix}job:{file_uuid}` | `momentry:job:abc123...` |
| Progress | `{prefix}progress:{file_uuid}` | `momentry:progress:abc123...` |
| Processor | `{prefix}job:{file_uuid}:processor:{type}` | `momentry:job:abc123:processor:face` |
| Health | `{prefix}health` | `momentry:health` |
## Namespace Isolation
### Production vs Playground
**Production (3002)**:
- Jobs created by production API → `momentry:job:*`
- Worker must run with production prefix
- Production worker sees only production jobs
**Playground (3003)**:
- Jobs created by playground API → `momentry_dev:job:*`
- Worker must run with playground prefix
- Playground worker sees only playground jobs
### Cross-Namespace Access
❌ **Cannot access**:
- Production API cannot see playground jobs
- Playground API cannot see production jobs
- Worker with wrong prefix will not process jobs
✅ **Design intent**:
- Complete isolation between environments
- No accidental cross-contamination
- Safe testing in playground without affecting production
## Worker Configuration
Workers must match the Redis prefix of the server that creates jobs:
```bash
# Production worker
./target/release/momentry worker
# Uses: momentry: prefix (default)
# Playground worker
./target/debug/momentry_playground worker
# Uses: momentry_dev: prefix (from .env.development)
```
### Worker Redis Connection
Workers read Redis prefix from environment:
1. Check `MOMENTRY_REDIS_PREFIX` environment variable
2. If not set, use default prefix:
- `momentry` binary → `momentry:`
- `momentry_playground` binary → `momentry_dev:`
## Common Issues
### Issue: Jobs Not Being Processed
**Symptoms**:
- API returns "Processing triggered"
- Worker shows no activity
- Redis job key created but not consumed
**Cause**: Worker running with wrong Redis prefix
**Solution**:
```bash
# Check worker prefix
redis-cli keys "momentry*"
# If jobs in momentry: namespace
# Production worker needed
./target/release/momentry worker
# If jobs in momentry_dev: namespace
# Playground worker needed
./target/debug/momentry_playground worker
```
### Issue: Progress API Returns Empty
**Symptoms**:
- Progress API returns empty response
- Job exists but progress not visible
**Cause**: Progress key in different namespace
**Solution**:
- Ensure worker prefix matches server prefix
- Check Redis keys: `redis-cli keys "{prefix}progress:*"`
## Redis CLI Examples
```bash
# List all production jobs
redis-cli -a accusys keys "momentry:job:*"
# List all playground jobs
redis-cli -a accusys keys "momentry_dev:job:*"
# Check progress for specific file (production)
redis-cli -a accusys HGETALL "momentry:progress:{file_uuid}"
# Check progress for specific file (playground)
redis-cli -a accusys HGETALL "momentry_dev:progress:{file_uuid}"
# Delete all production jobs (⚠️ destructive)
redis-cli -a accusys keys "momentry:job:*" | xargs redis-cli -a accusys del
# Delete all playground jobs (⚠️ destructive)
redis-cli -a accusys keys "momentry_dev:job:*" | xargs redis-cli -a accusys del
```
## Best Practices
1. **Always match worker to server**: Production worker for production server, playground worker for playground server
2. **Check Redis keys**: Before debugging worker issues, verify namespace alignment
3. **Document in AGENTS.md**: Update Redis prefix documentation when configuration changes
4. **Never mix namespaces**: Keep production and playground completely isolated
5. **Use environment variables**: Configure prefix via `.env` files, not hardcoded values
## Related Documentation
- `docs_v1.0/DESIGN/Redis_Progress_Reporting_V1.0.md` - Progress reporting design
- `docs_v1.0/M4_workspace/2026-06-21_issue_report.md` - Issue report with Redis prefix problem
- `AGENTS.md` - Environment configuration reference
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2026-06-21 | Initial documentation for Redis prefix configuration |
docs_v1.0/DESIGN/Redis_Progress_Reporting_V1.0.md
+270
View File
@@ -0,0 +1,270 @@
---
document_type: "design_doc"
service: "MOMENTRY_CORE"
title: "Redis Progress Reporting V1.0"
version: "V1.0"
date: "2026-05-17"
author: "M5"
status: "draft"
---
# Redis Progress Reporting V1.0
| 項目 | 內容 |
|------|------|
| Service | `MOMENTRY_CORE` |
| Version | V1.0 |
| Date | 2026-05-17 |
| Author | M5 (OpenCode) |
| Status | Draft |
## 1. Overview
This document defines the standardized progress reporting architecture for Momentry Core processors. It replaces the inconsistent ad-hoc progress patterns found across `scripts/`, `src/worker/`, and `src/api/`.
### 1.1 Problems Addressed
| # | Problem | Detail |
|---|---------|--------|
| 1 | Worker Redis key does not match `OPERATIONS/MOMENTRY_CORE_REDIS_KEYS.md` V1.0 spec | Worker writes `worker:job:{uuid}:processor:{name}` instead of spec `job:{uuid}:processor:{name}` |
| 2 | Progress API reads wrong key | `get_progress()` reads `worker:job:{uuid}:processor:{name}` — unresolved with Playground subscriber which writes `job:{uuid}:processor:{name}` |
| 3 | Swift processors (Face/OCR/Pose) lack RedisPublisher | Progress lost — only stdout text |
| 4 | ASRX/Story/Visual chunk have no incremental progress | Start + complete only, no `current/total` updates |
| 5 | `frames_processed` / `chunks_produced` never updated in real-time | Worker only writes processor hash at start and exit |
| 6 | No `output_count` / `output_type` fields | Impossible to know how many faces/objects/segments were produced |
### 1.2 Key Design Decisions
| Decision | Rationale |
|----------|-----------|
| Progress unit = frames for video processors | All media-level processors work frame by frame |
| Output count separate from progress | Processors may produce N outputs per frame (multiple faces, objects) |
| Pub/sub for real-time, Hash for final state | Pub/sub is transient; Hash persists for API queries |
---
## 2. Redis Key Architecture
### 2.1 Key Patterns
All keys use the configured `REDIS_KEY_PREFIX` (default: `momentry:` for production, `momentry_dev:` for playground).
| Pattern | Type | TTL | Purpose | Owner |
|---------|------|-----|---------|-------|
| `{prefix}progress:{uuid}` | Pub/Sub | — | Real-time progress messages | Python scripts |
| `{prefix}job:{uuid}` | Hash | 24h | Per-video job state | Worker |
| `{prefix}job:{uuid}:processor:{name}` | Hash | 24h | Per-processor final state | Worker |
| `{prefix}job:{uuid}:processor:{name}:output_count` | String | 24h | Output count by type | Worker |
### 2.2 Processor Hash Fields
```
{prefix}job:{uuid}:processor:{name}
├── status String running / completed / failed / pending
├── current u32 Units processed (frames for video processors)
├── total u32 Total units
├── output_count u32 Output items produced (faces, objects, segments)
├── output_type String Type name of output: faces / objects / segments / cuts / etc.
├── pid i32 OS process ID (0 if not running)
├── error String Error message if failed
└── updated_at String ISO 8601 timestamp
```
### 2.3 Migrated Keys
The following key patterns from the original implementation are REMOVED:
| Old Key | Reason |
|---------|--------|
| `{prefix}worker:job:{uuid}:processor:{name}` | Non-standard prefix — not in `MOMENTRY_CORE_REDIS_KEYS.md` spec |
| `{prefix}job:{uuid}:processor:{name}:status` (flat) | Redundant — status stored in Hash |
| `{prefix}job:{uuid}:processor:{name}:progress` (flat) | Replaced by `current` + `total` for percent calculation |
| `{prefix}job:{uuid}:processor:{name}:current` (flat) | Replaced by Hash fields |
| `{prefix}job:{uuid}:processor:{name}:total` (flat) | Replaced by Hash fields |
| `{prefix}job:{uuid}:processor:{name}:started_at` (flat) | Replaced by Hash `updated_at` |
---
## 3. Pub/Sub Message Format
### 3.1 Channel
```
{prefix}progress:{uuid}
```
### 3.2 Message JSON
```json
{
"processor": "face",
"current": 150,
"total": 162696,
"output_count": 423,
"output_type": "faces",
"message": "Processing frame 150",
"timestamp": 1700000000
}
```
### 3.3 Field Definitions
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `processor` | String | ✅ | Processor name: asr / asrx / yolo / ocr / face / pose / cut / story / visual_chunk |
| `current` | u32 | ✅ | Units processed (frames for video processors) |
| `total` | u32 | ✅ | Total units |
| `output_count` | u32 | ❌ | Output items produced so far |
| `output_type` | String | ❌ | Type name: faces / objects / segments / cuts / text_regions / persons / speakers / stories / visual_chunks |
| `message` | String | ❌ | Human-readable progress description |
| `timestamp` | u64 | ✅ | Unix timestamp |
---
## 4. Per-Processor Metrics
| Processor | current/total Unit | output_type | When to Publish |
|-----------|-------------------|-------------|-----------------|
| ASR | frames | `segments` | Every 100 segments processed |
| ASRX | frames | `speakers` | Every processing stage |
| YOLO | frames | `objects` | Every 500 frames |
| OCR | frames | `text_regions` | Every 5% |
| Face | frames | `faces` | Every batch (5% of frames) |
| Pose | frames | `persons` | Every 10% |
| CUT | frames | `cuts` | Every scene detected |
| Story | chunks | `stories` | Every chunk processed |
| Visual chunk | frames | `visual_chunks` | Every chunk processed |
### 4.1 Output Type Enum
```rust
pub enum OutputType {
Segments, // ASR
Speakers, // ASRX
Objects, // YOLO
TextRegions, // OCR
Faces, // Face
Persons, // Pose
Cuts, // CUT
Stories, // Story
VisualChunks, // Visual chunk
}
```
---
## 5. Data Flow
```
┌──────────────────┐ Pub/Sub ┌──────────────────────┐
│ Python Processor │ ───────── progress:{uuid} ──────────→│ Worker (subscriber) │
│ (ASR/YOLO/Face) │ {current, total, │ │
│ │ output_count, output_type} │ ──→ HSET │
└──────────────────┘ │ job:{uuid}: │
│ processor:{name} │
┌──────────────────┐ │ │
│ Swift Processor │ ──→ Python wrapper ──→ pub/sub │ (status, current, │
│ (Face/OCR/Pose) │ (add RedisPublisher) │ total, output_count,│
└──────────────────┘ │ output_type) │
└──────────┬───────────┘
│ HGETALL
┌──────────▼───────────┐
│ Progress API │
│ GET /progress/:uuid │
│ │
│ ─→ compute % │
│ ─→ return JSON │
└─────────────────────┘
```
---
## 6. Implementation Plan
### Phase 1: Python Processor RedisPublisher
| Task | Files | Effort |
|------|-------|--------|
| Add `RedisPublisher` to `face_processor.py` | `scripts/face_processor.py` | Medium |
| Add `RedisPublisher` to `ocr_processor.py` | `scripts/ocr_processor.py` | Medium |
| Add `RedisPublisher` to `pose_processor.py` | `scripts/pose_processor.py` | Medium |
| Add incremental `.progress()` to `asrx_processor_custom.py` | `scripts/asrx_processor_custom.py` | Low |
| Standardize pub/sub message to include `output_count`, `output_type` | All processor scripts | Low |
### Phase 2: Worker
| Task | Files | Effort |
|------|-------|--------|
| Fix Redis key from `worker:job:` to `job:` | `src/worker/processor.rs`, `src/core/db/redis_client.rs` | Low |
| Subscribe to `progress:{uuid}` channel in `run_processor()` | `src/worker/processor.rs` | Medium |
| HSET Processor Hash on each progress message | `src/worker/processor.rs` | Medium |
| Set `output_count` and `output_type` from pub/sub message | `src/worker/processor.rs` | Low |
### Phase 3: Progress API
| Task | Files | Effort |
|------|-------|--------|
| Read `output_count`, `output_type` from Redis Hash | `src/api/server.rs` | Low |
| Compute percentage from `current` / `total` | `src/api/server.rs` | Low |
| Return `output_count`, `output_type` in response JSON | `src/api/server.rs` | Low |
| Remove `worker:` fallback path | `src/api/server.rs` | Low |
### Phase 4: Cleanup
| Task | Files | Effort |
|------|-------|--------|
| Remove old `worker:job:` keys from Redis | Deployment script | Low |
| Remove `update_processor_progress()` DB path (stale `processing_status` JSONB) | `src/core/db/postgres_db.rs` | Medium |
---
## 7. API Response Changes
### ProgressResponse (new fields)
```json
{
"processors": [
{
"name": "face",
"status": "running",
"current": 150,
"total": 162696,
"progress": 0,
"frames_processed": 150,
"output_count": 423,
"output_type": "faces"
}
]
}
```
---
## 8. Dependencies
| Component | Version | Role |
|-----------|---------|------|
| Redis | ≥ 6.0 | Pub/Sub + Hash storage |
| `redis_publisher.py` | Existing | Python → Redis pub/sub client |
| `redis_client.rs` | Existing | Rust Redis client for worker + API |
---
## 9. References
| Doc | Relation |
|-----|----------|
| `OPERATIONS/MOMENTRY_CORE_REDIS_KEYS.md` | Parent spec — this doc supersedes sections 4, 7, 8 |
| `DESIGN/VIDEO_PROCESSING_SPEC.md` §2.3 | Original progress design (ProcessProgress struct) |
| `src/worker/processor.rs` | Worker progress write implementation |
| `scripts/redis_publisher.py` | Python pub/sub client |
| `src/api/server.rs` (get_progress) | Progress API handler |
---
## Version History
| Version | Date | Author | Change |
|---------|------|--------|--------|
| V1.0 | 2026-05-17 | M5 (OpenCode) | Initial draft — replaces ad-hoc progress patterns |
docs_v1.0/DESIGN/TKG_FORMATION_V1.0.md
+390
View File
@@ -0,0 +1,390 @@
---
title: TKG Formation V1.0
version: 1.0
date: 2026-06-25
author: OpenCode
status: draft
---
## Overview
Temporal Knowledge Graph (TKG) is built from multi-processor outputs to create a time-aligned knowledge graph. This document defines the formation phases, node/edge types, data flow, and integration with Identity Agent.
---
## Phase Definition
| Phase | Name | Trigger | Input | Output | Code Location |
|-------|------|---------|-------|--------|---------------|
| **Phase 0** | Populate | TKG rebuild | `face.json` | `PG face_detections.trace_id` | `tkg.rs:20-100` |
| **Phase 1** | Extract | Video register | Video frames | `Qdrant _faces` (512D embeddings) | `face_processor.py` |
| **Phase 2** | Build Nodes | TKG rebuild | `face.json`, PG tables | `tkg_nodes` (9 types) | `tkg.rs:506-515` |
| **Phase 3** | Build Edges | TKG rebuild | `tkg_nodes`, pose data | `tkg_edges` (9 types) | `tkg.rs:517-524` |
| **Phase 4** | Identity | Manual/API call | `Qdrant _faces`, `_seeds` | `tkg_nodes.status` updated | `identity_matcher.py` |
### Phase Details
#### Phase 0: Populate
**Purpose:** Assign `trace_id` to face detections.
**Flow:**
1. Check if `trace_id IS NOT NULL` already exists in `face_detections`
2. If not, call `store_traced_faces.py`
3. `store_traced_faces.py` runs `face_tracker.py` (IoU-only)
4. Update `face_detections.trace_id`
**Dependency:** `face.json` must exist (from Phase 1)
---
#### Phase 1: Extract
**Purpose:** Generate face embeddings and push to Qdrant.
**Flow:**
1. `face_processor.py` runs Vision detection (ANE)
2. Crop faces from video frames
3. CoreML FaceNet → 512D embedding
4. Push to Qdrant `_faces` collection
5. Write `face.json` (metadata only, no embedding)
**Output:**
- `face.json` in output directory
- Qdrant `_faces` collection with embeddings
---
#### Phase 2: Build Nodes
**Purpose:** Create TKG nodes from processor outputs.
**Node Builders:**
| Builder | Node Type | Data Source |
|---------|-----------|-------------|
| `build_face_track_nodes` | `face_track` | `face.json`, `face_detections` |
| `build_gaze_track_nodes` | `gaze_track` | `face.json` (pose data) |
| `build_lip_track_nodes` | `lip_track` | `face.json` (lips), `asrx.json` |
| `build_text_region_nodes` | `text_region` | `asrx.json` |
| `build_appearance_trace_nodes` | `appearance_trace` | `yolo.json` (person) |
| `build_accessory_nodes` | `accessory` | `yolo.json` |
| `build_yolo_object_nodes` | `object` | `yolo.json` |
| `build_hand_nodes` | `hand` | `pose.json` |
| `build_speaker_nodes` | `speaker` | `asrx.json` |
---
#### Phase 3: Build Edges
**Purpose:** Create TKG edges from node relationships.
**Edge Builders:**
| Builder | Edge Type | Source → Target |
|---------|-----------|-----------------|
| `build_co_occurrence_edges` | `co_occurs` | `object ↔ object` |
| `build_speaker_face_edges` | `speaker_face` | `speaker ↔ face_track` |
| `build_face_face_edges` | `face_face` | `face_track ↔ face_track` |
| `build_mutual_gaze_edges` | `mutual_gaze` | `gaze_track ↔ gaze_track` |
| `build_lip_sync_edges` | `lip_sync` | `lip_track ↔ text_region` |
| `build_has_appearance_edges` | `has_appearance` | `face_track ↔ appearance_trace` |
| `build_wears_edges` | `wears` | `face_track ↔ accessory` |
| `build_hand_object_edges` | `hand_object` | `hand ↔ object` |
---
#### Phase 4: Identity
**Purpose:** Mark face_track nodes with identity binding status.
**Flow:**
1. Identity Agent queries `_seeds` collection (TMDb/manual/propagation)
2. Queries `_faces` collection for trace representatives
3. Multi-angle matching (3 reps per trace)
4. Mark TKG nodes: `status='suggested'`, `confidence`, `pending_identity_name`
5. User confirms → update TKG, Qdrant, PG
6. Confirmed trace becomes propagation seed in `_seeds`
---
## Node Types (Naming Standardized)
**Naming Rule:**
- All trace types use `_track` suffix
- Text uses `_region` (non-temporal)
| Node Type | External ID Format | Key Properties |
|-----------|---------------------|----------------|
| `face_track` | `face_track_{trace_id}` | `trace_id`, `frame_count`, `start_frame`, `end_frame`, `avg_bbox`, `status`, `confidence`, `identity_uuid` |
| `gaze_track` | `gaze_track_{id}` | `direction` (frontal/left/right/up/down + diagonals) |
| `lip_track` | `lip_track_{id}` | `speaker_id`, `lip_area_range` |
| `text_region` | `text_region_{id}` | `speaker_id`, `text`, `start_time`, `end_time` |
| `appearance_trace` | `appearance_{trace_id}` | `clothing_color`, `upper_cloth`, `lower_cloth` |
| `accessory` | `accessory_{id}` | `type` (glasses/hat/etc.), `confidence` |
| `object` | `object_{class}_{id}` | `class`, `confidence`, `frame_count` |
| `speaker` | `speaker_{speaker_id}` | `speaker_id`, `segment_count`, `total_duration` |
### face_track Identity Properties
| Property | Type | Values |
|----------|------|--------|
| `status` | string | `pending` | `suggested` | `confirmed` | `stranger` |
| `pending_identity_name` | string/null | Suggested identity name |
| `pending_identity_uuid` | string/null | Suggested identity UUID |
| `suggested_by` | string/null | `tmdb` | `propagation` | `manual` |
| `confidence` | float/null | Matching score (0.0-1.0) |
| `identity_uuid` | string/null | Confirmed identity UUID |
| `identity_id` | integer/null | Confirmed PG identity.id |
| `identity_ref` | string/null | Reference string (e.g., `file_uuid:identity_1`) |
| `stranger_id` | integer/null | Stranger cluster ID |
| `stranger_ref` | string/null | Reference string (e.g., `stranger_1`) |
---
## Edge Types
| Edge Type | Storage Name | Source → Target | Properties |
|-----------|--------------|-----------------|------------|
| `co_occurs` | `CO_OCCURS_WITH` | `object ↔ object` | `frame_count`, `confidence` |
| `speaker_face` | `SPEAKS_AS` | `speaker → face_track` | `overlap_frames`, `confidence` |
| `face_face` | `INTERACTS_WITH` | `face_track ↔ face_track` | `co_occurrence_frames` |
| `mutual_gaze` | `MUTUAL_GAZE` | `gaze_track ↔ gaze_track` | `frame_count` |
| `lip_sync` | `LIP_SYNC` | `lip_track → text_region` | `speaker_id` |
| `has_appearance` | `HAS_APPEARANCE` | `face_track → appearance_trace` | `frame_count` |
| `wears` | `WEARS` | `face_track → accessory` | `confidence` |
| `hand_object` | `HOLDS` | `hand → object` | `frame_count`, `confidence` |
---
## Data Flow Diagram
```mermaid
graph TB
subgraph Phase0[Phase 0: Populate]
A[face.json] --> B[store_traced_faces.py]
B --> C[PG face_detections.trace_id]
end
subgraph Phase1[Phase 1: Extract]
D[Video Frames] --> E[face_processor.py]
E --> F[face.json metadata]
E --> G[Qdrant _faces 512D]
end
subgraph Phase2[Phase 2: Build Nodes]
C --> H[TKG Builder]
F --> H
I[pose.json] --> H
J[asrx.json] --> H
K[yolo.json] --> H
H --> L[tkg_nodes<br/>9 node types]
end
subgraph Phase3[Phase 3: Build Edges]
L --> M[TKG Builder]
M --> N[tkg_edges<br/>9 edge types]
end
subgraph Phase4[Phase 4: Identity]
G --> O[identity_matcher.py]
L --> O
P[Qdrant _seeds] --> O
O --> Q[mark_tkg_suggested]
Q --> L
O --> R[confirm_identity.py]
R --> L
R --> G
R --> P
end
style Phase0 fill:#e1f5fe
style Phase1 fill:#fff9c4
style Phase2 fill:#e8f5e9
style Phase3 fill:#f3e5f5
style Phase4 fill:#fce4ec
```
---
## SQL Query Examples
### Status Queries
```sql
-- Get all face_track nodes for a file
SELECT id, external_id, label, properties
FROM dev.tkg_nodes
WHERE node_type = 'face_track' AND file_uuid = 'xxx'
ORDER BY external_id;
-- Get pending faces (no identity suggestion)
SELECT id, external_id, properties->>'trace_id' as trace_id
FROM dev.tkg_nodes
WHERE node_type = 'face_track'
AND file_uuid = 'xxx'
AND (properties->>'status' IS NULL OR properties->>'status' = 'pending');
-- Get suggested faces (Identity Agent suggested)
SELECT id, external_id,
properties->>'pending_identity_name' as name,
properties->>'confidence' as confidence,
properties->>'suggested_by' as source
FROM dev.tkg_nodes
WHERE node_type = 'face_track'
AND file_uuid = 'xxx'
AND properties->>'status' = 'suggested'
ORDER BY (properties->>'confidence')::float DESC;
-- Get confirmed faces
SELECT id, external_id,
properties->>'identity_uuid' as identity_uuid,
properties->>'identity_name' as name
FROM dev.tkg_nodes
WHERE node_type = 'face_track'
AND file_uuid = 'xxx'
AND properties->>'status' = 'confirmed';
-- Get stranger cluster members
SELECT id, external_id, properties->>'stranger_id' as cluster
FROM dev.tkg_nodes
WHERE node_type = 'face_track'
AND file_uuid = 'xxx'
AND properties->>'status' = 'stranger'
ORDER BY (properties->>'stranger_id')::int;
```
### Identity Queries
```sql
-- Find all traces bound to an identity
SELECT id, external_id, properties->>'trace_id' as trace_id
FROM dev.tkg_nodes
WHERE node_type = 'face_track'
AND properties->>'identity_uuid' = 'xxx-xxx';
-- Count identities per file
SELECT properties->>'identity_uuid' as identity_uuid,
COUNT(*) as trace_count
FROM dev.tkg_nodes
WHERE node_type = 'face_track'
AND file_uuid = 'xxx'
AND properties->>'status' = 'confirmed'
GROUP BY properties->>'identity_uuid';
```
### Statistics Queries
```sql
-- Status distribution for a file
SELECT properties->>'status' as status, COUNT(*) as count
FROM dev.tkg_nodes
WHERE node_type = 'face_track' AND file_uuid = 'xxx'
GROUP BY properties->>'status';
-- Confidence distribution
SELECT
CASE
WHEN (properties->>'confidence')::float >= 0.9 THEN 'high'
WHEN (properties->>'confidence')::float >= 0.7 THEN 'medium'
ELSE 'low'
END as confidence_level,
COUNT(*) as count
FROM dev.tkg_nodes
WHERE node_type = 'face_track'
AND file_uuid = 'xxx'
AND properties->>'status' = 'suggested'
GROUP BY confidence_level;
-- Top suggested identities
SELECT properties->>'pending_identity_name' as name,
COUNT(*) as trace_count,
AVG((properties->>'confidence')::float) as avg_confidence
FROM dev.tkg_nodes
WHERE node_type = 'face_track'
AND properties->>'status' = 'suggested'
GROUP BY properties->>'pending_identity_name'
ORDER BY trace_count DESC
LIMIT 10;
```
### Cross-node Queries
```sql
-- Speaker ↔ face_track edges
SELECT
s.external_id as speaker,
f.external_id as face_track,
e.properties->>'overlap_frames' as overlap
FROM dev.tkg_edges e
JOIN dev.tkg_nodes s ON e.source_node_id = s.id
JOIN dev.tkg_nodes f ON e.target_node_id = f.id
WHERE e.file_uuid = 'xxx'
AND e.edge_type = 'SPEAKS_AS';
-- Objects co-occurrence
SELECT
o1.external_id as obj1,
o2.external_id as obj2,
e.properties->>'frame_count' as co_frames
FROM dev.tkg_edges e
JOIN dev.tkg_nodes o1 ON e.source_node_id = o1.id
JOIN dev.tkg_nodes o2 ON e.target_node_id = o2.id
WHERE e.file_uuid = 'xxx'
AND e.edge_type = 'CO_OCCURS_WITH'
ORDER BY (e.properties->>'frame_count')::int DESC;
```
---
## Integration with Identity Agent
### Identity Agent Flow
```
Identity Agent Pipeline:
│
├─ Round 1 (TH=0.55):
│ Query _seeds (source='tmdb')
│ Query _faces (file_uuid) → get trace representatives
│ Multi-angle match → suggestions
│ Mark TKG: status='suggested', confidence
│
├─ User Confirmation:
│ Update TKG: status='confirmed'
│ Update _faces: identity_uuid for all points
│ Update PG face_detections: identity_id
│ Add _seeds: source='propagation'
│
├─ Round 2 (TH=0.55):
│ Use confirmed traces as seeds
│ Match remaining pending traces
│
└─ Round 3+ (TH=0.50):
Continue propagation
Stranger clustering (TH=0.40)
```
### TKG Node Status Transitions
```
pending → suggested → confirmed → (final)
↘ stranger ↘ (final)
```
### Status Transition Triggers
| Transition | Trigger | Action |
|------------|---------|--------|
| `pending → suggested` | Identity Agent Round 1-3 | `mark_face_track_suggested()` |
| `suggested → confirmed` | User confirmation API | `mark_face_track_confirmed()` |
| `pending → stranger` | Stranger clustering | `mark_face_track_stranger()` |
| `confirmed → pending` | Undo binding | `clear_face_track_status()` |
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2026-06-25 | Initial version with Phase 0-4 definition, node naming, flow diagram, query examples |
docs_v1.0/DESIGN/TKG_MultiTrace_V1.0.md
+816
View File
@@ -0,0 +1,816 @@
# TKG Multi-Trace Design V1.0
**Date**: 2026-06-19
**Version**: 1.0.0
**Status**: Draft
---
## Overview
統一 8Hz 採樣框架，整合 face、appearance、gaze、lip 四條 trace，並接入 sentence/speaker/accessory 節點，構建完整的 Temporal Knowledge Graph (TKG)。
### 設計目標
1. **時間對齊**: 所有 trace 在同一 8Hz 網格上，edge 計算無需插值
2. **按需細化**: 特定特徵 (blink, lip-sync, mutual gaze) 可局部提高採樣率
3. **配件偵測**: 49 種配件分類 (頭部 12 + 脖子 5 + 手部 16 + 足部 8 + 攜帶 5 + 色彩 3)
4. **膚色 + 光源**: Fitzpatrick 分類 + 光照參數，支援可信度評估
5. **社交互動**: Mutual gaze (互相看), lip-sync (唇語同步), speaker-face 綁定
---
## 1. 8Hz 採樣框架
### 1.1 基本原理
```
影片 FPS: ~30
Sample Interval: round(fps / 8) = 4
Sample Frames: 0, 4, 8, 12, 16, ...
```
| 影片長度 | 總幀數 | 8Hz 樣本數 |
|----------|--------|------------|
| 5 分鐘 | 9,000 | ~2,250 |
| 10 分鐘 | 18,000 | ~4,500 |
| 30 分鐘 | 54,000 | ~13,500 |
### 1.2 按需細化機制
```
Layer 1: 8Hz 基底 (所有 processor)
↓
Layer 2: 細化 (特定特徵觸發)
細化場景:
- Blink 確認: 8Hz 發現 eye openness 突降 → 回頭抓前後 ±4 幀 (30Hz)
- Lip-sync: sentence chunk 覆蓋的時間段 → 16Hz
- Mutual Gaze: 兩人 gaze 方向接近 → 前後 ±2 幀 (30Hz) 確認
```
### 1.3 樣本幀計算
```rust
// worker/processor.rs
fn compute_sample_frames(total_frames: i64, fps: f64) -> Vec<i64> {
let interval = (fps / 8.0).round() as i64;
(0..total_frames).step_by(interval.max(1) as usize).collect()
}
fn merge_refine_frames(base: &[i64], refine: &HashSet<i64>) -> Vec<i64> {
let mut combined: HashSet<i64> = base.iter().cloned().collect();
combined.extend(refine.iter().cloned());
let mut sorted: Vec<i64> = combined.into_iter().collect();
sorted.sort();
sorted
}
```
---
## 2. Trace 類型
### 重要 Trace 總覽
| # | Trace 類型 | 來源 | 用途 |
|---|-----------|------|------|
| 1 | **face_trace** | face_detections + face.json | 人臉追蹤、身份識別 |
| 2 | **appearance_trace** | appearance.json | 服裝色彩、配件、膚色 |
| 3 | **gaze_trace** | face.json (pose_angle + landmarks) | 視線方向、互相看 |
| 4 | **lip_trace** | face.json (landmarks) | 唇型、說話同步 |
| 5 | **speaker_trace** | asrx.json (speaker diarization) | 說話者識別 |
| 6 | **text_trace** | dev.chunk (sentence chunks) | 文字內容、語意 |
| 7 | **skin_tone_trace** | face.json (ROI HSV) | 膚色分類、光源記錄 |
---
### 2.1 Face Trace (已有)
```json
{
"node_type": "face_trace",
"external_id": "trace_5",
"properties": {
"frame_count": 200,
"start_frame": 150,
"end_frame": 350,
"avg_bbox": { "x": 500, "y": 300, "width": 200, "height": 250 },
"avg_yaw": -0.15,
"avg_pitch": -0.08,
"avg_roll": -0.20,
"pose_count": 180,
"embedding": [...],
"skin_tone": {
"face_h_mean": 18.5,
"fitzpatrick": "Type IV - Medium",
"confidence": 0.82,
"lighting": {
"brightness": 0.65,
"color_temp": "warm",
"direction": "front",
"uniformity": 0.92,
"source": "indoor",
"quality": "good"
},
"sample_frames": 156
}
}
}
```
### 2.2 Appearance Trace (新增)
**綁定策略**: IoU 匹配 appearance person ↔ face detection，繼承 trace_id
```json
{
"node_type": "appearance_trace",
"external_id": "trace_5",
"properties": {
"trace_id": 5,
"frame_count": 400,
"start_frame": 100,
"end_frame": 500,
"face_overlap_frames": 200,
"confidence": 0.50,
"color_features": {
"dominant_colors": [[0.1, 0.6, 0.8], ...],
"upper_body_hsv": [[...], [...], [...]],
"lower_body_hsv": [[...], [...], [...]]
},
"accessories": {
"head": {
"hat": {"detected": true, "confidence": 0.82, "first_frame": 0},
"glasses": {"detected": true, "confidence": 0.67, "first_frame": 0},
"earrings": {"detected": false},
"mask": {"detected": false},
"hairstyle": {"type": "long", "confidence": 0.75},
"hair_accessory": {"detected": false},
"nose_ring": {"detected": false},
"lip_ring": {"detected": false},
"face_tattoo": {"detected": false},
"eyebrow_tattoo": {"detected": false},
"beard": {"detected": true, "confidence": 0.88},
"headscarf": {"detected": false}
},
"neck": {
"tie": {"detected": true, "confidence": 0.92, "first_frame": 0, "source": "hsv_color_block"},
"scarf": {"detected": false},
"shawl": {"detected": false},
"necklace": {"detected": true, "confidence": 0.71, "first_frame": 12, "source": "clip"},
"neck_tattoo": {"detected": false}
},
"hand": {
"ring": {"detected": false},
"bracelet": {"detected": false},
"watch": {"detected": true, "confidence": 0.63, "first_frame": 24},
"gloves": {"detected": false}
},
"hand_held": {
"phone": {"detected": true, "confidence": 0.88, "source": "hsv_color_block"},
"pen": {"detected": false},
"cup": {"detected": false},
"knife": {"detected": false},
"gun": {"detected": false}
},
"foot": {
"shoes": {"type": "sneaker", "confidence": 0.78, "source": "hsv_color_block"},
"socks": {"detected": false},
"barefoot": {"detected": false}
},
"vehicle": {
"bicycle": {"detected": false, "source": "hsv_color_block"},
"skateboard": {"detected": false},
"scooter": {"detected": false}
},
"carried": {
"backpack": {"detected": false},
"handbag": {"detected": true, "confidence": 0.85, "source": "hsv_color_block"},
"luggage": {"detected": false}
}
}
}
}
```
### 2.3 Speaker Trace (重要)
**來源**: ASRX speaker diarization + face trace 綁定
```json
{
"node_type": "speaker_trace",
"external_id": "SPEAKER_0",
"properties": {
"speaker_id": "SPEAKER_0",
"segment_count": 45,
"total_duration": 120.5,
"first_appearance": {"frame": 100, "time": 3.3},
"last_appearance": {"frame": 3600, "time": 120.0},
"full_text": "大家好 今天我們來討論... (完整語音轉文字)",
"segments": [
{"start_time": 0.1, "end_time": 2.0, "text": "大家好", "start_frame": 3, "end_frame": 60},
{"start_time": 5.2, "end_time": 8.5, "text": "今天我們來討論", "start_frame": 156, "end_frame": 255},
...
],
"face_trace_ids": [5, 12, 23],
"appearance_trace_ids": [5, 12],
"gaze_context": {
"looking_at_person": true,
"mutual_gaze_with": [12]
},
"lip_sync_quality": 0.85
}
}
```
**來源資料**:
```
ASRX → asrx.json (segments with speaker_id)
Face → face_detections (trace_id)
綁定 → SPEAKS_AS edge (speaker ↔ face_trace)
```
### 2.4 Text Trace (重要)
**來源**: dev.chunk (chunk_type='sentence') + ASRX text
```json
{
"node_type": "text_trace",
"external_id": "chunk_1",
"properties": {
"chunk_id": "chunk_1",
"text": "大家好，今天我們來討論這個話題",
"text_normalized": "大家好，今天我們來討論這個話題",
"start_time": 0.1,
"end_time": 5.2,
"start_frame": 3,
"end_frame": 156,
"speaker_id": "SPEAKER_0",
"language": "zh",
"confidence": 0.95,
"yolo_objects": ["person", "chair"],
"face_ids": ["face_100"],
"speaker_trace_id": "SPEAKER_0",
"face_trace_id": 5,
"lip_sync": {
"matched_frames": 120,
"total_frames": 153,
"quality": 0.85
},
"semantic_embedding": [0.12, -0.34, ...],
"sentiment": "neutral"
}
}
```
**來源資料**:
```
Rule 1 → dev.chunk (sentence chunks)
ASRX → asrx.json (speaker_id binding)
Face → face_detections (face_ids in chunk metadata)
YOLO → yolo.json (co-occurring objects)
```
**Edge 連接**:
- `SPEAKS_BY`: text_trace → speaker_trace
- `SPOKEN_WHILE`: text_trace → face_trace
- `LIP_SYNC`: text_trace → lip_trace
- `CONTAINS_OBJECT`: text_trace → object
### 2.5 Skin Tone Trace (重要)
**來源**: face.json ROI HSV + 光源分析
```json
{
"node_type": "skin_tone_trace",
"external_id": "trace_5",
"properties": {
"trace_id": 5,
"frame_count": 200,
"start_frame": 150,
"end_frame": 350,
"face_h_mean": 18.5,
"fitzpatrick": "Type IV - Medium",
"confidence": 0.82,
"lighting": {
"brightness": 0.65,
"color_temp": "warm",
"direction": "front",
"uniformity": 0.92,
"source": "indoor",
"quality": "good"
},
"sample_frames": 156,
"hand_h_mean": 17.8,
"arm_h_mean": 18.2
}
}
```
**Fitzpatrick 分類**:
| Type | 描述 | H 值 (HSV) |
|------|------|------------|
| I | 非常淺 | 0–5 |
| II | 淺 | 5–12 |
| III | 中等偏淺 | 12–18 |
| IV | 中等 | 18–25 |
| V | 深 | 25–35 |
| VI | 很深 | 35+ |
**光源品質**:
| Quality | 條件 | 膚色可信度 |
|---------|------|------------|
| good | brightness > 0.4, uniformity > 0.8, front light | 高 (×1.0) |
| fair | brightness > 0.3, uniformity > 0.6 | 中 (×0.7) |
| poor | brightness < 0.3 或 backlight | 低 (×0.5) |
### 2.6 Gaze Trace (新增)
```json
{
"node_type": "gaze_trace",
"external_id": "trace_5",
"properties": {
"trace_id": 5,
"frame_count": 200,
"start_frame": 150,
"end_frame": 350,
"avg_yaw": -0.15,
"avg_pitch": -0.08,
"avg_roll": -0.20,
"head_direction": "frontal",
"gaze_direction": "center-left",
"eye_openness": 0.85,
"blink_count": 12,
"blink_rate": 0.06,
"looking_at_person": true,
"looking_at_object": ["chair"],
"refined_ranges": [
{"start_frame": 200, "end_frame": 220, "hz": 30, "reason": "mutual_gaze"}
]
}
}
```
### 2.7 Lip Trace (重要)
**來源**: face.json → faces[].lips (inner_lips 6pts + outer_lips 14pts)
```json
{
"node_type": "lip_trace",
"external_id": "trace_5",
"properties": {
"trace_id": 5,
"frame_count": 180,
"start_frame": 160,
"end_frame": 340,
"avg_openness": 0.3,
"avg_width": 45.2,
"avg_height": 12.8,
"movement_variance": 0.15,
"speaking_frames": 95,
"silent_frames": 85,
"lip_landmark_samples": {
"inner_lips": [[x,y,z], ...],
"outer_lips": [[x,y,z], ...]
},
"speech_correlation": {
"text_trace_ids": ["chunk_1", "chunk_2", "chunk_3"],
"sync_quality": 0.85,
"matched_segments": [
{"start_frame": 160, "end_frame": 200, "text": "大家好"},
{"start_frame": 210, "end_frame": 250, "text": "今天我們來討論"}
]
},
"refined_ranges": [
{"start_frame": 160, "end_frame": 340, "hz": 30, "reason": "lip_sync"}
]
}
}
```
**Lip-sync 計算**:
```
Lip openness = inner_lips_area / outer_lips_area
Speaking detection:
- openness > threshold (動態調整)
- movement_variance > threshold (唇型變化)
- 持續 N 幀以上 (避免雜訊)
Sync with text:
- 比對 text_trace 的 start/end_time
- 計算 lip movement 與文字時間段的重疊率
- quality = matched_frames / total_text_frames
```
**Edge 連接**:
- `HAS_LIP`: face_trace → lip_trace
- `LIP_SYNC`: lip_trace → text_trace
- `GAZE_SYNC_SPEECH`: gaze_trace + lip_trace (說話時注視方向)
---
## 3. 配件偵測
### 3.1 偵測方式分工
| 方式 | 適用配件 | 速度 | 說明 |
|------|----------|------|------|
| **HSV 色塊** | tie, phone, watch, ring, bracelet, glasses, mask, hat, shoes, backpack, handbag, umbrella, pen, knife, cup, book, laptop, remote, baseball_bat | 快 | **主要方式** — 從 person crop 分析異色區塊 |
| **CLIP** | hairstyle, beard, face_tattoo, eyebrow_tattoo, earrings, nose_ring, lip_ring, neck_tattoo, headscarf, scarf, shawl, necklace, gloves, tool, gun, skateboard, scooter, roller_skates, socks, barefoot | 中 | zero-shot (YOLO 不可靠，色塊也不易區分時) |
| **MediaPipe** | gesture, arm_pose | 快 | 21 hand pts + 33 pose pts |
| **HSV** | upper_body_color, lower_body_color, skin_tone | 快 | 色彩特徵提取 |
### 3.2 Appearance 與 Landmark/Pose 緊密貼合
**核心原則**: Appearance 不獨立偵測 bbox，而是直接用 face/pose/mediapipe 的幾何結果裁切 ROI。
```
Face Landmarks (20pts) ──► 臉部 ROI ──► hat, glasses, mask, beard, earrings
Pose 33 Keypoints ───────► 身體 ROI ──► tie, necklace, upper/lower body HSV
MediaPipe Hands (21×2) ──► 手腕 ROI ──► watch, bracelet, ring, phone, glove
MediaPipe Pose Feet ─────► 腳部 ROI ──► shoes, socks, barefoot
```
**ROI 定位方式**:
```python
def get_accessory_rois(frame, face_data, pose_data, hand_data):
rois = {}
# 臉部區域 — 用 face bbox + landmarks
face_bbox = face_data['bbox']
landmarks = face_data['landmarks'] # nose, left_eye, right_eye
# 帽子 ROI: 臉部 bbox 上方延伸
rois['hat'] = expand_region(face_bbox, direction='up', factor=0.5)
# 眼鏡 ROI: 眼部 landmarks 水平帶
left_eye = landmarks['left_eye']
right_eye = landmarks['right_eye']
rois['glasses'] = bbox_around_points(left_eye, right_eye, padding=10)
# 口罩 ROI: 鼻子下方到下顎
nose = landmarks['nose']
rois['mask'] = region_below_point(nose, face_bbox.bottom)
# 脖子 ROI — 用 pose neck keypoints
if pose_data:
neck = pose_data['keypoints']['neck']
nose = pose_data['keypoints']['nose']
rois['neck'] = region_between(nose, neck, width=80)
# 手腕 ROI — 用 MediaPipe hand landmarks
if hand_data:
for side in ['left', 'right']:
wrist = hand_data[side]['wrist']
rois[f'{side}_wrist'] = circle_around(wrist, radius=30)
# 腳部 ROI — 用 pose ankle/toe keypoints
if pose_data:
for side in ['left', 'right']:
ankle = pose_data['keypoints'][f'{side}_ankle']
toe = pose_data['keypoints'][f'{side}_toe']
rois[f'{side}_foot'] = bbox_around_points(ankle, toe, padding=20)
return rois
```
### 3.3 HSV 色塊偵測流程
```python
def detect_accessories_tightly_coupled(frame, face_data, pose_data, hand_data):
# 1. 用 landmark/pose 精準定位各 ROI
rois = get_accessory_rois(frame, face_data, pose_data, hand_data)
results = {}
for roi_name, roi_bbox in rois.items():
roi_hsv = crop_and_convert(frame, roi_bbox, 'HSV')
# 2. 在精準 ROI 內找異色區塊
diff_mask = compute_color_diff(roi_hsv, main_colors, threshold=30)
blobs = find_connected_components(diff_mask)
for blob in blobs:
accessory = classify_accessory_by_position(blob, roi_name)
if accessory:
results[accessory] = {
"detected": True,
"confidence": blob.confidence,
"source": "hsv_color_block",
"roi": roi_name,
"first_frame": current_frame
}
# 3. 色塊不易判斷的項目 → CLIP
clip_only_items = ['hairstyle', 'beard', 'earrings', 'nose_ring', ...]
for item in clip_only_items:
confidence = clip_score(crop_person(frame, face_data['bbox']), CLIP_PROMPTS[item])
if confidence > 0.5:
results[item] = {"detected": True, "confidence": confidence, "source": "clip"}
return results
```
### 3.4 依賴關係
```
Face Detection ──► face_detections (trace_id, bbox, embedding)
│
▼
Face Landmarks ────► 臉部 ROI (hat, glasses, mask, beard)
│
▼
Pose 33pts ────────► 身體 ROI (neck, wrist, foot) ──► Appearance HSV
│
▼
MediaPipe Hands ───► 手腕 ROI (watch, bracelet, ring, phone)
│
▼
TKG appearance_trace
```
### 3.5 CLIP 提示詞 (僅用於色塊不易區分的配件)
```python
CLIP_PROMPTS = {
# 頭部 — 色塊不易判斷的項目
"hairstyle_short": "a person with short hair",
"hairstyle_long": "a person with long hair",
"hairstyle_braid": "a person with braided hair",
"hairstyle_bun": "a person with hair in a bun",
"face_tattoo": "a person with a visible face tattoo or face paint",
"eyebrow_tattoo": "a person with tattooed or styled eyebrows",
"beard": "a person with a beard or mustache",
# 耳朵/鼻子/嘴唇穿刺
"earrings": "a person wearing earrings",
"nose_ring": "a person wearing a nose ring or nose piercing",
"lip_ring": "a person wearing a lip ring or lip piercing",
# 脖子 — 項鍊等細小物件
"necklace": "a person wearing a necklace",
"neck_tattoo": "a person with a visible neck tattoo",
# 手部細小物件
"gloves": "a person wearing gloves",
"tool": "a person holding a tool like a wrench or screwdriver",
"gun": "a person holding a gun",
# 足部
"socks": "a person wearing visible socks",
"barefoot": "a barefoot person",
"roller_skates": "a person wearing roller skates",
}
```
---
## 4. 膚色 + 光源
### 4.1 Fitzpatrick 分類
| Type | 描述 | H 值 (HSV) |
|------|------|------------|
| I | 非常淺 | 0–5 |
| II | 淺 | 5–12 |
| III | 中等偏淺 | 12–18 |
| IV | 中等 | 18–25 |
| V | 深 | 25–35 |
| VI | 很深 | 35+ |
### 4.2 光源參數
| 參數 | 計算方式 | 範圍 |
|------|----------|------|
| brightness | V channel 平均 | 0.0–1.0 |
| color_temp | 白平衡估算 | warm/neutral/cool |
| direction | 陰影梯度 + yaw/pitch | front/side/back/top |
| uniformity | 臉部各區域 V 值標準差 | 0.0–1.0 |
| source | 亮度 + 色溫綜合判斷 | indoor/outdoor/flash |
### 4.3 光源品質
| Quality | 條件 | 膚色可信度 |
|---------|------|------------|
| good | brightness > 0.4, uniformity > 0.8, front light | 高 (×1.0) |
| fair | brightness > 0.3, uniformity > 0.6 | 中 (×0.7) |
| poor | brightness < 0.3 或 backlight | 低 (×0.5) |
---
## 5. TKG Node 類型
| node_type | external_id | 來源 | 重要性 | 屬性 |
|-----------|-------------|------|--------|------|
| `face_trace` | `trace_N` | face_detections | ★★★★ | frame_count, bbox, pose, embedding, skin_tone |
| `appearance_trace` | `trace_N` | appearance.json | ★★★★ | trace_id, color_features, accessories, confidence |
| `gaze_trace` | `trace_N` | face.json (pose_angle) | ★★★ | trace_id, gaze_direction, blink_count, looking_at |
| `lip_trace` | `trace_N` | face.json (lips) | ★★★★ | trace_id, avg_openness, speaking_frames, speech_correlation |
| `speaker_trace` | `SPEAKER_N` | asrx.json | ★★★★ | speaker_id, segments, face_trace_ids, full_text |
| `text_trace` | `chunk_N` | dev.chunk | ★★★★ | text, speaker_id, time_range, yolo_objects, lip_sync |
| `skin_tone_trace` | `trace_N` | face.json (ROI HSV) | ★★★ | trace_id, fitzpatrick, lighting, confidence |
| `object` | `class_name` | yolo.json | ★★ | total_detections, frames |
| `accessory` | `hat`, `glasses`, ... | appearance.json | ★★ | category, trace_ids, first/last_seen |
---
## 6. TKG Edge 類型
| Edge Type | Source → Target | 屬性 | 說明 |
|-----------|----------------|------|------|
| `SPEAKS_AS` | speaker_trace → face_trace | confidence, overlap_frames | 說話者綁定人臉 |
| `SPEAKS_BY` | text_trace → speaker_trace | — | 文字由誰說的 |
| `SPOKEN_WHILE` | text_trace → face_trace | frame_overlap | 說話時的人臉 |
| `HAS_APPEARANCE` | face_trace → appearance_trace | confidence, overlap_frames | 外觀特徵 |
| `HAS_GAZE` | face_trace → gaze_trace | overlap_frames | 視線方向 |
| `HAS_LIP` | face_trace → lip_trace | overlap_frames | 唇型資料 |
| `HAS_SKIN_TONE` | face_trace → skin_tone_trace | confidence, lighting_match | 膚色記錄 |
| `LIP_SYNC` | lip_trace → text_trace | time_alignment, openness_match | 唇語同步 |
| `WEARS` | appearance_trace → accessory | confidence, first_frame | 配件 |
| `LOOKING_AT` | gaze_trace → object | direction_match, distance | 注視物件 |
| `LOOKING_AT_PERSON` | gaze_trace → face_trace | direction_match | 注視他人 |
| `MUTUAL_GAZE` | face_trace ↔ face_trace | first_frame, last_frame, duration_frames, confidence | 互相看 |
| `CO_OCCURS_WITH` | object ↔ object | frame_count | 物件共現 |
| `SAME_SKIN_TONE` | face_trace ↔ face_trace | h_diff, lighting_match, confidence | 膚色相近 |
| `HOLDS` | appearance_trace → object | 手機等手持物品 |
---
## 7. Mutual Gaze 分析
### 7.1 計算邏輯
```
對每幀:
對每對 (person_A, person_B):
1. 計算 A 的 gaze vector (從 yaw/pitch/roll)
2. 計算 B 的 bbox center 在 A 座標系中的位置
3. 判斷 B 是否在 A 的 gaze cone 內 (threshold: ~15°)
4. 反向檢查 B → A
5. 雙向命中 → mutual_gaze
```
### 7.2 持續性確認
```
mutual_gaze 需要持續 N 幀以上才算有意義:
- 基底: 8Hz, 持續 ≥ 3 幀 (~0.375s) → 建立 edge
- 細化: 發現 candidate 後，回頭用 30Hz 確認
- confidence = 連續幀數 / 總可能幀數
```
### 7.3 Edge 屬性
```json
{
"edge_type": "MUTUAL_GAZE",
"source": "trace_5",
"target": "trace_12",
"properties": {
"first_frame": 150,
"last_frame": 280,
"duration_frames": 130,
"duration_seconds": 4.3,
"confidence": 0.85,
"context": "during_conversation"
}
}
```
---
## 8. 實作計畫
### Phase 0: 8Hz 採樣框架 (~100 行)
| 檔案 | 修改 |
|------|------|
| `worker/processor.rs` | 計算 8Hz sample frames + refine 框架 |
| `scripts/face_processor.py` | 接受 `--frames` 參數 |
| `scripts/appearance_processor.py` | bbox 來源改 yolo，接受 `--frames` |
| `scripts/mediapipe_holistic_processor.py` | 接受 `--frames` |
### Phase 1: Gaze + Mutual Gaze (~250 行)
| 模組 | 行數 |
|------|------|
| Gaze trace nodes | 150 |
| Mutual Gaze edges | 100 |
### Phase 2: Lip + Sentence + Speaker (~260 行)
| 模組 | 行數 |
|------|------|
| Lip trace nodes | 120 |
| Sentence nodes | 80 |
| Speaker 強化 | 60 |
### Phase 3: Appearance + Accessories (~280 行)
| 模組 | 行數 |
|------|------|
| Appearance traces (HSV + trace_id 綁定) | 120 |
| Accessories (CLIP detection) | 80 |
| Skin tone + lighting | 80 |
### Phase 4: TKG 整合 (~110 行)
| 模組 | 行數 |
|------|------|
| `build_tkg()` 統一呼叫 | 40 |
| Edge builders 更新 | 70 |
### 總計: ~1,000 行
---
## 9. 依賴關係圖
```
YOLO (全域) ──────────────────────────────────────────┐
│ │
▼ │
Face (8Hz) ──► trace_id ──┬──► Appearance (IoU 綁定) │
│ │ ├──► HSV 色彩 │
│ │ ├──► Accessories (CLIP) │
│ │ └──► Skin tone + light │
│ │ │
│ ├──► Gaze ──► Mutual Gaze ────┤
│ │ ──► Looking at YOLO │
│ │ │
│ └──► Lip ──► LIP_SYNC ◄──────┤
│ │
ASRX ──► Speaker ──► SPEAKS_AS ──► face_trace │
│ │ │
└──► Text (Rule 1) ────┴──► SPEAKS_BY │
├──► SPOKEN_WHILE │
└──► LIP_SYNC ────────────┘
所有 trace ──────────────────────────► TKG
```
---
## Appendix A: 配件完整清單 (49 種)
| 部位 | 配件 | 偵測方式 |
|------|------|----------|
| 頭部 (12) | hat, hairstyle, hair_accessory, earrings, nose_ring, lip_ring, face_tattoo, eyebrow_tattoo, glasses, mask, beard, headscarf | HSV 色塊 + CLIP |
| 脖子 (5) | tie, scarf, shawl, necklace, neck_tattoo | HSV 色塊 + CLIP |
| 手部/手臂 (16) | ring, bracelet, watch, gloves, phone, pen, laptop, book, cup, remote, tool, knife, gun, baseball_bat, gesture, arm_pose | HSV 色塊 + CLIP + MP |
| 足部/載具 (8) | shoes, socks, barefoot, skateboard, scooter, bicycle, motorbike, roller_skates | HSV 色塊 + CLIP |
| 攜帶/環境 (5) | backpack, handbag, luggage, chair, diningtable | HSV 色塊 + CLIP |
| 色彩 (3) | upper_body_hsv, lower_body_hsv, skin_tone | HSV |
> **註**: YOLO 不可靠，不再作為主要偵測方式。大部分配件改用 HSV 色塊分析，CLIP 僅用於色塊不易區分的項目 (如穿刺、紋身、髮型等)。
## Appendix B: DB Schema 變更
```sql
-- appearance_detections (新增)
CREATE TABLE appearance_detections (
id BIGSERIAL PRIMARY KEY,
file_uuid VARCHAR NOT NULL,
frame_number BIGINT NOT NULL,
person_id INTEGER NOT NULL,
x INTEGER, y INTEGER, width INTEGER, height INTEGER,
trace_id INTEGER,
confidence REAL,
hsv_histogram JSONB,
dominant_colors JSONB,
upper_body_hsv JSONB,
lower_body_hsv JSONB,
accessories JSONB,
skin_tone JSONB,
lighting JSONB,
created_at TIMESTAMPTZ DEFAULT NOW()
);
-- tkg_nodes (擴充 node_type)
-- 新增: appearance_trace, gaze_trace, lip_trace, sentence, accessory
-- tkg_edges (擴充 edge_type)
-- 新增: HAS_APPEARANCE, HAS_GAZE, HAS_LIP, WEARS, LOOKING_AT,
-- LOOKING_AT_PERSON, MUTUAL_GAZE, LIP_SYNC, SPEAKS_BY,
-- SAME_SKIN_TONE, HAS_NECK_ACCESSORY, HAS_HEAD_ACCESSORY, HOLDS
```
---
## Version History
| Version | Date | Author | Description |
|---------|------|--------|-------------|
| 1.0.0 | 2026-06-19 | OpenCode | Initial design: 8Hz sampling, 7 traces (face/appearance/gaze/lip/speaker/text/skin_tone), 49 accessories, skin tone + lighting, mutual gaze, lip-sync |
| 1.1.0 | 2026-06-19 | OpenCode | Added speaker_trace, text_trace, skin_tone_trace as important traces; enhanced lip_trace with speech_correlation; updated node/edge tables |
| **1.2.0** | **2026-06-19** | **OpenCode** | **Implementation complete: build_tkg() integrates all node/edge builders. 9 node types, 14 edge types. ~1500 lines added to tkg.rs** |
docs_v1.0/DESIGN/TKG_PHASE2_6_EDGES_MIGRATION.md
+257
View File
@@ -0,0 +1,257 @@
---
title: TKG Phase 2.6 Edges Migration Plan
version: 1.0
date: 2026-06-21
author: OpenCode
status: Draft
---
## Phase 2.6 Overview
迁移 TKG edges 从 PostgreSQL face_detections 到 Qdrant payload。
## Current Implementation Analysis
### 2.6.1: co_occurrence_edges (CO_OCCURS_WITH)
**Current Code** (`tkg.rs:932-1039`):
```rust
let face_rows = sqlx::query_as::<_, FaceDetectionRow>(&format!(
"SELECT trace_id::bigint, frame_number::bigint, x::float8, y::float8, width::float8, height::float8
FROM {} WHERE file_uuid = $1 AND trace_id IS NOT NULL
ORDER BY frame_number",
face_table
))
.bind(file_uuid)
.fetch_all(pool)
.await?;
```
**Dependencies**:
- `face_detections.trace_id`
- `face_detections.frame_number`
- `face_detections.x, y, width, height`
**Migration Strategy**:
```rust
// 从 Qdrant payload 获取
let embeddings = face_db.get_all_embeddings_for_file(file_uuid).await?;
// 按 frame 分组
let mut frame_map: HashMap<i64, Vec<(i64, f64, f64, f64, f64)>> = HashMap::new();
for emb in embeddings {
let frame = emb.payload.frame_number;
let trace_id = emb.payload.trace_id;
frame_map.entry(frame).or_default().push((
trace_id,
emb.payload.bbox_x,
emb.payload.bbox_y,
emb.payload.bbox_width,
emb.payload.bbox_height,
));
}
```
### 2.6.2: face_face_edges (MUTUAL_GAZE)
**Current Code** (`tkg.rs:1171-1320`):
```rust
let rows: Vec<(i64, i64, i64)> = sqlx::query_as(&format!(
"SELECT a.trace_id::bigint AS tid_a, b.trace_id::bigint AS tid_b, a.frame_number::bigint
FROM {} a
JOIN {} b ON a.file_uuid = b.file_uuid AND a.frame_number = b.frame_number AND a.trace_id < b.trace_id
WHERE a.file_uuid = $1 AND a.trace_id IS NOT NULL AND b.trace_id IS NOT NULL",
face_table, face_table
))
.bind(file_uuid)
.fetch_all(pool)
.await?;
```
**Dependencies**:
- `face_detections` self-join for co-occurrence
- `face_detections.trace_id`
- `face_detections.frame_number`
**Migration Strategy**:
```rust
// 从 Qdrant 获取所有 embeddings
let embeddings = face_db.get_all_embeddings_for_file(file_uuid).await?;
// 按 frame 分组
let mut frame_faces: HashMap<i64, Vec<FaceEmbeddingPayload>> = HashMap::new();
for emb in embeddings {
frame_faces.entry(emb.payload.frame_number).or_default().push(emb.payload);
}
// 找同 frame 的 face pairs
let mut pairs: Vec<(i64, i64, i64)> = Vec::new();
for (frame, faces) in frame_faces.iter() {
for i in 0..faces.len() {
for j in (i+1)..faces.len() {
let tid_a = faces[i].trace_id.min(faces[j].trace_id);
let tid_b = faces[i].trace_id.max(faces[j].trace_id);
pairs.push((tid_a, tid_b, *frame));
}
}
}
```
### 2.6.3: speaker_face_edges (SPEAKS_AS)
**Current Code** (`tkg.rs:1045-1169`):
```rust
let traces = sqlx::query_as::<_, (i64, i64, i64)>(&format!(
"SELECT trace_id::bigint, MIN(frame_number)::bigint as start_f, MAX(frame_number)::bigint as end_f
FROM {} WHERE file_uuid = $1 AND trace_id IS NOT NULL
GROUP BY trace_id",
face_table
))
.bind(file_uuid)
.fetch_all(pool)
.await?;
```
**Dependencies**:
- `face_detections.trace_id`
- `face_detections.frame_number` (MIN/MAX)
**Migration Strategy**:
```rust
// 从 Qdrant 获取所有 embeddings
let embeddings = face_db.get_all_embeddings_for_file(file_uuid).await?;
// 计算每个 trace_id 的 frame range
let mut trace_ranges: HashMap<i64, (i64, i64)> = HashMap::new();
for emb in embeddings {
let trace_id = emb.payload.trace_id;
let frame = emb.payload.frame_number;
let entry = trace_ranges.entry(trace_id).or_insert((frame, frame));
entry.0 = entry.0.min(frame);
entry.1 = entry.1.max(frame);
}
```
### 2.6.4: mutual_gaze_edges (MUTUAL_GAZE)
**Already in face_face_edges**:
- face_face_edges 包含 mutual_gaze 检测逻辑
- 不需要单独迁移
### 2.6.5: lip_sync_edges (LIP_SYNC)
**Already migrated in Phase 2.5.2**:
- `build_lip_trace_nodes_from_qdrant()` 已完成
- lip_sync_edges 已使用 Qdrant payload
## Migration Priority
| Priority | Edge Type | Complexity | Impact |
|----------|-----------|-------------|--------|
| P1 | co_occurrence_edges | Low | High (关系图) |
| P1 | face_face_edges | Medium | High (face 关系) |
| P2 | speaker_face_edges | Low | Medium (speaker 关系) |
| N/A | mutual_gaze_edges | - | 已包含在 face_face_edges |
| N/A | lip_sync_edges | - | 已迁移 Phase 2.5.2 |
## Performance Estimate
| Edge Type | Current (PG) | After Migration | Speedup |
|-----------|--------------|-----------------|---------|
| co_occurrence_edges | ~120ms | ~30ms | 4x |
| face_face_edges | ~90ms | ~25ms | 3.6x |
| speaker_face_edges | ~60ms | ~20ms | 3x |
| **Total** | **~270ms** | **~75ms** | **3.6x** |
## Implementation Steps
### Step 1: Add helper functions in `face_embedding_db.rs`
```rust
// Get all embeddings grouped by frame
pub async fn get_embeddings_by_frame(&self, file_uuid: &str) -> Result<HashMap<i64, Vec<FaceEmbeddingPayload>>>;
// Get trace_id frame ranges
pub async fn get_trace_frame_ranges(&self, file_uuid: &str) -> Result<HashMap<i64, (i64, i64)>>;
```
### Step 2: Create migration functions in `tkg.rs`
```rust
// Phase 2.6.1
async fn build_co_occurrence_edges_from_qdrant(
pool: &PgPool,
file_uuid: &str,
output_dir: &str,
face_db: &FaceEmbeddingDb,
) -> Result<usize>;
// Phase 2.6.2
async fn build_face_face_edges_from_qdrant(
pool: &PgPool,
file_uuid: &str,
pose_data: &[FacePose],
face_db: &FaceEmbeddingDb,
) -> Result<usize>;
// Phase 2.6.3
async fn build_speaker_face_edges_from_qdrant(
pool: &PgPool,
file_uuid: &str,
output_dir: &str,
face_db: &FaceEmbeddingDb,
) -> Result<usize>;
```
### Step 3: Replace in `build_tkg.rs`
```rust
// Old
let e_co = build_co_occurrence_edges(pool, file_uuid, output_dir).await?;
// New
let e_co = build_co_occurrence_edges_from_qdrant(pool, file_uuid, output_dir, face_db).await?;
```
### Step 4: Add feature flag (optional)
```rust
#[cfg(feature = "qdrant-edges")]
let e_co = build_co_occurrence_edges_from_qdrant(...).await?;
#[cfg(not(feature = "qdrant-edges"))]
let e_co = build_co_occurrence_edges(...).await?;
```
## Verification Plan
1. Run TKG rebuild on test file
2. Compare edge counts (PG vs Qdrant)
3. Verify edge properties match
4. Performance benchmark
5. Integration test with Rule2
## Risks & Mitigations
| Risk | Mitigation |
|------|------------|
| Qdrant collection empty | Fallback to PostgreSQL |
| Performance regression | Benchmark before merge |
| Edge count mismatch | Validate with test suite |
| Data inconsistency | Add reconciliation job |
## Success Criteria
- [ ] All edges use Qdrant payload (no face_detections queries)
- [ ] Edge counts match PostgreSQL version
- [ ] Performance improvement >= 2x
- [ ] Rule2/Rule3 work correctly
- [ ] No regressions in existing tests
## Timeline
- Phase 2.6.1 (co_occurrence): 1 day
- Phase 2.6.2 (face_face): 1 day
- Phase 2.6.3 (speaker_face): 0.5 day
- Testing & verification: 0.5 day
- **Total: 3 days**
docs_v1.0/DESIGN/TKG_PHASE2_7_IDENTITY_RESOLUTION.md
+165
View File
@@ -0,0 +1,165 @@
---
title: TKG Phase 2.7 Identity Resolution for Edges
version: 1.0
date: 2026-06-21
author: OpenCode
status: Draft
---
## Phase 2.7 Overview
为 gaze_trace 和 lip_trace nodes 添加 identity_id 属性，实现完整的 edge identity resolution。
## Current Implementation Analysis
### Rule2 Identity Resolution
**Location**: `src/core/chunk/rule2_ingest.rs`
**Current Logic** (lines 102-131):
```rust
// Only resolves face_trace nodes
let src_identity: Option<String> = if src_type == "face_trace" {
sqlx::query_scalar("SELECT i.name FROM tkg_nodes n
JOIN identities i ON i.id = (n.properties->>'identity_id')::bigint
WHERE n.node_type = 'face_trace' AND n.properties->>'identity_id' IS NOT NULL")
}
```
**Problem**:
- Only handles `face_trace` node type
- `gaze_trace` and `lip_trace` nodes lack identity_id
### Node Type Properties
| Node Type | external_id | identity_id | 状态 |
|-----------|-------------|-------------|------|
| **face_trace** | trace_{id} | ✓ 有 | ✅ Phase 2.3 |
| **gaze_trace** | gaze_{id} | ❌ 无 | 需要添加 |
| **lip_trace** | lip_{id} | ❌ 无 | 需要添加 |
## Solution Design
### Approach 1: Extend Rule2 Logic (Complex)
修改 Rule2 支持 gaze_trace/lip_trace node types：
```rust
let src_identity: Option<String> = if src_type == "face_trace" || src_type == "gaze_trace" || src_type == "lip_trace" {
// Parse trace_id from external_id
let trace_id = src_ext_id.split('_').last()?;
// Query face_trace node
sqlx::query_scalar("SELECT i.name FROM tkg_nodes n
JOIN identities i ON i.id = (n.properties->>'identity_id')::bigint
WHERE n.node_type = 'face_trace' AND n.external_id = 'trace_' || $1")
.bind(trace_id)
}
```
**优点**: 不需要修改 TKG builders
**缺点**: Rule2 逻辑复杂，查询效率低
### Approach 2: Add identity_id in TKG Builders (Recommended)
在创建 gaze_trace/lip_trace nodes 时直接设置 identity_id：
```rust
// Step 1: Query face_trace node's identity_id
let face_identity_id: Option<i64> = sqlx::query_scalar(
"SELECT (properties->>'identity_id')::bigint FROM tkg_nodes
WHERE file_uuid=$1 AND node_type='face_trace' AND external_id=$2"
)
.bind(file_uuid)
.bind(&format!("trace_{}", trace_id))
.fetch_optional(pool)
.await?;
// Step 2: Add to gaze/lip node properties
let props = serde_json::json!({
"trace_id": tid,
"identity_id": face_identity_id, // <-- NEW
...
});
```
**优点**:
- 性能最优（一次查询）
- Rule2 无需修改
- 逻辑清晰
**缺点**: 需要修改 TKG builders
### Recommended: Approach 2
## Implementation Plan
### Step 1: Modify build_gaze_trace_nodes_from_qdrant()
**Location**: `src/core/processor/tkg.rs:1859-1975`
**Add**:
```rust
// Query face_trace identity_id
let face_ext_id = format!("trace_{}", tid);
let face_identity_id: Option<i64> = sqlx::query_scalar(&format!(
"SELECT (properties->>'identity_id')::bigint FROM {}
WHERE file_uuid=$1 AND node_type='face_trace' AND external_id=$2",
nodes_table
))
.bind(file_uuid)
.bind(&face_ext_id)
.fetch_optional(pool)
.await?;
// Add to properties
let props = serde_json::json!({
"trace_id": tid,
"identity_id": face_identity_id, // <-- NEW
"frame_count": frame_count,
...
});
```
### Step 2: Modify build_lip_trace_nodes_from_qdrant()
**Location**: `src/core/processor/tkg.rs` (lip_trace builder)
**Add**: Same logic as gaze_trace
### Step 3: Update PostgreSQL fallback versions
Also update:
- `build_gaze_trace_nodes_from_pg()`
- `build_lip_trace_nodes_from_pg()`
### Step 4: Update Rule2 (Optional)
If desired, extend Rule2 to support gaze_trace/lip_trace:
```rust
let src_identity: Option<String> = if src_type == "face_trace" || src_type == "gaze_trace" || src_type == "lip_trace" {
// Query identity from node properties
...
}
```
**Note**: With Approach 2, Rule2 already works correctly!
## Verification Plan
1. TKG rebuild → check gaze/lip nodes have identity_id
2. Rule2 test → verify identity resolution works
3. Edge count comparison → ensure no regression
4. Performance benchmark → measure impact
## Success Criteria
- [ ] gaze_trace nodes have identity_id in properties
- [ ] lip_trace nodes have identity_id in properties
- [ ] Rule2 identity resolution works for all node types
- [ ] No regressions in edge counts
- [ ] Performance acceptable (<10ms added)
## Timeline
- Implementation: 1 day
- Testing: 0.5 day
- **Total: 1.5 days**
docs_v1.0/DESIGN/TKG_PHASE2_NONFACE_MIGRATION_V1.0.md
+186
View File
@@ -0,0 +1,186 @@
---
title: TKG Phase 2-4 Migration Plan (Non-Face Nodes)
version: 1.0
date: 2026-06-21
author: OpenCode
status: Draft
---
## 概览
Phase 2-3 已完成 face_trace_nodes 的 Qdrant 迁移。其他 node types 需要类似迁移。
## 当前状态
| Node Type | 数据源 | PostgreSQL 依赖 | 迁移状态 |
|-----------|--------|-----------------|----------|
| **face_trace_nodes** | Qdrant embeddings | ❌ 无 | ✅ Phase 2.1 完成 |
| **gaze_trace_nodes** | face.json | ✅ face_detections.trace_id | 🔄 待迁移 |
| **lip_trace_nodes** | face.json + lip.json | ✅ face_detections.trace_id | 🔄 待迁移 |
| **text_trace_nodes** | chunk table | ✅ chunk.sentence | ⏸️ 保持现状 |
| **yolo_object_nodes** | .yolo.json | ❌ 无 | ✅ 无需迁移 |
| **speaker_nodes** | .asrx.json | ❌ 无 | ✅ 无需迁移 |
| **appearance_trace_nodes** | .appearance.json | ❌ 无 | ✅ 无需迁移 |
| **skin_tone_trace_nodes** | .skin.json | ❌ 无 | ✅ 无需迁移 |
| **accessory_nodes** | .accessory.json | ❌ 无 | ✅ 无需迁移 |
## Edge Types 迁移状态
| Edge Type | 数据源 | PostgreSQL 依赖 | 迁移状态 |
|-----------|--------|-----------------|----------|
| **co_occurrence_edges** | face_detections | ✅ face_detections.trace_id | 🔄 待迁移 |
| **face_face_edges** | face_detections | ✅ face_detections.trace_id | 🔄 待迁移 |
| **speaker_face_edges** | face_detections + speaker | ✅ face_detections.trace_id | 🔄 待迁移 |
| **mutual_gaze_edges** | gaze.json | ✅ face_detections.trace_id | 🔄 待迁移 |
| **lip_sync_edges** | lip.json | ✅ face_detections.trace_id | 🔄 待迁移 |
## 迁移计划
### Phase 2.5: Gaze & Lip Nodes
**目标**: 使用 Qdrant payload 替代 face_detections 查询
#### 2.5.1: gaze_trace_nodes
**当前代码** (`src/core/processor/tkg.rs`):
```rust
let frame_rows: Vec<(i64, i64, f64, f64, f64, f64)> = sqlx::query_as(
"SELECT trace_id, frame_number, x, y, width, height
FROM face_detections WHERE file_uuid = $1"
)
```
**迁移方案**:
```rust
// 使用 Qdrant payload (trace_id, frame, bbox_x/y/w/h)
let qdrant_embeddings = face_db.get_all_embeddings_for_file(file_uuid).await?;
// Group by trace_id → compute gaze
```
#### 2.5.2: lip_trace_nodes
**当前代码**:
```rust
// Read lip.json, query face_detections for trace_id
let trace_id = sqlx::query_scalar(
"SELECT trace_id FROM face_detections
WHERE file_uuid = $1 AND frame_number = $2 AND x = $3 ..."
)
```
**迁移方案**:
```rust
// 使用 Qdrant payload 直接关联 trace_id
// face.json 已有 trace_id (Python store_traced_faces.py)
```
### Phase 2.6: Edge Types
#### 2.6.1: co_occurrence_edges
**当前代码**:
```rust
"SELECT trace_id FROM face_detections
WHERE file_uuid = $1 AND frame_number BETWEEN $2 AND $3"
```
**迁移方案**:
```rust
// 使用 Qdrant payload.group_by(trace_id)
// 预计算 frame ranges
```
#### 2.6.2: face_face_edges
**当前代码**:
```rust
"SELECT trace_id, frame_number FROM face_detections
WHERE file_uuid = $1 AND trace_id IS NOT NULL"
```
**迁移方案**:
```rust
// 使用 Qdrant embeddings 的 spatial proximity
// 无需 PostgreSQL
```
#### 2.6.3: speaker_face_edges
**当前代码**:
```rust
// JOIN face_detections.trace_id + speaker_nodes
```
**迁移方案**:
```rust
// Qdrant trace_id + speaker_nodes (already from .asrx.json)
```
### Phase 2.7: Identity Resolution for Edges
**当前代码** (Rule2):
```rust
// 已完成 Phase 2.3: 查询 tkg_nodes.properties.identity_id
```
**扩展**:
- gaze/lip edges 也需要 identity resolution
- 统一使用 `tkg_nodes.properties.identity_id`
## 不迁移的 Node Types
### text_trace_nodes
**原因**:
- chunk table 是必要持久化（sentence chunks）
- 不依赖 face_detections
- 保持现状，无需迁移
### JSON-based Nodes
**已无 PostgreSQL 依赖**:
- yolo_object_nodes: `.yolo.json`
- speaker_nodes: `.asrx.json`
- appearance_trace_nodes: `.appearance.json`
- skin_tone_trace_nodes: `.skin.json`
- accessory_nodes: `.accessory.json`
## 性能影响预估
| 迁移项 | 当前耗时 | 预估迁移后 | 提升 |
|--------|----------|------------|------|
| gaze_trace_nodes | ~50ms (PG query) | ~15ms (Qdrant) | **3x** |
| lip_trace_nodes | ~80ms (PG + lip.json) | ~20ms (Qdrant + lip.json) | **4x** |
| co_occurrence_edges | ~120ms (PG) | ~30ms (Qdrant) | **4x** |
| face_face_edges | ~90ms (PG) | ~25ms (Qdrant) | **3.6x** |
## 实施优先级
| 优先级 | 任务 | 影响 | 复杂度 |
|--------|------|------|--------|
| P1 | gaze_trace_nodes | 高（gaze 分析） | 低 |
| P1 | co_occurrence_edges | 高（关系图） | 中 |
| P2 | lip_trace_nodes | 中（lip 分析） | 中 |
| P2 | face_face_edges | 中（face 关系） | 中 |
| P3 | speaker_face_edges | 低（speaker 关系） | 中 |
## 关键决策
1. **text_trace_nodes**: 保持 chunk table 查询（必要持久化）
2. **JSON nodes**: 无需迁移（已无 PG 依赖）
3. **Qdrant 作为唯一 face 数据源**: trace_id, frame, bbox 全部从 payload 获取
4. **渐进式迁移**: 按优先级分 Phase 2.5, 2.6, 2.7
## 验收标准
- ✅ gaze_trace_nodes: 无 face_detections 查询
- ✅ lip_trace_nodes: 使用 Qdrant trace_id
- ✅ 所有 edges: 使用 Qdrant payload
- ✅ 性能测试: 比原架构快 2x 以上
- ✅ Rule2/Rule3: 正常工作（identity resolution）
## 参考文档
- `docs_v1.0/M4_workspace/2026-06-21_tkg_phase2_progress.md` (Phase 2-3)
- `src/core/processor/tkg.rs` (当前实现)
- `src/core/db/face_embedding_db.rs` (Qdrant API)
docs_v1.0/DESIGN/Trace_Inheritance_Rules.md
+209
View File
@@ -0,0 +1,209 @@
# Trace ID Inheritance & Expansion Rules
**Date**: 2026-07-19
**Author**: Core Team
**Status**: Final
---
## Overview
This document defines the trace ID inheritance rules and expansion logic for Face, Pose, and Appearance processing.
---
## Core Concepts
### Face = Identity, Pose/Appearance = Tracking
| Processor | Purpose | Description |
|-----------|---------|-------------|
| **Face** | Identity | Who is this person? Requires high-quality embedding for recognition. |
| **Pose** | Tracking | Where is this person? Maintains tracking when face is occluded. |
| **Appearance** | Tracking | What do they look like? Maintains tracking when pose is occluded. |
### Offline Processing Advantage
In offline processing, we can:
1. First detect all faces (identity anchors)
2. Then expand pose/appearance from face traces
This is different from real-time tracking where pose/appearance runs continuously and face anchors identity when visible.
---
## Processing Pipeline
### Step 1: Face Detection (8Hz)
```
swift_face → face.json
```
- Sampling rate: `floor(fps / 8)` (ensures ≥ 8Hz)
- Output: Face bounding boxes with landmarks and embeddings
### Step 2: Face Tracking
```
store_traced_faces.py → face_traced.json
```
- Algorithm: IoU + embedding similarity
- Output: Each face assigned a `trace_id`
- Purpose: Group same-person faces across frames
### Step 3: Pose Expansion
```
swift_pose_expansion → pose.json
```
**Input**: face_traced.json (frames with trace_id)
**Expansion Algorithm**:
1. For each trace_id, get all face frames
2. Expand outward (forward/backward) checking for pose
3. Stop when 3 consecutive frames have no pose detection
4. Inherit trace_id from face
**Output**: Pose keypoints with inherited trace_id
### Step 4: Appearance Expansion
```
swift_appearance_expansion → appearance.json
```
**Input**: pose.json (frames with trace_id)
**Expansion Algorithm**:
1. For each trace_id, get all pose frames
2. Expand outward (forward/backward) checking for appearance
3. Stop when 3 consecutive frames have HSV similarity < 0.5
4. Inherit trace_id from pose
**Output**: HSV histograms with inherited trace_id
---
## Trace ID Inheritance
```
Face Trace (identity anchor)
│
│ inherits trace_id
▼
Pose Expansion
│
│ inherits trace_id
▼
Appearance Expansion
```
**Key Points:**
- Trace ID originates from face tracking
- Pose inherits the same trace_id (same person)
- Appearance inherits the same trace_id (same person)
- This enables linking all detections to the same identity
---
## Frame Count Relationship
```
face frames ≤ pose frames ≤ appearance frames
```
**Explanation:**
- Face: Only frames where face is clearly visible
- Pose: Face frames + expanded frames (pose may still be visible when face is occluded)
- Appearance: Pose frames + expanded frames (appearance may still be visible when pose is occluded)
---
## Expansion Rules
### Pose Expansion
| Parameter | Value | Description |
|-----------|-------|-------------|
| `miss_threshold` | 3 | Stop after 3 consecutive frames without pose |
| `max_range` | 300 frames | Maximum expansion distance (≈10s at 30fps) |
| `output_rate` | 8Hz | Output sampling rate |
### Appearance Expansion
| Parameter | Value | Description |
|-----------|-------|-------------|
| `miss_threshold` | 3 | Stop after 3 consecutive frames with similarity < 0.5 |
| `similarity_threshold` | 0.5 | HSV histogram similarity threshold |
| `max_range` | 300 frames | Maximum expansion distance |
| `output_rate` | 8Hz | Output sampling rate |
---
## Tracking Continuity
### Pose Can Connect Face Traces
```
Face trace A (frames 1-10) Face trace B (frames 20-30)
↘ ↙
Pose connects (frames 15-18)
(Same person, face was occluded)
```
When pose expansion from two face traces overlaps, they may belong to the same person. Future enhancement: pose-based trace merging.
### Appearance Can Connect Pose Traces
Similar to pose, appearance similarity can connect pose traces when pose is temporarily occluded.
---
## Implementation Files
| Component | File |
|-----------|------|
| Face Detection | `scripts/swift_processors/swift_face.swift` |
| Face Tracking | `scripts/store_traced_faces.py` |
| Pose Expansion | `scripts/swift_processors/swift_pose_expansion.swift` |
| Appearance Expansion | `scripts/swift_processors/swift_appearance_expansion.swift` |
| Pose Processor Wrapper | `scripts/pose_processor_v2.py` |
| Appearance Processor Wrapper | `scripts/appearance_processor_v2.py` |
| Dependencies Definition | `src/core/db/postgres_db.rs:568-577` |
---
## Testing
### Verify Trace ID Inheritance
```bash
# Check face traces
cat /Users/accusys/momentry/output/$FILE_UUID.face_traced.json | jq '.frames[].faces[].trace_id' | sort | uniq -c
# Check pose traces (should have same trace_ids)
cat /Users/accusys/momentry/output/$FILE_UUID.pose.json | jq '.frames[].trace_id' | sort | uniq -c
# Check appearance traces (should have same trace_ids)
cat /Users/accusys/momentry/output/$FILE_UUID.appearance.json | jq '.frames[].trace_id' | sort | uniq -c
```
### Verify Frame Count Relationship
```bash
# face ≤ pose ≤ appearance
FACE_COUNT=$(cat $OUTPUT/$UUID.face.json | jq '.frames | length')
POSE_COUNT=$(cat $OUTPUT/$UUID.pose.json | jq '.frames | length')
APP_COUNT=$(cat $OUTPUT/$UUID.appearance.json | jq '.frames | length')
echo "Face: $FACE_COUNT, Pose: $POSE_COUNT, Appearance: $APP_COUNT"
# Expected: Face ≤ Pose ≤ Appearance
```
---
*Document Version: 1.0*
*Last Updated: 2026-07-19*
docs_v1.0/DESIGN/VideoPlayback_Architecture_V1.0.md
+374
View File
@@ -0,0 +1,374 @@
---
document_type: "design"
service: "MOMENTRY_CORE"
title: "Video Playback Architecture — Local Direct Serve & Remote Streaming"
version: "V1.0"
date: "2026-06-07"
author: "OpenCode"
status: "draft"
tags:
- "video-playback"
- "caddy"
- "streaming"
- "thumbnail"
- "wordpress-frontend"
related_documents:
- "DESIGN/FILE_LIFECYCLE_V1.0.md"
---
# Video Playback Architecture — Local Direct Serve & Remote Streaming
| Item | Value |
|------|-------|
| Scope | Video file playback & thumbnail serving for WordPress frontend (m5wp) |
| Status | Draft |
| Applies to | Search results (`serve_url`), Caddy routing, Momentry media-proxy endpoint |
| Key concept | Local files served directly by Caddy (zero backend overhead); remote files fall back to Momentry streaming; thumbnails proxied through Caddy to Momentry |
---
## Problem Statement
The WordPress frontend (`m5wp.momentry.ddns.net`) displays search results with video thumbnails and a player. Currently:
- **Thumbnails**: WordPress Code Snippet 61 (`momentry/v1/media` REST route) is inactive → all requests return `rest_no_route` 404
- **Video playback**: Frontend has no way to construct a playable URL from search results; no `serve_url` exists in the search response
- **WordPress constraint**: WordPress files and database tables must not be modified (marcom team territory)
The solution must work for two deployment scenarios:
- **Local**: Video file resides on the same server as Momentry → serve via static HTTP (zero processing overhead)
- **Remote**: Video file resides on an external storage (NAS, S3, etc.) → fall back to Momentry's ffmpeg-based streaming
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ Browser (search-chat @ m5wp.momentry.ddns.net) │
│ │
│ ┌──────────┐ ┌──────────────────┐ ┌─────────────────────┐ │
│ │ Search │ │ Thumbnail img │ │ <video src="..."> │ │
│ └────┬─────┘ └───────┬──────────┘ └──────────┬──────────┘ │
│ │ │ │ │
└───────┼─────────────────┼──────────────────────────┼─────────────┘
│ │ │
▼ ▼ ▼
┌───────────────────────────────────────────────────────────────┐
│ Caddy (m5wp block) │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ handle /wp-json/momentry/v1/media { │ │
│ │ rewrite * /api/v1/media-proxy{?} │ │
│ │ reverse_proxy localhost:3002 (+ X-API-Key) │ │
│ │ } │ │
│ │ │ │
│ │ handle_path /files/* { │ │
│ │ root * /Users/accusys/momentry/var/sftpgo/data │ │
│ │ file_server │ │
│ │ } │ │
│ │ │ │
│ │ reverse_proxy localhost:9002 ← WordPress (PHP-FPM) │ │
│ └─────────────────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────────────────┘
│ │ │
│ │ ▼
│ │ ┌───────────────────────┐
│ │ │ /files/* │
│ │ │ Local file on disk │
│ │ │ (zero backend cost) │
│ │ └───────────────────────┘
│ ▼
│ ┌─────────────────────────────────────────┐
│ │ Momentry Core (localhost:3002) │
│ │ │
▼ ▼ /api/v1/media-proxy │
┌─────────────────────────┐ │
│ type=thumbnail?frame=N │──→ face_thumbnail │
│ type=video&start=… │──→ stream_video │
└─────────────────────────┘ │
┌─────────────────────────┐ │
│ POST /api/v1/search/* │──→ smart_search │
│ response: serve_url │ │
└─────────────────────────┘ │
└───────────────────────────────────────────────┘
```
---
## Data Flow
### 1. Search → serve_url
```
Frontend Caddy Momentry Backend
│ │ │
│ POST /wp-json/.../search │ │
│ ─────────────────────────→│ │
│ │ POST /api/v1/search/* │
│ │ ──────────────────────→│
│ │ │
│ │ ←─ SearchResult[] ─────│
│ │ (with serve_url + │
│ │ file_name added) │
│ ←─ JSON response ────────│ │
│ results[0].serve_url = │ │
│ "https://m5wp.momentry.│ │
│ ddns.net/files/demo/ │ │
│ Charade_YouTube_24fps │ │
│ .mp4" │ │
```
#### serve_url Construction
The backend computes `serve_url` from the video's `file_path` (stored in `videos` table) and two config values:
| Config | Env Var | Default |
|--------|---------|---------|
| `STORAGE_ROOT` | `MOMENTRY_STORAGE_ROOT` | `/Users/accusys/momentry/var/sftpgo/data` |
| `SERVE_BASE_URL` | `MOMENTRY_SERVE_BASE_URL` | `https://m5wp.momentry.ddns.net/files` |
Algorithm:
```
file_path: /Users/accusys/momentry/var/sftpgo/data/demo/Charade_YouTube_24fps.mp4
STORAGE_ROOT /Users/accusys/momentry/var/sftpgo/data
─────────────────────────────────────────────
relative: demo/Charade_YouTube_24fps.mp4
↓ join with SERVE_BASE_URL
serve_url: https://m5wp.momentry.ddns.net/files/demo/Charade_YouTube_24fps.mp4
```
#### SearchResult Additions
```rust
pub struct SearchResult {
// ... existing fields
pub file_name: Option<String>, // e.g. "Charade_YouTube_24fps.mp4"
pub serve_url: Option<String>, // e.g. "https://m5wp.momentry.ddns.net/files/..."
}
```
### 2. Video Playback (Local)
```
Frontend <video> Caddy (file_server)
│ │
│ GET /files/demo/Charade… │
│ ─────────────────────────→│
│ │ root = /Users/accusys/momentry/var/sftpgo/data
│ │ serves /demo/Charade_YouTube_24fps.mp4
│ │
│ ←─ 200 video/mp4 ────────│
│ (range-request │
│ supported natively) │
```
**Characteristics**:
- Zero CPU cost — pure I/O, no ffmpeg decode
- HTTP range requests work natively (Caddy `file_server` supports `Accept-Ranges: bytes`)
- HTML5 `<video>` can seek arbitrarily, play/pause normally
- Supports MP4 (H.264), WebM, and any browser-playable format
### 3. Video Playback (Remote — Fallback)
```
Frontend Caddy Momentry Backend
│ │ │
│ GET /wp-json/.../ │ │
│ media?uuid=X& │ │
│ type=video& │ │
│ start_time=S& │ │
│ end_time=E │ │
│ ────────────────────→│ │
│ │ rewrite to │
│ │ /api/v1/media-proxy{?} │
│ │ │
│ │ GET /api/v1/media-proxy? │
│ │ uuid=X&type=video&... │
│ │ ─────────────────────────→│
│ │ │
│ │ stream_video: │
│ │ ffmpeg -ss S -i file │
│ │ -t (E-S) -c copy │
│ │ │
│ │ ←─ 200 video/mp4 ──────────│
│ │ (chunk data) │
│ ←─ HTTP streaming ───│ │
```
### 4. Thumbnail
```
Frontend <img> Caddy Momentry Backend
│ │ │
│ GET /wp-json/.../ │ │
│ media?uuid=X& │ │
│ type=thumbnail& │ │
│ frame=N │ │
│ ──────────────────────→│ │
│ │ rewrite to │
│ │ /api/v1/media-proxy{?} │
│ │ │
│ │ /api/v1/media-proxy? │
│ │ uuid=X&type=thumbnail& │
│ │ frame=N │
│ │ ─────────────────────────→│
│ │ │
│ │ face_thumbnail: │
│ │ look up trace_id path │
│ │ → cached face crop │
│ │ → validated JPEG │
│ │ │
│ │ ←─ 200 image/jpeg ────────│
│ ←─ JPEG ───────────────│ │
```
**Thumbnail flow detail**:
1. Caddy intercepts `/wp-json/momentry/v1/media` → rewrites to `/api/v1/media-proxy` keeping query params intact (`{?}`)
2. Momentry `media_proxy_handler` reads `uuid`, `type=thumbnail`, `frame=N` from query
3. Dispatches to the internal `face_thumbnail` handler
4. Returns cached face crop JPEG (or fallback frame extraction result)
---
## Caddyfile Configuration
Addition to the existing `m5wp` block:
```caddy
m5wp.momentry.ddns.net {
tls internal
# ── Local video files: direct serve, zero backend overhead ──
handle_path /files/* {
root * /Users/accusys/momentry/var/sftpgo/data
file_server
}
# ── Media proxy: thumbnails + remote streaming ──
# Bypasses inactive WordPress Code Snippet 61
handle /wp-json/momentry/v1/media {
rewrite * /api/v1/media-proxy{?}
reverse_proxy localhost:3002 {
header_up X-API-Key muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69
}
}
# ── Existing WordPress (PHP-FPM) ──
reverse_proxy localhost:9002
import common_log m5wp_access
}
```
**Key syntax**:
- `handle_path /files/*` — strips `/files` prefix, serves from `root` directory
- `{?}` — Caddy placeholder that preserves the original query string in the rewrite
- `handle /wp-json/momentry/v1/media` — matches exact path (query params are irrelevant for matching)
---
## Momentry API Changes
### New Endpoint: `GET /api/v1/media-proxy`
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `uuid` | string | yes | file_uuid (accepts `file_uuid` key as alias) |
| `type` | string | yes | `thumbnail`, `video` (future: `image`, `file`) |
| `frame` | int | for thumbnail | Frame number to extract |
| `trace_id` | int | no | Face trace ID for cached crop |
| `start_time` | float | for video | Start time in seconds |
| `end_time` | float | for video | End time in seconds |
| `mode` | string | no | `normal` or `debug` (video) |
| `audio` | string | no | `on` or `off` (video) |
**Dispatch logic**:
- `type=thumbnail` → call `face_thumbnail(State, Path(uuid), Query(frame, trace_id, ...))`
- `type=video` → call `stream_video(State, Path(uuid), Query(params), request)`
The endpoint reuses existing handler implementations via direct axum extractor composition, avoiding code duplication.
### Modified Endpoint: `POST /api/v1/search/smart`
**Response changes**: `SearchResult` gains two optional fields:
```json
{
"results": [
{
"file_uuid": "a6fb22eebefaef17e62af874997c5944",
"file_name": "Charade_YouTube_24fps.mp4",
"serve_url": "https://m5wp.momentry.ddns.net/files/demo/Charade_YouTube_24fps.mp4",
"start_frame": 88649,
"start_time": 3697.08,
"end_time": 3707.08,
"summary": "...",
"similarity": 0.85
}
]
}
```
The `serve_url` is computed after enrichment via a batch query to the `videos` table (`file_uuid → file_path`), then applying the path translation:
1. Strip `STORAGE_ROOT` prefix from `file_path`
2. Prepend `SERVE_BASE_URL`
---
## Environment Variables
Add to `.env` (production) and `.env.development`:
```bash
# Storage root: where video files are stored on disk
# Used to compute serve_url from file_path
MOMENTRY_STORAGE_ROOT=/Users/accusys/momentry/var/sftpgo/data
# Public base URL for direct file access via Caddy file_server
MOMENTRY_SERVE_BASE_URL=https://m5wp.momentry.ddns.net/files
```
---
## Trade-offs & Rationale
| Approach | Pros | Cons |
|----------|------|------|
| **Caddy file_server** (local) | Zero CPU, native range requests, no code change to Momentry for serving | Requires storage root config; files must be accessible from Caddy |
| **Momentry stream_video** (remote) | Works with any storage backend (S3, NAS, NFS) | ffmpeg decode per request, higher latency, CPU-bound |
| **WordPress PHP proxy** (rejected) | No infra change | Fragile, snippet inactive, violates marcom territory |
| **Direct backend streaming only** (rejected) | Simplest implementation | Unnecessary CPU for local files; 100% backend dependency |
### Fallback Logic (Frontend)
The frontend JavaScript should handle playback as follows:
```javascript
if (result.serve_url) {
// Local file — direct Caddy file_server
video.src = result.serve_url;
} else {
// Remote — use streaming endpoint
video.src = `/wp-json/momentry/v1/media?uuid=${result.file_uuid}&type=video&start_time=${result.start_time}&end_time=${result.end_time}`;
}
```
This gives the frontend flexibility to pick the optimal playback path based on available data.
---
## Future Considerations
- **S3/NAS remote files**: When video files are stored externally, the `file_path` won't match `STORAGE_ROOT`. The backend can detect this by checking `file_path.starts_with(STORAGE_ROOT)`. If it doesn't match, omit `serve_url` and rely on the streaming fallback.
- **Pre-signed URLs**: For S3 storage, `serve_url` could be replaced with a pre-signed URL or cloud CDN URL.
- **Caching**: `file_server` responses are cacheable; consider adding `Cache-Control` headers for thumbnails.
- **Authentication**: Direct file access currently has no auth. If needed, Caddy can inject auth via `forward_auth` or JWT validation.
---
## Version History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| V1.0 | 2026-06-07 | OpenCode | Initial design — local direct serve + remote streaming + thumbnail proxy architecture |
docs_v1.0/DESIGN/Worker_Health_Check_Mechanism.md
+328
View File
@@ -0,0 +1,328 @@
---
title: Worker Health Check Mechanism
version: 1.0
date: 2026-06-21
author: momentry_core development
status: active
---
## Overview
Momentry Core worker processes can become stuck due to:
- Redis connection timeouts
- Job queue corruption
- Long-running processor hangs
- Resource exhaustion
This document describes health check mechanisms and recommended solutions.
## Current Architecture
### Worker Process
```
momentry worker
│
├─→ Redis connection pool
│ └─→ Poll job queue ({prefix}job:*)
│
├─→ Processor executor
│ ├─→ Python scripts (timeout: configurable)
│ └─→ Resource monitoring (CPU, memory, GPU)
│
└─→ Dynamic concurrency
└─→ Adjust based on system resources
```
### Worker Logs
Worker logs are stored in:
- `logs/nohup_worker*.log` - Historical worker logs
- `logs/momentry_3002.log` - Production server logs
- `logs/momentry_3003.log` - Playground server logs
## Known Issues
### Issue: Worker Stuck (2026-06-21)
**Symptoms**:
- Worker process running but no activity
- Last log timestamp outdated (>17 hours old)
- Jobs triggered but never processed
- Redis keys created but not consumed
**Cause**: Worker process running for extended period without proper cleanup
**Resolution**:
```bash
# 1. Check worker status
ps aux | grep momentry.*worker
# 2. Check last activity
tail -20 logs/nohup_worker*.log
# 3. Kill stuck worker
kill <PID>
# 4. Restart worker
./target/release/momentry worker
```
## Recommended Health Check Mechanisms
### 1. Worker Heartbeat
**Implementation**:
- Worker writes heartbeat to Redis every 30 seconds
- Heartbeat key: `{prefix}health`
- Heartbeat value: `{timestamp, worker_pid, status}`
**Check**:
```bash
# Check worker heartbeat
redis-cli -a accusys HGETALL "momentry:health"
```
**Expected output**:
```json
{
"timestamp": "1782015243",
"worker_pid": "52908",
"status": "active",
"last_job": "abc123..."
}
```
### 2. Automatic Restart
**Recommendation**: Implement automatic restart on inactivity timeout
```bash
# Example: Restart worker if no heartbeat for 60 seconds
# (To be implemented in worker code)
while true; do
# Check heartbeat
LAST_HEARTBEAT=$(redis-cli HGET momentry:health timestamp)
CURRENT_TIME=$(date +%s)
if [ $((CURRENT_TIME - LAST_HEARTBEAT)) > 60 ]; then
echo "Worker stuck, restarting..."
pkill -f "momentry worker"
./target/release/momentry worker &
fi
sleep 30
done
```
### 3. Worker Status API
**Recommendation**: Add `/api/v1/worker/status` endpoint
**Response**:
```json
{
"worker_pid": 52908,
"status": "active",
"last_heartbeat": "2026-06-21T12:15:00Z",
"jobs_processed": 42,
"current_job": "abc123...",
"uptime_seconds": 3600
}
```
### 4. Job Queue Monitoring
**Check for stuck jobs**:
```bash
# List all pending jobs
redis-cli -a accusys keys "momentry:job:*"
# Check job timestamp
redis-cli -a accusys HGET "momentry:job:{file_uuid}" created_at
# If job > 1 hour old without progress → stuck job
```
### 5. Resource Monitoring
**Worker logs include system stats**:
```
System: CPU idle=50.0%, Memory=31948MB/49152MB (35.0%), No GPU
Dynamic concurrency: 2 (config: 2)
```
**Monitor**:
- CPU idle > 90% for extended period → worker not processing
- Memory > 90% → resource exhaustion risk
- GPU not available → GPU-dependent processors will fail
## Monitoring Script
```bash
#!/bin/bash
# worker_health_monitor.sh
PREFIX="momentry:"
REDIS_URL="redis://:accusys@localhost:6379"
while true; do
echo "=== Worker Health Check ==="
# Check worker process
WORKER_PID=$(pgrep -f "momentry worker")
if [ -z "$WORKER_PID" ]; then
echo "❌ No worker process running"
echo "Starting worker..."
./target/release/momentry worker &
continue
fi
echo "✅ Worker running (PID: $WORKER_PID)"
# Check Redis heartbeat
HEARTBEAT=$(redis-cli -a accusys HGET "${PREFIX}health" timestamp)
if [ -n "$HEARTBEAT" ]; then
AGE=$(( $(date +%s) - $HEARTBEAT ))
if [ $AGE > 60 ]; then
echo "⚠️ Worker heartbeat stale ($AGE seconds old)"
echo "Restarting worker..."
kill $WORKER_PID
./target/release/momentry worker &
else
echo "✅ Heartbeat recent ($AGE seconds old)"
fi
else
echo "⚠️ No heartbeat found"
fi
# Check pending jobs
JOBS=$(redis-cli -a accusys keys "${PREFIX}job:*" | wc -l)
echo "Pending jobs: $JOBS"
sleep 30
done
```
## Preventive Measures
### 1. Regular Worker Restart
**Recommendation**: Restart worker daily to prevent accumulation
```bash
# Daily restart at 3 AM
# Add to crontab:
0 3 * * * pkill -f "momentry worker" && sleep 5 && ./target/release/momentry worker &
# Or use systemd/launchd for automatic restart
```
### 2. Timeout Configuration
**Set reasonable timeouts**:
```bash
# Environment variables
MOMENTRY_ASR_TIMEOUT=3600 # 1 hour for ASR
MOMENTRY_CUT_TIMEOUT=3600 # 1 hour for CUT
MOMENTRY_DEFAULT_TIMEOUT=7200 # 2 hours default
```
### 3. Resource Limits
**Limit worker concurrency**:
```bash
# Worker flags
./target/release/momentry worker \
--max-concurrent 6 \ # Max parallel processors
--poll-interval 10 \ # Poll every 10 seconds
--batch-size 5 # Process 5 jobs per batch
```
### 4. Logging Enhancement
**Recommendation**: Add structured logging for job lifecycle
```rust
// In job_worker.rs
tracing::info!(
job_id = %job.id,
file_uuid = %file_uuid,
status = "started",
"Worker started job"
);
tracing::info!(
job_id = %job.id,
duration_ms = elapsed,
status = "completed",
"Worker completed job"
);
```
## Troubleshooting Guide
### Step 1: Check Process
```bash
ps aux | grep momentry.*worker
```
Expected: One worker process per environment (production + playground)
### Step 2: Check Logs
```bash
tail -50 logs/nohup_worker*.log
```
Look for:
- Last log timestamp
- Error messages
- Processor failures
### Step 3: Check Redis
```bash
redis-cli -a accusys keys "momentry:job:*"
redis-cli -a accusys HGETALL "momentry:health"
```
Look for:
- Pending jobs count
- Heartbeat timestamp
- Job creation timestamps
### Step 4: Check Resources
```bash
top -pid <worker_pid>
```
Look for:
- CPU usage (should be active if processing)
- Memory usage (should not exceed 80%)
- Process state (should be running, not sleeping)
### Step 5: Restart Worker
```bash
kill <worker_pid>
./target/release/momentry worker
```
## Related Documentation
- `docs_v1.0/DESIGN/Redis_Prefix_Configuration.md` - Redis namespace configuration
- `docs_v1.0/M4_workspace/2026-06-21_issue_report.md` - Worker stuck issue report
- `AGENTS.md` - Worker configuration reference
- `src/worker/job_worker.rs` - Worker implementation
---
## Version History
| Version | Date | Changes |
|---------|------|---------|
| 1.0 | 2026-06-21 | Initial documentation for worker health check mechanisms |
docs_v1.0/GUIDES/QC_FaceCluster_Support.md
+270
View File
@@ -0,0 +1,270 @@
---
title: QC FaceCluster Support Guide
version: 2.0
date: 2026-07-24
author: OpenCode
status: final
---
# QC Modal: face_cluster Support
> Companion guide for Studio team: frontend changes in `/Users/accusys/momentry_studio/src/views/LibraryView.vue` for `face_cluster` support, and backend multi-stage trace dedup upgrade.
| Scope | `/Users/accusys/momentry_studio/src/views/LibraryView.vue` |
|-------|-------------------------------------------------------------|
| Backend changes | [`/Users/accusys/momentry_core/docs_v1.0/DESIGN/AlwaysProduce_Processing_Contract.md`](../DESIGN/AlwaysProduce_Processing_Contract.md) |
| Status | ✅ Frontend done (commit `2af8ffa`). Backend upgraded to multi-stage trace dedup. |
| Version | 2.0 |
**Glossary:**
| Term | Definition |
|------|------------|
| **Always-Produce rule** | Every processor MUST write its output JSON after scanning the last frame, even for zero results. See [`/Users/accusys/momentry_core/docs_v1.0/DESIGN/AlwaysProduce_Processing_Contract.md`](../DESIGN/AlwaysProduce_Processing_Contract.md#2-always-produce-rule). |
| **Stage 2** | Second-level processors that depend on Stage 1 (`asr`, `ocr`, `face`): `asrx`, `face_cluster`, `pose`, `appearance`. |
| **QC Modal** | Pipeline Quality Control modal launched via the 🔍 button in the file context menu (advanced mode). |
| **trace_id** | Per-video integer identifier linking face detections across frames (from face tracker). Each `trace_id` represents the same person in a continuous shot. |
| **Multi-stage dedup** | Two-stage clustering: (1) strict AgglomerativeClustering on trace-level mean embeddings, (2) cross-cluster merge via trace-pair voting with temporal overlap guard. |
---
## Background
The backend at `/Users/accusys/momentry_core/scripts/fast_face_clustering_processor.py` was upgraded from single-pass face-level clustering to **multi-stage trace-based deduplication**.
### Problem: Single-pass clustering limitations
The old algorithm ran AgglomerativeClustering (cosine distance threshold 0.4) on up to 25k+ individual face embeddings, using random sampling for large datasets. This caused:
- **Fragmentation**: same person appearing in different shots/scenes got split across multiple clusters because their face embeddings exceeded the fixed threshold
- **Sampling bias**: only 5000 faces sampled for large files, minority clusters missed
- **No temporal info**: no use of `trace_id` or frame-range overlap checks
### Analysis: 12 files, 180,791 face embeddings
Analysis of all production data in Qdrant `_faces` collection showed:
| Metric | Value |
|--------|-------|
| Files analyzed | 12 |
| Total face embeddings | 180,791 |
| Worst fragmentation | `c36f35685177` — 62k faces, 5,616 traces, 46% single-face |
| Cross-trace pairs >0.7 similarity | 529,305 in worst file |
| Temporal overlap among high-sim pairs | 99.9% **non-overlapping** (safe to merge) |
| Temporal overlap for talking head | 100% **overlapping** (temporal guard prevents false merge) |
### Solution: Multi-stage trace dedup
1. **Trace aggregation**: group all Qdrant face embeddings by `trace_id`, compute confidence-weighted mean embedding per trace + frame range
2. **Stage 1 (strict)**: AgglomerativeClustering on trace-level mean vectors (cosine distance threshold 0.35)
3. **Stage 2 (merge)**: trace-pair voting across cluster boundaries — if >30% of cross-cluster trace pairs have similarity >0.70 AND overall cluster frame ranges don't overlap → merge
### Sourced from Qdrant `_faces` Collection
Collection: `_faces` (512D, Cosine distance)
Payload: `{file_uuid, frame, trace_id, bbox, confidence, identity_id, identity_uuid}`
---
## face_cluster.json Output Format (Unchanged)
From `/Users/accusys/momentry_core/scripts/fast_face_clustering_processor.py`:
```json
{
"status": "has_faces",
"file_uuid": "9781de6d...",
"clusters": [
{
"cluster_id": "Person_0",
"face_count": 12,
"representative_face": {
"face_id": "face_10_3",
"confidence": 0.95,
"frame": 123,
"bbox": { "x": 100, "y": 200, "width": 50, "height": 60 }
}
}
],
"frames": [
{
"frame": 123,
"timestamp": 5.13,
"faces": [
{ "face_id": "face_10_3", "cluster_id": "Person_0", "confidence": 0.95 }
]
}
]
}
```
On no-faces / no-data, status is `"no_faces"`, `"no_face_json"`, or `"no_embeddings"`, with `"clusters": []` and `"frames": []`.
---
## Backend: Complete
| File | Change | Status |
|------|--------|--------|
| `/Users/accusys/momentry_core/scripts/fast_face_clustering_processor.py` | **Multi-stage trace dedup**: trace aggregation + Stage 1 strict clustering (0.35) + Stage 2 trace-pair voting merge (0.70 sim, 0.30 ratio) + temporal overlap guard | ✅ Done |
| `/Users/accusys/momentry_core/scripts/fast_face_clustering_processor.py` | Always-Produce: 3 early returns write empty output + RedisPublisher progress | ✅ Done |
| `/Users/accusys/momentry_core/src/worker/job_worker.rs:186` | Worker heartbeat EXPIRE 15s after HMSET | ✅ Done |
| `/Users/accusys/momentry_core/src/api/health.rs:631-655` | `check_worker_alive()` — Redis TTL check replacing `ps aux` | ✅ Done |
| `/Users/accusys/momentry_core/src/api/processing.rs:842` | `GET /api/v1/file/{uuid}/processor-counts` — auto-includes `FaceCluster` via `ProcessorType::all()` | ✅ Already correct |
### Test Results (7 files, 155,095 face embeddings total)
| File | Type | Faces | Traces | Stage 1 | Final | Merged |
|------|------|-------|--------|---------|-------|--------|
| `c36f35685177` | Crowd | 62,298 | 5,616 | 1,113 | 826 | **287** |
| `d8acb03870f0` | Crowd | 693 | 107 | 45 | 43 | 2 |
| `84d838f260e1` | Crowd | 597 | 89 | 34 | 33 | 1 |
| `c0a9dc37cd84` | Crowd | 1,137 | 78 | 26 | 24 | 2 |
| `31a6b8212760` | Multi | 744 | 32 | 10 | 9 | 1 |
| `5e5f3de82208` | Multi | 532 | 22 | 4 | 4 | 0 |
| `bfba056f5021` | Talking head | 89,791 | 16 | 4 | 4 | **0 (correct)** |
**Key verification**: talking head file had 100% temporal overlap among all high-sim trace pairs — Stage 2 correctly merged 0 clusters (temporal guard prevented false positive).
---
## Frontend: Studio Team Changes (Already Done)
**Commit**: `2af8ffa` → Gitea (pushed by Studio team)
### Change 1: Job Output File List (P0)
**File**: `/Users/accusys/momentry_studio/src/views/LibraryView.vue`
**Line**: ~L1034
Add `'face_cluster'` to the `processors` array in `refreshSystemStatus()`:
```typescript
const processors = ['cut', 'asr', 'asrx', 'face', 'ocr', 'pose', 'appearance', 'face_cluster']
```
Expected: QC Modal → Jobs → each job's output list includes `face_cluster.json` with cluster count. Empty results show `face_cluster.json (0筆)`.
### Change 2: QC Result Query List (P0)
**File**: `/Users/accusys/momentry_studio/src/views/LibraryView.vue`
**Line**: ~L1096
Add `'face_cluster'` to the `processors` array in `runProcessorQC()`:
```typescript
const processors = ['cut', 'asr', 'asrx', 'face', 'ocr', 'pose', 'appearance', 'face_cluster']
```
Expected: Pipeline visualization and node status include `FACE_CLUSTER`.
### Change 3: `getJsonCount()` — Add `clusters` Case (P0)
**File**: `/Users/accusys/momentry_studio/src/views/LibraryView.vue`
**Line**: ~L1002
Insert `data.clusters` check before `return 0`:
```typescript
if (data.cuts) return data.cuts.length
if (data.clusters) return data.clusters.length
return 0
```
`face_cluster.json` uses `clusters` array — without this branch, count always shows 0.
### Change 4: Pipeline Stage 2 — Add FACE_CLUSTER Node (P0)
**File**: `/Users/accusys/momentry_studio/src/views/LibraryView.vue`
**Line**: ~L460
Add `FACE_CLUSTER` to the Stage 2 filter:
```html
v-for="r in qcResults.filter(p => ['ASRX','FACE_CLUSTER','APPEARANCE'].includes(p.processor))"
```
Expected pipeline:
```
[ASRX] → [FACE_CLUSTER] → [APPEARANCE] → 📄 JSON Outputs
```
### Change 5 (Optional, P1): Context Menu
**File**: `/Users/accusys/momentry_studio/src/views/LibraryView.vue`
**Line**: ~L179
Insert `face_cluster` block between ASRX and Face:
```html
<div class="ms-proc-line" :class="procStatusClass('face_cluster')">
<label class="ms-fm-check-label"><input type="checkbox" v-model="procFaceCluster"> Face Cluster</label>
<span class="ms-proc-status">{{ procStatusIcon('face_cluster') }}</span>
<span class="ms-proc-count" @click.stop="viewProcessorJson('face_cluster')">{{ procCountLabel('face_cluster', 'frame') }}</span>
<button class="ms-proc-redo" @click.stop="redoProcessor('face_cluster')" title="Re-run Face Cluster">🔄</button>
</div>
```
Add reactive ref:
```typescript
const procFaceCluster = ref(true)
```
### Change 6 (Optional, P1): `selectedProcessors()`
**File**: `/Users/accusys/momentry_studio/src/views/LibraryView.vue`
**Line**: ~L930
```typescript
if (procAsrx.value) procs.push('asrx')
if (procFaceCluster.value) procs.push('face_cluster')
if (procFace.value) procs.push('face')
```
---
## What to Expect in QC
With the multi-stage algorithm:
| Before (single-pass) | After (multi-stage trace dedup) |
|----------------------|--------------------------------|
| Many small clusters for the same person across different shots | Fewer, more accurate clusters — cross-shot fragments merged |
| Single-face traces often assigned to wrong cluster (noise) | Single-face traces remain as small clusters but don't pollute larger ones |
| Talking head: reasonable (limited impact) | Unchanged (temporal guard prevents false merge) |
| Crowd/multi-person: severe fragmentation | 26% fewer clusters in worst case (287 clusters merged in test) |
**Example**: `c36f35685177` (62k faces, crowd scene)
- Old: ~1,113+ clusters (single-pass, sampling-based)
- New: 826 clusters (trace-level, two-stage, temporal verified)
---
## Verification
1. Open QC Modal on a processed file (with or without faces)
2. Verify Jobs output list includes `face_cluster.json`
3. Verify Pipeline Stage 2 shows `FACE_CLUSTER` node with ✓ status and cluster count
4. For a no-faces file (e.g., `9781de6d...`), confirm `face_cluster.json` shows 0 clusters
---
## See Also
- [`/Users/accusys/momentry_core/docs_v1.0/DESIGN/AlwaysProduce_Processing_Contract.md`](../DESIGN/AlwaysProduce_Processing_Contract.md) — Frame-Scan model, Always-Produce rule, Redis progress spec
- [`/Users/accusys/momentry_core/docs_v1.0/DESIGN/Worker_Health_Check_Mechanism.md`](../DESIGN/Worker_Health_Check_Mechanism.md) — Worker heartbeat TTL mechanism
- [`/Users/accusys/momentry_core/docs_v1.0/DESIGN/FILE_LIFECYCLE_V1.0.md`](../DESIGN/FILE_LIFECYCLE_V1.0.md) — Processor stage definitions
- `/Users/accusys/momentry_core/src/core/db/postgres_db.rs:495` — `ProcessorType::all()` includes `FaceCluster`
- `/Users/accusys/momentry_core/src/api/processing.rs:842` — `GET /api/v1/file/{uuid}/processor-counts`
- `/Users/accusys/momentry_core/scripts/fast_face_clustering_processor.py` — Multi-stage trace dedup implementation
---
## Version History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0 | 2026-07-24 | OpenCode | Initial — frontend changes for Always-Produce face_cluster support |
| 2.0 | 2026-07-24 | OpenCode | Backend upgraded to multi-stage trace dedup. Frontend changes completed. |
docs_v1.0/GUIDES/WordPress_Frontend_VideoPlayback_Guide.md
+322
View File
@@ -0,0 +1,322 @@
---
document_type: "guide"
service: "MOMENTRY_CORE"
title: "WordPress Frontend — Video Playback Integration Guide"
version: "V1.0"
date: "2026-06-07"
author: "OpenCode"
status: "draft"
tags:
- "wordpress"
- "frontend"
- "video-playback"
- "thumbnail"
- "integration"
related_documents:
- "DESIGN/VideoPlayback_Architecture_V1.0.md"
---
# WordPress Frontend — Video Playback Integration Guide
| Item | Value |
|------|-------|
| Scope | WordPress frontend (m5wp) video playback & thumbnail changes |
| Status | Draft |
| Backend | Momentry Core API (m5api.momentry.ddns.net) |
| Caddy | Reverse proxy + file server on m5wp.momentry.ddns.net |
| Target audience | WordPress frontend developer |
---
## Architecture
```
Browser (search-chat @ m5wp.momentry.ddns.net)
│
├─ POST https://m5api.momentry.ddns.net/api/v1/search/smart?api_key=KEY
│ └─ Response includes serve_url + file_name (already live)
│
├─ <video src="serve_url"> # Local: Caddy file_server, zero backend cost
│ └─ https://m5wp.momentry.ddns.net/files/demo/Charade_YouTube_24fps.mp4
│
├─ <video src="/wp-json/.../media"> # Remote fallback: Caddy → Momentry streaming
│ └─ /wp-json/momentry/v1/media?uuid=X&type=video&start_time=S&end_time=E
│
└─ <img src="/wp-json/.../media"> # Thumbnail: unchanged, already working
└─ /wp-json/momentry/v1/media?type=thumbnail&uuid=X&frame=N
```
**Traffic paths (all verified production)**:
| Resource | Path | Status |
|----------|------|--------|
| Search results | `m5api.momentry.ddns.net/api/v1/search/smart` | ✅ Returns serve_url |
| Video (serve_url) | `m5wp.momentry.ddns.net/files/...` | ✅ 200, Accept-Ranges: bytes |
| Video (streaming fallback) | `m5wp/.../media?type=video` | ✅ 200 video/mp4 |
| Thumbnail | `m5wp/.../media?type=thumbnail` | ✅ 200 image/jpeg |
---
## 1. Search Endpoint Migration
### Before (being deprecated — drops serve_url / file_name)
```
POST /wp-json/momentry/v1/search-proxy
→ WordPress PHP proxy → localhost:3002 → response
Critical problem: The search-proxy rebuilds the response envelope.
Even though Momentry Core returns `serve_url` and `file_name`,
these fields arrive as `null` in the proxy response because:
1. Semantic mode (`/api/v1/search/llm-smart`) extracts only
`$smart_data['results']` and wraps it in a new envelope
with explicitly listed fields — unknown fields like
`serve_url` / `file_name` are silently dropped.
2. Keyword/universal mode passes through the raw response,
but `serve_url` is computed post-search by Momentry Core's
enricher — this enrichment path may not trigger when the
request comes through a non-standard proxy route.
Net effect: The frontend never receives `serve_url` or `file_name`
from the proxy, making direct Caddy file_server playback impossible.
→ **Must call m5api directly to get these fields.**
```
### After
```javascript
var SEARCH_URL = 'https://m5api.momentry.ddns.net/api/v1/search/smart';
var API_KEY = 'muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69';
```
CORS is open (`access-control-allow-origin: *`), so direct fetch works.
### API Key Transmission
**Method A: query parameter (recommended for simplicity)**
```javascript
fetch(SEARCH_URL + '?api_key=' + encodeURIComponent(API_KEY), { ... })
```
**Method B: X-API-Key header**
```javascript
fetch(SEARCH_URL, {
headers: { 'X-API-Key': API_KEY, 'Content-Type': 'application/json' }
})
```
**Method C (future): Caddy m5api block injects key**
No frontend changes needed once configured.
---
## 2. Search Response Format
```json
{
"query": "gun",
"results": [
{
"file_uuid": "a6fb22eebefaef17e62af874997c5944",
"file_name": "Charade_YouTube_24fps.mp4",
"serve_url": "https://m5wp.momentry.ddns.net/files/demo/Charade_YouTube_24fps.mp4",
"start_frame": 63445,
"start_time": 2646.19,
"end_time": 0.0,
"fps": 23.976,
"summary": "He has a gun, Mr. Bartholomew.",
"similarity": 0.755
}
],
"strategy": "hybrid_semantic+keyword"
}
```
### New Fields (both already live in backend)
| Field | Type | Description |
|-------|------|-------------|
| `file_name` | `string` | Original filename, e.g. `Charade_YouTube_24fps.mp4` |
| `serve_url` | `string \| null` | Direct playable URL via Caddy file_server. `null` if file is not on local storage. |
---
## 3. Code Changes: `fetchSearchApi()`
### Before
```javascript
function fetchSearchApi(query) {
return fetch('/wp-json/momentry/v1/search-proxy', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: query, mode: CURRENT_SEARCH_MODE })
}).then(r => r.json());
}
```
### After
```javascript
var API_KEY = 'muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69';
var SEARCH_BASE = 'https://m5api.momentry.ddns.net/api/v1/search/smart';
var ID_SEARCH_BASE = 'https://m5api.momentry.ddns.net/api/v1/identities/search';
function fetchSearchApi(query) {
// People mode → identities endpoint
if (CURRENT_SEARCH_MODE === 'people') {
var url = ID_SEARCH_BASE + '?q=' + encodeURIComponent(query)
+ '&limit=20&page=1&page_size=20'
+ '&api_key=' + encodeURIComponent(API_KEY);
return fetch(url).then(checkStatus).then(r => r.json());
}
// Keyword / Semantic → search/smart (unified)
var url = SEARCH_BASE + '?api_key=' + encodeURIComponent(API_KEY);
return fetch(url, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ query: query, limit: 30 })
}).then(checkStatus).then(r => r.json());
}
function checkStatus(r) {
if (!r.ok) throw new Error('API error: ' + r.status + ' ' + r.statusText);
return r;
}
```
### Key Changes
| Item | Before | After |
|------|--------|-------|
| URL | WordPress search-proxy | m5api direct |
| API Key | In PHP (hidden) | URL query param (exposed) |
| Mode param | Sent to proxy | Only used for people vs smart routing |
| limit | 20 | 30 |
| Error handling | Silent failure | Explicit throw |
---
## 4. Code Changes: `mapMomentToCard()` — serve_url Support
### Before
```javascript
function mapMomentToCard(m) {
var videoId = m.file_uuid;
var tStart = m.start_time;
var tEnd = m.end_time;
var fps = m.fps;
return {
id: m.id || m.file_uuid,
url: '/wp-json/momentry/v1/media?uuid=' + encodeURIComponent(videoId)
+ '&type=video&start_time=' + encodeURIComponent(tStart)
+ '&end_time=' + encodeURIComponent(tEnd),
thumbnailUrl: buildThumbUrl(videoId, m.start_frame || tStart),
title: m.summary || 'Untitled',
fileUuid: videoId,
startTime: tStart,
endTime: tEnd,
fps: fps,
momentId: m.id
};
}
```
### After
```javascript
function mapMomentToCard(m) {
var videoId = m.file_uuid;
var tStart = m.start_time;
var tEnd = m.end_time;
var fps = m.fps;
// 1. Prefer serve_url (local file, Caddy direct serve)
var videoUrl = m.serve_url || null;
// 2. Fall back to streaming endpoint
if (!videoUrl) {
videoUrl = '/wp-json/momentry/v1/media?uuid=' + encodeURIComponent(videoId)
+ '&type=video&start_time=' + encodeURIComponent(tStart)
+ '&end_time=' + encodeURIComponent(tEnd);
}
return {
id: m.id || m.file_uuid,
url: videoUrl,
thumbnailUrl: buildThumbUrl(videoId, m.start_frame || tStart),
title: m.summary || 'Untitled',
fileUuid: videoId,
startTime: tStart,
endTime: tEnd,
fps: fps,
momentId: m.id,
serveUrl: m.serve_url
};
}
```
Note: `openMM()` and `openVideo()` use `card.url` which is now already set to `serve_url` by `mapMomentToCard()`. No changes needed in those functions.
---
## 5. Thumbnails (No Change)
Thumbnail URL format stays the same:
```
/wp-json/momentry/v1/media?type=thumbnail&uuid={uuid}&frame={frame}
```
Caddy proxy + Momentry Core `media-proxy` endpoint are deployed and verified (`200 image/jpeg`).
---
## 6. Implementation Summary
| # | Task | Location | Change | Depends On |
|---|------|----------|--------|------------|
| 1 | Update `fetchSearchApi()` | post_content ID=523 | Direct call to m5api, api_key query param | None |
| 2 | Update `mapMomentToCard()` | post_content ID=523 | Read `m.serve_url`, use as `url` when present | Task 1 |
| 3 | Add error handling | post_content ID=523 | `checkStatus()` helper | Task 1 |
| 4 | Keep thumbnails | post_content ID=523 | No change needed | None |
| 5 | Update `send()` | post_content ID=523 | Remove mode param for search/smart | Task 1 |
---
## 7. Testing
Open the browser console on search-chat page:
```javascript
// 1. Confirm search returns serve_url
fetch('https://m5api.momentry.ddns.net/api/v1/search/smart?api_key=muser_68600856036340bcafc01930eb4bd839_1774418104_97221b69', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({query: 'gun', limit: 1})
})
.then(r => r.json())
.then(d => console.log('serve_url:', d.results[0]?.serve_url, 'file_name:', d.results[0]?.file_name));
// 2. Test serve_url direct playback
var vid = document.createElement('video');
vid.src = 'https://m5wp.momentry.ddns.net/files/demo/Charade_YouTube_24fps.mp4#t=10,20';
vid.controls = true;
document.body.appendChild(vid);
// 3. Test thumbnail (unchanged)
var img = new Image();
img.onload = () => console.log('Thumbnail OK');
img.onerror = () => console.error('Thumbnail failed');
img.src = '/wp-json/momentry/v1/media?uuid=a6fb22eebefaef17e62af874997c5944&type=thumbnail&frame=0';
```
---
## Architecture Reference
See `DESIGN/VideoPlayback_Architecture_V1.0.md` for Caddyfile configuration and `media-proxy` endpoint details.
---
## Version History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| V1.0 | 2026-06-07 | OpenCode | Initial version — search endpoint migration, serve_url support, thumbnail unchanged |
docs_v1.0/M4_workspace/2026-05-27_charade_pipeline_checklist.md
+242
View File
@@ -0,0 +1,242 @@
---
title: Charade Full Movie Pipeline Checklist
version: 1.0
date: 2026-05-27
author: M5Max48
status: in_progress
---
# Charade Full Movie Pipeline Checklist
**File UUID**: `c3c635e3641da80dde10cc555ffcdda5`
**File Name**: Charade (1963) Cary Grant & Audrey Hepburn | Comedy Mystery Romance Thriller | Full Movie.mp4
**Duration**: 6785 seconds (113 minutes)
**Total Frames**: 169,625
---
## P0: Processor Outputs
### Purpose
原始處理器輸出檔案，存放在 `/Users/accusys/momentry/output_dev/`。這些是後續 ingestion 的資料來源。
### Processor Details
| Processor | Expected Output | Size Estimate | Purpose | Status |
|-----------|-----------------|---------------|---------|--------|
| CUT | `c3c635e3641da80dde10cc555ffcdda5.cut.json` | ~170KB | Scene boundary detection，切割點用於 Rule 3 chunking | ✅ Done |
| YOLO | `c3c635e3641da80dde10cc555ffcdda5.yolo.json` | ~50-80MB | Object detection，每幀的物件類別與位置 | 🔄 Running |
| Face | `c3c635e3641da80dde10cc555ffcdda5.face.json` | ~1.5GB | Face detection + 512-dim embedding (FaceNet CoreML) | 🔄 44% |
| Face Traced | `c3c635e3641da80dde10cc555ffcdda5.face_traced.json` | ~1.2GB | Face tracking，同一人物的連續出現 → trace_id | ⏳ Pending (after Face) |
| OCR | `c3c635e3641da80dde10cc555ffcdda5.ocr.json` | ~50KB | Text recognition from frames | ❌ Skipped |
| Pose | `c3c635e3641da80dde10cc555ffcdda5.pose.json` | ~20MB | Body pose estimation | 🔄 Running |
| ASRX | `c3c635e3641da80dde10cc555ffcdda5.asrx.json` | ~8MB | Speaker diarization，語者分段 | ✅ Done (reuse from public) |
| Visual Chunk | `c3c635e3641da80dde10cc555ffcdda5.visual_chunk.json` | ~60KB | Visual scene chunk metadata | ✅ Done |
| Scene | `c3c635e3641da80dde10cc555ffcdda5.scene.json` | ~300B | Scene list from CUT | ✅ Done |
| Scene Meta | `c3c635e3641da80dde10cc555ffcdda5.scene_meta.json` | ~50KB | Heuristic scene metadata (人物 + 物件統計) | ⏳ Pending |
| Story LLM | `c3c635e3641da80dde10cc555ffcdda5.story_llm.json` | ~800KB | LLM-generated story summaries per chunk | ✅ Done |
| Story Story | `c3c635e3641da80dde10cc555ffcdda5.story_story.json` | ~800KB | Story parent-child relationships | ✅ Done |
| TMDb | `c3c635e3641da80dde10cc555ffcdda5.tmdb.json` | ~5KB | TMDb cast list with face embeddings | ⏳ Pending |
| 5W1H | `c3c635e3641da80dde10cc555ffcdda5.5w1h.json` | ~500KB | 5W1H agent output (who/when/where/what/why/how) | ✅ Done |
### Key Dependencies
- Face Traced 需要 Face 完成後才能執行 (face_traced.json = face.json + tracking)
- Scene Meta 需要 Face + YOLO 完成
- TMDb 需要 Face Traced 完成後執行 matching
---
## P1: Database Records
### Purpose
將 processor outputs 存入 PostgreSQL，供 API query 使用。
### Table Details
| Table | Expected Records | Purpose | Verification Query | Status |
|-------|------------------|---------|-------------------|--------|
| `dev.videos` | 1 row | Video metadata (duration, fps, status) | `SELECT file_uuid, status FROM dev.videos WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | ✅ Registered |
| `dev.monitor_jobs` | 1 row | Processing job state machine | `SELECT uuid, status, completed_processors FROM dev.monitor_jobs WHERE uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | 🔄 Running |
| `dev.pre_chunks` | ~7,000 rows | Raw processor outputs (ASR sentences, YOLO objects, etc.) | `SELECT COUNT(*) FROM dev.pre_chunks WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | ⏳ Pending |
| `dev.face_detections` | ~70,000 rows | Face detection records (每幀每張臉) | `SELECT COUNT(*) FROM dev.face_detections WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | ⏳ Pending |
| `dev.face_detections.embedding` | ~70,000 non-NULL | 512-dim FaceNet embedding (用於 identity matching) | `SELECT COUNT(embedding) FROM dev.face_detections WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | ⏳ Pending |
| `dev.face_detections.trace_id` | ~70,000 non-NULL | Face tracking ID (同一人物跨幀連續出現) | `SELECT COUNT(trace_id) FROM dev.face_detections WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | ⏳ Pending |
| `dev.face_detections.identity_id` | ~50,000 non-NULL | TMDb identity binding (Audrey, Cary, etc.) | `SELECT COUNT(identity_id) FROM dev.face_detections WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | ⏳ Pending |
### Key Points
- `embedding` 必須非 NULL 才能進行 TMDb matching (之前 store_traced_faces.py bug 修復)
- `trace_id` 由 `store_traced_faces.py` 從 face_traced.json 計算
- `identity_id` 由 `match_faces_to_tmdb.py` 計算 (cosine similarity > 0.5)
---
## P2: Chunk Ingestion
### Purpose
將 raw processor outputs 轉換為 searchable chunks，用於 RAG query。
### Chunk Types
| Chunk Type | Expected Count | Purpose | Source | Verification Query | Status |
|------------|----------------|---------|--------|-------------------|--------|
| sentence (Rule 1) | ~1,700 | Sentence-level chunks for text search | ASR output → sentence split | `SELECT COUNT(*) FROM dev.chunk WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5' AND chunk_type = 'sentence'` | ⏳ Pending |
| llm_parent | ~800 | LLM-generated summary parent chunks | Story LLM output | `SELECT COUNT(*) FROM dev.chunk WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5' AND chunk_type = 'llm_parent'` | ⏳ Pending |
| story_parent | ~800 | Story parent chunks (narrative segments) | Story processor | `SELECT COUNT(*) FROM dev.chunk WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5' AND chunk_type = 'story_parent'` | ⏳ Pending |
| story_child | ~1,700 | Story child chunks (linked to sentence) | Story processor | `SELECT COUNT(*) FROM dev.chunk WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5' AND chunk_type = 'story_child'` | ⏳ Pending |
| cut (Rule 3) | ~500 | Scene-level chunks for scene search | CUT output → scene boundaries | `SELECT COUNT(*) FROM dev.chunk WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5' AND chunk_type = 'cut'` | ⏳ Pending |
| trace | ~3,600 | Face trace chunks (identity-centric) | Face Traced output | `SELECT COUNT(*) FROM dev.chunk WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5' AND chunk_type = 'trace'` | ⏳ Pending |
### Ingestion Pipeline
1. **Rule 1**: ASR → sentence split → chunk + embedding → Qdrant
2. **Rule 3**: CUT + ASR → scene chunks → chunk + embedding → Qdrant
3. **Trace**: Face Traced → trace chunks → TKG nodes → Qdrant
### Key Points
- `start_frame` / `end_frame` 必須正確計算 (之前 bug: frame=0)
- Chunks 必須有 `embedding` 才能 search
---
## P3: Vector Embeddings
### Purpose
將 chunks 的 text 轉換為 768-dim embeddings，存入 PostgreSQL + Qdrant，用於 semantic search。
### Embedding Targets
| Target | Expected Count | Model | Purpose | Verification | Status |
|--------|----------------|-------|---------|--------------|--------|
| PostgreSQL `dev.chunk.embedding` | ~5,000 | Gemma-2-9B (768-dim) | Text semantic search | `SELECT COUNT(embedding) FROM dev.chunk WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | ⏳ Pending |
| Qdrant `momentry_dev_rule1_v2` | ~5,000 points | Gemma-2-9B | Fast vector similarity search | `curl -H "api-key: Test3200Test3200Test3200" "http://localhost:6333/collections/momentry_dev_rule1_v2"` | ⏳ Pending |
| Qdrant `_face` collection | ~70,000 points | FaceNet-512 (512-dim) | Face identity search | Face embeddings sync via `sync_face_embeddings()` | ⏳ Pending |
### Embedding Pipeline
1. **Text chunks**: `embeddinggemma_server.py` (port 11436) → 768-dim embedding
2. **Face embeddings**: FaceNet CoreML (from face.json) → 512-dim embedding (已在 P0 產生)
3. **Sync to Qdrant**: `sync_face_embeddings()` function in Rust
### Key Points
- Text embeddings 使用 Gemma-2-9B (local LLM server)
- Face embeddings 使用 FaceNet-512 (CoreML ANE accelerated)
- Qdrant 提供 fast similarity search (cosine similarity)
---
## P4: Identity Binding
### Purpose
將 detected faces 綁定到 TMDb identities (Audrey Hepburn, Cary Grant, etc.)，用於 identity_text search。
### Identity Matching Pipeline
| Step | Expected Result | Method | Verification | Status |
|------|-----------------|--------|--------------|--------|
| TMDb seeds loaded | 23 identities | `tmdb_embed_extractor.py` → TMDb profile face embeddings | `SELECT COUNT(*) FROM dev.identities WHERE source = 'tmdb' AND face_embedding IS NOT NULL` | ✅ Done |
| Face matching | ~50,000 bindings | `match_faces_to_tmdb.py` → cosine similarity > 0.5 | `SELECT COUNT(identity_id) FROM dev.face_detections WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5' AND identity_id IS NOT NULL` | ⏳ Pending |
| Audrey Hepburn faces | ~16,000 | Highest similarity match | `SELECT COUNT(*) FROM dev.face_detections fd JOIN dev.identities i ON fd.identity_id = i.id WHERE fd.file_uuid = 'c3c635e3641da80dde10cc555ffcdda5' AND i.name = 'Audrey Hepburn'` | ⏳ Pending |
| Cary Grant faces | ~5,000 | Second highest match | Same query for Cary Grant | ⏳ Pending |
### Matching Algorithm
```python
# match_faces_to_tmdb.py
for trace_id in traces:
for face_embedding in trace_faces:
for tmdb_identity in tmdb_identities:
similarity = cosine_similarity(face_embedding, tmdb_identity.face_embedding)
if similarity >= 0.5:
match trace_id → tmdb_identity
```
### Key Points
- TMDb seeds 需要 `face_embedding` (之前已驗證: 23 identities with embeddings)
- Face `embedding` 必須非 NULL (之前 store_traced_faces.py bug 修復)
- Threshold: 0.5 (可調整)
---
## P5: API Endpoints
### Purpose
驗證 API endpoints 可以正確返回 identity_text search results。
### API Tests
| Endpoint | Purpose | Expected Response | Test Command | Status |
|----------|---------|-------------------|--------------|--------|
| `/api/v1/search/identity_text` | Search chunk text → identities | Results with `identity_name`, `trace_id`, `identity_source` | `curl "http://localhost:3003/api/v1/search/identity_text?file_uuid=c3c635e3641da80dde10cc555ffcdda5&q=Regina&limit=5"` | ⏳ Pending |
| `/api/v1/identities` | List identities with TMDb | Identity list with `tmdb_id`, `face_embedding` | `curl "http://localhost:3003/api/v1/identities?name=Audrey"` | ⏳ Pending |
| `/api/v1/progress/:file_uuid` | Check processing progress | JSON with `status`, `completed_processors` | `curl "http://localhost:3003/api/v1/progress/c3c635e3641da80dde10cc555ffcdda5"` | ⏳ Pending |
### Expected API Response Example
```json
{
"success": true,
"total": 5,
"results": [
{
"chunk_id": "sentence_123",
"start_time": 355.0,
"text_content": "Oh, mine's Regina Lampert.",
"identity_id": 9,
"identity_name": "Audrey Hepburn",
"identity_source": "tmdb",
"trace_id": 169
}
]
}
```
### Key Points
- `identity_text` API 需要 `chunk.start_frame` / `chunk.end_frame` 正確 (之前 bug: frame=0)
- `identity_id` 必須非 NULL 才能返回 identity_name
---
## P6: Completion Criteria
### Purpose
驗證 pipeline 完整完成，所有 ingestion steps 成功。
### Final Verification Checklist
| Criteria | Purpose | Check Command | Expected Result | Status |
|----------|---------|---------------|-----------------|--------|
| All processor outputs exist | 確認所有 processor JSON 檔案產生 | `ls -la output_dev/c3c635e3641da80dde10cc555ffcdda5.*` | 14+ files with size > 0 | ⏳ Pending |
| Job status = completed | 確認 worker 完成 job | `SELECT status FROM dev.monitor_jobs WHERE uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | `completed` | ⏳ Pending |
| Video status = completed | 確認 video state 更新 | `SELECT status FROM dev.videos WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | `completed` | ⏳ Pending |
| All chunks have embeddings | 確認 text embeddings 完成 | `SELECT COUNT(*) = COUNT(embedding) FROM dev.chunk WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | `true` (all chunks have embedding) | ⏳ Pending |
| Face traces assigned | 確認 face tracking 完成 | `SELECT COUNT(*) = COUNT(trace_id) FROM dev.face_detections WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | `true` (all faces have trace_id) | ⏳ Pending |
| TMDb matching done | 確認 identity binding 完成 | `SELECT COUNT(identity_id) > 40000 FROM dev.face_detections WHERE file_uuid = 'c3c635e3641da80dde10cc555ffcdda5'` | `true` (> 40K identity bindings) | ⏳ Pending |
| Qdrant synced | 確認 vector search ready | Check Qdrant points count | Points increased by ~5,000 | ⏳ Pending |
### Success Thresholds
- **Face detections**: ~70,000 (169K frames / 3 sample interval)
- **Identity bindings**: > 40,000 (60% match rate)
- **Chunks with embeddings**: > 4,000 (all chunk types)
- **Qdrant points**: > 90,000 (current) → > 95,000 (after Charade)
---
## Verification Script
```bash
# Run after completion
./scripts/verify_charade_pipeline.sh c3c635e3641da80dde10cc555ffcdda5
```
---
## Notes
- OCR processor failed, skipped
- Face detection using SwiftFace (ANE accelerated)
- TMDb matching using `scripts/match_faces_to_tmdb.py`
- Expected total processing time: ~2-3 hours
---
## Version History
| Version | Date | Author | Changes |
|---------|------|--------|---------|
| 1.0 | 2026-05-27 | M5Max48 | Initial checklist |
docs_v1.0/M4_workspace/2026-05-29_identity_sync_and_wp_fixes.md
+49
View File
@@ -0,0 +1,49 @@
# Session Summary: Identity Fixes + WP Proxy Fixes + Data Sync
**Date**: 2026-05-29
**Author**: OpenCode
**Status**: Completed (marcom team testing)
## What Was Done (Chronological)
### 1. Production Identity Fixes (3002)
- **James Coburn restored** (id=18738, confirmed)
- **Chantal Goya restored** (id=18737, confirmed)
- **Louis Viret name/status fixed**
- **Sequences fixed**: `identities_id_seq` (48→18734), `face_detections_id_seq` (141383→932413), `identity_history_id_seq`, `identity_bindings_id_seq`, `pre_chunks_id_seq`, `file_identities_id_seq`
- **COALESCE fix** for `reference_data` NULL crash (`postgres_db.rs:3198`, `storage.rs:196`)
### 2. Bug Fixes
- **DELETE identity**: Fixed binding order bug + removed `identity_confidence` column reference
- **PATCH identity**: `jsonb_deep_merge` Nested JSON metadata
- **mergeinto UNDO/REDO**: MongoDB deserialization fix (`Collection<Document>`)
### 3. Library Page Infinite Load Fix
- **Root cause**: WP scan proxy (snippet 48) didn't forward query params → infinite pagination loop
- **Fix**: Added `$request->get_query_params()` forwarding in scan proxy
- **Safety**: Added `maxPages = 10` limit in JS pagination
### 4. Identity Data Sync (Dev → Production)
- **Full replacement** of `public.identities`, `public.identity_bindings`, `public.identity_history` with dev data
- James Coburn id: 18738 → 11
- Bindings: 11,892 → 12,834 (+942)
- **Verification**: 0 differences between schemas
### 5. Snippet 55 Filter
- Added `.filter(f => f.is_registered)` to show only registered files on library page
- Changed `status:'unregistered'` → `status: f.status || 'unregistered'`
## Key Decisions
- Library page filter: default show registered files only
- Identity sync: full DELETE + INSERT (not UPDATE) to ensure consistency
- No user-defined metadata fields (starred/notes/role) preserved — matches dev exactly
## Handoff to Marcom
- `/people/` page should show correct identity state
- `/library/` page should show only registered files (4 currently)
- Login required for `/library/` — redirects to `/login/` if not authenticated
## Files Modified
- `snippet 48` (/scan WP proxy — query param forwarding)
- `snippet 55` (library page JS — registered-only filter, maxPages safety)
- `docs_v1.0/M4_workspace/2026-05-29_identity_sync_prod.md` (sync record)

Some files were not shown because too many files have changed in this diff Show More