1. Add Redis PipelineProgress cleanup in unregister_internal
- Deletes progress key when file is unregistered
2. Fix ingestion_complete asr_status query
- Query ASRX first, then ASR
- Filter out NULL values to avoid false negatives
3. Fix TKG 0 nodes/edges support
- TKG is considered done if face traces are complete
- TKG may create 0 nodes/edges for videos with minimal content
4. Add ASRX fallback to ASR segments
- When ASRX has 0 segments but ASR has segments, use ASR segments
- Ensures at least one ASRX output when ASR has content
When a job completes via the essential_completed branch (all essential
processors done but some non-essential failed), the PipelineProgress
was not being updated to 100%, causing a discrepancy where the status
showed 'completed' but progress showed <100%.
Now PipelineProgress.mark_completed() is called in both completion paths.
1. trace_done now checks for 'no_faces' status in face_traced.json
- Videos with no detected faces now complete correctly
- Previously stuck because trace_count=0 returned false
2. ASRX fallback to ASR segments includes start_frame/end_frame
- Added _convert_asr_segments_to_asrx helper function
- TKG can now process fallback segments correctly
This allows processing of:
- Videos with no audio track (ASR: no_audio_track)
- Videos with no faces (face_traced.json: no_faces)
When a file is re-registered, the old PipelineProgress in Redis
was not cleared, causing the Portal to show 'completed' status
even though the new job was still running.
Now PipelineProgress is deleted when a new job starts processing.
Previously, ingestion_complete only checked ASR and face traces,
causing jobs to complete before TKG was triggered. This resulted in
TKG nodes being 0 even after job completion.
Now ingestion_complete also checks if TKG nodes exist for the file.
The job stays in 'running' status until TKG completes.
The TKG guard was checking if nodes exist before spawning TKG build.
This caused a deadlock when:
1. TKG spawns (async)
2. ingestion_complete checks → false (nodes not yet created)
3. Returns Ok(false), job stays running
4. Next poll: guard checks nodes → no nodes yet (TKG still running)
5. Skips TKG → deadlock!
Since build_tkg uses ON CONFLICT (idempotent), it's safe to call
multiple times. Removed the guard to fix the deadlock.
When a job is deleted from the database (e.g., by unregister),
the worker now checks if the job still exists before processing.
If the job no longer exists, it skips it instead of getting stuck.
This fixes the issue where unregistering a file would leave the
worker stuck trying to process a non-existent job.
Identity agent now checks Qdrant _seeds collection for the file's
seed identity photos before running. If no seeds exist, the agent
is skipped to avoid unnecessary processing.
Flow:
1. Job completes (Face + ASRX done)
2. Check Qdrant _seeds for file_uuid
3. If seeds exist → run identity agent
4. If no seeds → skip identity agent
Bug: ASRX handler stored pre_chunks as BOTH 'asrx' and 'asr' types.
This caused confusion because Rule 1 queries 'asrx' type, but only
'asr' type existed in the database (asrx type was deleted or never stored).
Fix: Remove the duplicate 'asr' storage (lines 530-542).
ASRX handler now only stores 'asrx' type pre_chunks to workspace SQLite.
PostgreSQL pre_chunks are stored by processor.rs with correct 'asrx' type.
This ensures Rule 1 can find ASRX pre_chunks correctly.
Bug: all_completed only checked existing results, not missing processors.
If a processor (like pose) never created a result row, all_completed would
still return true and mark the job as completed.
Fix: all_completed now checks that every processor in job_processors has
a corresponding completed result. Added logging for missing processors.
Also fixed:
- any_pending now checks all expected processors, not just existing results
- Added missing_processors detection and logging
- Skip chunks where both ASRX text and OCR text are empty
- Use count-based chunk_id instead of index to avoid gaps
- This ensures PostgreSQL and Qdrant chunk counts match
- ingestion_complete query used file_uuid column which is always NULL
- Changed to JOIN processor_results with monitor_jobs on job_id
- All stuck jobs now complete successfully
- Add Queued variant to VideoStatus enum
- Trigger sets videos.status='queued' instead of staying 'pending'
- Worker sets videos.status='processing' on pickup
- list_monitor_jobs_by_status ORDER BY created_at ASC (FIFO)
- queue_position counts both 'pending' and 'queued' jobs
- Identity agent: per-face max matching, multi-round with derived
seeds from high-confidence faces, angle diversity filter (cosine sim < 0.90)
- Pending person API: POST /file/:file_uuid/pending-person
+ GET /file/:file_uuid/pending-persons with status=pending, source=manual
- Update API docs (07_identity.md)
- Remove Rule 3 (Scene Chunking) from worker auto-trigger
- Remove rule3_ingest.rs and related imports
- Remove Story/Caption from playground module parsing
- Clean up scan.rs Rule 3 display
- Fix ASRX field name conversion (start_time -> start)
Reason: Story/5W1H/Scene accuracy too poor - will redesign later
- store_vector: stub -> actual PG embedding storage
- search_parent_chunks_semantic: include sentence chunks
- Remove early return in check_and_complete_job
- Python: asr_processor.py detects FPS from CUT/ffprobe (no fallback), outputs start_frame/end_frame
- Rust: All AsrSegment structs use start_time/end_time with #[serde(alias)] for backward compat
- store_asr_chunks: prefers ASR output frames, falls back to time-based conversion
- Added backward compatibility test for old JSON format (start/end)
Breaking change: ffprobe/CUT FPS failure now aborts instead of using default 24fps
Bug fixes from M4 report:
- 4 remaining dev.chunks → dev.chunk in SQL queries
- search_path includes public for pgvector extension
- get_chunk_by_chunk_id_and_uuid: uuid → file_uuid
- New endpoint: GET /api/v1/file/:uuid/chunk/:chunk_id
- New src/verification/ module: verify_output() checks JSON structure/completeness per processor type
- Worker: after processor succeeds, verification agent gates the result
- Passed -> mark completed + cleanup_temp_files (remove .tmp/.partial/.err/timestamp backups)
- Failed -> mark failed with verification details, preserve files for inspection
- cleanup_temp_files() keeps only the canonical {uuid}.{proc}.json
- Auto-run tkg_builder.py after face trace store + Qdrant sync + trace chunks
- Add face-face CO_OCCURS_WITH edges (two traces in same frame)
- docs: TKG integration report for M4
- New trace_ingest module: creates chunks for each face trace (time + bbox + ASR text)
- Computes pairwise time overlaps between traces -> co_appearances in metadata
- Worker auto-triggers after face trace store + Qdrant sync
- SearchFilters: chunk_type filter (sentence/cut/trace/visual)
- SearchFilters: co_appears_with_trace_id filter
- Add helper functions to extract text from nested content structure
- Update SearchResult to include uuid field
- Add PostgreSQL function get_chunk_by_chunk_id_and_uuid to handle duplicate chunk_ids
- Update Qdrant search functions to extract uuid from payload
- Change embedding model to nomic-embed-text-v2-moe:latest
- Update Qdrant collection name to momentry_rule1
- Fix MongoDB authentication and disable cache for development
- Improve error handling in processor.rs
- Update documentation with new embedding model
Accusys