docs: update for new architecture (Probe API, Job Worker)
- VIDEO_REGISTRATION.md: Add Probe API reference and comparison table - JOB_WORKER_IMPLEMENTATION_PLAN.md: Update status to implemented - MOMENTRY_CORE_MONITORING.md: Add Job Worker monitoring section - monitor_jobs and processor_results table docs - Worker status check commands - Redis job monitoring commands
This commit is contained in:
accusys
682
docs/JOB_WORKER_IMPLEMENTATION_PLAN.md
Normal file
682
docs/JOB_WORKER_IMPLEMENTATION_PLAN.md
Normal file
@@ -0,0 +1,682 @@
|
||||
# Job Worker 實作計畫
|
||||
|
||||
| 項目 | 內容 |
|
||||
|------|------|
|
||||
| 建立者 | Warren / OpenCode |
|
||||
| 建立時間 | 2026-03-24 |
|
||||
| 文件版本 | V1.1 |
|
||||
| 狀態 | ✅ 已實作 |
|
||||
|
||||
---
|
||||
|
||||
## 版本歷史
|
||||
|
||||
| 版本 | 日期 | 目的 | 操作人 |
|
||||
|------|------|------|--------|
|
||||
| V1.0 | 2026-03-24 | 建立實作計畫 | OpenCode |
|
||||
| V1.1 | 2026-03-25 | 實作完成，更新狀態 | OpenCode |
|
||||
|
||||
---
|
||||
|
||||
## 實作狀態
|
||||
|
||||
### ✅ 已完成
|
||||
|
||||
| 元件 | 檔案 | 狀態 |
|
||||
|------|------|------|
|
||||
| MonitorJob 結構 | `src/core/db/postgres_db.rs` | ✅ |
|
||||
| ProcessorResult 結構 | `src/core/db/postgres_db.rs` | ✅ |
|
||||
| Worker 配置 | `src/worker/config.rs` | ✅ |
|
||||
| Job Worker | `src/worker/job_worker.rs` | ✅ |
|
||||
| Processor Pool | `src/worker/processor.rs` | ✅ |
|
||||
| Worker 模組 | `src/worker/mod.rs` | ✅ |
|
||||
| PostgreSQL 表格 | `monitor_jobs`, `processor_results` | ✅ |
|
||||
| 類型修復 | `i32`, `NaiveDateTime` | ✅ |
|
||||
|
||||
### 待整合
|
||||
|
||||
| 項目 | 說明 |
|
||||
|------|------|
|
||||
| Worker 服務啟動 | 需要加入 launchd plist |
|
||||
| 監控整合 | 需要加入 MOMENTRY_CORE_MONITORING.md |
|
||||
| 備份涵蓋 | 需要確認備份包含新表格 |
|
||||
|
||||
---
|
||||
|
||||
## 1. 設計決策
|
||||
|
||||
### 1.1 確認的設計決策
|
||||
|
||||
| 項目 | 決策 | 理由 |
|
||||
|------|------|------|
|
||||
| 觸發方式 | 輪詢（Job Worker） | 暫無可靠的 API 觸發機制 |
|
||||
| 並行處理 | 最多 2 個 | 可根據 CPU/GPU 能力調整 |
|
||||
| 失敗處理 | 獨立模組，部分完成可接續 | 任何模組失敗都產出狀態記錄 |
|
||||
| Worker 啟動 | 獨立進程 | 隔離、易管理 |
|
||||
| 並行上限調整 | 環境變數 + 預設值 | 靈活、可調整 |
|
||||
| 狀態同步 | PostgreSQL + Redis | 可靠 + 即時 |
|
||||
|
||||
### 1.2 環境變數
|
||||
|
||||
| 變數 | 預設值 | 說明 |
|
||||
|------|--------|------|
|
||||
| `MOMENTRY_MAX_CONCURRENT` | 2 | 最大並行 processor 數 |
|
||||
| `MOMENTRY_POLL_INTERVAL` | 5 | 輪詢間隔（秒） |
|
||||
| `MOMENTRY_WORKER_ENABLED` | true | 是否啟用 worker |
|
||||
|
||||
---
|
||||
|
||||
## 2. 系統架構
|
||||
|
||||
### 2.1 完整流程圖
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ 檔案註冊觸發處理流程 │
|
||||
├─────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ 1. SFTPGo 上傳 │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ 2. Hook 呼叫 Register API │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ 3. Register API │
|
||||
│ ├─► ffprobe 提取 metadata │
|
||||
│ ├─► 寫入 videos 表 │
|
||||
│ └─► 建立 monitor_jobs 記錄 (status=pending) │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ 4. Job Worker (獨立進程，輪詢機制) │
|
||||
│ ├─► 輪詢 pending jobs │
|
||||
│ ├─► 檢查 videos 表 fs_json 決定需要處理什麼 │
|
||||
│ ├─► 並行執行 processors (最多 2 個) │
|
||||
│ └─► 更新 videos, monitor_jobs, processor_results 表 │
|
||||
│ │ │
|
||||
│ ▼ │
|
||||
│ 5. 處理結果 │
|
||||
│ ├─► 更新 videos 表 (fs_json, psql_chunk, qvector_chunk) │
|
||||
│ ├─► 更新 monitor_jobs 表 (status, progress) │
|
||||
│ ├─► 更新 processor_results 表 (每個模組狀態) │
|
||||
│ └─► Redis Pub/Sub 即時進度 │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 2.2 Job Worker 架構
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ Job Worker 架構 │
|
||||
├─────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ PostgreSQL │ ───▶ │ Worker │ ───▶ │ Processor │ │
|
||||
│ │ Job Queue │ │ Loop │ │ Pool │ │
|
||||
│ └─────────────┘ └──────┬──────┘ └──────┬──────┘ │
|
||||
│ │ │ │
|
||||
│ ▼ ▼ │
|
||||
│ ┌─────────────┐ ┌─────────────┐ │
|
||||
│ │ Video State │ │ Processor 1 │ │
|
||||
│ │ Check │ │ (ASR/YOLO) │ │
|
||||
│ └─────────────┘ ├─────────────┤ │
|
||||
│ │ Processor 2 │ │
|
||||
│ │ (CUT/OCR) │ │
|
||||
│ └─────────────┘ │
|
||||
│ │
|
||||
│ Redis ──── Pub/Sub ──── 即時進度 │
|
||||
│ │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 資料庫結構
|
||||
|
||||
### 3.1 Migration 檔案
|
||||
|
||||
**檔案**: `migrations/003_job_worker.sql`
|
||||
|
||||
```sql
|
||||
-- ================================================================
|
||||
-- Migration 003: Job Worker System
|
||||
-- ================================================================
|
||||
|
||||
-- 3.1.1 更新 videos 表
|
||||
ALTER TABLE videos ADD COLUMN IF NOT EXISTS status VARCHAR(20) DEFAULT 'pending';
|
||||
ALTER TABLE videos ADD COLUMN IF NOT EXISTS user_id BIGINT;
|
||||
ALTER TABLE videos ADD COLUMN IF NOT EXISTS job_id INTEGER REFERENCES monitor_jobs(id);
|
||||
|
||||
COMMENT ON COLUMN videos.status IS 'pending, processing, completed, failed';
|
||||
COMMENT ON COLUMN videos.user_id IS 'WordPress user ID';
|
||||
COMMENT ON COLUMN videos.job_id IS 'Associated monitor_jobs ID';
|
||||
|
||||
-- 3.1.2 更新 monitor_jobs 表
|
||||
ALTER TABLE monitor_jobs ADD COLUMN IF NOT EXISTS video_id BIGINT REFERENCES videos(id);
|
||||
ALTER TABLE monitor_jobs ADD COLUMN IF NOT EXISTS user_id BIGINT;
|
||||
ALTER TABLE monitor_jobs ADD COLUMN IF NOT EXISTS processors VARCHAR(20)[];
|
||||
ALTER TABLE monitor_jobs ADD COLUMN IF NOT EXISTS completed_processors VARCHAR(20)[];
|
||||
ALTER TABLE monitor_jobs ADD COLUMN IF NOT EXISTS failed_processors VARCHAR(20)[];
|
||||
|
||||
COMMENT ON COLUMN monitor_jobs.processors IS 'Processors to run: asr, cut, yolo, ocr, face, pose, asrx';
|
||||
COMMENT ON COLUMN monitor_jobs.completed_processors IS 'Successfully completed processors';
|
||||
COMMENT ON COLUMN monitor_jobs.failed_processors IS 'Failed processors';
|
||||
|
||||
-- 3.1.3 新增 processor_results 表
|
||||
CREATE TABLE IF NOT EXISTS processor_results (
|
||||
id SERIAL PRIMARY KEY,
|
||||
job_id INTEGER REFERENCES monitor_jobs(id) ON DELETE CASCADE,
|
||||
video_id BIGINT REFERENCES videos(id) ON DELETE CASCADE,
|
||||
processor VARCHAR(20) NOT NULL,
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'pending',
|
||||
output_path TEXT,
|
||||
started_at TIMESTAMP,
|
||||
completed_at TIMESTAMP,
|
||||
error_message TEXT,
|
||||
progress_total INT DEFAULT 0,
|
||||
progress_current INT DEFAULT 0,
|
||||
last_checkpoint JSONB,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
UNIQUE(job_id, processor)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_processor_results_job ON processor_results(job_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_processor_results_video ON processor_results(video_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_processor_results_status ON processor_results(status);
|
||||
|
||||
COMMENT ON TABLE processor_results IS 'Tracks individual processor execution status';
|
||||
COMMENT ON COLUMN processor_results.status IS 'pending, running, completed, failed, skipped';
|
||||
|
||||
-- 3.1.4 更新 videos 表標記欄位用途
|
||||
COMMENT ON COLUMN videos.fs_video IS 'Video file exists on filesystem';
|
||||
COMMENT ON COLUMN videos.fs_json IS 'All processor JSON files generated';
|
||||
COMMENT ON COLUMN videos.fs_chunks IS 'Chunk files generated';
|
||||
COMMENT ON COLUMN videos.fs_vectors IS 'Vector files generated';
|
||||
COMMENT ON COLUMN videos.psql_chunk IS 'Chunks stored in PostgreSQL';
|
||||
COMMENT ON COLUMN videos.pvector_chunk IS 'Vectors stored in PostgreSQL';
|
||||
COMMENT ON COLUMN videos.qvector_chunk IS 'Vectors stored in Qdrant';
|
||||
```
|
||||
|
||||
### 3.2 表關係圖
|
||||
|
||||
```
|
||||
videos monitor_jobs
|
||||
┌──────────────────────┐ ┌──────────────────────┐
|
||||
│ id (PK) │◄────────│ video_id (FK) │
|
||||
│ uuid │ │ user_id │
|
||||
│ status │ │ processors[] │
|
||||
│ fs_video │ │ completed_processors[]│
|
||||
│ fs_json │ │ failed_processors[] │
|
||||
│ job_id (FK)─────────┼────────►│ status │
|
||||
│ user_id │ │ id (PK) │
|
||||
└──────────────────────┘ └──────────────────────┘
|
||||
│
|
||||
│
|
||||
processor_results
|
||||
┌──────────────────────┐
|
||||
│ job_id (FK) │
|
||||
│ video_id (FK) │
|
||||
│ processor │
|
||||
│ status │
|
||||
│ progress_current │
|
||||
│ last_checkpoint │
|
||||
│ id (PK) │
|
||||
└──────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 模組並行策略
|
||||
|
||||
### 4.1 模組分類
|
||||
|
||||
| 模組 | 資源需求 | 獨立性 | 建議並行 |
|
||||
|------|----------|--------|----------|
|
||||
| ASR | GPU/CPU | 高 | ✅ 可並行 |
|
||||
| CUT | CPU | 高 | ✅ 可並行 |
|
||||
| YOLO | GPU | 中 | ✅ 可並行 |
|
||||
| OCR | GPU/CPU | 高 | ✅ 可並行 |
|
||||
| Face | GPU | 中 | ✅ 可並行 |
|
||||
| Pose | GPU | 中 | ✅ 可並行 |
|
||||
| ASRX | GPU/CPU | 高 | ✅ 可並行 |
|
||||
|
||||
### 4.2 建議並行組合
|
||||
|
||||
| 組合 | 模組 1 | 模組 2 | 說明 |
|
||||
|------|---------|---------|------|
|
||||
| GPU+CPU | YOLO/Pose/Face | ASR/CUT/OCR | 平衡負載 |
|
||||
| 雙GPU | YOLO | Pose | 雙 GPU 卡片 |
|
||||
| 雙CPU | ASR | CUT/OCR | 無 GPU 時 |
|
||||
|
||||
### 4.3 Worker 配置
|
||||
|
||||
```rust
|
||||
// src/worker/config.rs
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct WorkerConfig {
|
||||
pub max_concurrent: usize, // 預設 2
|
||||
pub poll_interval_secs: u64, // 預設 5
|
||||
pub enabled: bool, // 預設 true
|
||||
}
|
||||
|
||||
impl Default for WorkerConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
max_concurrent: 2,
|
||||
poll_interval_secs: 5,
|
||||
enabled: true,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl WorkerConfig {
|
||||
pub fn from_env() -> Self {
|
||||
Self {
|
||||
max_concurrent: std::env::var("MOMENTRY_MAX_CONCURRENT")
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(2),
|
||||
poll_interval_secs: std::env::var("MOMENTRY_POLL_INTERVAL")
|
||||
.ok()
|
||||
.and_then(|v| v.parse().ok())
|
||||
.unwrap_or(5),
|
||||
enabled: std::env::var("MOMENTRY_WORKER_ENABLED")
|
||||
.ok()
|
||||
.map(|v| v != "false")
|
||||
.unwrap_or(true),
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. 失敗處理機制
|
||||
|
||||
### 5.1 設計原則
|
||||
|
||||
```
|
||||
每個模組獨立處理：
|
||||
- 成功 → 產出完整 .json，status=completed
|
||||
- 失敗 → 產出 .json 包含 error 狀態，status=failed
|
||||
- 部分完成 → 可從 checkpoint 繼續，status=running
|
||||
```
|
||||
|
||||
### 5.2 Processor 輸出格式
|
||||
|
||||
```json
|
||||
{
|
||||
"processor": "asr",
|
||||
"status": "completed|failed|partial",
|
||||
"completed_at": "2026-03-24T12:00:00Z",
|
||||
"result": { ... },
|
||||
"error": null,
|
||||
"last_checkpoint": {
|
||||
"frame": 5000,
|
||||
"timestamp": 180.5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 失敗處理流程
|
||||
|
||||
```rust
|
||||
async fn run_processor(&self, module: &str, video: &Video) -> Result<()> {
|
||||
let output_path = self.get_output_path(video, module);
|
||||
|
||||
match self.execute_processor(module, video, &output_path).await {
|
||||
Ok(result) => {
|
||||
// 成功：更新狀態
|
||||
self.db.update_processor_status(job_id, module, "completed").await?;
|
||||
self.publish_progress(job_id, module, 100).await?;
|
||||
}
|
||||
Err(e) => {
|
||||
// 失敗：仍然保存部分結果
|
||||
let partial_result = self.get_partial_result(&output_path);
|
||||
self.db.update_processor_status(job_id, module, "failed").await?;
|
||||
self.db.save_error_message(job_id, module, &e.to_string()).await?;
|
||||
|
||||
// 記錄錯誤但不中斷其他模組
|
||||
tracing::warn!("Processor {} failed: {}", module, e);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. 實作結構
|
||||
|
||||
### 6.1 目錄結構
|
||||
|
||||
```
|
||||
src/
|
||||
├── worker/
|
||||
│ ├── mod.rs # Worker 模組導出
|
||||
│ ├── config.rs # Worker 配置
|
||||
│ ├── worker.rs # Worker 主邏輯
|
||||
│ ├── processor.rs # Processor 執行器
|
||||
│ ├── queue.rs # Job 佇列管理
|
||||
│ └── progress.rs # 進度追蹤
|
||||
├── api/
|
||||
│ └── server.rs # 更新 Register API
|
||||
└── main.rs # 新增 worker 命令
|
||||
```
|
||||
|
||||
### 6.2 核心模組
|
||||
|
||||
#### 6.2.1 Worker Config (`src/worker/config.rs`)
|
||||
|
||||
```rust
|
||||
pub struct WorkerConfig {
|
||||
pub max_concurrent: usize,
|
||||
pub poll_interval_secs: u64,
|
||||
pub enabled: bool,
|
||||
}
|
||||
|
||||
impl WorkerConfig {
|
||||
pub fn from_env() -> Self { ... }
|
||||
}
|
||||
```
|
||||
|
||||
#### 6.2.2 Worker Loop (`src/worker/worker.rs`)
|
||||
|
||||
```rust
|
||||
pub struct JobWorker {
|
||||
db: PostgresDb,
|
||||
redis: RedisCache,
|
||||
config: WorkerConfig,
|
||||
semaphore: Arc<Semaphore>,
|
||||
}
|
||||
|
||||
impl JobWorker {
|
||||
pub async fn run(&self) -> Result<()> {
|
||||
loop {
|
||||
if self.config.enabled {
|
||||
self.process_pending_jobs().await?;
|
||||
}
|
||||
tokio::time::sleep(Duration::from_secs(self.config.poll_interval_secs)).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn process_pending_jobs(&self) -> Result<()> {
|
||||
// 1. 檢查並發數
|
||||
// 2. 取得 pending jobs
|
||||
// 3. 分配給 worker pool
|
||||
// 4. 並行執行 processors
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 6.2.3 Processor Pool (`src/worker/processor.rs`)
|
||||
|
||||
```rust
|
||||
pub struct ProcessorPool {
|
||||
max_concurrent: usize,
|
||||
}
|
||||
|
||||
impl ProcessorPool {
|
||||
pub async fn execute(&self, job: &Job, video: &Video) -> Result<ProcessorResult> {
|
||||
// 根據 videos 表決定需要執行哪些 processor
|
||||
// 並行執行最多 2 個
|
||||
// 處理失敗但不中斷其他 processor
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. API 端點設計
|
||||
|
||||
### 7.1 新增端點
|
||||
|
||||
| 端點 | 方法 | 說明 |
|
||||
|------|------|------|
|
||||
| `/api/v1/jobs` | GET | 列出所有 jobs |
|
||||
| `/api/v1/jobs/:uuid` | GET | 取得特定 job 詳細 |
|
||||
| `/api/v1/jobs/:uuid/retry` | POST | 重試失敗的 processor |
|
||||
| `/api/v1/jobs/:uuid/cancel` | POST | 取消 job |
|
||||
|
||||
### 7.2 端點詳情
|
||||
|
||||
#### GET /api/v1/jobs
|
||||
|
||||
```json
|
||||
Response:
|
||||
{
|
||||
"jobs": [
|
||||
{
|
||||
"id": 1,
|
||||
"uuid": "abc123def456",
|
||||
"status": "running",
|
||||
"progress": 60,
|
||||
"processors": ["asr", "cut", "yolo", "ocr", "face", "pose"],
|
||||
"completed": ["asr", "cut", "yolo"],
|
||||
"failed": []
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
#### GET /api/v1/jobs/:uuid
|
||||
|
||||
```json
|
||||
Response:
|
||||
{
|
||||
"id": 1,
|
||||
"uuid": "abc123def456",
|
||||
"video_id": 10,
|
||||
"status": "running",
|
||||
"processors": {
|
||||
"asr": {"status": "completed", "progress": 100},
|
||||
"cut": {"status": "completed", "progress": 100},
|
||||
"yolo": {"status": "running", "progress": 45, "current": 5000, "total": 11000},
|
||||
"ocr": {"status": "pending"},
|
||||
"face": {"status": "pending"},
|
||||
"pose": {"status": "pending"}
|
||||
},
|
||||
"created_at": "2026-03-24T12:00:00Z",
|
||||
"started_at": "2026-03-24T12:01:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Redis Key 設計
|
||||
|
||||
### 8.1 現有 Key 保持
|
||||
|
||||
```bash
|
||||
momentry:job:{uuid} # Job Hash
|
||||
momentry:job:{uuid}:processor:{name} # Processor Hash
|
||||
momentry:progress:{uuid} # Pub/Sub Channel
|
||||
momentry:jobs:active # Set: 運行中 UUIDs
|
||||
momentry:jobs:completed # Set: 完成 UUIDs
|
||||
momentry:jobs:failed # Set: 失敗 UUIDs
|
||||
```
|
||||
|
||||
### 8.2 進度更新時序
|
||||
|
||||
```
|
||||
Processor 執行
|
||||
│
|
||||
├─► 每秒更新 Redis Hash (即時)
|
||||
│
|
||||
├─► 每 10% 或完成時更新 PostgreSQL (持久)
|
||||
│
|
||||
└─► 失敗時立即更新 PostgreSQL (錯誤記錄)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. 實作順序
|
||||
|
||||
### Phase 1: 資料庫遷移
|
||||
|
||||
| 任務 | 說明 |
|
||||
|------|------|
|
||||
| 1.1 | 建立 `migrations/003_job_worker.sql` |
|
||||
| 1.2 | 更新 `postgres_db.rs` 對應的 struct |
|
||||
| 1.3 | 執行 migration 驗證 |
|
||||
|
||||
### Phase 2: Worker 框架
|
||||
|
||||
| 任務 | 說明 |
|
||||
|------|------|
|
||||
| 2.1 | 建立 `src/worker/mod.rs` |
|
||||
| 2.2 | 建立 `src/worker/config.rs` |
|
||||
| 2.3 | 建立 `src/worker/worker.rs` |
|
||||
| 2.4 | 建立 `src/worker/processor.rs` |
|
||||
|
||||
### Phase 3: Register API 整合
|
||||
|
||||
| 任務 | 說明 |
|
||||
|------|------|
|
||||
| 3.1 | 修改 `src/api/server.rs` 的 register 函數 |
|
||||
| 3.2 | 加入建立 monitor_jobs 的邏輯 |
|
||||
| 3.3 | 更新 videos 表 status 欄位 |
|
||||
|
||||
### Phase 4: Processor 執行
|
||||
|
||||
| 任務 | 說明 |
|
||||
|------|------|
|
||||
| 4.1 | 實作 processor 並行執行（最多 2 個） |
|
||||
| 4.2 | 實作失敗處理（保存部分結果） |
|
||||
| 4.3 | 實作 checkpoint 恢復 |
|
||||
|
||||
### Phase 5: 進度追蹤
|
||||
|
||||
| 任務 | 說明 |
|
||||
|------|------|
|
||||
| 5.1 | Redis Pub/Sub 整合 |
|
||||
| 5.2 | PostgreSQL 定期同步 |
|
||||
| 5.3 | API 進度端點更新 |
|
||||
|
||||
### Phase 6: API 端點
|
||||
|
||||
| 任務 | 說明 |
|
||||
|------|------|
|
||||
| 6.1 | GET /api/v1/jobs |
|
||||
| 6.2 | GET /api/v1/jobs/:uuid |
|
||||
| 6.3 | POST /api/v1/jobs/:uuid/retry |
|
||||
| 6.4 | POST /api/v1/jobs/:uuid/cancel |
|
||||
|
||||
### Phase 7: CLI 命令
|
||||
|
||||
| 任務 | 說明 |
|
||||
|------|------|
|
||||
| 7.1 | `cargo run -- worker` 命令 |
|
||||
| 7.2 | Worker 啟動/停止/狀態顯示 |
|
||||
| 7.3 | launchd plist 設定 |
|
||||
|
||||
### Phase 8: 測試
|
||||
|
||||
| 任務 | 說明 |
|
||||
|------|------|
|
||||
| 8.1 | 單元測試 |
|
||||
| 8.2 | 端到端測試 |
|
||||
| 8.3 | 失敗處理測試 |
|
||||
| 8.4 | 並行執行測試 |
|
||||
|
||||
---
|
||||
|
||||
## 10. CLI 命令
|
||||
|
||||
### 10.1 Worker 命令
|
||||
|
||||
```bash
|
||||
# 啟動 worker
|
||||
cargo run -- worker
|
||||
|
||||
# 顯示 worker 幫助
|
||||
cargo run -- worker --help
|
||||
```
|
||||
|
||||
### 10.2 環境變數
|
||||
|
||||
```bash
|
||||
# Worker 配置
|
||||
export MOMENTRY_MAX_CONCURRENT=2
|
||||
export MOMENTRY_POLL_INTERVAL=5
|
||||
export MOMENTRY_WORKER_ENABLED=true
|
||||
|
||||
# 現有環境變數
|
||||
export DATABASE_URL=postgres://accusys@localhost:5432/momentry
|
||||
export REDIS_URL=redis://:accusys@localhost:6379
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. 預估工時
|
||||
|
||||
| Phase | 任務 | 預估工時 |
|
||||
|-------|------|----------|
|
||||
| 1 | 資料庫遷移 | 2h |
|
||||
| 2 | Worker 框架 | 4h |
|
||||
| 3 | Register API 整合 | 2h |
|
||||
| 4 | Processor 執行 | 4h |
|
||||
| 5 | 進度追蹤 | 2h |
|
||||
| 6 | API 端點 | 3h |
|
||||
| 7 | CLI 命令 | 2h |
|
||||
| 8 | 測試 | 4h |
|
||||
| **總計** | | **23h** |
|
||||
|
||||
---
|
||||
|
||||
## 12. 參考文件
|
||||
|
||||
| 文件 | 用途 |
|
||||
|------|------|
|
||||
| `docs/MOMENTRY_CORE_MONITORING.md` | 監控系統規範 |
|
||||
| `docs/MOMENTRY_CORE_REDIS_KEYS.md` | Redis Key 設計 |
|
||||
| `docs/PROCESSING_PIPELINE.md` | 處理流程 |
|
||||
| `docs/CHUNK_DESIGN.md` | 資料庫設計 |
|
||||
| `docs/API_REFERENCE.md` | API 參考 |
|
||||
|
||||
---
|
||||
|
||||
## 13. 附錄
|
||||
|
||||
### A. 狀態機
|
||||
|
||||
```
|
||||
┌──────────────┐
|
||||
│ PENDING │
|
||||
└──────┬───────┘
|
||||
│ register 後
|
||||
▼
|
||||
┌──────────────┐
|
||||
┌─────▶│ PROCESSING │◀──────┐
|
||||
│ └──────┬───────┘ │
|
||||
│ │ │
|
||||
部分失敗 all completed 全部失敗
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ PARTIAL │ │COMPLETED │ │ FAILED │
|
||||
└──────────┘ └──────────┘ └──────────┘
|
||||
```
|
||||
|
||||
### B. videos 表 status 欄位
|
||||
|
||||
| 值 | 說明 |
|
||||
|------|------|
|
||||
| `pending` | 已註冊，等待處理 |
|
||||
| `processing` | 處理中 |
|
||||
| `completed` | 所有處理完成 |
|
||||
| `failed` | 處理失敗 |
|
||||
|
||||
### C. processor_results 表 status 欄位
|
||||
|
||||
| 值 | 說明 |
|
||||
|------|------|
|
||||
| `pending` | 等待執行 |
|
||||
| `running` | 執行中 |
|
||||
| `completed` | 執行成功 |
|
||||
| `failed` | 執行失敗 |
|
||||
| `skipped` | 跳過（如檔案已存在） |
|
||||
@@ -14,6 +14,7 @@
|
||||
|------|------|------|--------|-----------|
|
||||
| V1.0 | 2026-03-17 | 創建文件 | Warren | OpenCode / MiniMax M2.5 |
|
||||
| V1.1 | 2026-03-25 | 新增可配置 Redis Key Prefix 說明 | Warren | OpenCode / GLM-5 |
|
||||
| V1.2 | 2026-03-25 | 新增 Job Worker 監控、processor_results 表 | Warren | OpenCode / GLM-5 |
|
||||
|
||||
---
|
||||
|
||||
@@ -522,6 +523,33 @@ CREATE INDEX idx_monitor_jobs_status ON monitor_jobs(status);
|
||||
CREATE INDEX idx_monitor_jobs_created_at ON monitor_jobs(created_at);
|
||||
```
|
||||
|
||||
### processor_results 表
|
||||
|
||||
```sql
|
||||
CREATE TABLE processor_results (
|
||||
id SERIAL PRIMARY KEY,
|
||||
job_id INTEGER REFERENCES monitor_jobs(id) ON DELETE CASCADE,
|
||||
video_id BIGINT REFERENCES videos(id),
|
||||
processor VARCHAR(20) NOT NULL,
|
||||
status VARCHAR(20) NOT NULL DEFAULT 'pending',
|
||||
started_at TIMESTAMP,
|
||||
completed_at TIMESTAMP,
|
||||
error_message TEXT,
|
||||
created_at TIMESTAMP DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_processor_results_job ON processor_results(job_id);
|
||||
CREATE INDEX idx_processor_results_video ON processor_results(video_id);
|
||||
CREATE INDEX idx_processor_results_status ON processor_results(status);
|
||||
```
|
||||
|
||||
**processor 狀態值**:
|
||||
- `pending` - 等待處理
|
||||
- `running` - 處理中
|
||||
- `completed` - 已完成
|
||||
- `failed` - 處理失敗
|
||||
- `skipped` - 跳過（已在其他處理中完成）
|
||||
|
||||
---
|
||||
|
||||
## 8. 環境變數
|
||||
@@ -553,8 +581,62 @@ REDIS_PASSWORD=accusys
|
||||
| 6 | SFTPGo API 管理工具 | ✅ 已實作 | 2026-03-22 |
|
||||
| 7 | SFTPGo Hook 自動註冊 | ✅ 已實作 | 2026-03-22 |
|
||||
| 8 | 文檔化監控規範 | ✅ 已實作 | 2026-03-22 |
|
||||
| 9 | 整合測試 | 🔜 待實作 | - |
|
||||
| 10 | 生產環境部署驗證 | ⏳ 待開始 | - |
|
||||
| 9 | Job Worker 輪詢機制 | ✅ 已實作 | 2026-03-25 |
|
||||
| 10 | processor_results 表 | ✅ 已實作 | 2026-03-25 |
|
||||
| 11 | Probe API 端點 | ✅ 已實作 | 2026-03-25 |
|
||||
| 12 | 整合測試 | 🔜 待實作 | - |
|
||||
| 13 | 生產環境部署驗證 | ⏳ 待開始 | - |
|
||||
|
||||
### Job Worker 監控 (2026-03-25 新增)
|
||||
|
||||
**Worker 服務狀態檢查**:
|
||||
```bash
|
||||
# 檢查 Worker 程序
|
||||
ps aux | grep momentry
|
||||
|
||||
# 查看 Worker 日誌
|
||||
tail -f /Users/accusys/momentry/log/worker.log
|
||||
```
|
||||
|
||||
**monitor_jobs 狀態查詢**:
|
||||
```bash
|
||||
# 查看待處理工作
|
||||
psql -U accusys -d momentry -c "SELECT * FROM monitor_jobs WHERE status = 'pending';"
|
||||
|
||||
# 查看執行中工作
|
||||
psql -U accusys -d momentry -c "SELECT * FROM monitor_jobs WHERE status = 'running';"
|
||||
|
||||
# 查看失敗工作
|
||||
psql -U accusys -d momentry -c "SELECT * FROM monitor_jobs WHERE status = 'failed';"
|
||||
```
|
||||
|
||||
**processor_results 狀態查詢**:
|
||||
```bash
|
||||
# 查看特定工作的處理器狀態
|
||||
psql -U accusys -d momentry -c "
|
||||
SELECT pr.*, mj.uuid
|
||||
FROM processor_results pr
|
||||
JOIN monitor_jobs mj ON pr.job_id = mj.id
|
||||
WHERE mj.uuid = 'a1b10138a6bbb0cd';
|
||||
"
|
||||
|
||||
# 查看所有失敗的處理器
|
||||
psql -U accusys -d momentry -c "
|
||||
SELECT pr.processor, COUNT(*) as failures
|
||||
FROM processor_results pr
|
||||
WHERE pr.status = 'failed'
|
||||
GROUP BY pr.processor;
|
||||
"
|
||||
```
|
||||
|
||||
**Redis 工作狀態**:
|
||||
```bash
|
||||
# 查看活躍工作
|
||||
redis-cli SMEMBERS momentry:jobs:active
|
||||
|
||||
# 查看工作詳情
|
||||
redis-cli HGETALL momentry:job:{uuid}
|
||||
```
|
||||
|
||||
### 已完成實作 (2026-03-22)
|
||||
|
||||
|
||||
233
docs/VIDEO_REGISTRATION.md
Normal file
233
docs/VIDEO_REGISTRATION.md
Normal file
@@ -0,0 +1,233 @@
|
||||
# Video Registration
|
||||
|
||||
## 概述
|
||||
|
||||
影片註冊 API (`POST /api/v1/register`) 用於將影片加入 Momentry Core 系統進行處理。
|
||||
|
||||
## 路徑格式
|
||||
|
||||
### 支援的路徑格式
|
||||
|
||||
| 格式 | 範例 | 說明 |
|
||||
|------|------|------|
|
||||
| 相對路徑 | `./demo/video.mp4` | 推薦格式 |
|
||||
| 相對路徑（無 ./） | `demo/video.mp4` | 自動加上 `./` |
|
||||
| 絕對路徑 | `/Users/.../sftpgo/data/demo/video.mp4` | 支援但不推薦 |
|
||||
|
||||
### 路徑結構
|
||||
|
||||
```
|
||||
./username/filepath
|
||||
│ │ │
|
||||
│ │ └── 檔案路徑（可以是多層目錄）
|
||||
│ └── 使用者名稱（SFTPgo 用戶目錄名稱）
|
||||
└── 相對路徑前綴
|
||||
```
|
||||
|
||||
**範例**：
|
||||
- `./demo/video.mp4` → username=`demo`, filepath=`video.mp4`
|
||||
- `./demo/movies/2024/video.mp4` → username=`demo`, filepath=`movies/2024/video.mp4`
|
||||
- `./warren/project1/interview.mp4` → username=`warren`, filepath=`project1/interview.mp4`
|
||||
|
||||
## UUID 計算
|
||||
|
||||
### 計算規則
|
||||
|
||||
```
|
||||
UUID = SHA256(username/filepath)[0:16]
|
||||
```
|
||||
|
||||
**範例**：
|
||||
```rust
|
||||
// 路徑: ./demo/video.mp4
|
||||
// username: "demo"
|
||||
// filepath: "video.mp4"
|
||||
// key: "demo/video.mp4"
|
||||
// UUID: SHA256("demo/video.mp4")[0:16]
|
||||
```
|
||||
|
||||
### 特性
|
||||
|
||||
| 特性 | 說明 |
|
||||
|------|------|
|
||||
| 用戶隔離 | 不同用戶的相同檔名會產生不同 UUID |
|
||||
| 一致性 | 相同相對路徑一定產生相同 UUID |
|
||||
| 遷移安全 | SFTPgo 資料路徑變更後 UUID 保持一致 |
|
||||
|
||||
### 範例
|
||||
|
||||
```rust
|
||||
// 用戶 demo 的影片
|
||||
compute_uuid_from_relative_path("./demo/video.mp4")
|
||||
// → "9760d0820f0cf9a7"
|
||||
|
||||
// 用戶 warren 的相同檔名影片
|
||||
compute_uuid_from_relative_path("./warren/video.mp4")
|
||||
// → "a1b2c3d4e5f6g7h8" (不同的 UUID)
|
||||
```
|
||||
|
||||
## 重複註冊檢查
|
||||
|
||||
### 行為
|
||||
|
||||
1. 系統檢查 UUID 是否已存在於資料庫
|
||||
2. 如果存在，返回 `already_exists: true` 和現有影片資訊
|
||||
3. 如果不存在，創建新的影片記錄
|
||||
|
||||
### API 回應
|
||||
|
||||
**新註冊**：
|
||||
```json
|
||||
{
|
||||
"uuid": "9760d0820f0cf9a7",
|
||||
"video_id": 18,
|
||||
"job_id": 2,
|
||||
"file_name": "video.mp4",
|
||||
"duration": 159.637188,
|
||||
"width": 640,
|
||||
"height": 360,
|
||||
"already_exists": false
|
||||
}
|
||||
```
|
||||
|
||||
**重複註冊**：
|
||||
```json
|
||||
{
|
||||
"uuid": "9760d0820f0cf9a7",
|
||||
"video_id": 18,
|
||||
"job_id": 2,
|
||||
"file_name": "video.mp4",
|
||||
"duration": 159.637188,
|
||||
"width": 640,
|
||||
"height": 360,
|
||||
"already_exists": true
|
||||
}
|
||||
```
|
||||
|
||||
## SFTPgo 整合
|
||||
|
||||
### 目錄結構
|
||||
|
||||
SFTPgo 的用戶目錄結構：
|
||||
|
||||
```
|
||||
/Users/accusys/momentry/var/sftpgo/data/
|
||||
├── demo/ ← 用戶目錄
|
||||
│ ├── video.mp4
|
||||
│ └── movies/
|
||||
│ └── movie1.mp4
|
||||
├── warren/ ← 用戶目錄
|
||||
│ └── project1/
|
||||
│ └── interview.mp4
|
||||
└── momentry/ ← 用戶目錄
|
||||
└── presentation.mp4
|
||||
```
|
||||
|
||||
### 註冊流程
|
||||
|
||||
1. SFTPgo 用戶上傳檔案到各自的目錄
|
||||
2. n8n 或其他服務調用註冊 API
|
||||
3. 使用相對路徑格式：`./username/filepath`
|
||||
4. 系統計算 UUID 並檢查重複
|
||||
5. 創建處理任務
|
||||
|
||||
## 程式碼範例
|
||||
|
||||
### 註冊影片
|
||||
|
||||
```bash
|
||||
# 使用相對路徑註冊
|
||||
curl -X POST http://localhost:3002/api/v1/register \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"path": "./demo/video.mp4"}'
|
||||
|
||||
# 或使用多層目錄
|
||||
curl -X POST http://localhost:3002/api/v1/register \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"path": "./demo/movies/2024/video.mp4"}'
|
||||
```
|
||||
|
||||
### UUID 計算函數
|
||||
|
||||
```rust
|
||||
// 使用相對路徑計算 UUID
|
||||
pub fn compute_uuid_from_relative_path(relative_path: &str) -> String {
|
||||
let (username, filepath) = extract_user_from_relative_path(relative_path);
|
||||
compute_uuid(&username, &filepath)
|
||||
}
|
||||
|
||||
// 從相對路徑提取用戶名和檔案路徑
|
||||
pub fn extract_user_from_relative_path(relative_path: &str) -> (String, String) {
|
||||
let path = relative_path.strip_prefix("./").unwrap_or(relative_path);
|
||||
let path_buf = PathBuf::from(path);
|
||||
|
||||
let mut components = path_buf.components();
|
||||
let username = components
|
||||
.next()
|
||||
.map(|c| c.as_os_str().to_string_lossy().to_string())
|
||||
.unwrap_or_default();
|
||||
|
||||
let filepath: String = components
|
||||
.map(|c| c.as_os_str().to_string_lossy().to_string())
|
||||
.collect::<Vec<_>>()
|
||||
.join("/");
|
||||
|
||||
(username, filepath)
|
||||
}
|
||||
```
|
||||
|
||||
## 相關 API
|
||||
|
||||
### Probe API（僅探測，不註冊）
|
||||
|
||||
如果只需要取得影片資訊而不註冊，可以使用 Probe API：
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:3002/api/v1/probe \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"path": "./demo/video.mp4"}'
|
||||
```
|
||||
|
||||
**回應範例**：
|
||||
```json
|
||||
{
|
||||
"uuid": "a1b10138a6bbb0cd",
|
||||
"file_name": "video.mp4",
|
||||
"duration": 120.5,
|
||||
"width": 1920,
|
||||
"height": 1080,
|
||||
"fps": 30.0,
|
||||
"cached": false,
|
||||
"format": {...},
|
||||
"streams": [...]
|
||||
}
|
||||
```
|
||||
|
||||
**與 Register API 的差異**：
|
||||
|
||||
| 功能 | Probe API | Register API |
|
||||
|------|-----------|---------------|
|
||||
| 計算 UUID | ✓ | ✓ |
|
||||
| 執行 ffprobe | ✓ | ✓ |
|
||||
| 儲存 probe.json | ✓ | ✓ |
|
||||
| 寫入 videos 表 | ✗ | ✓ |
|
||||
| 建立 monitor_job | ✗ | ✓ |
|
||||
| 返回 job_id | ✗ | ✓ |
|
||||
| 適用場景 | 預覽影片資訊 | 註冊並處理影片 |
|
||||
|
||||
## 相關檔案
|
||||
|
||||
| 檔案 | 說明 |
|
||||
|------|------|
|
||||
| `src/core/storage/uuid.rs` | UUID 計算邏輯 |
|
||||
| `src/api/server.rs` | 註冊與 Probe API 實現 |
|
||||
| `src/core/probe/ffprobe.rs` | ffprobe 整合 |
|
||||
| `docs/SFTPGO_DEMO_USER.md` | SFTPgo 用戶設置 |
|
||||
| `docs/API_ENDPOINTS.md` | API 端點總覽 |
|
||||
|
||||
## 歷史
|
||||
|
||||
| 日期 | 變更 |
|
||||
|------|------|
|
||||
| 2026-03-25 | 初始版本 - 新增 UUID 計算規則和重複註冊檢查 |
|
||||
| 2026-03-25 | 新增 Probe API 說明 |
|
||||
Reference in New Issue
Block a user