warren/momentry_core
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases 2 Wiki Activity

feat: Initial v0.9 release with API Key authentication

Browse Source 
## v0.9.20260325_144654

### Features
- API Key Authentication System
- Job Worker System
- V2 Backup Versioning

### Bug Fixes
- get_processor_results_by_job column mapping

Co-authored-by: OpenCode
This commit is contained in:
accusys
2026-03-25 14:52:51 +08:00
parent 47e86b696f
commit 383201cacd
193 changed files with 40268 additions and 422 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/N8N_DEMO_WORKFLOW.md
+677
View File
@@ -0,0 +1,677 @@
# n8n Video RAG Workflow - Node 設計
> 建立時間: 2026-03-22
> 目標: 讓 marcom 團隊能夠複製、貼上、修改使用的完整操作指南
---
## 完整 Workflow 架構
```
┌─────────────────────────────────────────────────────────────────────────────┐
│ n8n Workflow: Video RAG Demo │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Phase 1: SFTPGo 準備 (全部在 n8n Node 內執行) │ │
│ │ │ │
│ │ ① Webhook Trigger │ │
│ │ ↓ │ │
│ │ ② Set Variables (解析 file_name, query) │ │
│ │ ↓ │ │
│ │ ③ Get SFTPGo Token │ │
│ │ ↓ │ │
│ │ ④ Upload to SFTPGo │ │
│ │ ↓ │ │
│ │ ⑤ Create Share Link │ │
│ │ ↓ │ │
│ │ ⑥ Verify Upload (List Files + List Shares) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Phase 2: Momentry 註冊 (只處理 ASR, ASRX, STORY) │ │
│ │ │ │
│ │ ⑦ Register Video (modules=asr,asrx,story) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Phase 3: Progress Loop (n8n Logs 記錄) │ │
│ │ │ │
│ │ ⑧ Wait 10s ─────────────────────────────────────────────────┐ │ │
│ │ ↓ │ │
│ │ ⑨ Check Progress (API) │ │
│ │ ↓ │ │
│ │ ⑩ Log Progress (Code Node → n8n Logs) │ │
│ │ ↓ │ │
│ │ ⑪ Is Complete? (IF) │ │
│ │ │ │ │
│ │ ├── NO ──────────────────────────────── Loop Back ─────────┘ │ │
│ │ └── YES ────────────────────────────────────────────── Exit ──┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Phase 4: 搜尋與回應 │ │
│ │ │ │
│ │ ⑫ Hybrid Search (Vector + BM25) │ │
│ │ ↓ │ │
│ │ ⑬ Build Response │ │
│ │ ↓ │ │
│ │ ⑭ Respond to Webhook │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
### 模組說明
| 模組 | 用途 | 輸出 |
|------|------|------|
| `asr` | 語音轉文字 (Whisper) | 字幕/文字稿 |
| `asrx` | 說話者分離 (WhisperX) | 誰在什麼時候說什麼 |
| `story` | 故事線生成 (Parent-Child Chunks) | 敘事結構 + 父子區塊關聯 |
**注意**: 只處理語音和故事相關模組，跳過 YOLO、OCR、Face、Pose 等視覺分析。
┌─────────────────────────────────────────────────────────────────────────────┐
│ n8n Workflow: Video RAG Demo │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Phase 1: SFTPGo 準備 (全部在 n8n Node 內執行) │ │
│ │ │ │
│ │ ① Webhook Trigger │ │
│ │ ↓ │ │
│ │ ② Set Variables (解析 file_name, query) │ │
│ │ ↓ │ │
│ │ ③ Get SFTPGo Token │ │
│ │ ↓ │ │
│ │ ④ Upload to SFTPGo │ │
│ │ ↓ │ │
│ │ ⑤ Create Share Link │ │
│ │ ↓ │ │
│ │ ⑥ Verify Upload (List Files + List Shares) │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Phase 2: Momentry 註冊 │ │
│ │ │ │
│ │ ⑦ Register Video │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Phase 3: Progress Loop (n8n Logs 記錄) │ │
│ │ │ │
│ │ ⑧ Wait 10s ─────────────────────────────────────────────────┐ │ │
│ │ ↓ │ │ │
│ │ ⑨ Check Progress (API) │ │ │
│ │ ↓ │ │ │
│ │ ⑩ Log Progress (Code Node → n8n Logs) │ │ │
│ │ ↓ │ │ │
│ │ ⑪ Is Complete? (IF) │ │ │
│ │ │ │ │ │
│ │ ├── NO ──────────────────────────────── Loop Back ─────────┘ │ │
│ │ └── YES ────────────────────────────────────────────── Exit ──┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Phase 4: 搜尋與回應 │ │
│ │ │ │
│ │ ⑫ Natural Language Search │ │
│ │ ↓ │ │
│ │ ⑬ Get Media URL (含 media_url) │ │
│ │ ↓ │ │
│ │ ⑭ Build Response │ │
│ │ ↓ │ │
│ │ ⑮ Respond to Webhook │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────┘
```
---
## Node 詳細配置
### Node ①: Webhook Trigger (觸發器)
```yaml
Node Name: "Webhook Trigger"
Node Type: "Webhook"
Configuration:
HTTP Method: POST
Path: "video-rag"
Response Mode: "Response Node"
Response Node: "Respond to Webhook"
Input JSON Example:
{
"file_name": "Old_Time_Movie_Show_-_Charade_1963.HD.mov",
"query": "What is the movie about?"
}
```
---
### Node ②: Set Variables (變數設定)
```yaml
Node Name: "Set Variables"
Node Type: "Set"
Configuration:
Keep Only Set: true
Variables:
- Name: "file_name"
Value: "{{ $json.body.file_name }}"
- Name: "query"
Value: "{{ $json.body.query }}"
- Name: "sftpgo_path"
Value: "/{{ $json.body.file_name }}"
- Name: "register_path"
Value: "/Users/accusys/sftpgo_test/demo/{{ $json.body.file_name }}"
```
---
### Node ③: Get SFTPGo Token (取得權杖)
```yaml
Node Name: "Get SFTPGo Token"
Node Type: "HTTP Request"
Configuration:
Method: GET
URL: "http://localhost:8080/api/v2/user/token"
Authentication: "Basic Auth"
User: "demo"
Password: "demopassword123"
Output:
{
"access_token": "eyJhbGci...",
"expires_at": "2026-03-22T07:00:00Z"
}
```
---
### Node ④: Upload to SFTPGo (上傳檔案)
```yaml
Node Name: "Upload to SFTPGo"
Node Type: "HTTP Request"
Configuration:
Method: POST
URL: "http://localhost:8080/api/v2/user/files"
Authentication: "Bearer Token"
Bearer Token: "{{ $json.access_token }}"
Body Content Type: "Form-Data Multipart"
Body:
path: /demo
mkdir_parents: true
filenames: @{{ $json.file_name }}
Output:
{"message":"Upload completed"}
```
**檔案來源選項:**
1. **Webhook 接收**: 從 Webhook 的 binary data 取得
2. **固定路徑**: 指定本地檔案路徑
3. **URL 下載**: 先下載遠端檔案再上傳
---
### Node ⑤: Create Share Link (建立分享連結)
```yaml
Node Name: "Create Share Link"
Node Type: "HTTP Request"
Configuration:
Method: POST
URL: "http://localhost:8080/api/v2/user/shares"
Authentication: "Bearer Token"
Bearer Token: "{{ $json.access_token }}"
Body Content Type: "JSON"
Body:
{
"name": "{{ $json.file_name }}_share",
"paths": ["/{{ $json.file_name }}"],
"scope": 1,
"expires_at": 0
}
Output:
{
"id": "CjmQfrkXY5qDtC46WVZY2S",
"name": "Charade_share"
}
```
---
### Node ⑥: Verify Upload (驗證上傳)
```yaml
Node Name: "Verify Upload - List Shares"
Node Type: "HTTP Request"
Configuration:
Method: GET
URL: "http://localhost:8080/api/v2/user/shares"
Authentication: "Bearer Token"
Bearer Token: "{{ $json.access_token }}"
Output:
[
{
"id": "CjmQfrkXY5qDtC46WVZY2S",
"name": "Charade_share",
"paths": ["/Old_Time_Movie_Show_-_Charade_1963.HD.mov"]
}
]
```
---
### Node ⑦: Register Video (註冊影片)
**說明**: 只註冊 ASR、ASRX、STORY 模組處理
```yaml
Node Name: "Register Video"
Node Type: "HTTP Request"
Configuration:
Method: POST
URL: "http://localhost:3002/api/v1/register"
Body Content Type: "JSON"
Body:
{
"path": "{{ $json.register_path }}",
"modules": "asr,asrx,story"
}
Output:
{
"uuid": "a1b10138a6bbb0cd",
"video_id": 7,
"file_name": "Old_Time_Movie_Show_-_Charade_1963.HD.mov",
"duration": 6879.33,
"width": 1920,
"height": 1080
}
```
**可用模組**:
| 模組 | 說明 |
|------|------|
| `asr` | 語音轉文字 (Whisper) |
| `asrx` | 說話者分離 (WhisperX) |
| `story` | 故事線生成 (Parent-Child) |
| `yolo` | 物體偵測 (可選) |
| `cut` | 場景偵測 (可選) |
| `ocr` | 文字辨識 (可選) |
| `face` | 人臉偵測 (可選) |
| `pose` | 姿態估計 (可選) |
---
### Node ⑧: Wait 10 Seconds (輪詢間隔)
```yaml
Node Name: "Wait 10 Seconds"
Node Type: "Wait"
Configuration:
Amount: 10
Unit: "Seconds"
```
---
### Node ⑨: Check Progress (檢查進度)
```yaml
Node Name: "Check Progress"
Node Type: "HTTP Request"
Configuration:
Method: GET
URL: "http://localhost:3002/api/v1/progress/{{ $('Register Video').item.json.uuid }}"
Output:
{
"uuid": "a1b10138a6bbb0cd",
"processors": [
{"name": "asr", "status": "complete", "message": "1867 segments"},
{"name": "asrx", "status": "progress", "message": "ASRX_TRANSCRIBING"},
{"name": "story", "status": "pending", "message": ""}
]
}
```
**注意**: 只有 asr、asrx、story 三個模組
---
### Node ⑩: Log Progress (記錄進度)
```yaml
Node Name: "Log Progress"
Node Type: "Code"
Configuration:
Language: "JavaScript"
Code:
```javascript
const progress = $input.first().json;
const processors = progress.processors;
const totalProcessors = processors.length;
const completedProcessors = processors.filter(p => p.status === 'complete').length;
const overallProgress = Math.round((completedProcessors / totalProcessors) * 100);
const currentProcessor = processors.find(p =>
p.status === 'progress' || p.status === 'info'
);
const progressMessage = `
═══════════════════════════════════════════════
📹 Video RAG Processing: ${overallProgress}%
UUID: ${progress.uuid}
${processors.map(p => {
const icon = p.status === 'complete' ? '✅' :
p.status === 'progress' || p.status === 'info' ? '🔄' : '⏳';
return ` ${icon} ${p.name.padEnd(6)} ${p.message || p.status}`;
}).join('\n')}
${currentProcessor ? `Current: ${currentProcessor.name}` : 'All complete!'}
═══════════════════════════════════════════════
`.trim();
console.log(progressMessage);
return {
json: {
uuid: progress.uuid,
overall_progress: overallProgress,
completed_processors: completedProcessors,
total_processors: totalProcessors,
current_processor: currentProcessor?.name || 'idle',
processors: processors,
log_message: progressMessage
}
};
```
Output:
{
"uuid": "a1b10138a6bbb0cd",
"overall_progress": 33,
"log_message": "📹 Video RAG Processing: 33%..."
}
```
---
### Node ⑪: Is Complete? (判斷分支)
```yaml
Node Name: "Is Complete?"
Node Type: "IF"
Configuration:
Condition:
$json.processors.every(p => p.status === 'complete')
Connections:
TRUE (完成): → Node ⑫ Natural Language Search
FALSE (未完成): → Node ⑧ Wait 10 Seconds (Loop)
```
---
### Node ⑫: Natural Language Search (RAG 搜尋)
```yaml
Node Name: "Natural Language Search"
Node Type: "HTTP Request"
Configuration:
Method: POST
URL: "http://localhost:3002/api/v1/search"
Body Content Type: "JSON"
Body:
{
"query": "{{ $('Set Variables').item.json.query }}",
"limit": 10,
"uuid": "{{ $('Register Video').item.json.uuid }}"
}
Output:
{
"results": [
{
"uuid": "a1b10138a6bbb0cd",
"chunk_id": "c_001",
"text": "Hello and welcome to the old-time movie show...",
"score": 0.92
}
]
}
```
---
### Node ⑫B: Hybrid Search (Vector + BM25)
**說明**: 使用混合搜尋，結合向量相似度和全文檢索
```yaml
Node Name: "Hybrid Search"
Node Type: "HTTP Request"
Configuration:
Method: POST
URL: "http://localhost:3002/api/v1/search/hybrid"
Body Content Type: "JSON"
Body:
{
"query": "{{ $('Set Variables').item.json.query }}",
"limit": 10,
"uuid": "{{ $('Register Video').item.json.uuid }}",
"vector_weight": 0.7,
"bm25_weight": 0.3
}
Output:
{
"query": "What is the movie about?",
"results": [
{
"uuid": "a1b10138a6bbb0cd",
"chunk_id": "c_001",
"chunk_type": "sentence",
"start_time": 0.0,
"end_time": 5.0,
"text": "Hello and welcome to the old-time movie show...",
"vector_score": 0.85,
"bm25_score": 0.75,
"combined_score": 0.80
}
]
}
```
**權重建議**:
| 查詢類型 | vector_weight | bm25_weight |
|----------|---------------|-------------|
| 主題查詢 | 0.8 | 0.2 |
| 事實查找 | 0.5 | 0.5 |
| 平衡查詢 | 0.7 | 0.3 |
---
### Node ⑬: Get Media URL (取得媒體連結)
```yaml
Node Name: "Get Media URL"
Node Type: "HTTP Request"
Configuration:
Method: POST
URL: "http://localhost:3002/api/v1/n8n/search"
Body Content Type: "JSON"
Body:
{
"query": "{{ $('Set Variables').item.json.query }}",
"limit": 10,
"uuid": "{{ $('Register Video').item.json.uuid }}"
}
Output:
{
"count": 10,
"hits": [
{
"id": "c_001",
"vid": "a1b10138a6bbb0cd",
"text": "Hello and welcome to the old-time movie show...",
"score": 0.92,
"media_url": "https://wp.momentry.ddns.net/Old_Time_Movie_Show_-_Charade_1963.HD.mov"
}
]
}
```
---
### Node ⑭: Build Response (組合結果)
```yaml
Node Name: "Build Response"
Node Type: "Set"
Configuration:
Keep Only Set: true
Variables:
- Name: "ok"
Value: true
- Name: "uuid"
Value: "{{ $('Register Video').item.json.uuid }}"
- Name: "file_name"
Value: "{{ $('Set Variables').item.json.file_name }}"
- Name: "query"
Value: "{{ $('Set Variables').item.json.query }}"
- Name: "count"
Value: "{{ $('Get Media URL').item.json.count }}"
- Name: "results"
Value: "{{ $('Get Media URL').item.json.hits }}"
- Name: "overall_progress"
Value: "{{ $('Log Progress').item.json.overall_progress }}"
```
---
### Node ⑮: Respond to Webhook (回傳結果)
```yaml
Node Name: "Respond to Webhook"
Node Type: "Respond to Webhook"
Configuration:
Respond With: "JSON"
Response Body:
{
"ok": true,
"uuid": "{{ $json.uuid }}",
"file_name": "{{ $json.file_name }}",
"query": "{{ $json.query }}",
"count": {{ $json.count }},
"results": {{ $json.results }},
"overall_progress": {{ $json.overall_progress }},
"message": "Video RAG completed successfully"
}
```
---
## 快速複製所需資訊
### SFTPGo 設定
| 項目 | 值 |
|------|-----|
| API Base | `http://localhost:8080/api/v2` |
| Demo User | `demo` |
| Demo Password | `demopassword123` |
| Demo Home | `/Users/accusys/sftpgo_test/demo` |
| Token Endpoint | `/api/v2/user/token` |
| Upload Endpoint | `/api/v2/user/files` |
| Share Endpoint | `/api/v2/user/shares` |
### Momentry 設定
| 項目 | 值 |
|------|-----|
| API Base | `http://localhost:3002` |
| Register | `POST /api/v1/register` |
| Progress | `GET /api/v1/progress/{uuid}` |
| Search | `POST /api/v1/search` |
| n8n Search | `POST /api/v1/n8n/search` |
| Hybrid Search | `POST /api/v1/search/hybrid` |
| Media Base | `https://wp.momentry.ddns.net` |
### Demo 測試資料
**Charade (1963) Demo Video**
- UUID: `a1b10138a6bbb0cd`
- 位置: `/Users/accusys/test_video/Old_Time_Movie_Show_-_Charade_1963.HD.mov`
- 時長: 6872 秒 (~1.9 小時)
**已處理檔案**:
| 檔案 | 大小 | 內容 |
|------|------|------|
| `asr.json` | 210KB | 1867 語音區段 |
| `cut.json` | 220KB | 1331 場景 |
| `story.json` | 1.8MB | 641 父子區塊 |
| `transcript.txt` | 40KB | 可讀文字稿 |
**Output 目錄**: `/Users/accusys/momentry_core_0.1/output`
---
## 版本歷史
| 日期 | 版本 | 變更 |
|------|------|------|
| 2026-03-22 | v1.0 | 初始建立 |
| 2026-03-22 | v1.1 | 新增 Hybrid Search (Vector + BM25) 節點 |
| 2026-03-22 | v1.2 | 簡化為只處理 ASR、ASRX、STORY 模組 |