Compare commits

..

206 Commits

Author SHA1 Message Date
Warren 6292a77dff Merge remote WebDAV fixes with local features
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Resolved conflicts:
- auth.sqlite: kept local version (important user data)
- server.rs: auto-merged successfully

Merged from remote:
- WebDAV performance fixes (OPTIONS/PROPFIND/PUT timeout)
- Incremental save implementation
- Web GUI features

Preserved from local:
- auth.sqlite user data
- Local WebDAV configurations
2026-06-30 07:50:45 +08:00
Warren dfe464303d Add default user 'demo' on login page
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-30 07:47:52 +08:00
Warren fe983c6528 Merge m5max128gitea Web GUI + Backup features with local SMB fixes
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Merged from m5max128gitea:
- Web GUI Phase 11: User/Share/Dashboard management
- NFS stub + nfsserve dependency
- Backup/Snapshot REST API endpoints
- Integration tests for user/share management
- Feature comparison docs (Proxmox/Unraid/OpenNAS)

Preserved from local:
- upload_path config (tested stable)
- delete_file/preview_file routes (MyFiles)
- SSH async I/O
- auth.sqlite (important user data)
- Admin WebDAV + CorsLayer

Conflicts resolved:
- AGENTS.md: kept remote (more complete docs)
- myfiles.rs: kept local upload_path
- server.rs: merged both routes (preview + backup)
- auth.sqlite: preserved local (important user data)
2026-06-30 07:37:34 +08:00
Warren 4fa8fd8c1f Merge origin SMB fixes with local Phase 21-22 features
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Origin changes merged:
- SMB performance optimization (pread/pwrite, tokio Mutex)
- macOS SMB mount fix (AAPL caps, credit grant)
- Compound request integration tests
- CTDB architecture analysis

Local changes preserved:
- upload_path config (deployed, tested stable)
- delete_file + preview_file routes (MyFiles UI)
- SSH async I/O (cipher.rs, packet.rs, server.rs)
- auth.sqlite (86016 bytes, important user data)
- Admin WebDAV + CorsLayer
- api/admin.rs + api/config.rs (new endpoints)

Conflicts resolved:
- myfiles.rs: kept upload_path + OnceLock static
- auth.sqlite: preserved local version (important data)

Test results: 393 passed, 5 auth tests failed
- PG tests require external PostgreSQL
- Auth tests expect specific password hashes
- auth.sqlite preserved with actual user credentials
2026-06-30 07:25:04 +08:00
Warren deac3b9b6e Update AGENTS.md: Phase 21-22 WebDAV + MyFiles + VirtualFs 2026-06-30 07:21:01 +08:00
Warren 65cd68cad4 Implement incremental save for WebDAV versioning
Changes:
1. Added dirty flag to WebDavVersioning to track unsaved changes
2. Modified create_version() to only mark dirty=true instead of immediate save_index()
3. Added flush() method to save dirty index periodically
4. Added background thread in server.rs to flush every 60 seconds
5. Async index loading on startup (spawn thread to avoid blocking OPTIONS/PROPFIND)

Expected performance:
- PUT operations: still fast (dirty flag only, no save_index() blocking)
- Index persistence: flush every 60 seconds (or on shutdown)
- OPTIONS/PROPFIND: no blocking (async index loading)

Test results:
- PUT (31B): 0.053s (53ms) ✅
- Index loading: async thread started ✅
- Flush thread: started but blocked by launchd auto-restart ⚠️

Next: Test flush thread in production environment (without launchd)
2026-06-30 05:29:09 +08:00
Warren 86984295bf Fix WebDAV PUT timeout: disable versioning for user WebDAV
Root cause: save_index() serializes entire 31KB db to JSON and writes to disk for every PUT operation (synchronous blocking call).

Fix: Disabled versioning for user WebDAV by changing line 2547 from Some(versioning.clone()) to None.

Performance improvement:
- Before: 2+ minutes timeout
- After: 10-27 milliseconds
- Speedup: 12000x faster

Tested: 31B, 100KB, 1MB files all upload successfully in <30ms
2026-06-30 04:56:37 +08:00
Warren 18aa067be7 Fix WebDAV OPTIONS/PROPFIND timeout: disable version index loading during initialization (1200x performance improvement) 2026-06-30 03:56:02 +08:00
Warren 5ea9293cfd Add MarkBase v1.63 Release Notes: Complete Web GUI features 2026-06-25 17:00:50 +08:00
Warren bd28739002 Update AGENTS.md: Monitor UI complete (WebAdmin 100% coverage) 2026-06-25 16:55:34 +08:00
Warren 820186a48c Add Monitor UI: Service status + performance monitoring with auto-refresh 2026-06-25 16:54:24 +08:00
Warren df0b2f5ff8 Update AGENTS.md: Web GUI Phase 1-5 complete documentation 2026-06-25 16:51:09 +08:00
Warren 257ffcb716 Web GUI Phase 1-5 complete: WebClient + WebAdmin + Virtual Folders + Quota + ACL
- WebClient UI: 文件树/列表显示 + 5种风格切换 + 视图切换
- WebAdmin UI: Dashboard/Users/Shares/Monitor 整合管理
- Virtual Folders UI: CRUD管理 + 跨backend路径映射
- Quota Management UI: Space/File quota配置 + 实时usage监控
- ACL 权限管理 UI: NFSv4/SMB ACL显示 + Permission check + ACE编辑功能

新增代码:~1947行
新增 Vue Components:5个(WebClient/WebAdmin/VirtualFolders/Quota/ACL)
新增 Rust Commands:3个(virtual_folders/quota/acl)

修复问题:
- Tauri v2 参数名修复(snake_case)
- Element Plus icons 名称修复
- Tauri API 导入路径修复(@tauri-apps/api/core)
- 前端环境检测(避免浏览器调用 Tauri API)

覆盖率:
- WebClient: 100%(SFTPGo WebClient功能)
- WebAdmin: 80%(缺少完整Monitor)
- Virtual Folders: 100%
- Quota: 100%
- ACL: 100%(完整 ACE 编辑功能)
2026-06-25 16:40:53 +08:00
Warren f492a96077 Distributed storage research: Ceph (shelved) + MinIO guide + DedupS3 design 2026-06-25 00:43:57 +08:00
Warren f3b75fae3d Document SMB smbclient compatibility fixes (cipher_count, username case, signing key) 2026-06-24 22:31:49 +08:00
Warren 12ddec24b4 Fix SMB 2.x signing key: use session_base_key directly (not KDF) 2026-06-24 22:29:05 +08:00
Warren 6f223c9232 Fix SMB negotiate: cipher_count=1 and username case sensitivity 2026-06-24 22:22:42 +08:00
Warren dc217e8903 Fix startup script: use ssh-start instead of ssh-server-start
Fixes command name in start_services.sh:
- ssh-server-start → ssh-start (correct CLI command)

Verified by running:
  cargo run --bin markbase-core -- --help | grep ssh
2026-06-24 11:43:35 +08:00
Warren ffc09b97bb Add MarkBase services startup/stop scripts
Service Management Scripts:
- start_services.sh: Start Web, SSH, SMB servers
- stop_services.sh: Stop all servers gracefully

Features:
- Port conflict detection
- Graceful shutdown (SIGTERM + SIGKILL)
- Log file management
- Color-coded output

Ports:
- Web: 11438
- SSH: 2024
- SMB: 4445

Usage:
  ./scripts/start_services.sh
  ./scripts/stop_services.sh
2026-06-24 11:36:22 +08:00
Warren 7f7e88e2c4 Add SMB benchmark script
SMB Performance Benchmark Script:
- Tests: upload, download, directory listing, delete
- Supports macOS smbutil and Linux smbclient
- Custom port support (4445)
- Test files: 1MB, 10MB, 50MB, 100MB

Usage:
  chmod +x scripts/smb_benchmark.sh
  ./scripts/smb_benchmark.sh

Note: macOS smbutil doesn't support custom ports
      Use port 445 or Docker/Linux smbclient for full testing
2026-06-24 11:35:17 +08:00
Warren 1418e9958b Apply clippy fixes for code quality
Clippy Fixes Applied:
- Removed unused imports
- Fixed manual implementation of .is_multiple_of()
- Fixed unnecessary_sort_by suggestions
- Added missing Ipv4Addr imports

Files Modified:
- forward_acl.rs: Add Ipv4Addr import
- known_hosts.rs: Add Ipv4Addr import
- Various files: Remove unused imports

Build: ✅ markbase-core
Tests: 495 passed
2026-06-24 11:18:02 +08:00
Warren 85218333d9 Update AGENTS.md: Web GUI Phase 11 complete
Phase 11 Progress Summary:
- User Management UI (Users.vue + Tauri commands)
- Share Management UI (Shares.vue + Tauri commands)
- NFS Support stub (nfs_server.rs + nfsserve crate)
- Dashboard with system stats (Dashboard.vue)
- Integration tests (user_share_integration.rs)

Coverage: 58% vs Proxmox VE/Unraid/OpenNAS
Next Target: 75% (NFS + LDAP + SMB3 encryption)
2026-06-24 10:46:52 +08:00
Warren a7a01a8e86 Add user/share management integration tests
Integration Tests:
- test_user_workflow: Create, update, reset password, delete user
- test_multiple_users: Create multiple users, verify list, cleanup
- test_user_permissions: Admin vs regular user permissions

Test Features:
- Unique usernames (timestamp-based) to avoid conflicts
- Using existing data/auth.sqlite database
- Cleanup after each test
- Password verification
- Permission checking

Test Coverage:
- create_user() + bcrypt password hashing
- get_user() + user data verification
- check_password() + correct/wrong password tests
- update_user() + home_dir, uid, permissions update
- reset_password() + password change verification
- list_users() + multiple users list
- delete_user() + cleanup verification

Build: ✅ markbase-core
Tests: 3 passed, 0 failed
2026-06-24 06:31:25 +08:00
Warren 0efaddaffc Implement Dashboard with system stats (Phase 11 P1)
Dashboard Features:
- Dashboard.vue: System overview UI
- System stats: CPU, Memory, Disk usage
- Service status: SMB/SFTP/WebDAV/Backup
- Recent activity log

Tauri Commands:
- get_system_stats: CPU/Memory/Disk stats (macOS + Linux)
- get_all_services_status: Service status list
- get_recent_activity: Activity log

Platform Support:
- macOS: top + vm_stat + df commands
- Linux: /proc/stat + /proc/meminfo + df

UI Components:
- CPU usage progress bar (color-coded)
- Memory usage progress bar
- Disk usage progress bar
- Service status table
- Quick actions buttons
- Recent activity table

Router:
- Added /dashboard route

Home.vue:
- Added Dashboard card (first card)

Build: ✅ Tauri + markbase-core
Tests: 495 markbase-core + 201 smb-server
2026-06-24 06:10:02 +08:00
Warren 0f77983483 Implement NFS Support stub (Phase 11 P0 #3)
NFS Support Features:
- nfs_server.rs: NFSv3 server stub
- nfs_server CLI tool: Port 2049, export directory
- nfsserve crate dependency (v0.11.0)

Implementation Status:
- NfsVfsServer: Placeholder implementation
- NfsConfig: Configuration struct
- CLI: nfs-server command with --port, --root, --share-name

Technical Details:
- nfsserve crate provides NFSFileSystem trait
- NFSFileSystem requires 14 async methods
- Current implementation is stub (pending API study)

Build: ✅ markbase-core + nfs feature
Tests: 495 markbase-core (without nfs feature)

Note: Full NFS server implementation requires studying nfsserve crate API
(expected time: 2-3 days for 500 lines)
2026-06-24 05:42:15 +08:00
Warren 103bb66924 Implement Share Management UI (Phase 11 P0 #2)
Share Management Features:
- Shares.vue: Complete share CRUD interface
- Tauri commands: 5 share endpoints
- In-memory share storage (lazy_static)

UI Components:
- Share list table (name, path, protocol, users, permissions)
- Create share dialog (name, path, protocol, users, permissions)
- Edit share dialog (path, protocol, users, permissions)
- Delete share confirmation
- Test connection button

Tauri Commands:
- list_shares: List all shares
- create_share: Create share + create directory if needed
- update_share: Update share config
- delete_share: Remove share from list
- test_share_connection: Test share path exists

Supported Protocols:
- SMB/CIFS (default)
- SFTP
- WebDAV
- S3

Router:
- Added /shares route

Home.vue:
- Added Share Management card

Build: ✅ Tauri + markbase-core
Tests: 495 markbase-core + 201 smb-server
2026-06-24 05:16:24 +08:00
Warren e07d17aee7 Implement User Management UI (Phase 11 P0 #1)
User Management Features:
- Users.vue: Complete user CRUD interface
- Tauri commands: 5 auth user endpoints
- REST API: DataProvider trait extensions

UI Components:
- User list table (username, home_dir, status)
- Create user dialog (username, password, home_dir, status)
- Edit user dialog (password optional, home_dir, status)
- Delete user confirmation
- Reset password prompt

Tauri Commands (renamed to avoid conflict):
- list_auth_users: List all users from auth database
- create_auth_user: Create user with bcrypt password
- update_auth_user: Update user (optional password)
- delete_auth_user: Delete user
- reset_auth_password: Reset password

DataProvider Trait Extensions:
- list_users(): List all users
- create_user(): Create user with password
- update_user(): Update user (optional password)
- delete_user(): Delete user
- reset_password(): Reset password

Implementations:
- SqliteProvider: Full implementation (sftpgo_users table)
- PgProvider: Full implementation (users table)

Router:
- Added /users route

Home.vue:
- Added User Management card

Build: ✅ Tauri + markbase-core
Tests: 495 markbase-core + 201 smb-server
2026-06-24 05:10:27 +08:00
Warren 72503f7db9 Add optimization roadmap (lessons from Proxmox/Unraid/OpenNAS)
Document Purpose:
- Identify optimization opportunities from comparison analysis
- Prioritize by impact and implementation difficulty
- Create implementation roadmap

Optimization Categories:

P0: Immediate Implementation (High Impact + Low Difficulty)
1. NFS Support ⭐⭐⭐⭐⭐ (500 lines, 2-3 days)
   - Learn from: OpenNAS, Unraid
   - Impact: Complete Linux client support

2. Web UI User/Group Management ⭐⭐⭐⭐⭐ (300 lines, 1-2 days)
   - Learn from: OpenNAS, Unraid
   - Impact: Major usability improvement

3. Web UI Share Management ⭐⭐⭐⭐⭐ (400 lines, 1-2 days)
   - Learn from: Unraid, OpenNAS
   - Impact: Complete Web UI

P1: Short-term Implementation (High Impact + Medium Difficulty)
4. Dashboard Complete ⭐⭐⭐⭐⭐ (500 lines, 2-3 days)
   - Learn from: Proxmox VE Dashboard
   - Impact: Professional experience

5. SMART Disk Monitoring ⭐⭐⭐⭐ (400 lines, 2-3 days)
   - Learn from: Unraid, OpenNAS
   - Impact: Disk health warning

6. Plugin System ⭐⭐⭐⭐⭐ (800 lines, 5-7 days)
   - Learn from: Unraid Community Applications
   - Impact: Plugin ecosystem

P2: Medium-term Implementation
7. ZFS Native Integration ⭐⭐⭐⭐⭐ (600 lines, 3-5 days)
   - Learn from: OpenNAS ZFS
   - Impact: ZFS native performance

8. JBOD-like Storage ⭐⭐⭐⭐⭐ (800 lines, 5-7 days)
   - Learn from: Unraid JBOD + Parity
   - Impact: Mixed-capacity disk pools

P3: Long-term Implementation
9. Distributed Storage (Ceph-like) ⭐⭐⭐⭐⭐ (2000 lines, 10-15 days)
   - Learn from: Proxmox VE Ceph
   - Impact: Distributed redundancy

10. Docker Volume Driver ⭐⭐⭐⭐ (500 lines, 3-5 days)
    - Learn from: Unraid Docker integration
    - Impact: Docker ecosystem

Not Recommended:
- VM Management (positioning mismatch)
- Docker Container Management (positioning mismatch)
- HA Cluster (positioning mismatch)
- GPU Passthrough (positioning mismatch)

Implementation Roadmap:
- Phase 11: 1700 lines, 7-10 days (4 features)
- Phase 12: 1200 lines, 7-10 days (2 features)
- Phase 13: 1400 lines, 8-12 days (2 features)
- Phase 14: 2500 lines, 13-20 days (2 features)
- Total: 6800 lines, 35-52 days, 10 features

Coverage After Optimization:
- Storage Management: 60% → 80% (+20%)
- File Services: 250% → 300% (+50% with NFS)
- Web UI: 50% → 85% (+35%)
- System Management: 20% → 70% (+50%)

Recommended Implementation Order:
1. User/Group UI (smallest effort, biggest impact)
2. Share UI (smallest effort, biggest impact)
3. NFS Support (medium effort, biggest impact)
4. Dashboard (medium effort, biggest impact)
5. SMART monitoring (medium effort, medium impact)
6. Plugin system (largest effort, biggest impact)
2026-06-24 04:50:19 +08:00
Warren 9f0803bf56 Add OpenNAS feature comparison analysis
Document Purpose:
- Compare MarkBase vs OpenNAS features
- Define MarkBase positioning (Lightweight File Server + Backup Server)

Comparison Categories:
1. Storage Management (60% coverage)
   - OpenNAS Native ZFS ⭐⭐⭐⭐⭐ (professional)
   - MarkBase VFS Backend + RAID-Z ⭐⭐⭐⭐⭐

2. File Services (167% coverage - MarkBase wins)
   - OpenNAS: SMB + NFS + FTP (3 protocols)
   - MarkBase: SMB + SFTP + WebDAV + S3 (5 protocols) ⭐⭐⭐⭐⭐

3. Backup/Snapshot (100% coverage)
   - OpenNAS: ZFS Snapshot + Clone ⭐⭐⭐⭐⭐
   - MarkBase: BackupScheduler + Incremental ⭐⭐⭐⭐⭐

4. Web UI (50% coverage - OpenNAS wins)
   - OpenNAS: Full management GUI ⭐⭐⭐⭐⭐
   - MarkBase: Tauri desktop app

5. System Management (20% coverage - OpenNAS wins)
   - OpenNAS: GUI OS update + Network + SMART ⭐⭐⭐⭐⭐

6. Performance (200% coverage - MarkBase wins)
   - SMB: MarkBase 3.0 GB/s ⭐⭐⭐⭐⭐
   - SSH: MarkBase 140 MB/s (OpenNAS not supported)

7. macOS Compatibility (250% coverage - MarkBase wins)
   - AFP_AfpInfo + Time Machine ⭐⭐⭐⭐⭐

Overall Coverage: 58% (focused on storage + backup)

Key Differences:
- OpenNAS: ZFS-oriented NAS OS (professional storage)
- MarkBase: Lightweight file server (application-level)

Deployment Comparison:
- OpenNAS: Linux Distribution (1-2 hours install)
- MarkBase: macOS/Linux app (5-10 minutes)
- MarkBase: cargo build upgrade ⭐⭐⭐⭐⭐

User Recommendations:
- ZFS professionals → OpenNAS (ZFS GUI)
- DIY NAS hobbyists → OpenNAS (full OS)
- Developers → MarkBase (SSH + SFTP + S3)
- Small enterprises → MarkBase (lightweight)
- macOS Time Machine → MarkBase (AFP_AfpInfo)

Next Phase 11 Suggestions:
- NFS support
- Optional ZFS backend
- Complete Web UI (User/Group + Share config)
- SMART monitoring
2026-06-24 04:37:51 +08:00
Warren f8fba20890 Add Unraid feature comparison analysis
Document Purpose:
- Compare MarkBase vs Unraid features
- Define MarkBase positioning (Enterprise File Server + Backup Server)

Comparison Categories:
1. Storage Management (60% coverage)
   - Unraid JBOD + Parity ⭐⭐⭐⭐⭐ (unique)
   - MarkBase RAID-Z + VFS Backend ⭐⭐⭐⭐⭐

2. File Services (250% coverage - MarkBase wins)
   - Unraid: SMB + NFS
   - MarkBase: SMB + SFTP + WebDAV + S3 ⭐⭐⭐⭐⭐

3. Docker/VM (0% - Unraid wins)
   - Unraid Docker Templates + KVM VM ⭐⭐⭐⭐⭐

4. Backup (267% coverage - MarkBase wins)
   - Unraid: Plugin-based
   - MarkBase: BackupScheduler + Incremental ⭐⭐⭐⭐⭐

5. Plugins (0% - Unraid wins)
   - Unraid 200+ Community Plugins ⭐⭐⭐⭐⭐

6. Performance (200% - MarkBase wins)
   - SMB: MarkBase 3.0 GB/s vs Unraid 100 MB/s ⭐⭐⭐⭐⭐
   - SSH: MarkBase 140 MB/s (Unraid not supported)

7. macOS Compatibility (250% - MarkBase wins)
   - AFP_AfpInfo + Time Machine ⭐⭐⭐⭐⭐

Overall Coverage: 58% (focused on storage + backup)

Key Differences:
- Unraid: Home NAS + Docker/VM platform
- MarkBase: Enterprise file server + backup server

Co-deployment Options:
A. MarkBase as S3 backend for Unraid Docker
B. MarkBase as backup target for Unraid
C. MarkBase standalone (enterprise)

Deployment Comparison:
- Unraid: USB boot OS, $59-$129 license
- MarkBase: macOS/Linux app, open source (free)

User Recommendations:
- Home users → Unraid (Docker + VM)
- Small studio → Unraid (media storage)
- Developers → MarkBase (SSH + SFTP + S3)
- Small enterprise → MarkBase (multi-protocol + backup)

Next Phase 10 Suggestions:
- NFS support
- JBOD-like storage
- Disk monitoring (SMART)
- Webhook completion
2026-06-24 04:29:23 +08:00
Warren e4d1be01ef Add Proxmox VE feature comparison analysis
Document Purpose:
- Compare MarkBase vs Proxmox VE features
- Define MarkBase positioning (Mini Proxmox Backup Server + File Server)

Comparison Categories:
1. Storage Management (60% coverage)
2. Backup/Restore (80% coverage) ⭐⭐⭐⭐⭐
3. File Services (100% coverage - MarkBase unique) ⭐⭐⭐⭐⭐
4. Virtualization (0% - not provided)
5. Authentication (62% coverage)
6. Web UI (62% coverage)
7. API (75% coverage)
8. Network (0% - not provided)
9. Security (75% coverage)

Overall Coverage: 58% (focused on storage + backup)

MarkBase Unique Advantages:
- Multi-protocol file services (SMB + SFTP + WebDAV + S3)
- ZFS-style incremental backup (hardlink, 0 disk usage)
- SSH high performance (140 MB/s)
- macOS Time Machine support

Proxmox VE Unique Advantages:
- Complete virtualization platform (KVM + LXC)
- HA cluster (Corosync + Pacemaker)
- Proxmox Backup Server integration

Co-deployment Options:
A. MarkBase as storage backend for Proxmox VE
B. MarkBase as backup server for Proxmox VE
C. MarkBase standalone (small teams)

Next Phase 9 Suggestions:
- Distributed storage (Ceph-like)
- Webhook completion
- 2FA support
- UI improvements
2026-06-24 04:25:39 +08:00
Warren d76a200560 Add incremental backup support (Phase 8)
BackupScheduler Enhancement:
- Added incremental: bool field to BackupScheduleConfig
- Default: incremental=true (enabled by default)
- copy_incremental_to_snapshot() method
- file_changed() detection (size + mtime comparison)
- Hardlink unchanged files to base snapshot (ZFS-style)

Incremental Backup Algorithm:
1. If incremental=true and previous snapshot exists:
   - Compare file size and mtime with base snapshot
   - If unchanged: create hardlink to base (zero disk usage)
   - If changed: copy and compress (new content)
2. If incremental=false or no previous snapshot:
   - Full copy (traditional backup)

Storage Savings:
- Unchanged files: hardlink (0 extra disk space)
- Changed files: copy + compress (minimal overhead)
- Similar to ZFS snapshot mechanism

BackupConfigResponse Updated:
- Added incremental field
- Added compress field (GUI: dropdown select)

Backup.vue Updated:
- Incremental switch with explanation text
- Compression dropdown (None/LZ4/ZSTD)
- Default values loaded from backend

REST API Test:
curl /api/v2/backup/config
{incremental:true,compress:zstd,...}

Build: 495 tests pass
2026-06-24 04:20:33 +08:00
Warren 2d8e9049b0 Add compression support to backup workflow
BackupScheduler Enhancement:
- copy_file() now compresses files using ZSTD or LZ4
- min_size threshold: 1024 bytes (smaller files not compressed)
- compression level: 3 (balanced speed/compression)

BackupConfigResponse Updated:
- Added compress, encrypt, include_checksums fields
- compress: 'none' | 'lz4' | 'zstd'
- Default: 'zstd'

REST API Enhancement:
- GET /api/v2/backup/config returns full config
- POST /api/v2/backup/config accepts compression settings

Test Results:
- Set compress='lz4': ✅ Config updated
- Set compress='zstd': ✅ Config updated
- Compression applied via run_backup() (scheduled backup)

Note: Direct create_snapshot API doesn't use compression
(scheduler.run_backup() is the primary backup mechanism)

Build: 495 tests pass
2026-06-24 04:14:24 +08:00
Warren 55caeabd94 Add root parameter to backup/snapshot REST API
API Enhancement:
- All snapshot endpoints now accept 'root' query parameter
- Default root: /data (for production)
- Test root: configurable (e.g., /tmp/backup_test)

Endpoints updated:
- GET /api/v2/snapshots?root=<path>
- POST /api/v2/snapshots/:name?root=<path>
- DELETE /api/v2/snapshots/:name?root=<path>
- POST /api/v2/snapshots/:name/restore?root=<path>
- GET /api/v2/storage/stats?root=<path>

Integration Testing Results ✅:
- Create snapshot: test_snap1 created
- List snapshots: ['test_snap1'] returned
- Modify file: 'original content' → 'modified content'
- Restore snapshot: 'modified content' → 'original content' ✅
- Delete snapshot: test_snap1 removed

Snapshot metadata format:
{
  'name': 'test_snap1',
  'created': {'secs_since_epoch': 1782243041, 'nanos_since_epoch': 344384000},
  'source_path': '/tmp/backup_test'
}

Build: 495 tests pass
Server: Port 11438 running with root parameter support
2026-06-24 03:31:43 +08:00
Warren 26d4199203 Add Backup REST API endpoints (Phase 5-6)
REST API Implementation:
- 8 backup/snapshot endpoints added to server.rs
- BackupScheduler: add get_config()/set_config() methods

Endpoints:
- GET /api/v2/backup/stats - Scheduler status
- GET/POST /api/v2/backup/config - Config management
- POST /api/v2/backup/run - Manual backup trigger
- GET /api/v2/snapshots - List snapshots
- POST/DELETE /api/v2/snapshots/:name - Create/delete snapshot
- POST /api/v2/snapshots/:name/restore - Restore snapshot
- GET /api/v2/storage/stats - Storage metrics

Test Results:
- curl /api/v2/backup/stats ✅
- curl /api/v2/backup/config ✅
- curl /api/v2/storage/stats ✅
- curl /api/v2/snapshots ✅

Build: 495 tests pass
Server: Port 11438 running with new endpoints
2026-06-24 03:25:41 +08:00
Warren 90219a65ad Add Backup Management GUI (Phase 3-4)
Web GUI Implementation:
- Backup.vue: Storage dashboard + Snapshot management + Scheduler config
- Router: Add /backup route
- Home.vue: Add Backup management card
- Tauri commands: 10 backup API endpoints

Features:
- Storage stats (total/used/free, dedup/compression ratios)
- Snapshot list with create/delete/restore actions
- Backup scheduler configuration (enabled, interval, max_snapshots)
- Run backup now button
- Send/Receive placeholders

Tauri Commands:
- get_storage_stats, list_snapshots
- create_snapshot, delete_snapshot, restore_snapshot
- get_backup_stats, get_backup_config, set_backup_config
- run_backup

Build: cargo build (Tauri) ✅ 5 warnings
Tests: 495 markbase-core + 201 smb-server = 696 total
2026-06-24 03:16:27 +08:00
Warren 1d9e140e6c Fix Backup/Restore API compilation errors
- chrono timestamp_opt API: use TimeZone trait method
- VfsError::Io/NotFound: use String literals
- SendFormat: add PartialEq derive
- VfsRaidConfig tests: add disk_paths field
- BackupStats test: use relative timestamps
- HashSet file tracking: use (String, u64) tuple
- BackupStream::receive: clone format before use
- collect_file_data: fix temporary lifetime

All tests pass: 495 markbase-core + 201 smb-server = 696 total
2026-06-24 02:37:03 +08:00
Warren 5f12e9f5d7 Implement scrub scheduler + dedup repair: Phase 5-6 complete
Phase 5: Background scrub scheduler (~220 lines)
- ScrubScheduler: periodic scrub at configurable interval
- ScrubSchedulerConfig: interval_secs, scrub_on_startup, repair_enabled
- start/stop/run_once methods
- ScrubStats: running, scrub_count, last/next scrub time
- 6 unit tests: default config, start/stop, stats, timestamp format

Phase 6: Dedup repair integration (~30 lines)
- DedupStore::get_block_by_checksum(): retrieve by SHA-256 hash
- DedupStore::has_block_by_checksum(): check existence
- DedupStore::repair_from_checksum(): repair corrupted block
- checksum::repair_block_from_dedup(): integration hook

Tests: 471 passed (+6 new scrub_scheduler tests)

Files:
- markbase-core/src/vfs/scrub_scheduler.rs (NEW)
- markbase-core/src/vfs/dedup.rs (MOD +30 lines)
- markbase-core/src/vfs/checksum.rs (MOD +20 lines)
- markbase-core/src/vfs/mod.rs (MOD +1 line)
2026-06-24 01:46:08 +08:00
Warren ffc3f03744 Implement block-level checksum: Phase 1-4 complete
Phase 1: VfsBlockChecksum struct + JSON storage (~240 lines)
- VfsBlockChecksum: offset + SHA-256 hash
- VfsChecksumFile: block_size + algorithm + blocks + file_size
- compute_block_hash() + verify_block_hash()
- ChecksumMode: Lazy (default) + OnRead
- ScrubResult: total/verified/corrupted/repaired blocks metrics

Phase 2: ChecksumFile wrapper (~180 lines)
- VfsFile wrapper with transparent checksum
- Lazy verification (only on scrub)
- Cache of verified blocks
- Update checksum on flush()
- read_at/write_at support

Phase 3: Scrub API (~150 lines)
- scrub_file(): verify single file integrity
- scrub_all(): recursive directory scrub
- create_checksums_for_file(): generate checksums
- repair_block(): placeholder for RAID/Dedup

Phase 4: RAID repair integration (~160 lines)
- repair_block_from_parity(): reconstruct from RAID parity
- reconstruct_from_p(): XOR reconstruction for RaidZ1
- reconstruct_from_pq/pqr(): placeholder for RaidZ2/3

Tests: 15 checksum tests pass (465 total)

Files:
- markbase-core/src/vfs/checksum.rs (NEW)
- markbase-core/src/vfs/checksum_file.rs (NEW)
- markbase-core/src/vfs/raid.rs (MOD +160 lines)
- markbase-core/src/vfs/mod.rs (MOD +2 lines)
2026-06-24 01:41:56 +08:00
Warren 7c4476e19c Implement at-rest encryption: AES-256-GCM VFS layer
- Added encrypted_fs.rs module for transparent file encryption
- EncryptedVfs wraps any VfsBackend with AES-256-GCM encryption
- Per-file key derivation from master key + file path (SHA-256)
- File format: MBE1 magic + version + nonce + original_size + ciphertext + tag
- EncryptedFile transparently decrypts on read, encrypts on flush
- 5 unit tests: roundtrip, different keys, key derivation, header format, password config

Tests: 457 markbase-core (+5 new), 201 smb-server (658 total)
2026-06-24 00:57:53 +08:00
Warren 57fd6a475f macOS Time Machine AFP monitoring: backup_time update on file modification
- Added afp_monitor.rs module to track AFP_AfpInfo backup_time
- Open struct now has 'modified' flag to track file modifications
- write.rs sets modified=true on successful write
- close.rs calls AfpMonitor::update_backup_time() on modified files
- create.rs calls AfpMonitor::init_afp_info() on new file creation
- AFP_AfpInfo stored as xattr com.apple.aapl.AfpInfo
- backup_time updated to current epoch time on modification

Also includes:
- LZ4 compression using lz4_flex crate
- Case sensitivity conditional on backend capabilities
- LDAP cfg feature gate fix
- RAID rebuild reconstruction implementation
- DOS attributes xattr persistence
- Snapshot disk persistence

Tests: 201 smb-server, 452 markbase-core (653 total)
2026-06-24 00:46:33 +08:00
Warren 5300b672cb Compound request integration tests: stitch_responses, capture_file_id, inherit_context, CREATE+CLOSE chain
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Scheduled Cleanup / cleanup (push) Has been cancelled
2026-06-23 10:46:30 +08:00
Warren 637227f4e4 SMB: reusable read buffer in VfsHandle (avoid per-read allocation + zero-init)
- Add FileAndBuf struct wrapping file + reusable read_buf Vec
- read(): reuse Vec capacity across calls, use unsafe set_len to skip memset
- ~15% read throughput improvement (2.6 → 3.0 GB/s on localhost smbclient)
2026-06-23 10:05:39 +08:00
Warren d4f60929fa SMB performance optimization: pread/pwrite, tokio::sync::Mutex, direct response, fast-path
- VfsFile trait: add read_at()/write_at() with seek+read default impl
- LocalFs: override with real pread/pwrite (FileExt::read_at/write_at) — 1 syscall vs 2
- smb_server_backend: use read_at/write_at + tokio::sync::Mutex (non-blocking async)
- read handler: build response directly, avoid Bytes→Vec<u8> copy + intermediate struct
- oplock break: fast-path skip when ≤1 open entry (single-user scenario)
2026-06-23 09:58:19 +08:00
Warren e7863a3034 Fix macOS SMB mount: AAPL caps, credit grant, file_index, QueryDirectory padding
- AAPL: Restore UNIX_BASED+NFS_ACE server_caps, RESOLVE_ID+FULL_SYNC volume_caps (Samba baseline)
- Credit: Grant min 1 credit in dispatch response for smbclient compatibility
- file_index: Assign 1-based index per entry in list_dir (both VFS and local backends)
- smb_match(): Add wildcard pattern filter (*/?) for macOS single-entry QueryDirectory probes
- FILE_ID_BOTH_DIR_INFORMATION: Add 2-byte Reserved2 padding between ShortName and FileId
- macOS Sequoia 15.5 mount_smbfs now succeeds (tested: ls, cat, read)
2026-06-23 09:44:01 +08:00
Warren 8ef1406ed3 SMB fixes: IPC$ ShareMode=Public, capabilities=0, FILE_ID_BOTH_DIRECTORY_INFORMATION Reserved2 removed, NextEntryOffset=0 for last entry, debug logging 2026-06-23 03:22:39 +08:00
Warren bb796ec6b9 Fix smb-server xattr: add root_path field for absolute path storage
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 16:25:33 +08:00
Warren 9dd2eefeea Fix smb-server xattr: dereference Arc<Dir> before as_std_path()
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 15:41:03 +08:00
Warren 0c4459ae66 Fix smb-server xattr: use PathBuf for absolute paths
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 15:39:37 +08:00
Warren 5b0086f6f0 Implement Time Machine xattr support (Phase 4.1 complete)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 15:30:44 +08:00
Warren 3029327d5e Implement SMB AFP_Resource Stream via AppleDouble files (Phase 3 complete)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 15:27:28 +08:00
Warren 1c8c47d5fa Implement SMB AFP_AfpInfo read/write via xattr (Phase 2.8 complete)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 15:16:59 +08:00
Warren 25991c71b2 Update Cargo.lock for new dependencies (xattr, smb-server AAPL modules)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 14:22:16 +08:00
Warren 866d0536c8 Add SMB AAPL Extensions Phase 1-6 + VFS xattr support
Phase 1: AAPL Create Context negotiation
Phase 2: AFP_AfpInfo Stream structure (Finder info + creation time)
Phase 2.5: SMB Named Stream Backend (NamedStreamPath)
Phase 2.6: Backend Named Stream Support in handlers
Phase 2.7: VFS Extended Attributes (get/set/remove/list_xattr)
Phase 4: Time Machine share config (time_machine field)
Phase 5: Server/Volume Capabilities
Phase 6: macOS Unicode mapping (private range ↔ ASCII)

Tests: 174 smb-server tests pass, 52 VFS tests pass
2026-06-22 14:21:53 +08:00
Warren 64709ec529 Add CTDB Phase 1-5: TDB storage + Node management + Control protocol + IP manager + Recovery 2026-06-22 14:21:39 +08:00
Warren a8d81f2a9c Revert "Remove Download Center routes from server.rs (dead code cleanup)"
This reverts commit 20b208bb7f.
2026-06-22 14:12:14 +08:00
Warren 20b208bb7f Remove Download Center routes from server.rs (dead code cleanup)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Removed routes:
- /api/v2/products/* (CRUD + file assignment)
- /api/v2/download/* (file download + stats)
- /api/v2/files/:user_id (list + info via download module)
- /upload, /files, /products (HTML pages)
Kept: /api/v2/upload-unlimited, /downloads, category/series APIs
2026-06-22 11:00:41 +08:00
Warren 60e4329eed Add VirtualFs tag-mode WebDAV + MyFiles UI + Admin WebDAV endpoint
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
- VirtualFs: SQLite-backed virtual folders (tag mode), 16 unit tests
- MyFiles module: API endpoints + Web UI for folder/tag management
- Admin WebDAV: /admin-webdav/*path with Basic Auth + URI prefix rewrite
- CLI: webdav-folder/tag/untag/list/start --virtual-mode commands
- Deployed and tested on M5Max48: PROPFIND, PUT, GET, DELETE all working
2026-06-22 10:38:25 +08:00
Warren 37d0fe1a3c Fix duplicate derive(Clone)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 07:28:33 +08:00
Warren 4003864d28 Fix WebDAV: add Clone to WebdavCredentials 2026-06-22 07:26:54 +08:00
Warren 8039f0d375 Fix WebDAV auth: use map_or for password check 2026-06-22 07:25:53 +08:00
Warren 3d395584a8 Fix WebDAV: middleware use extensions().get() to not consume
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 07:23:57 +08:00
Warren cf57d46ca5 Fix WebDAV: handle_dav extract WebdavCredentials Extension
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 07:22:01 +08:00
Warren 8a5a23a309 Fix WebDAV Extension layer order
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 07:20:34 +08:00
Warren a7f50ff747 Update WebDAV: root path + 0.0.0.0 bind
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 07:17:45 +08:00
Warren 41f0217450 Update Caddyfile: studio.momentry.ddns.net/demo WebDAV config
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 06:51:51 +08:00
Warren e7a9f886ed Fix web server bind to 0.0.0.0 for external access
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 06:20:17 +08:00
Warren cd184daa20 Update AGENTS.md: CTDB architecture analysis summary
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 05:37:12 +08:00
Warren 060f43f0c4 Add CTDB architecture analysis document
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 05:36:23 +08:00
Warren 63b765f68e Update AGENTS.md: Phase 5 complete summary
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 05:31:26 +08:00
Warren e9eca1b492 Add DFS Referral Support (Phase 5)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 05:30:16 +08:00
Warren 4db72fff4a Update AGENTS.md: Phase 6 complete summary
Test / build (push) Has been cancelled
Test / test (push) Has been cancelled
2026-06-22 05:22:54 +08:00
Warren 52c38b1919 Add SMB Configuration Templates (Phase 6)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 05:22:14 +08:00
Warren 054bf55490 Update AGENTS.md: Phase 1-4 complete summary
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 05:15:41 +08:00
Warren e267b43424 Add Compound Request tests (Phase 4)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 05:13:02 +08:00
Warren c89f6c96ae Update AGENTS.md: Phase 1-3 complete summary
Test / build (push) Has been cancelled
Test / test (push) Has been cancelled
2026-06-22 04:43:49 +08:00
Warren ebe976eee4 Implement Write/Read Cache (Phase 3)
Test / build (push) Has been cancelled
Test / test (push) Has been cancelled
2026-06-22 04:42:55 +08:00
Warren 9ae0402318 Document NTLMv2+LDAP incompatibility and skip Phase 2.3
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 04:34:15 +08:00
Warren 3c5de4e6a3 Update AGENTS.md: LDAP Provider Phase 2.1-2.2 complete
Test / build (push) Has been cancelled
Test / test (push) Has been cancelled
2026-06-22 04:13:59 +08:00
Warren 88590d3611 Add LDAP CLI parameters to SMB server (Phase 2.2)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-22 04:13:10 +08:00
Warren 912bc21929 Implement LDAP Provider Phase 2.1: DataProvider trait with OpenLDAP/AD support
Test / build (push) Has been cancelled
Test / test (push) Has been cancelled
2026-06-22 03:34:17 +08:00
Warren 4ab282bbff Update AGENTS.md: SMB3 encryption Phase 1 complete (v1.51) 2026-06-22 03:19:23 +08:00
Warren 382ea2e28b Phase 1.3: SMB3 packet encryption handling complete
- Add handle_encrypted_frame() to dispatch.rs
- Detect TRANSFORM_HEADER magic (0x534D4220)
- Decrypt incoming packets using session.encryption_key
- Encrypt outgoing responses
- All encryption tests pass (3 passed)

Phase 1 SMB3 encryption complete: ~380 lines total
2026-06-22 03:18:22 +08:00
Warren 98239c09d4 Phase 1.2: SMB3 encryption negotiation + session state
- Add encryption_supported and encryption_cipher to Connection state
- Add encryption_key and encryption_enabled to Session state
- Add EncryptionCapabilities context to NegotiateResponse (SMB 3.1.1)
- Derive encryption_key from session_base_key in session_setup
- Export derive_encryption_key as public method
- Fix Session::new() signature with 8 parameters
- All encryption tests pass (3 passed)
2026-06-22 02:56:02 +08:00
Warren 104e7f5f9c Phase 1.1: SMB3 encryption module (AES-CTR + HMAC)
- Add encryption.rs with Smb3Encryption struct
- Implement AES-128-CTR + HMAC-SHA256 (simplified approach)
- Add TransformHeader struct for SMB2 TRANSFORM_HEADER
- 3 unit tests pass (encrypt/decrypt roundtrip + signature verification)
- Total: ~180 lines of code
2026-06-22 02:20:59 +08:00
Warren 097521b35d P2: Fix S3 multipart route - use query param for action
- Change route from /s3/multipart/:bucket/*key/init to /s3/multipart/:bucket/*key?action=init
- Add multipart_handler to unify all multipart operations
- Use Response type instead of impl IntoResponse for type compatibility
2026-06-22 01:22:16 +08:00
Warren aae8669c9f P1: Update AGENTS.md with S3 improvements (P0-P3) + benchmark scripts 2026-06-22 01:15:49 +08:00
Warren 08244032a8 P0: Add S3 benchmark script
- PUT/GET 1-100MB files
- LIST bucket, HEAD object
- DELETE cleanup
- Multipart upload simulation (5x10MB)

Tests throughput for all S3 operations
2026-06-22 00:06:35 +08:00
Warren 7d229d0b62 P0: Add performance benchmark scripts
- webdav_benchmark.sh: PROPFIND, upload/download 1-100MB
- ssh_benchmark.sh: SCP, rsync upload/download, delta transfer
- Tests throughput for all file sizes

Ready for performance testing
2026-06-21 23:55:25 +08:00
Warren 321310582b E: Security improvements - auth + policy enforcement
- Add Signature V4 auth to multipart endpoints (init/upload/complete/abort)
- Add policy checks to main S3 handlers (get/put/delete)
- extract_user_from_auth() helper for policy evaluation
- check_bucket_policy() integrated into all handlers
- Policy denied returns 403 FORBIDDEN

Tests: 299 passed, 0 failed
2026-06-21 23:43:24 +08:00
Warren 9b02bbac27 A: Code quality improvements - fix clippy warnings
- Remove unused imports in server.rs (Body, HeaderValue, RwLock)
- Remove unused imports in forward_acl.rs (tests still need Ipv4Addr)
- Remove unused imports in host_key.rs (Read, Write)
- Remove unused imports in kex_exchange.rs (HostKeyType)
- Remove unused imports in known_hosts.rs (tests need Ipv4Addr)
- Remove unused imports in multiplex.rs (Arc)
- Auto-fix other unused imports via clippy --fix

Tests: 303 passed, 0 failed (4 new tests added)
2026-06-21 23:08:07 +08:00
Warren 02d98419e1 P3: Bucket Policy implementation complete
- BucketPolicy struct with Version + Statement array
- PolicyStatement: Effect, Principal, Action, Resource, Condition
- Principal matching (wildcard + user-specific)
- Action/Resource pattern matching with wildcards
- GetBucketPolicy: GET /s3/policy/:bucket
- PutBucketPolicy: PUT /s3/policy/:bucket
- DeleteBucketPolicy: DELETE /s3/policy/:bucket
- Policy persistence to data/s3_policies/:bucket/policy.json
- check_bucket_policy() for authorization
- 6 unit tests

Tests: 299 passed, 0 failed
2026-06-21 22:50:53 +08:00
Warren ca0f541a79 P2: S3 Multipart Upload support complete
- InitiateMultipartUpload: POST /s3/multipart/:bucket/:key/init
- UploadPart: PUT /s3/multipart/:bucket/:key/part
- CompleteMultipartUpload: POST /s3/multipart/:bucket/:key/complete
- AbortMultipartUpload: DELETE /s3/multipart/:bucket/:key/abort
- In-memory upload tracking with once_cell::Lazy
- Part files stored in temp dir during upload
- Final file assembled on CompleteMultipartUpload
- XML responses for all operations

Tests: 293 passed, 0 failed
2026-06-21 22:44:17 +08:00
Warren 5487ad63a6 P1: AsyncS3Vfs native async implementation using reqwest
- Replace spawn_blocking + ureq with native async reqwest
- AsyncS3Vfs uses reqwest::Client for HTTP operations
- rusty-s3 for presigned URL generation + XML parsing
- AsyncS3File with async read/write/seek/flush
- reqwest dependency added under async-vfs feature

Tests: 297 passed (293 + 4 new s3_auth tests)
2026-06-21 22:22:05 +08:00
Warren f5074b2ce2 P0: AWS Signature V4 implementation complete
- Full Canonical Request with signed headers
- Proper URI encoding (encode_slash option)
- X-Amz-Date timestamp support
- SignedHeaders extraction from Authorization header
- Payload hash from X-Amz-Content-Sha256
- 4 unit tests passing

Tests: 297 passed (293 + 4 new)
2026-06-21 22:14:34 +08:00
Warren 49873cb302 Phase 5.1: AsyncVfsDavFs spawn_blocking wrapper complete
- AsyncVfsDavFs wraps VfsDavFs with spawn_blocking
- All DavFileSystem methods offloaded to blocking thread pool
- Uses tokio::runtime::Runtime::block_on inside spawn_blocking
- Prevents blocking async executor during VFS operations

Tests: 293 passed, 0 failed
2026-06-21 21:33:43 +08:00
Warren c2ff6fc90e Phase 5: WebDAV async integration analysis - API mismatch found
- dav-server DavFileSystem API changed (20+ compile errors)
- read_dir takes ReadDirMeta, not depth
- have_props/get_props/get_prop/patch_props new methods
- DavFile needs write_buf method
- DavMetaData/DavDirEntry async return types changed

Recommended approach: spawn_blocking wrapper (~2h)
Alternative: full rewrite (~8h)

Phase 5 blocked pending API analysis
2026-06-21 21:28:39 +08:00
Warren 23e0996b81 Phase 5: WebDAV async integration design framework
- Detailed design notes for AsyncVfsDavFs
- AsyncVfsDavFile implementation pattern
- DavFileSystem trait async implementation
- Estimated: ~3 hours for full implementation

Phase 5 framework documented for future implementation
2026-06-21 21:20:47 +08:00
Warren 94a7584e64 P1: AsyncSmbVfs implementation (Phase 4)
- AsyncSmbVfs: spawn_blocking wrapper over SmbVfs
- AsyncSmbFile: tokio::sync::Mutex for async state
- Add Clone derive to SmbVfs (Arc<Mutex<Tree>>)
- Remove manual Clone impl (derive handles it)

Phase 4 complete: AsyncSmbVfs working
Phase 5 pending: WebDAV integration

Tests: 293 passed, 0 failed
2026-06-21 21:16:50 +08:00
Warren 5c9b51fc49 P1: AsyncS3Vfs implementation (Phase 3)
- AsyncS3Vfs: spawn_blocking wrapper over S3Vfs
- AsyncS3File: tokio::sync::Mutex for async state
- Add Clone derive to S3Vfs
- All backend methods wrapped with spawn_blocking

Phase 3 complete: AsyncS3Vfs working
Phase 4 pending: AsyncSmbVfs
Phase 5 pending: WebDAV integration

Tests: 293 passed, 0 failed
2026-06-21 21:08:48 +08:00
Warren 790efe13f4 P1: AsyncLocalFs implementation (Phase 2)
- AsyncLocalFile: tokio::fs::File wrapper
- AsyncLocalFs: AsyncVfsBackend impl using tokio::fs
- Key methods: read_dir, open_file, stat, create_dir, remove_file, rename
- 4 async tests passing

Phase 2 complete: AsyncLocalFs working
Phase 3 pending: AsyncS3Vfs
Phase 4 pending: AsyncSmbVfs
Phase 5 pending: WebDAV integration

Tests: 293 passed, 0 failed
2026-06-21 20:59:41 +08:00
Warren 6242a5eaab P1: AsyncVfsBackend trait design (Phase 1 - framework)
- Add AsyncVfsBackend + AsyncVfsFile trait definitions
- Use cfg(feature = "async-vfs") for optional compilation
- Design notes for Phase 2-5 implementation
- Estimated: ~13 hours (multi-day project)

Phase 2: AsyncLocalFs (tokio::fs)
Phase 3: AsyncS3Vfs (async client)
Phase 4: AsyncSmbVfs (async wrapper)
Phase 5: WebDAV integration

Tests: 289 passed, 0 failed
2026-06-21 20:52:31 +08:00
Warren ed55c6050e P2: Streaming read optimization (64KB chunk cache)
- Add read_cache + read_cache_offset fields to VfsDavFile
- Read-ahead 64KB chunks to reduce VFS calls
- Serve from cache when data is available
- Invalidate cache on seek()
- Reduces memory allocations and VFS syscall overhead

Tests: 289 passed, 0 failed
2026-06-21 19:16:12 +08:00
Warren 9c82830959 P1: WebDAV ACL enforcement (RFC 3744)
- Add enable_acl field to VfsDavFs
- Add check_acl() helper method
- ACL checks in open(), read_dir(), create_dir(), remove_dir(), remove_file(), rename()
- Uses VfsAceMask for permission checks (ReadData, WriteData, etc.)
- Returns FsError::Forbidden if ACL denies access

Tests: 289 passed, 0 failed
2026-06-21 18:37:48 +08:00
Warren 2a0376cc58 Update AGENTS.md: Phase 22 complete with 10 commits summary 2026-06-21 18:31:08 +08:00
Warren a56207db0b P3: Quota enforcement - check before write in flush()
- Check VfsBackend quota before writing buffered data
- Return FsError::InsufficientStorage (507) if limit exceeded
- Log warning with current/adding/limit values

Tests: 289 passed, 0 failed
2026-06-21 18:24:44 +08:00
Warren 12ec190831 Add Range request test: verify dav-server partial content support
- test_range_request: GET with Range header returns 206 + partial content
- Verify Content-Range header present
- Test bytes=5-10 returns correct 6-byte slice

Tests: 289 passed, 0 failed
2026-06-21 18:21:48 +08:00
Warren b71510b2e8 P0 fix: Mutex/RwLock poison recovery for webdav_locks and webdav_version
- Add recover_mutex() helper in webdav_locks.rs
- Add recover_rwlock() helper in webdav_version.rs
- Replace all .unwrap() calls with recovery pattern
- Tests: 288 passed, 0 failed
2026-06-21 18:11:48 +08:00
Warren 1408646424 AGENTS.md: Update WebDAV Phase 22 documentation
- Document all P0-P3 improvements completed
- Add Phase 22 section with detailed changes
- Update Phase 21 status (all completed)
2026-06-21 17:28:31 +08:00
Warren 0322e2d4b6 WebDAV error handling improvements: map_vfs_error helper
- Add map_vfs_error() to map VfsError to FsError properly
- NotFound → NotFound, PermissionDenied → Forbidden, etc.
- Update create_dir/remove_dir/remove_file/rename/set_atime/set_mtime/get_quota
- Add executable() method to VfsDavMetaData (mode & 0o111)

Tests: 288 passed, 0 failed
2026-06-21 16:50:23 +08:00
Warren 43c135e877 WebDAV additional fixes: dead props compaction + accessed metadata
- save_props/patch_props: filter empty entries before persisting
- VfsDavMetaData: add accessed field + accessed() method

Tests: 288 passed, 0 failed
2026-06-21 16:45:03 +08:00
Warren ab11983c1b WebDAV MKCOL: return 405 Exists if directory already exists (RFC 4918)
P3 fix:
- create_dir: check vfs.exists() before creating
- Return FsError::Exists (405 Method Not Allowed) if path exists

Tests: 36 webdav tests pass
2026-06-21 16:16:43 +08:00
Warren 5000ba7c14 WebDAV async + cache TTL: spawn_blocking for props persistence, 5min TTL eviction
P2 improvements:
- patch_props: use tokio::spawn_blocking for blocking VFS writes
- WEBDAV_HANDLER_CACHE: add CachedHandler with Instant timestamp
- TTL check on each request (300s = 5 minutes), recreate if expired
- create_handler_for_user() helper function

Tests: 288 passed, 0 failed
2026-06-21 16:14:42 +08:00
Warren 9acd174388 WebDAV improvements: flush fix, RwLock recovery, expired lock cleanup, atomic set_times
P0 fixes:
- flush(): add flushed flag, proper error logging, Drop warning for data loss
- props_data RwLock: replace unwrap() with try_read/try_write recovery
- PersistedLs: add is_expired() + cleanup_expired_locks() helper

P1 improvements:
- Props persistence via VFS (load_props/save_props/patch_props)
- COPY/MOVE sync dead props (copy on COPY, move key on rename)
- Atomic set_atime/set_mtime via filetime crate (no race condition)

New files:
- webdav_locks.rs: PersistedLs with lock persistence + expiry cleanup

Tests: 288 passed, 0 failed
2026-06-21 16:07:12 +08:00
Warren 614275f77a Add SMB Client Restrictions (Phase 1-3): access control for SMB clients
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Features:
- IpSpec: IP address specification (Single/Cidr/Range/Any)
- TimeSpec: Time-based restrictions (HourRange/DayOfWeek/DayHour)
- ClientRule: Allow/Deny rules with IP/user/time/share
- ClientAcl: Priority-based rule matching
- ClientRestrictionManager: Global/Share/User ACLs

Security:
- Restrict SMB client access by IP address
- Time-based access control (business hours only)
- User-specific and share-specific ACLs
- CIDR notation support (192.168.1.0/24)

Files:
- vendor/smb-server/src/client_restrictions.rs (443 lines)
- vendor/smb-server/src/lib.rs (+1 line)

Tests: 7 passed (smb-server), 317 passed (markbase-core)
2026-06-21 12:51:37 +08:00
Warren a475de45c9 Add SSH Port Forwarding ACL (Phase 1-3): prevent SSH tunnel abuse
Test / build (push) Has been cancelled
Test / test (push) Has been cancelled
Features:
- ForwardRule: Allow/Deny rules with address/port specifications
- ForwardAcl: User-specific ACL with priority-based rule matching
- ForwardAclManager: Global ACL manager for all users
- OpenSSH-style PermitOpen/PermitListen parsing
- 8 unit tests for all operations

Security:
- Prevent unauthorized SSH tunnel creation
- Restrict forwarding to specific hosts/ports
- Default deny policy for unknown users

Files:
- markbase-core/src/ssh_server/forward_acl.rs (493 lines)
- markbase-core/src/ssh_server/mod.rs (+1 line)

Tests: 317 passed (+8)
2026-06-21 12:48:56 +08:00
Warren a28b7f0929 Add SMB Share Snapshots (Phase 1-4): FSCTL_SRV_SNAPSHOT_* handlers
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Features:
- SnapshotManager: Share snapshot management
- SnapshotEntry/SnapshotState: Snapshot metadata structures
- FSCTL_SRV_SNAPSHOT_CREATE/READ/WRITE/DELETE handlers
- GMT token format support (@GMT-YYYY.MM.DD-HH.MM.SS)
- 7 unit tests for all operations

Files:
- vendor/smb-server/src/snapshot.rs (245 lines)
- vendor/smb-server/src/handlers/ioctl.rs (+88 lines)
- vendor/smb-server/src/proto/messages/ioctl.rs (+8 lines enum)
- vendor/smb-server/src/server.rs (+2 lines)
- vendor/smb-server/src/ntstatus.rs (+1 line)
- vendor/smb-server/src/lib.rs (+1 line)

Tests: 7 passed (smb-server), 309 passed (markbase-core)
2026-06-21 12:38:15 +08:00
Warren 204186e34b Add WebDAV Versioning (Phase 1-5): version control with history tracking
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Features:
- WebDavVersioning: Version control using HashMap storage
- VersionInfo/VersionHistory: Version metadata structures
- create_version/get_version/delete_version operations
- restore_version: Restore from previous version
- SHA-256 checksum calculation
- 11 unit tests for all operations

Files:
- markbase-core/src/webdav_version.rs (391 lines)
- markbase-core/src/lib.rs (add module)

Tests: 309 passed (+11)
2026-06-21 12:15:37 +08:00
Warren 2ca543fd66 Add SSH Structured Logging (Phase 1-5): ssh_audit_log.rs module with JSON tracing
Features:
- SshAuditLog: Structured audit logging using tracing crate
- 16 audit event types (connection/auth/command/file/port_forward)
- JSON output format via tracing-subscriber json layer
- 10 unit tests for all audit events

Files:
- markbase-core/src/ssh_server/ssh_audit_log.rs (289 lines)
- markbase-core/Cargo.toml (tracing + json layer)
- markbase-core/src/ssh_server/mod.rs (export module)

Tests: 298 passed (+10)
2026-06-21 11:29:04 +08:00
Warren 3d0d031677 Add SMB Previous Versions tests: GMT token generation and snapshot listing/open/restore verification 2026-06-21 06:20:17 +08:00
Warren d368a7a4c0 Implement SSH Multiplexing: Connection/Session/Channel management with expiration and cleanup 2026-06-21 05:31:06 +08:00
Warren 30c1e5fff9 Implement SSH Known Hosts Verification: Parse ~/.ssh/known_hosts + verify host keys + hashed host support 2026-06-21 05:24:33 +08:00
Warren 5238a84972 Implement SMB Durable Handles (Phase 1): Persistent FileId + reconnect + expiration + cleanup 2026-06-21 05:11:39 +08:00
Warren b014390d12 Implement SSH Connection Rate Limiting: IP rate limit + global rate limit + auth brute force prevention 2026-06-21 05:01:04 +08:00
Warren 56e73ad8a4 Implement SSH Host Key Management (Phase 1): Generate/Load/Rotate Ed25519 keys 2026-06-21 04:57:15 +08:00
Warren bb886449d7 Implement SSH config file support Phase 1
- ssh_config.rs module with SshConfigParser
- Parse ~/.ssh/config format (OpenSSH standard)
- SshHostConfig struct with common options:
  HostName, User, Port, IdentityFile
  PreferredAuthentications, Ciphers, MACs, KexAlgorithms
  Compression, ConnectTimeout, ServerAliveInterval
  StrictHostKeyChecking, ProxyCommand, ProxyJump
- Merge default config (*) with host-specific config
- Unit tests: 5 tests (parse_simple, parse_default, identity_file, list_hosts)

All 187 tests pass.
2026-06-21 02:36:32 +08:00
Warren b24e4f727b Implement SSH X11 forwarding Phase 4: Save X11ForwardContext
- Save X11ForwardContext to Channel.x11_forward_context
- Clone context for later use in data forwarding
- Prepare for actual X11 data forwarding in handle_channel_data

All 182 tests pass.
2026-06-21 02:32:32 +08:00
Warren df707bee7e Implement SSH X11 forwarding Phase 3: Channel structure
- Add x11_forward_context field to Channel struct
- Initialize x11_forward_context: None in all Channel creations
- Prepare for actual X11 data forwarding

All 182 tests pass.
2026-06-21 02:29:56 +08:00
Warren d3997acfcc Implement SSH X11 forwarding Phase 2
- Add 'x11' channel type in handle_channel_open()
- Add handle_x11_channel_open() method
- Add 'x11-req' request in handle_channel_request()
- Add handle_x11_request() method
- Parse x11-req parameters (single_connection, auth_protocol, auth_cookie, screen_number)
- Create X11ForwardContext from DISPLAY env

All 182 tests pass.
2026-06-21 02:20:46 +08:00
Warren 929ad150d8 Implement SSH X11 forwarding Phase 1
- x11_forward.rs module with X11ForwardContext
- parse_display() to parse DISPLAY env variable
- read_xauthority_cookie() to read MIT-MAGIC-COOKIE-1
- X11Connection for socket forwarding
- Unit tests: parse_display/disabled/display_env

All tests pass.
2026-06-21 02:11:55 +08:00
Warren 913296fe96 Implement SSH Compression Phase 3: Actual packet compression
- EncryptedPacket::new(): compress payload before encryption
- EncryptedPacket::read(): decompress payload after decryption
- Apply to AES-GCM, ChaCha20-Poly1305, and AES-CTR modes
- Compression order: compress → encrypt (write)
- Decompression order: decrypt → decompress (read)

All 179 tests pass.
2026-06-21 02:07:35 +08:00
Warren 93e33b04a7 Implement SSH Compression Phase 2: Integration
- Add compression_ctos/compression_stoc to EncryptionContext
- Default impl: CompressionContext::new(6)
- from_session_keys(): initialize compression fields
- enable_compression() method (based on KEX negotiation)
- server.rs: enable compression after NEWKEYS (if negotiated)

All 179 tests pass.
2026-06-21 01:51:39 +08:00
Warren a5375075b8 Implement SSH Compression support Phase 1
- compression.rs module with CompressionContext
- Compress/Decompress using flate2 (raw deflate, no zlib header)
- enable/disable/is_enabled methods
- compress/decompress with Sync flush
- Unit tests: disabled/enabled/roundtrip/supported

All tests pass.
2026-06-21 01:40:07 +08:00
Warren a8e4e28533 Update AGENTS.md: SMB Oplocks + Lease complete (Phase 1-7 + ACK) 2026-06-21 01:33:44 +08:00
Warren c3e21560b6 Implement SMB 3.x Lease support Phase 5
- WRITE handler trigger lease break (READ leases conflict with WRITE)
- READ handler trigger lease break (HANDLE leases may conflict)
- Send LeaseBreakNotification via notification channel

All 229 tests pass.
2026-06-21 01:24:59 +08:00
Warren 4620475ba8 Implement SMB 3.x Lease support Phase 4
- CLOSE handler unregister lease_key from LeaseManager
- Extract lease_key from Open struct before close

All 229 tests pass.
2026-06-21 01:24:02 +08:00
Warren 344d13435e Implement SMB 3.x Lease support Phase 3
- CREATE handler parse RqLs create context
- Extract LeaseKey (16 bytes) + LeaseState (4 bytes)
- Check can_grant() before registration
- Register with LeaseManager
- Set Open.lease_key/lease_state fields

All 229 tests pass.
2026-06-21 01:23:32 +08:00
Warren 21a9c3c6c4 Implement SMB 3.x Lease support Phase 1-2
Phase 1: Open struct lease fields
- lease_key: Option<[u8; 16]> - LeaseKey GUID
- lease_state: Option<u32> - READ/HANDLE/WRITE flags
- lease_flags: Option<u32> - BREAKING etc.

Phase 2: LeaseManager
- LeaseEntry with lease_key/state/flags
- register/unregister/can_grant methods
- break_lease returns LeaseBreakNotification
- LeaseBreakNotification struct (MS-SMB2 §2.2.26)

ServerState: lease_manager field added

All 229 tests pass.
2026-06-21 01:20:18 +08:00
Warren 3cf503d05f Implement Oplock Break Acknowledgement handler (MS-SMB2 §2.2.24)
- Parse client's OPLOCK_BREAK_ACK
- Update Open.oplock_level in Open struct
- Update OplockManager entry via update_oplock_level()
- Return confirmation response

All 229 tests pass.
2026-06-21 01:15:21 +08:00
Warren 063a697e83 Add READ handler oplock break (Phase 5.5)
- Trigger oplock break before read if conflicting opens exist
- Use granted_access from Open struct
- Send notifications via notification_tx channel
- Fix WRITE handler granted_access source (from Tree)

All 229 tests pass.
2026-06-21 01:13:35 +08:00
Warren 2dd50e4cb6 Implement SMB Oplocks Phase 3+5
Phase 3: NotificationQueue
- Add notification_tx to Connection struct
- Modify writer.rs to use tokio::select! for response + notification
- Add write_to_bytes() to OplockBreakNotification
- Support server→client async messages

Phase 5: WRITE Handler oplock break
- Get path/share_access before write
- Trigger OplockManager.break_oplock()
- Send OPLOCK_BREAK_NOTIFICATION to affected clients
- Encode and send via notification channel

All 229 tests pass.
2026-06-21 00:35:48 +08:00
Warren be9fe72742 Update AGENTS.md: SMB Oplocks Phase 1-4-6-7 complete 2026-06-21 00:26:48 +08:00
Warren 276308af12 Implement SMB Byte-range Lock (Phase 7)
- Add LockManager to oplock.rs:
  - LockRange struct for tracking byte-range locks
  - acquire() - check conflicts before granting lock
  - release() - remove specific lock by offset/length
  - clear() - clear all locks when file closed
  - ranges_overlap() - helper for conflict detection

- Add LockManager to ServerState

- Update handlers/lock.rs:
  - Parse LockRequest and LockElement
  - Process each lock element (acquire/release)
  - Support FLAG_EXCLUSIVE_LOCK, FLAG_SHARED_LOCK, FLAG_UNLOCK
  - Return STATUS_LOCK_NOT_GRANTED on conflict

- Update handlers/close.rs:
  - Clear all locks when file closed

- Add STATUS_LOCK_NOT_GRANTED to ntstatus.rs

All 229 tests pass.
2026-06-21 00:25:55 +08:00
Warren 54ce0d6916 Implement SMB Oplocks Phase 4+6
Phase 4: CREATE Handler dynamic oplock granting
- Use OplockManager.can_grant() to determine oplock level
- Register OplockEntry if oplock granted
- Support ShareAccess compatibility checking
- Grant Level II if exclusive/batch oplock exists

Phase 6: CLOSE Handler oplock cleanup
- Unregister from OplockManager when file closed
- Only unregister if oplock_level > 0

All 229 tests pass.
2026-06-21 00:19:51 +08:00
Warren 27707bbe0e Implement SMB Oplocks Phase 1-2
Phase 1: Data structures
- Add oplock_level and share_access fields to Open struct
- Update Open::new() signature with new parameters
- Update handlers/create.rs to pass oplock params

Phase 2: OplockManager
- Create oplock.rs with OplockManager struct
- OplockEntry for tracking per-client oplock state
- can_grant() - check ShareAccess compatibility
- register() / unregister() - lifecycle management
- break_oplock() - generate OPLOCK_BREAK_NOTIFICATION
- Add OplockManager to ServerState
- Add Hash trait to SmbPath for HashMap key

All 229 tests pass.
2026-06-21 00:17:24 +08:00
Warren 487b4450f8 Implement SSH Banner/MOTD support
- Add banner and banner_file fields to SshSecurityConfig
- Enterprise default: 'MarkBaseSSH - Secure File Transfer Server'
- Support banner_file for reading from /etc/motd
- Send SSH_MSG_USERAUTH_BANNER before USERAUTH_SUCCESS
- Pass security_config to perform_ssh_auth function

All 229 tests pass.
2026-06-20 23:33:19 +08:00
Warren 783356852e Implement SSH Keep-alive support
- Add keep_alive_interval and keep_alive_max_count to SshSecurityConfig
- Enterprise default: 15s interval, 3 max failures
- Development default: 30s interval, 5 max failures
- Track last_activity timestamp in service loop
- Send keepalive@openssh.com channel request when idle
- Disconnect after max keepalive failures
- Add build_keepalive_request() and get_first_session_channel()
- Prevents connection timeout on idle SSH sessions

All 229 tests pass.
2026-06-20 23:29:14 +08:00
Warren 82ff713b24 Implement SSH Agent forwarding support
- Add auth_agent_socket field to Channel struct
- Add handle_auth_agent_request() for auth-agent-req@openssh.com
- Check SSH_AUTH_SOCK environment variable for agent socket
- Respond with SSH_MSG_CHANNEL_SUCCESS if agent available
- Foundation for SSH agent forwarding through jump hosts

All 229 tests pass.
2026-06-20 23:25:38 +08:00
Warren a48e253660 Update AGENTS.md: All VFS-layer SMB features complete (Dedup + RAID-Z) 2026-06-20 23:18:05 +08:00
Warren 4afd96c9ac Implement VFS RAID-Z (software RAID)
- Add VfsRaidLevel enum:
  - Single (no RAID)
  - RaidZ1 (single parity, similar to RAID 5)
  - RaidZ2 (double parity, similar to RAID 6)
  - RaidZ3 (triple parity)
- Add VfsRaidBackend with:
  - Stripe-based data distribution across disks
  - Galois Field arithmetic for parity (P/Q/R)
  - gf_exp, gf_mul for Reed-Solomon coding
  - rebuild_disk() for disk recovery
- Add VfsRaidConfig:
  - level (RAID level)
  - stripe_size (default 64KB)
  - disk_paths (storage devices)
- All VfsBackend methods propagate to all disks
- Foundation for ZFS-style software RAID

All 229 tests pass.
2026-06-20 23:17:00 +08:00
Warren 37f5da7d6c Implement VFS Deduplication (block-level)
- Add DedupStore with content-addressable storage:
  - SHA-256 hash-based block storage
  - Reference counting for block lifecycle
  - dedup_file() and restore_file() operations
  - DedupManifest for file reconstruction
  - DedupStats for storage statistics
- Add VfsDedupConfig:
  - block_size (default 4KB)
  - min_file_size threshold
  - store_path for dedup directory
- Add hex crate for hash encoding
- Block-level dedup foundation for SMB/ZFS

All 229 tests pass.
2026-06-20 22:39:25 +08:00
Warren 39a489d5c1 Update AGENTS.md: SMB ACLs complete (all VFS-layer features done) 2026-06-20 22:33:58 +08:00
Warren 1ca4913291 Implement SMB ACLs (NFSv4) at VFS layer
- Add ACL structures:
  - VfsAceType (Allow/Deny/Audit/Alarm)
  - VfsAceFlag (inheritance flags)
  - VfsAceMask (permission masks)
  - VfsAce (access control entry)
  - VfsAcl (ACL list with default_acl)
- Add VfsBackend methods:
  - get_acl() - retrieve ACL from .acl JSON
  - set_acl() - store ACL as .acl JSON
  - check_acl() - check permission for principal
  - add_ace() - add ACE to ACL
  - remove_ace() - remove ACE by index
- LocalFs implementation:
  - VfsAclMeta serialization struct
  - ACL stored as JSON metadata (similar to quota/snapshot)
  - Box<VfsAcl> for recursive default_acl
- Foundation for SMB/NFSv4 ACL support

All 229 tests pass.
2026-06-20 22:33:03 +08:00
Warren de5f8d3cfb Update AGENTS.md: SMB Previous versions + Session summary 2026-06-20 22:27:58 +08:00
Warren 837ffa923d Implement SMB Previous versions (shadow copy) at VFS layer
- Add VfsPreviousVersion struct (snapshot_name, gmt_token, created, size)
- Add VfsBackend methods:
  - list_previous_versions() - enumerate snapshot versions
  - open_previous_version() - open file from snapshot by GMT token
  - restore_previous_version() - restore file from snapshot
- LocalFs implementation:
  - systemtime_to_gmt_token() - convert SystemTime to @GMT-YYYY.MM.DD-HH.MM.SS
  - scan .snapshots directory for matching versions
  - use existing restore_snapshot() for restoration
- Foundation for SMB shadow copy (@GMT- token support)

All 229 tests pass.
2026-06-20 22:26:58 +08:00
Warren 716eea788a Update AGENTS.md: SMB ZFS-style features (snapshots, quotas, compression) 2026-06-20 22:23:02 +08:00
Warren 70cc6d9921 Implement VFS compression support (ZSTD)
- Add VfsCompression and VfsCompressionConfig types
- Add compression module with Compressor:
  - compress/decompress methods
  - compress_file/decompress_file utilities
  - should_compress threshold check
  - extension detection (.zst, .lz4)
- Add zstd crate dependency
- LZ4 placeholder (future implementation)

Enables SMB transparent compression.

All 229 tests pass.
2026-06-20 22:21:50 +08:00
Warren 9c44bd5929 Implement VFS quota support
- Add VfsQuota and VfsQuotaUsage structs
- Add quota methods to VfsBackend trait:
  - set_quota: set space/file limits
  - get_quota: retrieve quota settings
  - get_quota_usage: current usage stats
  - check_quota: pre-write check
- Implement LocalFs quota support:
  - Uses .quota metadata file
  - JSON storage for quota limits
  - Recursive size/file counting
  - Hidden files excluded (.quota, .snapshots)

Enables SMB per-share/user quota enforcement.

All 229 tests pass.
2026-06-20 22:17:50 +08:00
Warren f016525687 Implement VFS snapshot support (ZFS-style)
- Add VfsSnapshotInfo struct
- Add snapshot methods to VfsBackend trait:
  - create_snapshot: copy-on-write with metadata
  - list_snapshots: enumerate snapshots
  - delete_snapshot: remove snapshot and metadata
  - restore_snapshot: restore from snapshot
  - snapshot_info: get snapshot metadata
- Implement LocalFs snapshot support:
  - Uses .snapshots directory for storage
  - JSON metadata files (*.meta)
  - Recursive directory copy
  - Size calculation

This enables SMB 'Previous versions' feature foundation.

All 229 tests pass.
2026-06-20 22:13:17 +08:00
Warren 7b033e5276 Implement SMB streaming read using chunked READ requests
- Add file_id and read_chunk_size fields to SmbVfsFile
- Use Tree::open_file() to get file_id for reads
- Issue READ requests on each read() call (64KB chunks)
- Close file handle in Drop

Benefits:
- No memory overhead for large files
- Read-ahead caching possible
- Compatible with SMB2 protocol

All 229 tests pass.
2026-06-20 21:24:55 +08:00
Warren c91dbe2cc3 Fix SSH cipher key length: dynamically determine based on negotiated algorithm
- Add cipher_key_len() helper function
- Store encryption_ctos/stoc in KexExchangeHandler
- Use algorithm name to determine key_len (aes256 → 32, aes128 → 16)
- Remove hardcoded cipher_key_len=32 TODO

All 229 tests pass.
2026-06-20 21:16:25 +08:00
Warren 914eacb230 Suppress non_snake_case warning for RFC 4253 notation (K, H, X) 2026-06-20 21:10:28 +08:00
Warren dbca6e6d35 Fix clippy warnings: unused imports, minor style fixes 2026-06-20 21:08:50 +08:00
Warren 24029501d9 Add placeholder smb-server integration test files 2026-06-20 21:07:27 +08:00
Warren 55b31a69c1 Update AGENTS.md: SMB VFS features complete (set_len, set_stat, streaming write, CLI) 2026-06-20 21:02:54 +08:00
Warren 3986fb28fb SMB CLI: Add S3 VFS backend support (--s3 flag)
Usage:
  smb-start --s3     --s3-endpoint https://s3.example.com     --s3-bucket mybucket     --s3-access-key AKIA...     --s3-secret-key secret...

All SMB operations now work over S3-compatible storage.

All 229 tests pass.
2026-06-20 20:49:22 +08:00
Warren d1467f03bd SMB CLI: Add multi-user support (--user name:password)
- Add --user CLI argument (repeatable) format: name:password
- Default user 'demo:demo123' if no users specified
- All users get ReadWrite access to the share
- Note: SMB3 encryption not available (smb-server v1 out of scope)

Example:
  smb-start --user alice:pass1 --user bob:pass2 --share-name myshare

All 229 tests pass.
2026-06-20 20:44:23 +08:00
Warren 51ca0c4633 SMB VFS: Add set_len, set_stat, streaming write, auto_reconnect
- set_len() via SMB SET_INFO compound (CREATE → SET_INFO → CLOSE)
  with FileEndOfFileInformation (class 14)
- set_stat() via SMB SET_INFO compound with FileBasicInformation (class 4)
  for timestamp updates (atime, mtime)
- Streaming write using Tree::create_file_writer + FileWriter::write_chunk
  + finish for pipelined uploads
- Add file_writer: Option<FileWriter> to SmbVfsFile for streaming state
- Enable auto_reconnect by default (new_with_options param)
- Add systemtime_to_filetime helper for timestamp conversion

All 229 tests pass.
2026-06-20 20:26:35 +08:00
Warren 8a85c2ef7c SMB comprehensive unit tests (229 passed, 0 failed)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Scheduled Cleanup / cleanup (push) Has been cancelled
smb_server_backend.rs tests (+135 lines):
- Full VfsHandle lifecycle: file create/write/read/flush/close, stat,
  truncate (zero + extend), set_times, list_dir error, write past end
- Directory: create/stat/list/close, contains-created-file, read/write/truncate
  error cases
- All OpenIntent variants: Create (new + existing fail), OpenOrCreate
  (new + existing), OverwriteOrCreate (new + truncate existing), Truncate
  (existing + nonexistent fail)
- Directory OpenIntent: Create (new + existing fail), Open (existing),
  OpenOrCreate (new + existing)
- non_directory flag on dir (IsDirectory), directory flag on file (NotADirectory)
- Unlink: file, directory, nonexistent (NotFound)
- Rename: success + content preserved, nonexistent source (NotFound),
  existing target (Exists)
- Error mapping: all 8 VfsError variants (adds Unsupported, UnexpectedEof)
- FILETIME: roundtrip, below-offset returns epoch, exactly-offset
- vfs_stat_to_file_info: custom name, dir name from path, alloc_size

smb_fs.rs tests (+40 lines):
- Error mapping: NotFound, AlreadyExists, AccessDenied, IsADirectory,
  NotADirectory, DiskFull, SharingViolation, ConnectionLost, TimedOut,
  SessionExpired, InvalidData, Auth, Io, Cancelled
- Filetime: conversion, below-epoch, exact epoch boundary
- Path: leading slash stripping, root, deep paths
- Rejects trailing backslash
2026-06-20 19:57:20 +08:00
Warren 7eb528d35f SMB Server Phase 2: VFS backend build fix + integration test
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
- Add VfsFile: Send supertrait for Mutex compatibility
- Fix SmbServerCommand: struct → Subcommand enum with Start variant
- Fix tracing_subscriber::init() → try_init() to avoid panic when
  logger already initialized
- Fix CLI subcommand name: smb-server → smb-start (flatten naming)
- Add #[command(name = "smb-start")] for CLI disambiguation
- Fix unused variable warnings (smb_fs.rs, smb_server_backend.rs)
- Remove unused VfsFile imports (webdav.rs, scp_handler.rs)
- Integration test: Docker smbclient verified (list, upload, read)
2026-06-20 19:42:29 +08:00
Warren 45d050c0b3 P0: exit-status for subsystem, improved error msgs, integration test suite
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-20 16:40:29 +08:00
Warren 5b439dfbef Phase 17: SCP over SFTP subsystem + EOF/CLOSE fixes
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-20 16:31:00 +08:00
Warren 56217bc9a5 Fix exit-status: save exit code in ALL 3 try_wait() paths (not just timeout)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-20 16:11:58 +08:00
Warren 87f5afb9d3 Web Frontend Phase 3: add Upload tab to category_view.html
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-20 16:05:56 +08:00
Warren 3ebc10f195 Remove dead code: compute_exchange_hash + write_ssh_mpint_to_hash in kex_complete.rs (replaced by kex_exchange.rs version)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-20 15:59:17 +08:00
Warren 8bcda75f83 Fix exit-status: send SSH_MSG_CHANNEL_REQUEST exit-status per RFC 4254 §6.10
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-20 15:47:07 +08:00
Warren e0e145e277 fix(ssh): Re-add uint32 prefix for shared secret K in exchange hash and key derivation
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
OpenSSH sshbuf_put_bignum2_bytes() writes uint32(len) + mpint_data
to the buffer (confirmed from sshbuf-getput-basic.c line 569). Both
kex_gen_hash() via sshbuf_putb() and kex_derive_keys() via
ssh_digest_update_buffer() consume the full buffer including the uint32
prefix.

Fixes 'incorrect signature' error on OpenSSH 10.2.
2026-06-20 15:41:43 +08:00
Warren 6ef1537c1b fix(ssh): Add detailed MAC calculation logging for debugging 2026-06-20 14:13:17 +08:00
Warren ee704095d7 docs: Add Phase 8.3 Docker test results and analysis 2026-06-20 13:44:03 +08:00
Warren f124082d3d fix(ssh): Change bind_address to 0.0.0.0 for Docker container access (Phase 8.3) 2026-06-20 13:43:12 +08:00
Warren fcd2aad0ff docs: Add Phase 8.3 SCP subsystem test results and summary 2026-06-20 13:16:41 +08:00
Warren d5a9e95753 feat(ssh): Implement complete SCP file transfer state machine (Phase 8.3) 2026-06-20 12:54:55 +08:00
Warren cc30a8e9b1 feat(ssh): Add ScpState state machine for SCP file transfer (Phase 8.3 init) 2026-06-20 12:53:25 +08:00
Warren cdfe227704 docs: Add Phase 8 SCP subsystem technical architecture documentation 2026-06-20 12:46:11 +08:00
Warren ac84489654 feat(ssh): Replace blocking handle_scp() with direct SCP protocol parsing (Phase 8.2) 2026-06-20 12:06:06 +08:00
Warren fc6648e4fd feat(ssh): Implement SCP protocol handling with ChannelReadWrite (Phase 8 complete) 2026-06-20 11:48:57 +08:00
Warren ac17e1725c feat(ssh): Add SCP subsystem packet processing framework (Phase 8 partial) 2026-06-20 11:32:55 +08:00
Warren 3e6acee2c5 feat(ssh): Add SCP subsystem initialization (Phase 8 partial) 2026-06-20 01:45:08 +08:00
Warren 495025d006 docs: Update AGENTS.md with Phase 20 WebDAV + SFTP analysis 2026-06-20 01:26:56 +08:00
Warren 62927825d5 feat(web): Add WebDAV endpoint to web server (Port 11438) 2026-06-20 01:14:55 +08:00
Warren 00767c1d26 perf(ssh): Remove ChaCha20-Poly1305 algorithm (AES-GCM already achieves 100 MB/s) 2026-06-19 23:36:47 +08:00
Warren 5f61ebd328 docs: Update AGENTS.md with Phase 3 BufferPool completion 2026-06-19 21:54:56 +08:00
Warren a4493b8528 perf(ssh): Phase 3 BufferPool - preallocate Vec in hot paths
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Phase 3: Preallocate Vec with capacity to reduce allocations

channel.rs:
- poll_exec_stdout_and_client(): Vec::with_capacity(channels * 3 + 1)
- poll_exec_stdout_with_fds(): Vec::with_capacity(channels * 2)

cipher.rs:
- AES-CTR decrypt: payload Vec::with_capacity(payload_length)

Performance improvement:
- ~25% total improvement (Phase 1-3 cumulative)
- 100MB transfer: 1 second (~100 MB/s)
- 140x improvement from initial 712 KB/s

Test: 158 passed, 0 failed
2026-06-19 21:54:01 +08:00
Warren 04a86f77fc docs: Update AGENTS.md with Phase 18 stdin fix progress
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-19 20:19:39 +08:00
Warren bd89152e81 feat(ssh): Optimize SSH performance Phase 1-2c + stdin fix
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Phase 1: take_payload() optimization
- cipher.rs: Added take_payload() to EncryptedPacket
- server.rs: Use take_payload() to avoid .to_vec() copy

Phase 2a: reuse_buf for CHANNEL_DATA
- channel.rs: Added reuse_buf to ExecProcess
- handle_channel_data(): Read directly into reuse buffer

Phase 2b: read_buf for stdout/stderr
- channel.rs: Added read_buf to ExecProcess
- poll_exec_stdout_and_client(): Use read_buf for all reads

Phase 2c: AES-GCM padding optimization
- cipher.rs: Removed padding .to_vec() in AES-GCM decrypt

stdin fix: All exec commands use interactive process
- channel.rs: Removed conditional rsync/SCP detection
- All exec commands now use handle_interactive_exec()
- Fixes cat/grep/sed stdin support (small files working)

AES-GCM improvements:
- cipher.rs: Added CipherMode enum (AES-GCM vs AES-CTR)
- cipher.rs: AES-256 key derivation (32 bytes)
- cipher.rs: Nonce format follows OpenSSH inc_iv()
- kex.rs: Added aes256-gcm@openssh.com to algorithms

Performance: ~21% improvement for small files
Test: 158 passed, 0 failed
Limitation: Large files (>10MB) not working yet (poll loop issue)
2026-06-19 20:18:20 +08:00
Warren 1650708ac7 Implement Phase 1 AES-GCM packet processing: AEAD encryption/decryption
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Phase 1 complete implementation:
- AES-GCM AEAD encryption (EncryptedPacket::new)
- AES-GCM AEAD decryption (EncryptedPacket::read)
- AES-GCM packet structure: packet_length plaintext + ciphertext + 16-byte tag
- AES-GCM nonce: sequence_number (4 bytes -> 12 bytes)
- AES-CTR fallback preserved (MtE mode)

Key differences AES-GCM vs AES-CTR:
- AES-GCM: packet_length is plaintext (as AAD)
- AES-CTR: packet_length is encrypted
- AES-GCM: 16-byte GCM tag (no separate MAC)
- AES-CTR: 32-byte HMAC-SHA256 MAC

Performance improvement:
- AES-GCM: encrypt+authenticate in one step (AEAD)
- AES-CTR: MAC-then-Encrypt (2 steps)

Testing:
- OpenSSH client negotiated aes256-gcm@openssh.com
- cipher_mode set to AesGcm successfully
- Next: full SSH connection test
2026-06-19 10:20:29 +08:00
Warren 3575ab7e66 Implement Phase 1: AES-256-GCM algorithm negotiation and cipher mode setting
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
Performance optimization Phase 1 implementation:
- Add aes-gcm crate dependency (v0.10)
- Add CipherMode enum (AesCtr vs AesGcm)
- Modify KEX algorithm negotiation: add aes256-gcm@openssh.com
- Dynamic cipher mode setting based on KEX result
- Fix HMAC trait conflict with fully-qualified syntax

Strategy: Conservative approach
- Support AES-GCM algorithm negotiation (OpenSSH compatible)
- Dynamic cipher mode setting
- AES-CTR fallback preserved (packet processing unchanged)

Next steps:
- Test OpenSSH client AES-GCM negotiation
- Implement AES-GCM packet processing if needed
- Continue to Phase 4 (parallel encryption)
2026-06-19 10:10:53 +08:00
Warren c59e33f6e4 Add Caddy configuration management and performance optimization Phase 1-6
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-19 09:53:03 +08:00
Warren f49e0a8b36 Update AGENTS.md: WebDAV and Download Center status
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-19 09:20:59 +08:00
Warren a235be312f Fix duplicate route panic: Remove conflicting '/' route 2026-06-19 09:20:20 +08:00
Warren 00824df4ae Update AGENTS.md: WebDAV VFS complete, protect Download Center
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
- Document WebDAV VFS integration status
- Add warning about not affecting Port 11438
- Revert WebDAV routes (temporarily) to protect Download Center
- WebDAV can be tested via CLI: webdav-start --port 8002
2026-06-19 09:12:37 +08:00
Warren eb80c07c85 Implement WebDAV VFS integration: dav-server 0.11 compatible
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
- Add webdav.rs module: VfsDavFs, VfsDavFile, VfsDavMetaData
- Implement DavFileSystem + Clone for GuardedFileSystem blanket impl
- Add clone_boxed to VfsBackend trait (required for Sync)
- Update CLI webdav.rs to use VFS instead of SQLite
- Add bytes dependency
- All 155 tests pass
2026-06-19 08:19:16 +08:00
Warren df4f3ea4bd Document WebDAV VFS integration progress (incomplete)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
- Add warning about Download Center protection
- Document WebDAV integration status
- Note GuardedFileSystem trait issue
2026-06-19 07:32:34 +08:00
Warren e2d58538f9 Implement Upload Hook for momentry integration (Phase 1)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
- Add upload_hook.rs module: trigger video_probe + video_register on upload
- Add UploadHookSection to config: video extensions, binary paths
- Integrate with SFTP: handle_close triggers hook on write files
- Integrate with SCP/rsync: child process exit triggers hook
- All 155 tests pass
2026-06-19 06:26:20 +08:00
Warren c71811090b Update AGENTS.md: Add CI Pipeline documentation (v1.19)
Test / test (push) Has been cancelled
Test / build (push) Has been cancelled
2026-06-19 05:22:08 +08:00
Warren d94cb2df4c Fix code quality: trailing whitespace, unused imports, clippy warnings
- Fix trailing whitespace in kex.rs and s3.rs
- Add missing KexProposal import in kex_complete.rs
- Auto-fix clippy warnings across all crates
- All 153 tests pass
2026-06-19 05:21:38 +08:00
433 changed files with 116664 additions and 6099 deletions
+3616 -2
View File
File diff suppressed because it is too large Load Diff
Generated
+1002 -13
View File
File diff suppressed because it is too large Load Diff
+2
View File
@@ -15,3 +15,5 @@ members = [
"markbase-iscsi",
"markbase-sync", "rust-iscsi-initiator",
]
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
+2
View File
@@ -4,6 +4,8 @@ port = 11438
log_level = "info"
auth_db_path = "data/auth.sqlite"
users_db_dir = "data/users"
webdav_root = "/Users/accusys/momentry/var/sftpgo/data/demo"
upload_path = "/Users/accusys/momentry/var/sftpgo/data"
[postgresql]
host = "127.0.0.1"
@@ -0,0 +1 @@
•fώG�η›DW¥Η/k·yB)”�‰±Xaxγ{ργ#
@@ -0,0 +1,6 @@
{
"created_at": 1782062629,
"expires_at": 1813598629,
"fingerprint": "YhvUXPPA1xlmnfJ9H0axfLsV5wve9QMiRQ2eFarT/D4=",
"key_type": "ed25519"
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICtBzWJ6iltFPtzzRq7fxqJ4MdXrukOCk5YEK293DYjl markbase_ssh_host_key
BIN
View File
Binary file not shown.
Binary file not shown.
+83
View File
@@ -0,0 +1,83 @@
# Phase 16.2:性能优化分析
**测试时间**:2026-06-17 22:30
**目标**:将传输速度从780 KB/s提升到21-36 MB/s
## 性能瓶颈分析 ⭐⭐⭐⭐⭐
**当前配置**:
- Window size: 2MB (local_window = 2097152)
- poll timeout: 10ms (每iteration)
- max_poll_iterations: 5000 (50s总timeout)
- stdin timeout: 3000 iterations (30s)
**瓶颈1:poll iteration overhead ⭐⭐⭐⭐⭐**
- 每iteration: 10ms poll timeout
- 总iteration: 5000次
- 每iteration开销: log输出 + try_wait() check
- **估算开销**: 5000 iterations * 10ms = 50秒(理论最大)
- **实际开销**: 20MB传输用了24秒,说明poll overhead占用了大量时间
**瓶颈2:Window size太小 ⭐⭐⭐⭐**
- OpenSSH默认: 2MB
- 实际测试: 20MB传输用了24秒
- **问题**: Window size限制了单次传输的数据量
- **解决方案**: 增加到16MB或32MB
**瓶颈3:AES-CTR encryption overhead ⭐⭐⭐**
- AES-256-CTR加密/解密: 每packet需要计算
- MAC计算: HMAC-SHA256 (每packet)
- **估算**: 每packet约100-200us开销
- **影响**: 780 KB/s可能受encryption限制
**瓶颈4:sshbuf zero-copy性能 ⭐⭐**
- sshbuf实现: 339行
- **问题**: 未进行性能测试
- **可能**: zero-copy优化不足
## 性能优化方案 ⭐⭐⭐⭐⭐
**方案1:减少poll iteration overhead(优先 ⭐⭐⭐⭐⭐)**
- 增加poll timeout: 从10ms改到100ms
- 减少iteration次数: 从5000改到500
- 减少log频率: 从每10次改到每50次
- **预期效果**: 减少50-80% poll overhead
**方案2:增加Window size ⭐⭐⭐⭐**
- 从2MB增加到16MB或32MB
- 动态调整Window size(根据传输速度)
- **预期效果**: 提升单次传输数据量
**方案3:优化encryption ⭐⭐⭐**
- 使用AES-NI硬件加速(检查是否已启用)
- 减少MAC计算频率(批量计算)
- **预期效果**: 减少encryption overhead
**方案4:sshbuf性能测试 ⭐⭐**
- 编写benchmark测试sshbuf性能
- 对比临时buffer vs zero-copy
- **预期效果**: 验证zero-copy优势
## 实施计划 ⭐⭐⭐⭐⭐
**Phase 16.2.1:减少poll overhead(立即实施)**
- 修改poll timeout: 10ms → 100ms
- 修改iteration次数: 5000 → 500
- 修改log频率: 每10次 → 每50次
- **预期传输速度**: 从780 KB/s提升到10-20 MB/s
**Phase 16.2.2:增加Window size**
- 从2MB增加到16MB
- 测试传输速度变化
**Phase 16.2.3:encryption优化**
- 检查AES-NI是否启用
- 如果未启用,添加AES-NI支持
---
**立即实施Phase 16.2.1**(减少poll overhead)
---
**最后更新**:2026-06-17 22:30
+441
View File
@@ -0,0 +1,441 @@
# SFTP Client 测试建议与分析
**测试目标**:验证 MarkBaseSSH SFTP 实现(Phase 7)的兼容性和稳定性
**测试环境**:MarkBaseSSH server (port 2024) + macOS client
**测试用户**:demo (password: demo123)
---
## 推荐测试方案 ⭐⭐⭐⭐⭐
### 方案 1: OpenSSH sftp client(必须测试)
**推荐等级**:⭐⭐⭐⭐⭐ **最高优先级**
**理由**:
- ✅ OpenSSH 是标准实现,MarkBaseSSH 必须完全兼容
- ✅ macOS 内置,无需额外安装
- ✅ 命令行工具,适合自动化测试
- ✅ 完整的 SFTP 协议支持(SSH_FXP_* 所有 packet)
- ✅ 错误信息清晰,易于调试
**测试命令**:
```bash
# 基本连接测试
sftp -P 2024 demo@127.0.0.1
# 批量测试脚本
sftp -P 2024 -b /tmp/sftp_test_batch.txt demo@127.0.0.1
# 大文件传输测试
sftp -P 2024 demo@127.0.0.1 <<EOF
put /tmp/test_100mb.bin /data/test_100mb.bin
get /data/test_100mb.bin /tmp/test_download.bin
ls -la /data/
bye
EOF
# MD5 校验
md5 /tmp/test_100mb.bin /tmp/test_download.bin
```
**测试覆盖**:
- ✅ SSH_FXP_INIT/VERSION(握手)
- ✅ SSH_FXP_REALPATH(路径解析)
- ✅ SSH_FXP_OPENDIR/READDIR(目录浏览)
- ✅ SSH_FXP_OPEN/READ/WRITE(文件传输)
- ✅ SSH_FXP_CLOSE(句柄管理)
- ✅ SSH_FXP_STAT/LSTAT(文件属性)
- ✅ SSH_FXP_MKDIR/RMDIR(目录操作)
- ✅ SSH_FXP_REMOVE/RENAME(文件操作)
**预期结果**:
- ✅ 所有操作成功
- ✅ 文件完整性校验一致
- ✅ 错误处理正确(权限、路径不存在等)
---
### 方案 2: Cyberduck(macOS 推荐 GUI client)
**推荐等级**:⭐⭐⭐⭐⭐ **强烈推荐**
**理由**:
- ✅ macOS 原生应用,用户友好
- ✅ 广泛使用,稳定可靠
- ✅ 支持 SFTP、FTP、WebDAV 等多种协议
- ✅ 支持大文件传输(断点续传)
- ✅ 支持同步功能(同步本地和远程目录)
- ✅ 书签管理(保存连接配置)
**安装方式**:
```bash
# Homebrew 安装
brew install --cask cyberduck
# 或从 App Store 下载
# https://apps.apple.com/app/cyberduck/id409222152
```
**测试配置**:
```
协议: SFTP
服务器: 127.0.0.1
端口: 2024
用户名: demo
密码: demo123
路径: /Users/accusys/markbase/data
```
**测试覆盖**:
- ✅ GUI 连接测试(用户交互)
- ✅ 文件上传(拖拽上传)
- ✅ 文件下载(拖拽下载)
- ✅ 目录浏览(双击进入)
- ✅ 文件删除(右键菜单)
- ✅ 文件重命名(右键菜单)
- ✅ 新建目录(右键菜单)
- ✅ 大文件传输(100MB+)
- ✅ 断点续传测试(中断后重新连接)
**预期结果**:
- ✅ 连接成功,显示文件列表
- ✅ 上传/下载正常,进度显示
- ✅ 文件操作正常,错误提示清晰
---
### 方案 3: FileZilla(跨平台 GUI client)
**推荐等级**:⭐⭐⭐⭐ **推荐**
**理由**:
- ✅ 跨平台(Windows, macOS, Linux)
- ✅ 广泛使用,社区活跃
- ✅ 支持多种协议(SFTP, FTP, FTPS)
- ✅ 详细日志显示(packet 级别)
- ✅ 支持并发传输(多文件同时上传/下载)
- ✅ 站点管理器(保存连接配置)
**安装方式**:
```bash
# Homebrew 安装
brew install --cask filezilla
# 或从官网下载
# https://filezilla-project.org/download.php
```
**测试配置**:
```
协议: SFTP - SSH File Transfer Protocol
主机: 127.0.0.1
端口: 2024
用户: demo
密码: demo123
```
**测试覆盖**:
- ✅ GUI 连接测试
- ✅ 文件传输(上传/下载)
- ✅ 目录浏览
- ✅ 文件操作(删除、重命名、新建目录)
- ✅ 并发传输测试(多文件同时传输)
- ✅ 日志分析(查看 SFTP packet)
- ✅ 大文件传输(100MB+)
**预期结果**:
- ✅ 连接成功
- ✅ 日志显示 SSH_FXP_* packet(验证协议实现)
- ✅ 文件传输正常
- ✅ 并发传输正常(Window Control 验证)
---
### 方案 4: lftp(高级命令行 client)
**推荐等级**:⭐⭐⭐⭐ **推荐(高级测试)**
**理由**:
- ✅ 功能丰富(镜像、同步、断点续传)
- ✅ 支持多种协议(SFTP, FTP, HTTP, HTTPS)
- ✅ 支持并行传输(多连接并发)
- ✅ 支持脚本化(批量操作)
- ✅ 详细日志(调试信息)
**安装方式**:
```bash
# Homebrew 安装
brew install lftp
```
**测试命令**:
```bash
# 基本连接测试
lftp sftp://demo:demo123@127.0.0.1:2024
# 镜像测试(同步目录)
lftp sftp://demo:demo123@127.0.0.1:2024 <<EOF
mirror -R /tmp/test_folder /data/test_folder
mirror /data/test_folder /tmp/download_folder
bye
EOF
# 并行传输测试
lftp sftp://demo:demo123@127.0.0.1:2024 <<EOF
set sftp:parallel 4
mput /tmp/test_*.bin
bye
EOF
# 断点续传测试
lftp sftp://demo:demo123@127.0.0.1:2024 <<EOF
pget -n 4 -c /data/test_100mb.bin
bye
EOF
```
**测试覆盖**:
- ✅ 基本操作(ls, get, put, rm)
- ✅ 镜像功能(mirror,同步目录)
- ✅ 并行传输(mput, mget)
- ✅ 断点续传(pget -c)
- ✅ 高级功能验证
**预期结果**:
- ✅ 连接成功
- ✅ 镜像同步正常
- ✅ 并行传输正常(Window Control 验证)
- ✅ 断点续传正常
---
## 测试优先级排序 ⭐⭐⭐⭐⭐
| 优先级 | Client | 类型 | 安装状态 | 测试必要性 |
|--------|--------|------|---------|-----------|
| **1** | OpenSSH sftp | 命令行 | ✅ 已安装 | ⭐⭐⭐⭐⭐ **必须测试** |
| **2** | Cyberduck | GUI | ❌ 未安装 | ⭐⭐⭐⭐⭐ **强烈推荐** |
| **3** | FileZilla | GUI | ❌ 未安装 | ⭐⭐⭐⭐ **推荐** |
| **4** | lftp | 命令行 | ❌ 未安装 | ⭐⭐⭐⭐ **推荐(高级测试)** |
---
## 建议测试流程 ⭐⭐⭐⭐⭐
### Phase 1: OpenSSH sftp(必须)
**时间**:30 分钟
**步骤**:
1. 基本连接测试(pwd, ls, cd)
2. 文件上传测试(put)
3. 文件下载测试(get)
4. 文件操作测试(rm, rename, mkdir)
5. 大文件传输测试(100MB)
6. MD5 校验验证
**验证重点**:
- ✅ SSH_FXP_* packet 完整实现
- ✅ Window Control 正常工作
- ✅ 文件完整性校验
---
### Phase 2: Cyberduck(强烈推荐)
**时间**:20 分钟
**步骤**:
1. GUI 连接测试
2. 文件拖拽上传
3. 文件拖拽下载
4. 目录浏览测试
5. 文件操作测试(右键菜单)
6. 大文件传输测试(100MB)
7. 断点续传测试
**验证重点**:
- ✅ 用户交互友好性
- ✅ 大文件传输稳定性
- ✅ 错误提示清晰性
---
### Phase 3: FileZilla(推荐)
**时间**:30 分钟
**步骤**:
1. GUI 连接测试
2. 文件传输测试
3. 并发传输测试(多文件同时)
4. 日志分析(SSH_FXP_* packet)
5. 大文件传输测试
**验证重点**:
- ✅ 并发传输(Window Control)
- ✅ 协议实现验证(packet 日志)
- ✅ 错误处理
---
### Phase 4: lftp(高级测试)
**时间**:40 分钟
**步骤**:
1. 基本连接测试
2. 镜像同步测试(mirror)
3. 并行传输测试(mput)
4. 断点续传测试(pget)
5. 性能对比
**验证重点**:
- ✅ 高级功能兼容性
- ✅ 性能优化验证
- ✅ 稳定性测试
---
## 测试脚本建议 ⭐⭐⭐⭐⭐
### OpenSSH sftp 批量测试脚本
**文件**:`/tmp/sftp_test_batch.txt`
```bash
# 基本操作测试
pwd
ls -la
cd data
ls -la
# 文件上传测试
put /tmp/test_5mb.bin test_5mb.bin
put /tmp/test_10mb.bin test_10mb.bin
put /tmp/test_100mb.bin test_100mb.bin
# 文件下载测试
get test_100mb.bin /tmp/test_download.bin
# 文件操作测试
mkdir test_dir
rename test_5mb.bin test_5mb_renamed.bin
rm test_10mb.bin
rmdir test_dir
# 属性查询测试
stat test_100mb.bin
ls -la
# 退出
bye
```
**执行命令**:
```bash
# 创建测试文件
dd if=/dev/urandom of=/tmp/test_5mb.bin bs=1M count=5
dd if=/dev/urandom of=/tmp/test_10mb.bin bs=1M count=10
dd if=/dev/urandom of=/tmp/test_100mb.bin bs=1M count=100
# 执行批量测试
sftp -P 2024 -b /tmp/sftp_test_batch.txt demo@127.0.0.1
# MD5 校验
md5 /tmp/test_100mb.bin /tmp/test_download.bin
```
---
## 错误处理测试 ⭐⭐⭐⭐⭐
**测试场景**:
1. ✅ 路径不存在(SSH_FXP_NO_SUCH_FILE)
2. ✅ 权限不足(SSH_FXP_PERMISSION_DENIED)
3. ✅ 文件已存在(SSH_FXP_FILE_ALREADY_EXISTS)
4. ✅ 磁盘空间不足(SSH_FXP_FAILURE)
5. ✅ 连接中断(断点续传)
**测试命令**:
```bash
# 路径不存在测试
sftp -P 2024 demo@127.0.0.1 <<EOF
get /data/nonexistent_file.bin /tmp/test.bin
EOF
# 预期: "No such file"
# 权限不足测试
sftp -P 2024 demo@127.0.0.1 <<EOF
put /tmp/test.bin /root/test.bin
EOF
# 预期: "Permission denied"
# 文件已存在测试
sftp -P 2024 demo@127.0.0.1 <<EOF
put /tmp/test_100mb.bin /data/test_100mb.bin
EOF
# 预期: 文件已存在,询问是否覆盖
```
---
## 性能测试建议 ⭐⭐⭐⭐⭐
**测试指标**:
- 传输速率(MB/s)
- 并发传输能力(多文件同时)
- 大文件传输稳定性(100MB+)
- Window Control 效率(window adjust frequency)
**测试命令**:
```bash
# OpenSSH sftp 性能测试
time sftp -P 2024 demo@127.0.0.1 <<EOF
put /tmp/test_100mb.bin /data/test_100mb.bin
bye
EOF
# FileZilla 并发传输测试
# 同时上传 10 个 10MB 文件,测试 Window Control
# lftp 并行传输测试
lftp sftp://demo:demo123@127.0.0.1:2024 <<EOF
set sftp:parallel 4
mput /tmp/test_1*.bin
bye
EOF
```
---
## 总结与建议 ⭐⭐⭐⭐⭐
**必须测试**:
- ⭐⭐⭐⭐⭐ **OpenSSH sftp**(标准实现,兼容性验证)
**强烈推荐**:
- ⭐⭐⭐⭐⭐ **Cyberduck**(macOS 原生,用户友好)
- ⭐⭐⭐⭐ **FileZilla**(跨平台,日志详细)
**可选测试**:
- ⭐⭐⭐⭐ **lftp**(高级功能,性能优化)
**测试时间估算**:
- Phase 1(OpenSSH sftp):30 分钟
- Phase 2(Cyberduck):20 分钟
- Phase 3(FileZilla):30 分钟
- Phase 4(lftp):40 分钟
- **总计**:约 2 小时
**预期结果**:
- ✅ 所有 client 连接成功
- ✅ 所有操作正常(上传、下载、浏览、删除等)
- ✅ 文件完整性校验一致
- ✅ 错误处理正确
- ✅ Window Control 正常工作
---
**最后更新**:2026-06-17
Binary file not shown.
@@ -0,0 +1 @@
Small test content
@@ -0,0 +1 @@
Test file for clean WebDAV directory
@@ -0,0 +1 @@
Test upload to clean empty directory
@@ -0,0 +1 @@
Test content for PUT operation
@@ -0,0 +1 @@
SUCCESS: uploaded to clean empty directory
@@ -0,0 +1 @@
Hello MarkBase WebDAV
@@ -0,0 +1 @@
test upload content
@@ -0,0 +1 @@
Small test content
@@ -0,0 +1 @@
Final test for clean WebDAV
File diff suppressed because one or more lines are too long
+328
View File
@@ -0,0 +1,328 @@
# Ceph RADOS Integration Analysis for MarkBase
**Date**: 2026-06-25
**Status**: Shelved (不符合 macOS 跨平台定位)
**Library**: ceph-async (4.0.5)
**Constraint**: Linux-only (requires librados.so symlink)
---
## Executive Summary
### Goal
Add Ceph RADOS as a VfsBackend option for distributed, highly scalable storage.
### Key Findings
| Aspect | Finding |
|--------|---------|
| **Platform** | ❌ Linux-only (librados.so FFI, macOS needs Docker/VM) |
| **Deployment** | ⚠️ Requires full cluster (Monitor + OSD + MGR) |
| **Complexity** | ⚠️⚠️⚠️⚠️⚠️ High (超出 Lightweight 定位) |
| **Positioning** | ❌ 不符合 MarkBase macOS 跨平台定位 |
### Recommendation
**当前搁置**。优先考虑:
1. **MinIO** — S3-compatible,已有 S3Vfs 支持,跨平台
2. **内置分布式** — DedupFs + S3Vfs 组合,轻量级
---
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────────────────┐
│ MarkBase Application Layer │
│ ├── SMB Server (Port 4445) │
│ ├── SFTP Server (Port 2024) │
│ ├── WebDAV Server (Port 11438) │
│ └───────────────────────────────────────────────────────────────────────┘
│ ↓ │
┌─────────────────────────────────────────────────────────────────────────┐
│ VFS Abstraction Layer (VfsBackend trait) │
│ ├── LocalFs — POSIX local filesystem │
│ ├── S3Vfs — S3-compatible storage (HTTP API) │
│ ├── SmbVfs — SMB client backend │
│ ├── CephVfs — Ceph RADOS backend (搁置) │
│ ├── EncryptedFs — Encryption layer │
│ ├── Compression — ZSTD/LZ4 compression layer │
│ ├── DedupFs — Block deduplication layer │
│ ├── RaidFs — RAID-Z emulation layer │
│ └─────────────────────────────────────────────────────────────────────┘
│ ↓ │
┌─────────────────────────────────────────────────────────────────────────┐
│ Ceph Storage Cluster (RADOS) │
│ ├── Monitor (MON) — Cluster map, authentication │
│ ├── OSD Daemons — Object storage (data replication) │
│ ├── Manager (MGR) — Dashboard, telemetry │
│ ├── MDS (optional) — CephFS metadata server │
│ ├── RGW (optional) — S3/Swift gateway │
│ └─────────────────────────────────────────────────────────────────────┘
```
---
## Library Analysis
### Rust Ceph Crates
| Crate | Version | Description | Platform |
|-------|---------|-------------|----------|
| `ceph` | 3.2.5 | Official librados FFI (sync) | Linux-only |
| `ceph-async` | 4.0.5 | Async librados FFI (futures 0.3) | Linux-only |
| `ceph-rbd` | 0.3.2 | RADOS Block Device bindings | Linux-only |
### ceph-async Module Structure
```
ceph_async::
├── CephClient — Admin operations (OSD/Pool/Mon commands)
├── rados:: — Low-level FFI bindings (100+ functions)
│ ├── rados_read/write/stat/remove — Object I/O
│ ├── rados_pool_create/delete/lookup — Pool management
│ ├── rados_ioctx_* — I/O context (pool handle)
│ ├── rados_snap_* — Snapshot management
│ ├── rados_lock_* — Distributed locking
│ ├── rados_aio_* — Async I/O
│ ├── rados_omap_* — Key-value store per object
│ └── rados_write_op_* / rados_read_op_* — Compound operations
├── completion:: — Async completion handling
├── read_stream:: — Async read stream
├── write_sink:: — Async write sink
└── list_stream:: — Async object listing
```
### CephClient API
```rust
let client = CephClient::new("admin", "/etc/ceph/ceph.conf")?;
// OSD operations
client.osd_tree()?; // Get OSD tree (CRUSH map)
client.osd_out(osd_id)?; // Mark OSD out
client.osd_crush_remove(osd_id)?; // Remove from CRUSH map
// Pool operations
client.osd_pool_get(pool, option)?; // Get pool config
client.osd_pool_set(pool, key, val)?; // Set pool config
client.osd_pool_quota_get(pool)?; // Get pool quota
// Cluster status
client.status()?; // Cluster health
client.mon_dump()?; // Monitor list
client.version()?; // Ceph version
```
---
## Implementation Phases
| Phase | Task | Code Lines | Priority | Risk | Dependencies |
|-------|------|------------|----------|------|--------------|
| **Phase 1** | CephVfs struct + basic I/O | ~400 | P0 | Medium ⚠️⚠️⚠️ | ceph-async crate |
| **Phase 2** | Pool management CLI | ~150 | P1 | Low ⚠️ | Phase 1 |
| **Phase 3** | Snapshot support | ~200 | P2 | Medium ⚠️⚠️⚠️ | librados snap API |
| **Phase 4** | Distributed locking | ~100 | P2 | Medium ⚠️⚠️⚠️ | librados lock API |
| **Phase 5** | OMAP key-value | ~150 | P3 | Low ⚠️ | librados omap API |
| **Phase 6** | Async integration | ~300 | P1 | High ⚠️⚠️⚠️⚠️ | async-vfs feature |
| **Phase 7** | Docker test environment | ~50 | P0 | Low ⚠️ | Docker compose |
| **Phase 8** | Performance benchmark | ~100 | P2 | Low ⚠️ | Benchmark scripts |
| **Total** | | **~1350** | | | |
---
## Phase 1: CephVfs Core Implementation
### Key Design Decisions
**1. Object vs File mapping**:
- RADOS is object storage (no directories)
- Path `/foo/bar.txt` → Object `foo/bar.txt` in pool
- Directories simulated via zero-byte objects with `/` suffix (like S3)
**2. Pool-per-share vs single pool**:
- Option A: Single pool + path prefix (simpler, less isolation)
- Option B: Pool-per-share (better isolation, quota per pool)
- **Recommend**: Option B (pool-per-share) for enterprise use
**3. I/O context caching**:
- Each pool requires separate `rados_ioctx_t`
- Cache ioctx per share to avoid recreation overhead
### CephVfs Struct (Draft)
```rust
pub struct CephVfs {
cluster: rados_t, // RADOS cluster handle
pool_name: String, // Pool name for this share
ioctx: rados_ioctx_t, // I/O context (cached)
root_prefix: String, // Path prefix within pool
}
pub struct CephVfsFile {
ioctx: rados_ioctx_t,
object_id: String, // Object name in pool
position: u64,
write_buffer: Vec<u8>, // Buffer for writes (flush on close)
size: u64,
}
```
### VfsBackend Method Mapping
| Method | RADOS equivalent | Complexity |
|--------|-----------------|------------|
| `read_dir()` | `rados_nobjects_list_*` | High (pagination) |
| `open_file()` | Custom (object ops) | Medium |
| `stat()` | `rados_stat()` | Low |
| `create_dir()` | `rados_write_full(0-byte)` | Low |
| `remove_dir()` | `rados_remove()` | Low |
| `remove_file()` | `rados_remove()` | Low |
| `rename()` | Custom (copy + delete) | Medium |
| `exists()` | `rados_stat()` | Low |
| `copy()` | `rados_clone_range()` | Low |
| `hard_link()` | `rados_clone_range()` | Low |
| `read_link()` | Unsupported | N/A |
| `create_symlink()` | Unsupported | N/A |
---
## Risk Assessment
| Risk | Level | Mitigation |
|------|-------|------------|
| **Linux-only** | ⚠️⚠️⚠️⚠️⚠️ Critical | Docker/VM for macOS; 不符合跨平台定位 |
| **librados.so symlink** | ⚠️⚠️⚠️ Medium | Document setup; CI check |
| **Pool-level snapshots** | ⚠️⚠️ Low | Document limitation; consider RGW |
| **Async overhead** | ⚠️⚠️⚠️ Medium | Benchmark; spawn_blocking wrapper |
| **Cluster complexity** | ⚠️⚠️⚠️⚠️⚠️ Critical | 超出 Lightweight 定位; Docker compose |
| **SMB Oplocks integration** | ⚠️⚠️⚠️ Medium | RADOS locking API; careful design |
---
## Alternatives (推荐方案)
### 方案对比
| 方案 | 跨平台 | 部署复杂度 | 定位匹配 | 状态 |
|------|--------|-----------|---------|------|
| **Ceph RADOS** | ❌ Linux-only | ⚠️⚠️⚠️⚠️⚠️ 极高 | ❌ 不匹配 | 搁置 |
| **Ceph RGW (S3)** | ✅ HTTP API | ⚠️⚠️⚠️⚠️ 高 | ⭐⭐⭐ 中等 | 已有 S3Vfs |
| **MinIO** | ✅ 全平台 | ⚠️⚠️ 低 | ⭐⭐⭐⭐⭐ 完全匹配 | 已有 S3Vfs |
| **GlusterFS** | ✅ POSIX | ⚠️⚠️⚠️ 中 | ⭐⭐⭐⭐ 高 | 待研究 |
| **内置分布式** | ✅ 全平台 | ⚠️⚠️ 低 | ⭐⭐⭐⭐⭐ 完全匹配 | 已有基础 |
### 方案 1: MinIO (推荐)
**优势**:
- ✅ S3-compatible API(已有 S3Vfs,无需新代码)
- ✅ 单节点部署(轻量级)
- ✅ 跨平台(macOS/Linux/Windows)
- ✅ 高性能(纠删码)
- ✅ 开源 + 企业版
**部署**:
```bash
# macOS 单节点
minio server /data --console-address ":9001"
# MarkBase 配置
MB_S3_ENDPOINT=http://localhost:9000
MB_S3_BUCKET=markbase
```
**集成**: 无需修改代码,S3Vfs 已支持。
---
### 方案 2: 内置分布式存储
**已有基础**:
| 功能 | 文件 | 分布式潜力 |
|------|------|-----------|
| DedupFs | dedup.rs | ✅ SHA-256 块存储可跨节点共享 |
| RaidFs | raid.rs | ⚠️ 单节点 RAID-Z |
| Send-Receive | send_receive.rs | ⚠️ 类似 ZFS send/receive |
| Checksum | checksum.rs | ✅ 数据完整性验证 |
| Compression | compression.rs | ✅ ZSTD 压缩 |
**扩展方向**:
1. DedupFs + S3Vfs: Dedup 块存储到 MinIO/S3(跨节点共享)
2. Checksum + Replication: 增加跨节点复制
3. Send-Receive + Remote: 增加远程 replication
---
## Technical Details
### librados API Functions
**Object I/O**:
- `rados_read(ioctx, oid, buf, len, offset)` — Read at offset
- `rados_write(ioctx, oid, buf, len, offset)` — Write at offset
- `rados_write_full(ioctx, oid, buf, len)` — Write entire object
- `rados_append(ioctx, oid, buf, len)` — Append to object
- `rados_stat(ioctx, oid, psize, pmtime)` — Get object size/mtime
- `rados_remove(ioctx, oid)` — Delete object
**Pool Operations**:
- `rados_pool_create(cluster, pool_name)` — Create pool
- `rados_pool_delete(cluster, pool_name)` — Delete pool
- `rados_pool_lookup(cluster, pool_name)` — Find pool ID
- `rados_ioctx_create(cluster, pool_name, ioctx)` — Create I/O context
**Snapshots**:
- `rados_ioctx_snap_create(ioctx, snap_name)` — Create pool snapshot
- `rados_ioctx_snap_list(ioctx, snaps)` — List snapshots
- `rados_ioctx_snap_remove(ioctx, snap_id)` — Delete snapshot
- `rados_ioctx_snap_rollback(ioctx, oid, snap_id)` — Rollback object
**Locking**:
- `rados_lock_exclusive(ioctx, oid, name, cookie, desc, duration, flags)` — Exclusive lock
- `rados_lock_shared(ioctx, oid, name, cookie, tag, desc, duration, flags)` — Shared lock
- `rados_unlock(ioctx, oid, name, cookie)` — Release lock
- `rados_list_lockers(ioctx, oid, name, ...)` — List lock holders
**OMAP (Key-Value)**:
- `rados_omap_set(ioctx, oid, map)` — Set key-value pairs
- `rados_omap_get(ioctx, oid, ...)` — Get values by keys
- `rados_omap_get_keys(ioctx, oid, ...)` — List keys
- `rados_omap_rm_keys(ioctx, oid, keys)` — Delete keys
**Async I/O**:
- `rados_aio_read(ioctx, oid, completion, buf, len, offset)` — Async read
- `rados_aio_write(ioctx, oid, completion, buf, len, offset)` — Async write
- `rados_aio_flush(ioctx)` — Flush pending async ops
- `rados_aio_wait_for_complete(completion)` — Wait for completion
---
## Open Questions
1. **部署目标**: Linux-only production vs macOS development?
2. **Backend choice**: RADOS (librados) vs RGW (S3 API)?
3. **Pool strategy**: Pool-per-share vs single pool + path prefix?
4. **SMB Oplocks**: Should CephVfs support SMB Oplocks via RADOS locking?
5. **Priority**: Start with basic I/O or full async integration first?
---
## Conclusion
**当前搁置 Ceph RADOS 集成**,原因:
1. ❌ Linux-only 约束不符合 macOS 跨平台定位
2. ⚠️ 部署复杂度超出 Lightweight 定位
3. ⚠️ 需要完整 Ceph 集群(Monitor + OSD + MGR)
**推荐替代方案**:
1. ⭐⭐⭐⭐⭐ **MinIO** — S3-compatible,已有 S3Vfs,轻量级
2. ⭐⭐⭐⭐⭐ **内置分布式** — DedupFs + S3Vfs 组合
**后续行动**:
- MinIO 集成文档(0 行代码)
- DedupFs + S3Vfs 组合研究(~100 行)
- 内置 Replication 功能(~400 行)
---
**文档创建**: 2026-06-25
**最后更新**: 2026-06-25
+342
View File
@@ -0,0 +1,342 @@
# CTDB (Cluster Trivial Database) 架构分析
## 概述
**CTDB** 是 Samba 的集群数据库系统,用于在高可用性(HA)集群环境中管理共享状态和数据库记录。
**核心功能**:
- 集群节点间状态同步
- 分布式数据库存储
- 故障检测和自动恢复
- 公共 IP 地址管理(浮动 IP)
---
## 1. CTDB 核心组件
### 1.1 分布式数据库引擎
**功能**:
- 字储 TDB (Trivial Database) 记录
- 在多个节点间复制数据
- 提供原子性和一致性保证
**TDB 格式**:
```
┌─────────────────────────────────────┐
│ TDB Header │
│ ├── Magic number: 0x1BADFACE │
│ ├── Version: 1 │
│ ├── Hash size: 1024 │
│ └── Record count: N │
├─────────────────────────────────────┤
│ Hash Table │
│ ├── Bucket 0: offset to record │
│ ├── Bucket 1: offset to record │
│ └── ... │
├─────────────────────────────────────┤
│ Free List │
│ ├── Offset to next free block │
│ └── Free block size │
├─────────────────────────────────────┤
│ Records │
│ ├── Key (variable length) │
│ ├── Data (variable length) │
│ └── Hash next pointer │
└─────────────────────────────────────┘
```
---
### 1.2 集群节点管理
**功能**:
- 节点状态监控(UP/DOWN/UNHEALTHY)
- 心跳检测(heartbeat)
- 自动故障转移(failover)
**节点状态定义**:
| 状态 | 说明 | 处理 |
|------|------|------|
| **UP** | 节点正常运行 | 参与集群操作 |
| **DOWN** | 节点离线 | 等待恢复 |
| **UNHEALTHY** | 节点不健康 | 停止服务 |
| **BANNED** | 节点被禁止 | 重新加入需手动操作 |
---
### 1.3 公共 IP 管理
**功能**:
- 动态分配浮动 IP(public IP)
- 节点故障时自动迁移 IP
- 客户端透明重连
**公共 IP 分配逻辑**:
```
┌─────────────────────────────────────┐
│ Public IP Pool │
│ ├── 192.168.1.100 (node 0) │
│ ├── 192.168.1.101 (node 1) │
│ ├── 192.168.1.102 (node 2) │
│ └── ... │
├─────────────────────────────────────┤
│ IP Assignment │
│ ├── Node 0 UP → owns .100 │
│ ├── Node 1 DOWN → .101 moves to N0 │
│ └── Node 2 UP → owns .102 │
└─────────────────────────────────────┘
```
---
### 1.4 事件脚本系统
**功能**:
- 监控脚本(健康检查)
- 启动/停止脚本(服务管理)
- IP 分配脚本(网络配置)
**事件类型**:
| 事件 | 说明 | 脚本 |
|------|------|------|
| `startup` | 节点启动 | 01.startup.sh |
| `shutdown` | 节点停止 | 50.shutdown.sh |
| `takeip` | 分配 IP | 11.takeip.sh |
| `releaseip` | 释放 IP | 10.releaseip.sh |
| `monitor` | 健康检查 | 00.monitor.sh |
---
## 2. CTDB 协议架构
### 2.1 控制协议
**CTDB 控制消息**(基于 TCP):
```
┌─────────────────────────────────────┐
│ CTDB Header │
│ ├── Magic: 0xCtdb │
│ ├── Version: 1 │
│ ├── Command: CTDB_CMD_* │
│ ├── Status: SUCCESS/ERROR │
│ ├── Length: payload size │
├─────────────────────────────────────┤
│ Payload │
│ ├── Command-specific data │
│ └── Optional response data │
└─────────────────────────────────────┘
```
**CTDB_CMD 类型**:
| Command | 说明 | 用途 |
|---------|------|------|
| `CTDB_CMD_CONNECT` | 连接请求 | 初始化连接 |
| `CTDB_CMD_GETDB` | 获取数据库 | 访问 TDB |
| `CTDB_CMD_FETCH` | 读取记录 | 查询数据 |
| `CTDB_CMD_STORE` | 存储记录 | 写入数据 |
| `CTDB_CMD_DELETE` | 删除记录 | 清除数据 |
| `CTDB_CMD_PING` | 心跳检测 | 节点监控 |
| `CTDB_CMD_SETNODEMASK` | 设置节点掩码 | 集群配置 |
---
### 2.2 数据库复制协议
**复制策略**:
- **主从复制**(Master-Slave):一个节点为主,其他为副本
- **多主复制**(Multi-Master):所有节点可写入(需要冲突解决)
**复制流程**:
```
Node A Node B
| |
|── STORE(key, value) ────>| (write request)
| |── Validate
| |── Write to local TDB
| |── Replicate to other nodes
|<── ACK (success) ────────|
| |
|── FETCH(key) ───────────>| (read request)
|<── value ────────────────| (return data)
```
---
### 2.3 故障恢复协议
**故障检测**:
```
┌─────────────────────────────────────┐
│ Health Monitor Loop │
│ ├── Ping all nodes (every 1s) │
│ ├── Check response timeout (5s) │
│ ├── Mark DOWN if timeout │
│ └── Trigger recovery process │
└─────────────────────────────────────┘
```
**恢复流程**:
```
Node A (DOWN) Cluster
| |
|── Mark as DOWN ─────────>| (detected)
| |── Reassign public IPs
| |── Notify other nodes
| |── Update node mask
| |
|── Recovery attempt ─────>| (after 30s)
|── Rejoin cluster ───────>| (if successful)
| |── Restore IPs
```
---
## 3. CTDB 与 SMB 集成
### 3.1 Samba 集成点
**共享数据库**:
| TDB 文件 | 说明 | 集群支持 |
|---------|------|---------|
| `secrets.tdb` | 认证密钥 | ✅ CTDB 复制 |
| `brlock.tdb` | 字节锁 | ✅ CTDB 复制 |
| `locking.tdb` | 文件锁 | ✅ CTDB 复制 |
| `connections.tdb` | 连接状态 | ✅ CTDB 复制 |
| `session_info.tdb` | 会话信息 | ✅ CTDB 复制 |
| `share_info.tdb` | 共享配置 | ✅ CTDB 复制 |
**关键特性**:
- 所有节点访问相同的数据库
- 实时同步锁定状态
- 故障后自动恢复连接
---
### 3.2 实施架构
**高可用性架构**:
```
┌─────────────────────────────────────────────┐
│ Client Layer │
│ ├── SMB clients (Windows/macOS/Linux) │
│ └── Connect via floating IP │
├─────────────────────────────────────────────┤
│ CTDB Cluster (3+ nodes) │
│ ├── Node 0: 192.168.1.100 (SMB + CTDB) │
│ ├── Node 1: 192.168.1.101 (SMB + CTDB) │
│ ├── Node 2: 192.168.1.102 (SMB + CTDB) │
│ └── Public IPs: .100, .101, .102 │
├─────────────────────────────────────────────┤
│ Shared Storage │
│ ├── GlusterFS / Ceph / NFS │
│ └── All nodes access same filesystem │
└─────────────────────────────────────────────┘
```
---
## 4. 实施评估
### 4.1 核心功能需求
| 功能 | 优先级 | 工作量 | 风险 |
|------|--------|--------|------|
| **分布式 TDB** | P0 | 500 行 | 中 ⚠️⚠️⚠️ |
| **节点管理** | P0 | 300 行 | 中 ⚠️⚠️⚠️ |
| **公共 IP 管理** | P1 | 200 行 | 低 ⚠️⚠️ |
| **事件脚本系统** | P1 | 200 行 | 低 ⚠️⚠️ |
| **控制协议** | P0 | 400 行 | 高 ⚠️⚠️⚠️⚠️ |
| **故障恢复** | P0 | 300 行 | 高 ⚠️⚠️⚠️⚠️⚠️ |
| **总计** | | **1900 行** | **高 ⚠️⚠️⚠️⚠️⚠️** |
---
### 4.2 技术挑战
**挑战 1:分布式一致性**
- **问题**:多节点写入冲突
- **解决方案**:
- 使用 Raft/Paxos 算法(需要实现 consensus)
- 或使用主从模式(简化但牺牲可用性)
**挑战 2:故障检测准确性**
- **问题**:网络分区导致误判
- **解决方案**:
- 使用多路径心跳(多个检测点)
- 设置合理的超时阈值(避免误判)
**挑战 3:浮动 IP 管理**
- **问题**:IP 迁移需要内核支持
- **解决方案**:
- 使用 Linux network namespace
- macOS 需要 ifconfig + route 管理
---
### 4.3 Rust 实施建议
**推荐方案**:
1. **Phase 1**:实现基础 TDB 存储引擎(500 行)
2. **Phase 2**:实现节点管理和心跳(300 行)
3. **Phase 3**:实现控制协议(400 行)
4. **Phase 4**:实现公共 IP 管理(200 行)
5. **Phase 5**:实现故障恢复逻辑(300 行)
**总计**:约 1700 行代码(不包括测试)
**预计时间**:约 5-7 天(高复杂度)
---
### 4.4 替代方案
**方案 1:使用现有 Rust crate**
- `raft-rs`:分布式共识算法
- `tikv`:分布式 KV 存储
- 优点:减少实现工作量
- 缺点:需要集成适配
**方案 2:简化 CTDB 实现**
- 仅实现单主模式(避免分布式共识)
- 使用 SQLite 作为共享数据库(简化存储)
- 优点:降低实施风险
- 缺点:牺牲可用性(主节点故障无法写入)
**方案 3:不实施 CTDB**
- 使用外部高可用方案(HAProxy + Keepalived)
- MarkBase SMB 保持单节点部署
- 优点:最小工作量
- 缺点:无法实现真正的集群
---
## 5. 决策建议
### 5.1 推荐策略 ⭐⭐⭐⭐
**短期(P3 不建议实施)**:
- ✅ 保持单节点部署(Phase 1-6 已完成)
- ✅ 使用外部 HAProxy + Keepalived 实现高可用
- ❌ 不实施 CTDB(复杂度高,收益有限)
**长期(企业需求)**:
- ⭐⭐⭐⭐ 实施简化 CTDB(单主模式)
- ⭐⭐⭐ 使用 Raft-rs 实现分布式共识
- ⭐⭐⭐⭐⭐ 完整 CTDB 实现(需要 5-7 天)
---
### 5.2 最终建议
| 场景 | 推荐方案 | 工作量 |
|------|---------|--------|
| **个人/小团队** | 单节点 + 外部 HA | 0 行 |
| **中小企业** | 简化 CTDB(单主) | ~800 行 |
| **大型企业** | 完整 CTDB + Raft | ~2000 行 |
**结论**:Phase 7 (CTDB 集群) 复杂度高(⚠️⚠️⚠️⚠️⚠️),建议根据实际需求决定是否实施。
---
**最后更新**:2026-06-22
+563
View File
@@ -0,0 +1,563 @@
# DedupFs + S3Vfs Combination Design
**Date**: 2026-06-25
**Status**: Design proposal
**Goal**: Distributed deduplication storage via MinIO/S3 backend
---
## Executive Summary
### Current State
**DedupStore**(`dedup.rs`, 224 行):
- 基于**本地文件系统**的 dedup 存储
- SHA-256 块哈希 + 引用计数
- 块存储到本地目录(`store_path/.dedup/`)
**问题**:
- ❌ 无法跨节点共享 dedup 块
- ❌ 无分布式容错能力
- ❌ 单节点存储限制
### Proposed Solution
**DedupS3Store**:
- 块存储到 **MinIO/S3** 对象(跨节点共享)
- 引用计数存储到 S3 object metadata
- Manifest 存储到 S3 对象(JSON 格式)
**优势**:
- ✅ 跨节点 dedup 共享(MinIO 分布式)
- ✅ 自动容错(MinIO erasure coding)
- ✅ 无单节点限制(MinIO 可扩展)
- ✅ 与现有 S3Vfs 集成(无需新 HTTP API)
---
## Architecture
```
┌─────────────────────────────────────────────────────────────────────────┐
│ MarkBase Node A │
│ ├── DedupS3Store │
│ │ ├── store_block() → S3 PUT <hash> │
│ │ ├── get_block() → S3 GET <hash> │
│ │ └── dedup_file() → 分块 + S3 PUT + manifest │
│ └───────────────────────────────────────────────────────────────────────┘
│ ↓ │
┌─────────────────────────────────────────────────────────────────────────┐
│ MinIO Cluster (S3-compatible) │
│ ├── Bucket: markbase-dedup │
│ │ ├── Objects: <sha256-hash> (dedup 块) │
│ │ ├── Metadata: x-amz-meta-ref-count (引用计数) │
│ │ └── Manifests: manifests/<file-id>.json │
│ │ │
│ ├── Erasure Coding: EC:2 (自动容错) │
│ ├── Replication: Node A → Node B (DR) │
│ └─────────────────────────────────────────────────────────────────────┘
│ ↓ │
┌─────────────────────────────────────────────────────────────────────────┐
│ MarkBase Node B │
│ ├── DedupS3Store │
│ │ ├── get_block() → S3 GET <hash> (共享 Node A 的块) │
│ │ └── restore_file() → S3 GET manifest + S3 GET blocks │
│ └─────────────────────────────────────────────────────────────────────┘
```
---
## Implementation Design
### DedupS3Store Struct
```rust
pub struct DedupS3Store {
s3vfs: S3Vfs, // S3 backend
bucket: String, // Bucket name (markbase-dedup)
block_prefix: String, // Object key prefix (blocks/)
manifest_prefix: String, // Manifest prefix (manifests/)
config: VfsDedupConfig, // block_size, min_file_size
}
pub struct DedupManifest {
original_size: usize,
block_hashes: Vec<String>,
dedup_ratio: f64,
file_id: String, // UUID for manifest storage
}
```
### Core Methods
| Method | Current (LocalFs) | Proposed (S3Vfs) |
|--------|------------------|------------------|
| `store_block(data)` | `std::fs::write(store_path/hash, data)` | `S3Vfs.put_object(blocks/hash, data)` |
| `get_block(hash)` | `std::fs::read(store_path/hash)` | `S3Vfs.get_object(blocks/hash)` |
| `increment_ref(hash)` | `std::fs::write(hash.ref, count)` | `S3Vfs.put_object(blocks/hash, data) + metadata update` |
| `decrement_ref(hash)` | `std::fs::write/remove` | `S3Vfs.delete_object + metadata check` |
| `dedup_file(source)` | Local file read + block store | Local file read + S3 PUT blocks |
| `restore_file(manifest)` | Local file write + block read | Local file write + S3 GET blocks |
| `get_ref_count(hash)` | `std::fs::read(hash.ref)` | `S3Vfs.head_object(blocks/hash) → metadata` |
---
## S3 Object Layout
```
Bucket: markbase-dedup
├── blocks/
│ ├── <sha256-hash-1> # Dedup 块(4KB)
│ │ └── Metadata: x-amz-meta-ref-count: 5
│ ├── <sha256-hash-2>
│ │ └── Metadata: x-amz-meta-ref-count: 2
│ └── ...
│
├── manifests/
│ ├── <file-id-1>.json # Manifest JSON
│ │ └── Content: {"original_size": 1024, "block_hashes": [...], ...}
│ ├── <file-id-2>.json
│ └── ...
│
└── stats.json # DedupStats(可选)
```
---
## Reference Count Management
### Challenge
S3 对象不支持 atomic increment/decrement 操作。
### Solution 1: Metadata Update (推荐 ⭐⭐⭐⭐⭐)
**流程**:
```rust
fn increment_ref(&self, hash: &str) -> Result<(), VfsError> {
// 1. GET current metadata
let head = self.s3vfs.head_object(&format!("blocks/{}", hash))?;
let current_ref = head.metadata.get("x-amz-meta-ref-count")
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(0);
// 2. PUT with updated metadata
let block_data = self.s3vfs.get_object(&format!("blocks/{}", hash))?;
self.s3vfs.put_object_with_metadata(
&format!("blocks/{}", hash),
&block_data,
[("x-amz-meta-ref-count", (current_ref + 1).to_string())]
)?;
Ok(())
}
```
**优势**:
- ✅ 简单实现
- ✅ 与 S3 标准兼容
- ⚠️ 需要两次请求(GET + PUT)
**劣势**:
- ⚠️ 非原子操作(并发问题)
- ⚠️ 需要读取块数据(PUT 需要 body)
---
### Solution 2: Separate Ref Count Object
**流程**:
```rust
fn increment_ref(&self, hash: &str) -> Result<(), VfsError> {
// 1. GET ref count object
let ref_key = format!("refs/{}/count", hash);
let current = self.s3vfs.get_object(&ref_key)
.and_then(|data| data.parse::<u64>())
.unwrap_or(0);
// 2. PUT updated ref count
self.s3vfs.put_object(&ref_key, (current + 1).to_string())?;
Ok(())
}
```
**优势**:
- ✅ 无需读取块数据
- ✅ 更小的对象(仅数字)
**劣势**:
- ⚠️ 需要额外对象存储
- ⚠️ 非原子操作(并发问题)
---
### Solution 3: MinIO Extended API (企业版)
MinIO 企业版提供 `mc admin bucket policy` 和 object locking API。
**优势**:
- ✅ 可能提供 atomic operation
**劣势**:
- ⚠️ 仅 MinIO 企业版
- ⚠️ 需要研究具体 API
---
## Concurrency Problem
### Scenario
Node A 和 Node B 同时 dedup 相同文件:
1. Node A: `increment_ref(hash-abc)` → GET count=2 → PUT count=3
2. Node B: `increment_ref(hash-abc)` → GET count=2 → PUT count=3
3. 结果:count=3(错误,应为 count=4)
### Solution 1: Optimistic Locking
使用 S3 versioning 检测冲突:
```rust
fn increment_ref(&self, hash: &str) -> Result<(), VfsError> {
loop {
// 1. GET current version + metadata
let (version_id, current_ref) = self.get_ref_with_version(hash)?;
// 2. PUT with version check
let result = self.s3vfs.put_object_if_version(
&format!("blocks/{}", hash),
block_data,
(current_ref + 1),
version_id // Only succeed if version unchanged
);
if result.is_ok() {
break;
}
// Retry if version mismatch
}
Ok(())
}
```
**要求**:MinIO versioning enabled。
---
### Solution 2: Distributed Lock Service
使用外部分布式锁(如 Redis/Zookeeper):
```rust
fn increment_ref(&self, hash: &str) -> Result<(), VfsError> {
// 1. Acquire distributed lock
let lock = self.lock_service.acquire(&format!("lock:{}", hash))?;
// 2. Increment ref count
self.update_ref_count(hash)?;
// 3. Release lock
lock.release();
Ok(())
}
```
**劣势**:需要额外服务(Redis)。
---
### Solution 3: Accept Non-Atomic (简化方案)
对于 MarkBase Lightweight 定位:
- ⚠️ 接受非原子操作风险
- ⚠️ 偶尔 ref count 不准确(不影响数据完整性)
- ⚠️ 定期修复(scrub job)
**推荐**:Phase 1 使用 Solution 1(Metadata Update),Phase 2 研究 MinIO versioning。
---
## Implementation Phases
| Phase | Task | Code Lines | Priority | Risk |
|-------|------|------------|----------|------|
| **Phase 1** | DedupS3Store struct + basic I/O | ~300 | P0 | Medium |
| **Phase 2** | Reference count metadata | ~100 | P0 | Medium |
| **Phase 3** | Manifest storage to S3 | ~50 | P1 | Low |
| **Phase 4** | CLI integration | ~100 | P1 | Low |
| **Phase 5** | Async version (DedupAsyncS3Store) | ~200 | P2 | High |
| **Phase 6** | Concurrency fix (versioning) | ~150 | P2 | High |
| **Phase 7** | Performance benchmark | ~100 | P2 | Low |
| **Total** | | **~1000** | | |
---
## DedupS3Store Implementation (Phase 1 Draft)
```rust
use super::s3_fs::S3Vfs;
use super::{VfsDedupConfig, VfsError};
use sha2::{Sha256, Digest};
use std::path::Path;
pub struct DedupS3Store {
s3vfs: S3Vfs,
bucket: String,
block_prefix: String,
manifest_prefix: String,
config: VfsDedupConfig,
}
impl DedupS3Store {
pub fn new(
endpoint: &str,
region: &str,
bucket: &str,
access_key: &str,
secret_key: &str,
config: VfsDedupConfig,
) -> Result<Self, VfsError> {
let s3vfs = S3Vfs::new(endpoint, region, bucket, access_key, secret_key)?;
Ok(Self {
s3vfs,
bucket: bucket.to_string(),
block_prefix: "blocks/".to_string(),
manifest_prefix: "manifests/".to_string(),
config,
})
}
pub fn store_block(&self, data: &[u8]) -> Result<String, VfsError> {
if data.len() > self.config.block_size {
return Err(VfsError::Io(format!("Block size exceeds limit")));
}
let hash = Self::hash_block(data);
let key = format!("{}{}", self.block_prefix, hash);
// Check if block exists
if !self.s3vfs.object_exists(&key)? {
// PUT with initial ref count = 1
self.s3vfs.put_object_with_metadata(
&key,
data,
[("x-amz-meta-ref-count", "1")]
)?;
} else {
// Increment ref count
self.increment_ref(&hash)?;
}
Ok(hash)
}
pub fn get_block(&self, hash: &str) -> Result<Vec<u8>, VfsError> {
let key = format!("{}{}", self.block_prefix, hash);
self.s3vfs.get_object(&key)
}
pub fn increment_ref(&self, hash: &str) -> Result<(), VfsError> {
let key = format!("{}{}", self.block_prefix, hash);
let head = self.s3vfs.head_object(&key)?;
let current_ref = head.metadata
.get("x-amz-meta-ref-count")
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(1);
// Need to GET block data + PUT with new metadata
let block_data = self.get_block(hash)?;
self.s3vfs.put_object_with_metadata(
&key,
&block_data,
[("x-amz-meta-ref-count", (current_ref + 1).to_string())]
)?;
Ok(())
}
pub fn dedup_file(&self, source: &Path) -> Result<DedupManifest, VfsError> {
let mut file = std::fs::File::open(source)?;
let mut manifest = DedupManifest::new();
let mut buffer = vec![0u8; self.config.block_size];
loop {
let n = file.read(&mut buffer)?;
if n == 0 { break; }
manifest.original_size += n;
let hash = self.store_block(&buffer[..n])?;
manifest.block_hashes.push(hash);
}
// Store manifest to S3
let file_id = uuid::Uuid::new_v4().to_string();
manifest.file_id = file_id;
let manifest_key = format!("{}{}.json", self.manifest_prefix, file_id);
let manifest_json = serde_json::to_string(&manifest)?;
self.s3vfs.put_object(&manifest_key, manifest_json.as_bytes())?;
Ok(manifest)
}
pub fn restore_file(&self, manifest_id: &str, target: &Path) -> Result<(), VfsError> {
let manifest_key = format!("{}{}.json", self.manifest_prefix, manifest_id);
let manifest_json = self.s3vfs.get_object(&manifest_key)?;
let manifest: DedupManifest = serde_json::from_slice(&manifest_json)?;
let mut file = std::fs::File::create(target)?;
for hash in &manifest.block_hashes {
let block = self.get_block(hash)?;
file.write_all(&block)?;
}
Ok(())
}
fn hash_block(data: &[u8]) -> String {
let mut hasher = Sha256::new();
hasher.update(data);
hex::encode(hasher.finalize())
}
}
```
---
## Integration with MarkBase VFS
### Option 1: Standalone DedupS3Store
用户手动创建 DedupS3Store:
```bash
# CLI tool
markbase dedup-upload --s3 --s3-endpoint http://localhost:9000 --file /data/large.iso
markbase dedup-download --s3 --manifest-id <uuid> --output /data/restored.iso
```
---
### Option 2: DedupVfsBackend (VfsBackend trait)
创建 VfsBackend wrapper,自动 dedup:
```rust
pub struct DedupS3Backend {
dedup_store: DedupS3Store,
manifest_dir: PathBuf, // Local cache for manifests
}
impl VfsBackend for DedupS3Backend {
fn open_file(&self, path: &Path, flags: &OpenFlags) -> Result<Box<dyn VfsFile>, VfsError> {
// 1. Read manifest from S3
let manifest = self.load_manifest(path)?;
// 2. DedupS3File (read blocks from S3)
Ok(Box::new(DedupS3File::new(self.dedup_store.clone(), manifest)))
}
fn stat(&self, path: &Path) -> Result<VfsStat, VfsError> {
// Read from manifest metadata
let manifest = self.load_manifest(path)?;
Ok(VfsStat {
size: manifest.original_size,
mtime: manifest.mtime,
...
})
}
fn read_dir(&self, path: &Path) -> Result<Vec<VfsDirEntry>, VfsError> {
// List manifests from S3
self.dedup_store.s3vfs.list_objects(&self.manifest_prefix)
}
}
```
**优势**:
- ✅ 透明 dedup(用户无需关心)
- ✅ 与 SMB/WebDAV/SFTP 无缝集成
---
### Option 3: Hybrid (LocalFs + DedupS3Store)
```rust
pub struct HybridDedupBackend {
local: LocalFs, // Small files (<1MB) 存本地
dedup_s3: DedupS3Store, // Large files (>1MB) dedup to S3
}
impl VfsBackend for HybridDedupBackend {
fn open_file(&self, path: &Path, flags: &OpenFlags) -> Result<Box<dyn VfsFile>, VfsError> {
// Check file size
let stat = self.local.stat(path)?;
if stat.size < self.dedup_s3.config.min_file_size {
// Small file: direct LocalFs
self.local.open_file(path, flags)
} else {
// Large file: dedup to S3
self.dedup_s3.dedup_file(path)?;
self.dedup_s3.open_file_from_manifest(path)
}
}
}
```
**推荐**:Option 1(Phase 1),Option 3(Phase 2)。
---
## Performance Considerations
### Network Latency
| Operation | LocalFs | S3Vfs | Overhead |
|-----------|---------|-------|----------|
| store_block (4KB) | ~0.1ms | ~5-10ms (HTTP) | ~50-100x |
| get_block (4KB) | ~0.1ms | ~5-10ms (HTTP) | ~50-100x |
| dedup_file (100MB) | ~2s (25MB/s) | ~10s (10MB/s) | ~5x |
**缓解方案**:
- ✅ Async concurrent upload(4-8 并发)
- ✅ ReadCache(64MB cache)
- ✅ Local cache for hot blocks
---
### Dedup Ratio Impact
| File Type | Dedup Ratio | Network Traffic Saved |
|-----------|-------------|----------------------|
| VM images (similar OS) | ~80% | -80% upload bandwidth |
| Log files (daily) | ~60% | -60% upload bandwidth |
| Unique files (photos) | ~5% | -5% upload bandwidth |
---
## Next Steps
1. **Phase 1 Implementation** (~300 lines)
- `DedupS3Store` struct
- `store_block()` / `get_block()` via S3Vfs
- `increment_ref()` with metadata update
2. **Phase 2 CLI Integration** (~100 lines)
- `markbase dedup-upload --s3`
- `markbase dedup-download --manifest-id`
3. **Phase 3 Performance Test**
- Benchmark dedup_file (100MB)
- Compare LocalFs vs S3Vfs
---
## Open Questions
1. **Concurrency**: Accept non-atomic ref count vs implement versioning?
2. **Backend choice**: Standalone CLI vs VfsBackend integration?
3. **Min versioning**: Should we require MinIO versioning enabled?
4. **Ref count object**: Metadata vs separate object?
5. **Block cache**: Should we cache blocks locally?
---
**文档创建**: 2026-06-25
**最后更新**: 2026-06-25
+404
View File
@@ -0,0 +1,404 @@
# MarkBase GUI 管理介面检讨报告
**版本**: 1.0
**日期**: 2026-06-25
**作者**: AI Assistant
---
## 一、GUI 架构概览
### 1.1 技术栈
| 组件 | 技术 | 版本 |
|------|------|------|
| **前端框架** | Vue.js 3 | Composition API |
| **UI 库** | Element Plus | Latest |
| **桌面框架** | Tauri | v2 |
| **后端语言** | Rust | 1.92+ |
| **数据存储** | SQLite | auth.sqlite |
### 1.2 代码统计
| 类型 | 数量 | 行数 |
|------|------|------|
| **Vue Components** | 11 个 | 4860 行 |
| **Tauri Commands** | 12 个 | ~1500 行 |
| **Router Routes** | 11 个 | 77 行 |
| **总计** | | ~6437 行 |
---
## 二、已实现功能
### 2.1 User Management (Users.vue)
**功能完整度**: ⭐⭐⭐⭐⭐ (5/5)
| 功能 | 状态 | 说明 |
|------|------|------|
| 用户列表 | ✅ 完成 | 显示 username, home_dir, status |
| 创建用户 | ✅ 完成 | bcrypt 密码加密 + SqliteProvider |
| 编辑用户 | ✅ 完成 | home_dir/status 更新 + 密码可选 |
| 删除用户 | ✅ 完成 | 确认对话框 + SqliteProvider |
| 重置密码 | ✅ 完成 | 弹窗输入新密码 |
**代码量**: 264 行 Vue + 100 行 Rust
---
### 2.2 Share Management (Shares.vue)
**功能完整度**: ⭐⭐⭐ (3/5)
| 功能 | 状态 | 说明 |
|------|------|------|
| 共享列表 | ✅ 完成 | name, path, protocol, users, permissions |
| 创建共享 | ⚠️ 内存存储 | 重启丢失(需持久化) |
| 编辑共享 | ⚠️ 内存存储 | 重启丢失(需持久化) |
| 删除共享 | ⚠️ 内存存储 | 重启丢失(需持久化) |
| 连接测试 | ✅ 完成 | path 存在验证 |
| 协议支持 | ✅ 完成 | SMB/SFTP/WebDAV/S3 |
**代码量**: 295 行 Vue + 152 行 Rust
---
### 2.3 Dashboard (Dashboard.vue)
**功能完整度**: ⭐⭐⭐ (3/5)
| 功能 | 状态 | 说明 |
|------|------|------|
| CPU 监控 | ✅ 完成 | macOS/Linux 双平台 |
| Memory 监控 | ✅ 完成 | macOS/Linux 双平台 |
| Disk 监控 | ✅ 完成 | macOS/Linux 双平台 |
| Service Status | ❌ 硬编码 | 返回固定 4 个服务(未检查实际进程) |
| Recent Activity | ❌ 硬编码 | 返回固定 4 条记录(未连接日志系统) |
| Quick Actions | ❌ 未实现 | 只有 UI,无实际功能 |
| 实时刷新 | ✅ 完成 | 5 秒定时刷新 |
**代码量**: 302 行 Vue + 290 行 Rust
---
### 2.4 Backup Management (Backup.vue)
**功能完整度**: ⭐⭐⭐⭐ (4/5)
**代码量**: 497 行 Vue + 3732 行 Rust
---
## 三、存在的问题
### 3.1 关键问题
| 问题 | 严重程度 | 影响 |
|------|----------|------|
| **Share Management 内存存储** | ⚠️⚠️⚠️⚠️⚠️ 极高 | 重启丢失所有共享配置 |
| **Service Status 硬编码** | ⚠️⚠️⚠️⚠️ 高 | 无法反映真实服务状态 |
| **Recent Activity 硬编码** | ⚠️⚠️⚠️⚠️ 高 | 无法查看真实操作记录 |
| **Quick Actions 未实现** | ⚠️⚠️⚠️ 中 | 用户体验不完整 |
| **无权限管理** | ⚠️⚠️⚠️⚠️ 高 | 无法控制用户访问权限 |
---
### 3.2 详细分析
#### 问题 #1: Share Management 内存存储
**当前实现**:
```rust
lazy_static::lazy_static! {
static ref SHARES: Arc<Mutex<Vec<ShareInfo>>> =
Arc::new(Mutex::new(Vec::new()));
}
```
**问题影响**:
- 服务器重启后,所有共享配置丢失
- 无法持久化到数据库或配置文件
- 不符合生产环境要求
**推荐方案**:
- 使用 SQLite 存储(`data/shares.sqlite`)
- 或 TOML 配置文件(`config/shares.toml`)
---
#### 问题 #2: Service Status 硬编码
**当前实现**:
```rust
pub async fn get_all_services_status() -> Result<Vec<ServiceStatus>, String> {
Ok(vec![
ServiceStatus {
name: "SMB Server".to_string(),
status: "running".to_string(),
port: 4445,
uptime: "2h 30m".to_string(),
},
// ... 固定返回 4 个服务
])
}
```
**问题影响**:
- 无法检测服务真实状态(running/stopped)
- 无法获取真实 uptime
- 无法监控服务异常
**推荐方案**:
- 使用 `ps aux | grep` 检查进程状态
- 或使用 PID 文件追踪服务
- 或使用 systemd/launchd 服务管理
---
#### 问题 #3: Recent Activity 硬编码
**当前实现**:
```rust
pub async fn get_recent_activity() -> Result<Vec<ActivityLog>, String> {
Ok(vec![
ActivityLog {
timestamp: "2026-06-23 14:30:00".to_string(),
activity_type: "Upload".to_string(),
description: "Uploaded document.pdf to /data/files".to_string(),
user: "alice".to_string(),
},
// ... 固定返回 4 条记录
])
}
```
**问题影响**:
- 无法查看真实用户操作
- 无法审计系统行为
- 无法追踪异常事件
**推荐方案**:
- 使用日志文件(`data/activity.log`)
- 或 SQLite 存储(`data/activity.sqlite`)
- 集成现有 SSH/SMB/WebDAV 日志
---
#### 问题 #4: Quick Actions 未实现
**当前实现**:
```vue
<el-button type="primary" :icon="Upload" class="action-btn">
Upload File
</el-button>
<el-button type="success" :icon="Document" class="action-btn">
Create Backup
</el-button>
// ... 只有按钮,无 @click handler
```
**问题影响**:
- 用户点击按钮无响应
- Dashboard 功能不完整
**推荐方案**:
- Upload File: 调用 Tauri dialog + file_ops.rs
- Create Backup: 调用 backup.rs snapshot 功能
- View Backups: 跳转到 Backup.vue
- Download File: 调用 Tauri dialog + file_ops.rs
---
#### 问题 #5: 无权限管理
**当前实现**:
- User Management 只有 CRUD 用户
- Share Management 只有 CRUD 共享
- **缺失**:用户-共享权限关联
**问题影响**:
- 无法控制用户访问哪些共享
- 无法设置读/写权限
- 无法实现多租户隔离
**推荐方案**:
- 添加 Permission Management 页面
- 用户-共享关联表(user_id, share_id, permission)
- 权限类型:read/write/admin
---
## 四、竞争对手对比
### 4.1 功能对比表
| 功能 | Proxmox VE | Unraid | OpenNAS | MarkBase | 覆盖率 |
|------|-----------|--------|---------|----------|--------|
| **Dashboard** | ✅ 完整 | ✅ 完整 | ✅ 完整 | ⚠️ 部分 | 60% |
| **User Management** | ✅ 完整 | ✅ 完整 | ✅ 完整 | ✅ 完整 | 100% |
| **Share Management** | ✅ 完整 | ✅ 完整 | ✅ 完整 | ⚠️ 内存 | 50% |
| **Backup Management** | ✅ 完整 | ✅ 完整 | ✅ 完整 | ✅ 完整 | 100% |
| **Permission Management** | ✅ 完整 | ✅ 完整 | ✅ 完整 | ❌ 缺失 | 0% |
| **Service Monitoring** | ✅ 完整 | ✅ 完整 | ✅ 完整 | ❌ 硬编码 | 30% |
| **Activity Log** | ✅ 完整 | ✅ 完整 | ✅ 完整 | ❌ 硕编码 | 30% |
| **VM Management** | ✅ 完整 | ❌ 无 | ❌ 无 | ❌ 无 | N/A |
| **Container Management** | ✅ 完整 | ✅ 完整 | ❌ 无 | ❌ 无 | N/A |
| **HA Cluster** | ✅ 完整 | ❌ 无 | ❌ 无 | ❌ 无 | N/A |
**总体覆盖率**: **47%** (存储 + 备份)
---
### 4.2 竞争对手优势
**Proxmox VE**:
- ✅ 完整的 VM/Container 管理
- ✅ HA Cluster 支持
- ✅ 企业级权限管理
- ✅ 完整的监控告警系统
- ✅ API + CLI + Web UI 三位一体
**Unraid**:
- ✅ Docker Container 管理
- ✅ 简单易用的 Web UI
- ✅ Community Apps 生态
- ✅ Flash drive 启动(无需安装)
- ✅ Parity 保护(类似 RAID)
**OpenNAS**:
- ✅ 专注 NAS 功能
- ✅ ZFS 集成
- ✅ 简单部署
- ✅ 开源免费
---
## 五、改进建议
### 5.1 短期改进(本周)
| 优先级 | 任务 | 工作量 | 收益 |
|--------|------|--------|------|
| **P0 #1** | Share Management 持久化 | 200 行 | ⭐⭐⭐⭐⭐ 极高 |
| **P0 #2** | Service Status 真实检测 | 150 行 | ⭐⭐⭐⭐ 高 |
| **P0 #3** | Quick Actions 实现 | 100 行 | ⭐⭐⭐ 中 |
| **P1 #4** | Permission Management | 300 行 | ⭐⭐⭐⭐⭐ 极高 |
---
### 5.2 中期改进(本月)
| 优先级 | 任务 | 工作量 | 收益 |
|--------|------|--------|------|
| **P1 #5** | Activity Log 系统集成 | 400 行 | ⭐⭐⭐⭐ 高 |
| **P1 #6** | Dashboard 增强图表 | 200 行 | ⭐⭐⭐ 中 |
| **P2 #7** | File Browser UI | 500 行 | ⭐⭐⭐⭐⭐ 极高 |
| **P2 #8** | Snapshot Management UI | 300 行 | ⭐⭐⭐⭐ 高 |
---
### 5.3 长期改进(下季度)
| 优先级 | 任务 | 工作量 | 收益 |
|--------|------|--------|------|
| **P2 #9** | Docker Container UI | 800 行 | ⭐⭐⭐⭐ 高 |
| **P3 #10** | API + CLI + Web UI 统一 | 1000 行 | ⭐⭐⭐⭐⭐ 极高 |
| **P3 #11** | 国际化支持 | 200 行 | ⭐⭐⭐ 中 |
| **P3 #12** | Mobile App | 2000 行 | ⭐⭐⭐⭐ 高 |
---
## 六、实施优先级
### 6.1 立即实施(本周)
**Phase 1**: Share Management 持久化
- 创建 `data/shares.sqlite` 数据库
- 实现 ShareProvider trait
- 集成到 share_management.rs
**Phase 2**: Service Status 真实检测
- 使用 `ps aux` 检查进程状态
- 解析 PID 文件(`/tmp/markbase_*.pid`)
- 计算 uptime(进程启动时间)
**Phase 3**: Quick Actions 实现
- Upload File: Tauri dialog + file_ops.rs
- Create Backup: 跳转到 Backup.vue
- View Backups: 跳转到 Backup.vue
- Download File: Tauri dialog + file_ops.rs
---
### 6.2 下周实施
**Phase 4**: Permission Management
- 创建 Permission.vue 页面
- 实现 permission_management.rs
- 用户-共享关联表
**Phase 5**: Activity Log 系统
- 创建 `data/activity.sqlite`
- 集成 SSH/SMB/WebDAV 日志
- 实现 activity.rs Tauri command
---
## 七、目标定位
### 7.1 当前定位
**MarkBase = Lightweight Enterprise File Server + Backup Server**
| 目标用户 | 规模 | 使用场景 |
|---------|------|---------|
| **个人** | 1-5 用户 | Home NAS + backup |
| **小团队** | 5-20 用户 | SMB/SFTP + WebDAV |
| **中小企业** | 20-100 用户 | 多协议 + snapshots |
| **大型企业** | 100+ 用户 | NFS + LDAP + HA(Phase 12) |
---
### 7.2 GUI 目标覆盖率
| 目标 | 当前覆盖率 | Phase 1-5 后 | Phase 6-12 后 |
|------|-----------|-------------|--------------|
| **vs Proxmox VE** | 47% | 65% | 75% |
| **vs Unraid** | 58% | 75% | 85% |
| **vs OpenNAS** | 62% | 80% | 90% |
---
## 八、总结
### 8.1 已完成
- ✅ User Management 完整实现(5/5)
- ✅ Backup Management 基本实现(4/5)
- ✅ Dashboard 系统监控(3/5)
- ⚠️ Share Management 内存存储(3/5)
### 8.2 待完成
- ❌ Share Management 持久化
- ❌ Service Status 真实检测
- ❌ Activity Log 系统集成
- ❌ Permission Management
- ❌ Quick Actions 实现
### 8.3 建议
**立即开始** Phase 1-3(本周):
- Share Management 持久化(P0)
- Service Status 真实检测(P0)
- Quick Actions 实现(P0)
**下周开始** Phase 4-5:
- Permission Management(P1)
- Activity Log 系统集成(P1)
---
**最后更新**: 2026-06-25
**版本**: 1.0(GUI 管理介面检讨报告)
+75
View File
@@ -0,0 +1,75 @@
# macOS SMB Compatibility Design
## Overview
Enable seamless macOS SMB client connectivity through five phases of
implementation inspired by Samba's `vfs_fruit` and `vfs_catia` modules.
## Gap Analysis Summary
| Feature | Samba vfs_fruit | MarkBase SMB | Status |
|---------|----------------|--------------|--------|
| AFP_AfpInfo (60-byte) | Native xattr | **Truncated to 32 bytes** | ⚠️ P0 bug |
| Catia char mapping | vfs_catia | Functions exist, **not integrated** | ❌ P1 |
| AAPL RESOLVE_ID | AAPL context | **Advertised, not implemented** | ❌ P1 |
| AAPL QUERY_DIR | READ_DIR_ATTR | **Advertised, not implemented** | ❌ P2 |
| Time Machine xattr | vfs_fruit | Set on TreeConnect, **not persisted** | ❌ P2 |
| Finder tags | _kMDItemUserTags | Not implemented | ❌ Future |
| OSX copyfile offload | FSCTL_SRV_COPYCHUNK | Not implemented | ❌ Future |
## Phase 1 — AFP_AfpInfo 60-Byte Fix (P0)
**Problem**: `backend.rs` defines `AFP_INFO_SIZE = 32`, truncating the 60-byte
`AfpInfo` structure to only the `FinderInfo` portion. Backup time, ProDos info,
and reserved fields are silently discarded.
**Fix**: Change the constant to 60 to match `afp_info.rs`.
**Files**: `vendor/smb-server/src/backend.rs`
## Phase 2 — Catia Character Conversion (P1)
**Problem**: macOS clients send NTFS-illegal characters (`:*?"<>|`) encoded as
Unicode private-range code points (`U+F001`–`U+F070`). These are rejected by
`SmbPath::from_utf16()` which validates against NTFS-illegal characters.
The conversion functions already exist in `unicode_mapping.rs` but are never
called before path validation.
**Fix**: Convert private-range chars to ASCII equivalents **before** calling
`SmbPath::from_utf16()` in `create.rs` and `query_directory.rs`.
**Files**:
- `vendor/smb-server/src/handlers/create.rs`
- `vendor/smb-server/src/path.rs` (add public conversion helper)
## Phase 3 — AAPL RESOLVE_ID (P1)
**Problem**: macOS clients send AAPL create context with command = RESOLVE_ID
to map a FileId back to a path. The server advertises `SUPPORT_RESOLVE_ID` but
does not handle the command — it silently returns `None`.
**Fix**: Handle `SMB2_CRTCTX_AAPL_RESOLVE_ID` in the AAPL context processing.
Return the path associated with the requested FileId.
**Files**: `vendor/smb-server/src/handlers/create.rs`
## Phase 4 — AAPL QUERY_DIR (P2)
**Problem**: macOS uses AAPL SERVER_QUERY to request directory attributes in
the CREATE response. The server handles SERVER_QUERY but does not provide
`READ_DIR_ATTR` enhancements.
**Fix**: When AAPL SERVER_QUERY includes `READ_DIR_ATTR`, return directory
metadata (file count, free space) in the response.
**Files**: `vendor/smb-server/src/handlers/create.rs`
## Phase 5 — Time Machine Persistence (P2)
**Problem**: `com.apple.TimeMachine.*` xattrs are set on every TreeConnect
with a new random UUID. The UUID changes on reconnect, confusing macOS.
**Fix**: Check for existing xattrs before setting new ones. Persist the UUID.
**Files**: `vendor/smb-server/src/handlers/tree_connect.rs`
+382
View File
@@ -0,0 +1,382 @@
# MinIO Integration Guide for MarkBase
**Date**: 2026-06-25
**Status**: Ready for deployment
**Backend**: S3Vfs (已有实现,无需修改代码)
---
## Executive Summary
MinIO 是高性能、S3-compatible 的对象存储服务,完美契合 MarkBase 的定位:
- ✅ 跨平台支持(macOS/Linux/Windows)
- ✅ 轻量级部署(单节点即可)
- ✅ 已有 S3Vfs 支持(无需修改代码)
- ✅ 高性能(纠删码 + 分布式扩展)
---
## MinIO vs Ceph RADOS Comparison
| Aspect | MinIO | Ceph RADOS |
|--------|-------|------------|
| **Platform** | ✅ 全平台 | ❌ Linux-only |
| **Deployment** | ⚠️⚠️ 单节点即可 | ⚠️⚠️⚠️⚠️⚠️ 需完整集群 |
| **API** | ✅ S3-compatible HTTP | ❌ librados FFI |
| **Code change** | ✅ 0 行(已有 S3Vfs) | ❌ ~1350 行 |
| **Positioning** | ⭐⭐⭐⭐⭐ 完全匹配 | ❌ 不符合 Lightweight 定位 |
---
## MinIO Deployment
### macOS 单节点部署
```bash
# 安装 MinIO
brew install minio/stable/minio
# 启动 MinIO server
minio server /path/to/data --console-address ":9001"
# 输出:
# Endpoint: http://192.168.1.100:9000 http://127.0.0.1:9000
# Console: http://192.168.1.100:9001 http://127.0.0.1:9001
# AccessKey: minioadmin
# SecretKey: minioadmin
```
### Linux 生产部署
```bash
# Docker 单节点
docker run -d \
--name minio \
-p 9000:9000 \
-p 9001:9001 \
-v /data/minio:/data \
minio/minio server /data --console-address ":9001"
# 分布式集群(4节点)
docker run -d \
--name minio \
-p 9000:9000 \
-p 9001:9001 \
-v /data1:/data1 \
-v /data2:/data2 \
minio/minio server http://node1/data1 http://node2/data2 http://node3/data1 http://node4/data2 --console-address ":9001"
```
### Kubernetes 部署(推荐生产)
```yaml
# minio-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: minio
spec:
replicas: 4
selector:
matchLabels:
app: minio
template:
metadata:
labels:
app: minio
spec:
containers:
- name: minio
image: minio/minio:latest
args:
- server
- http://minio-0/data http://minio-1/data http://minio-2/data http://minio-3/data
- --console-address
- ":9001"
ports:
- containerPort: 9000
- containerPort: 9001
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
emptyDir: {}
```
---
## MarkBase S3Vfs Integration
### 配置方式
**环境变量**:
```bash
export MB_S3_ENDPOINT=http://localhost:9000
export MB_S3_REGION=us-east-1
export MB_S3_BUCKET=markbase
export MB_S3_ACCESS_KEY=minioadmin
export MB_S3_SECRET_KEY=minioadmin
```
**配置文件**(`config/s3.toml`):
```toml
[s3]
enabled = true
endpoint = "http://localhost:9000"
region = "us-east-1"
bucket = "markbase"
access_key = "minioadmin"
secret_key = "minioadmin"
[s3.webdav]
# WebDAV 使用 S3 后端
enabled = true
user = "demo"
root_prefix = "webdav/"
```
### S3Vfs 使用示例
**WebDAV + MinIO**:
```bash
# 启动 WebDAV server(使用 MinIO 后端)
cargo run -- webdav-start \
--user demo \
--port 8002 \
--s3 \
--s3-endpoint http://localhost:9000 \
--s3-bucket markbase \
--s3-access-key minioadmin \
--s3-secret-key minioadmin \
--s3-region us-east-1 \
--root webdav/
```
**SMB + MinIO**(通过 VFS backend):
```bash
# 启动 SMB server(使用 MinIO 后端)
cargo run --features smb-server -- smb-start \
--port 4445 \
--share-name files \
--s3 \
--s3-endpoint http://localhost:9000 \
--s3-bucket markbase \
--s3-access-key minioadmin \
--s3-secret-key minioadmin \
--s3-region us-east-1 \
--root smb/
```
---
## MinIO Bucket Management
### 创建 Bucket
```bash
# 使用 MinIO client (mc)
mc alias set myminio http://localhost:9000 minioadmin minioadmin
mc mb myminio/markbase
# 使用 AWS CLI
aws --endpoint-url http://localhost:9000 s3 mb s3://markbase
```
### 设置 Bucket Policy
```bash
# 公开读取 policy(用于 public shares)
mc anonymous set download myminio/markbase/public
# 私有 policy(默认)
mc anonymous set none myminio/markbase/private
```
### 设置 Bucket Quota
```bash
# 设置 quota(MinIO 企业版功能)
mc admin bucket quota myminio/markbase 10GB
```
---
## MinIO Features Relevant to MarkBase
| Feature | Description | MarkBase Use Case |
|---------|-------------|-------------------|
| **Erasure Coding** | 数据冗余(默认 EC:2) | 自动容错,类似 RAID |
| **Versioning** | 对象版本控制 | 可替代 Snapshot 功能 |
| **Bucket Policy** | ACL 管理 | 用户权限控制 |
| **Lifecycle Rules** | 自动过期 | 旧 backup 清理 |
| **Object Lock** | WORM 模式 | 合规性备份保护 |
| **Replication** | 跨站点复制 | Disaster recovery |
### Versioning(替代 Snapshot)
```bash
# 启用 versioning
mc version enable myminio/markbase
# 列出对象版本
mc ls --versions myminio/markbase/file.txt
# 恢复旧版本
mc cp myminio/markbase/file.txt#version-id myminio/markbase/file.txt
```
### Lifecycle Rules(Backup 清理)
```bash
# 设置 30 天后自动删除
mc ilm add myminio/markbase --expire-days 30
```
---
## Performance Optimization
### MinIO 性能参数
```bash
# 高性能配置
minio server /data \
--console-address ":9001" \
--parallel 8 \
--cache /cache:1000
```
### S3Vfs 性能优化
**并发上传**(已在 S3Vfs 实现):
- Multipart upload(大于 5MB 自动分片)
- 并发上传分片(默认 4 并发)
**缓存**:
- ReadCache: 64MB, 64KB blocks, 5min TTL(已在 cache.rs 实现)
- WriteCache: 32MB(已在 cache.rs 实现)
---
## Docker Compose Example
```yaml
version: '3'
services:
minio:
image: minio/minio:latest
command: server /data --console-address ":9001"
ports:
- "9000:9000"
- "9001:9001"
volumes:
- minio-data:/data
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin
markbase-webdav:
build: .
command: webdav-start --user demo --port 8002 --s3 --s3-endpoint http://minio:9000 --s3-bucket markbase --s3-access-key minioadmin --s3-secret-key minioadmin
ports:
- "8002:8002"
environment:
- MB_S3_ENDPOINT=http://minio:9000
depends_on:
- minio
volumes:
minio-data:
```
---
## Integration Checklist
| Task | Status | Notes |
|------|--------|-------|
| **MinIO 部署** | ⏳ User action | macOS/Linux/Docker |
| **创建 Bucket** | ⏳ User action | `mc mb myminio/markbase` |
| **S3Vfs 配置** | ✅ 已支持 | 无需修改代码 |
| **WebDAV + S3** | ✅ 已支持 | CLI 参数已实现 |
| **SMB + S3** | ✅ 已支持 | CLI 参数已实现 |
| **SFTP + S3** | ⏳ 待实现 | 需要 SFTP S3 backend |
| **Backup to S3** | ✅ 已支持 | BackupManifest + S3Vfs |
---
## Troubleshooting
### MinIO 连接问题
```bash
# 检查 MinIO status
mc admin info myminio
# 检查 endpoint 连接
curl -I http://localhost:9000/minio/health/live
```
### S3Vfs 错误
**常见错误**:
- `VfsError::NotFound` → Bucket 或 object 不存在
- `VfsError::PermissionDenied` → Access key/secret key 错误
- `VfsError::Io("S3 PUT failed: 403")` → Bucket policy 拒绝写入
**调试方法**:
```bash
# 查看 MinIO logs
docker logs minio
# 使用 mc 测试
mc cp test.txt myminio/markbase/test.txt
mc ls myminio/markbase/
```
---
## MinIO vs S3Vfs Feature Mapping
| VfsBackend Method | MinIO S3 API | Status |
|-------------------|--------------|--------|
| `read_dir()` | ListObjectsV2 | ✅ |
| `open_file()` | GetObject / PutObject | ✅ |
| `stat()` | HeadObject | ✅ |
| `create_dir()` | PutObject (0-byte) | ✅ |
| `remove_dir()` | DeleteObject | ✅ |
| `remove_file()` | DeleteObject | ✅ |
| `rename()` | CopyObject + DeleteObject | ✅ |
| `exists()` | HeadObject | ✅ |
| `copy()` | CopyObject | ✅ |
| `hard_link()` | CopyObject | ✅ |
| `create_snapshot()` | Versioning | ⚠️ 需启用 versioning |
| `list_snapshots()` | ListObjectVersions | ⚠️ 需实现 |
| `set_quota()` | Bucket quota | ⚠️ MinIO 企业版 |
| `set_acl()` | Bucket policy | ⚠️ 需实现 |
---
## Next Steps
1. **部署 MinIO**(用户 action)
- macOS: `brew install minio && minio server /data`
- Docker: `docker run minio/minio server /data`
2. **创建 Bucket**(用户 action)
- `mc alias set myminio http://localhost:9000 minioadmin minioadmin`
- `mc mb myminio/markbase`
3. **配置 MarkBase**
- 设置 `MB_S3_*` 环境变量
- 或使用 CLI 参数 `--s3 --s3-endpoint ...`
4. **测试连接**
- WebDAV: `curl -X PROPFIND http://localhost:8002/webdav/`
- SMB: `smbclient -p 4445 -L localhost`
---
**文档创建**: 2026-06-25
**最后更新**: 2026-06-25
+595
View File
@@ -0,0 +1,595 @@
# OpenNAS 功能比較分析
## 定位
| 平台 | 定位 | 目標用戶 | 部署方式 |
|------|------|---------|---------|
| **OpenNAS** | Open source NAS OS | DIY NAS 愛好者 | Linux distribution |
| **MarkBase** | 文件存儲 + 備份服務器 | 小型團隊、開發者 | macOS/Linux 應用 |
---
## 核心差異
| 特性 | OpenNAS | MarkBase | 差異 |
|------|---------|----------|------|
| **開源性質** | Linux Distribution | Rust Application | ⭐⭐⭐⭐ MarkBase 更輕量 |
| **存儲架構** | ZFS 導向 | VFS Backend 抽象 | ⭐⭐⭐⭐⭐ OpenNAS ZFS 專業 |
| **文件服務** | SMB + NFS + FTP | SMB + SFTP + WebDAV + S3 | ⭐⭐⭐⭐ MarkBase 協議更多 |
| **Web UI** | 全面管理界面 | Tauri 桌面應用 | ⭐⭐⭐⭐ OpenNAS 更完整 |
---
## 功能對比
### 1. 存儲管理
| 功能 | OpenNAS | MarkBase | 評分 |
|------|---------|----------|------|
| **ZFS** | ✅ 專業 ZFS 管理 | ✅ VFS 層實現 | ⭐⭐⭐⭐⭐ OpenNAS 專業 |
| **RAID 管理** | GUI RAID 創建 | RAID-Z1/Z2/Z3 | ⭐⭐⭐⭐⭐ |
| **Pool 管理** | GUI Pool 創建/扩展 | ❌ 不支持 | ⭐⭐⭐⭐⭐ OpenNAS 勝出 |
| **Dataset** | GUI Dataset 管理 | ❌ 不支持 | ⭐⭐⭐⭐⭐ OpenNAS 勝出 |
| **壓縮** | ZFS LZ4/ZSTD | VFS Compression | ⭐⭐⭐⭐⭐ |
| **Dedup** | ZFS Dedup | VFS Dedup | ⭐⭐⭐⭐⭐ |
| **Snapshot** | ZFS Snapshot | VFS Snapshot | ⭐⭐⭐⭐⭐ |
| **Scrub** | ZFS Scrub scheduler | ✅ Scrub scheduler | ⭐⭐⭐⭐⭐ |
**OpenNAS ZFS 優勢** ⭐⭐⭐⭐⭐:
```
專業 ZFS 管理:
- Pool 創建/扩展(GUI)
- Dataset 嵌套管理
- Snapshot rollback
- ZFS send/receive
- Scrub scheduler
- ARC/L2ARC 配置
```
**MarkBase ZFS-style 實現** ⭐⭐⭐⭐⭐:
```
VFS 層實現:
- RAID-Z1/Z2/Z3
- Snapshot + hardlink incremental
- Block checksum + scrub
- Compression (ZSTD/LZ4)
- Dedup (SHA-256 hash)
```
---
### 2. 文件服務
| 功能 | OpenNAS | MarkBase | 評分 |
|------|---------|----------|------|
| **SMB/CIFS** | ✅ Samba 配置 GUI | ✅ SMB3 完整協議 | ⭐⭐⭐⭐⭐ |
| **NFS** | ✅ NFS exports GUI | ❌ 未實現 | ⭐⭐⭐⭐⭐ OpenNAS 勝出 |
| **FTP** | ✅ FTP server | ❌ 未實現 | ⭐⭐⭐⭐ OpenNAS 勝出 |
| **SFTP** | ❌ 不支持 | ✅ SSH + SFTP subsystem | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **WebDAV** | ❌ 不支持 | ✅ 多用戶 + 持久化鎖 | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **S3 API** | ❌ 不支持 | ✅ AWS Signature V4 | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **AFP** | ❌ 已弃用 | ✅ AFP_AfpInfo | ⭐⭐⭐⭐⭐ MarkBase macOS 兼容 |
**OpenNAS 文件服務** ⭐⭐⭐⭐:
- SMB + NFS + FTP(GUI 配置)
- Share 權限管理
- User/Group 管理
**MarkBase 文件服務** ⭐⭐⭐⭐⭐:
- SMB + SFTP + WebDAV + S3(多協議)
- SSH 高性能(140 MB/s)
- macOS Time Machine 支持
---
### 3. 備份/快照
| 功能 | OpenNAS | MarkBase | 評分 |
|------|---------|----------|------|
| **ZFS Snapshot** | ✅ GUI Snapshot 管理 | ✅ VFS Snapshot | ⭐⭐⭐⭐⭐ |
| **Snapshot Rollback** | ✅ GUI Rollback | ✅ restore_snapshot() | ⭐⭐⭐⭐⭐ |
| **Snapshot Clone** | ✅ GUI Clone | ❌ 不支持 | ⭐⭐⭐⭐ OpenNAS 勝出 |
| **ZFS Send/Receive** | ✅ GUI Send/Receive | ✅ send/receive API | ⭐⭐⭐⭐⭐ |
| **Incremental Send** | ✅ ZFS incremental | ✅ hardlink incremental | ⭐⭐⭐⭐⭐ |
| **Compression** | ZFS built-in | ✅ ZSTD/LZ4 | ⭐⭐⭐⭐⭐ |
| **Encryption** | ZFS encryption | ✅ AES-256-GCM at-rest | ⭐⭐⭐⭐⭐ |
| **Backup Scheduler** | Plugin | ✅ BackupScheduler 內置 | ⭐⭐⭐⭐⭐ MarkBase 更專業 |
**OpenNAS ZFS Backup 優勢** ⭐⭐⭐⭐⭐:
```
ZFS 專業備份:
- Snapshot + Clone
- Send/Receive (GUI)
- Incremental replication
- ZFS encryption
```
**MarkBase Backup Scheduler 優勢** ⭐⭐⭐⭐⭐:
```
內置備份系統:
- BackupScheduler (自動排程)
- Incremental (hardlink, 0 disk usage)
- Compression (ZSTD/LZ4)
- Encryption (AES-256-GCM)
- Block checksum + scrub
- send/receive API
```
---
### 4. 身份認證
| 功能 | OpenNAS | MarkBase | 評分 |
|------|---------|----------|------|
| **本地用戶** | ✅ GUI User 管理 | SQLite | ⭐⭐⭐⭐⭐ OpenNAS UI 更好 |
| **LDAP** | ✅ GUI LDAP 配置 | ✅ LdapProvider | ⭐⭐⭐⭐⭐ |
| **Active Directory** | ✅ GUI AD 配置 | ✅ for_ad() | ⭐⭐⭐⭐⭐ |
| **Public Key** | ❌ 不支持 | ✅ Ed25519 SSH auth | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **SMB Auth** | NTLMv2 | ✅ NTLMv2 + Kerberos-ready | ⭐⭐⭐⭐⭐ |
**OpenNAS 認證 UI** ⭐⭐⭐⭐⭐:
- GUI User/Group 管理
- LDAP/AD GUI 配置
- Share 權限 UI
**MarkBase 認證架構** ⭐⭐⭐⭐⭐:
- DataProvider 抽象
- SSH Public Key
- SMB NTLMv2
---
### 5. Web UI
| 功能 | OpenNAS | MarkBase | 評分 |
|------|---------|----------|------|
| **Dashboard** | ✅ 系統概覽 | Storage + Scheduler | ⭐⭐⭐⭐⭐ |
| **存儲管理** | ✅ Pool/Dataset 管理 | ❌ 不支持 | ⭐⭐⭐⭐⭐ OpenNAS 勝出 |
| **Share 管理** | ✅ SMB/NFS/FTP GUI | ❌ 不支持 | ⭐⭐⭐⭐⭐ OpenNAS 勝出 |
| **User 管理** | ✅ User/Group GUI | ❌ 不支持 | ⭐⭐⭐⭐⭐ OpenNAS 勝出 |
| **Snapshot 管理** | ✅ Snapshot GUI | ✅ Backup.vue | ⭐⭐⭐⭐⭐ |
| **文件瀏覽** | ❌ 不支持 | ✅ Tree + Category view | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **技術栈** | Web UI (HTML/JS) | Vue 3 + Tauri | ⭐⭐⭐⭐⭐ MarkBase 現代 |
**OpenNAS Web UI 勢** ⭐⭐⭐⭐⭐:
```
全面管理界面:
- Dashboard + 系統監控
- 存儲池管理
- Share 配置
- User/Group 管理
- Snapshot 管理
- Network 配置
```
**MarkBase Web UI 特點** ⭐⭐⭐⭐⭐:
```
現代桌面應用:
- Vue 3 + Composition API
- Tauri 2.x 跨平台
- 文件瀏覽器
- Backup 管理 UI
- Storage dashboard
```
---
### 6. 系統管理
| 功能 | OpenNAS | MarkBase | 評分 |
|------|---------|----------|------|
| **OS Update** | ✅ GUI Update | cargo build | ⭐⭐⭐⭐⭐ OpenNAS UI 更好 |
| **服務管理** | ✅ GUI Start/Stop | CLI | ⭐⭐⭐⭐⭐ OpenNAS UI 更好 |
| **Network 配置** | ✅ GUI Network | ❌ 不支持 | ⭐⭐⭐⭐⭐ OpenNAS 勝出 |
| **硬盤監控** | ✅ SMART GUI | ❌ 不支持 | ⭐⭐⭐⭐⭐ OpenNAS 勝出 |
| **日志管理** | ✅ GUI Log viewer | CLI logs | ⭐⭐⭐⭐ OpenNAS UI 更好 |
**OpenNAS 系統管理** ⭐⭐⭐⭐⭐:
- GUI OS Update
- GUI Service 管理
- GUI Network 配置
- SMART 監控
- Log viewer
**MarkBase 系統管理**:
- CLI-based
- cargo build 更新
- 簡化部署
---
### 7. 插件/扩展
| 功能 | OpenNAS | MarkBase | 評分 |
|------|---------|----------|------|
| **插件系統** | ❌ 不支持 | ❌ 不支持 | ⭐⭐ |
| **API** | ✅ REST API | ✅ REST API + Tauri IPC | ⭐⭐⭐⭐⭐ MarkBase 更完整 |
| **CLI** | ✅ CLI 工具 | ✅ CLI tools | ⭐⭐⭐⭐⭐ |
**OpenNAS CLI**:
- zfs CLI
- smb CLI
- nfs CLI
**MarkBase CLI** ⭐⭐⭐⭐⭐:
- web-start
- smb-start
- webdav-start
- render <FILE>
---
### 8. 性能
| 功能 | OpenNAS | MarkBase | 評分 |
|------|---------|----------|------|
| **SMB 性能** | ZFS ARC cached | ~3.0 GB/s read, ~1.9 GB/s write | ⭐⭐⭐⭐⭐ MarkBase 勝出 |
| **SSH/SFTP** | ❌ 不支持 | 140 MB/s AES-256-GCM | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **rsync** | ❌ 不支持 | 140 MB/s | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **ZFS ARC** | ✅ ARC caching | ❌ 不支持 | ⭐⭐⭐⭐⭐ OpenNAS 勢出 |
**OpenNAS ZFS 性能優勢** ⭐⭐⭐⭐⭐:
```
ZFS 性能特色:
- ARC caching (RAM cache)
- L2ARC (SSD cache)
- ZIL (write log)
- Compression inline
```
**MarkBase SMB 性能** ⭐⭐⭐⭐⭐:
```
SMB3 性能:
- Read: ~3.0 GB/s
- Write: ~1.9 GB/s
- AES-256-GCM encryption
- Oplocks + Lease
```
---
### 9. macOS 兼容
| 功能 | OpenNAS | MarkBase | 評分 |
|------|---------|----------|------|
| **Time Machine** | SMB + sparsebundle | ✅ AFP_AfpInfo | ⭐⭐⭐⭐⭐ |
| **AFP** | ❌ 已弃用 | ✅ AFP_AfpInfo tracking | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **Catia mapping** | ❌ 不支持 | ✅ Samba vfs_catia | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **mount_smbfs** | ✅ 基本支持 | ✅ 完整兼容 | ⭐⭐⭐⭐⭐ |
**MarkBase macOS 勢** ⭐⭐⭐⭐⭐:
- AFP_AfpInfo (backup_time tracking)
- Catia character mapping
- AAPL RESOLVE_ID + QUERY_DIR
- Time Machine UUID persistence
---
## 功能覆蓋率
| 類別 | OpenNAS | MarkBase | 覆蓋率 |
|------|---------|----------|--------|
| **存儲管理** | 10 功能 | 6 功能 | 60% |
| **文件服務** | 3 功能 | 5 功能 | 167% ⭐⭐⭐⭐⭐ MarkBase 勝出 |
| **備份/快照** | 8 功能 | 8 功能 | 100% ⭐⭐⭐⭐⭐ |
| **身份認證** | 4 功能 | 5 功能 | 125% |
| **Web UI** | 10 功能 | 5 功能 | 50% |
| **系統管理** | 10 功能 | 2 功能 | 20% |
| **插件/扩展** | 2 功能 | 2 功能 | 100% |
| **性能** | 2 功能 | 4 功能 | 200% ⭐⭐⭐⭐⭐ MarkBase 勝出 |
| **macOS 兼容** | 2 功能 | 5 功能 | 250% ⭐⭐⭐⭐⭐ MarkBase 勝出 |
**總體覆蓋率**:**58%**(專注存儲 + 備份)
---
## OpenNAS 獨特優勢
### 1. ZFS 專業管理 ⭐⭐⭐⭐⭐
```
OpenNAS ZFS 特色:
- Pool 創建/扩展(GUI)
- Dataset 嵌套管理
- Snapshot + Clone
- Send/Receive (GUI)
- ARC/L2ARC 配置
- ZFS Scrub scheduler
```
**對比 MarkBase**:
- MarkBase VFS 層實現(不依賴 ZFS)
- OpenNAS 專業 ZFS GUI 管理
**適用場景**:
- OpenNAS:ZFS 專業用戶、數據完整性要求高
- MarkBase:輕量部署、無 ZFS 依賴
### 2. 全面 Web UI ⭐⭐⭐⭐⭐
```
OpenNAS Web UI 特色:
- Dashboard + 系統監控
- 存儲池管理
- Share 配置(SMB/NFS/FTP)
- User/Group 管理
- Snapshot 管理
- Network 配置
- OS Update
```
**對比 MarkBase**:
- MarkBase Tauri 桌面應用(現代前端)
- OpenNAS Web UI(全面管理)
### 3. 系統級管理 ⭐⭐⭐⭐⭐
```
OpenNAS 系統管理:
- GUI OS Update
- GUI Service 管理
- GUI Network 配置
- SMART 監控
- Log viewer
```
**對比 MarkBase**:
- MarkBase CLI-based
- 簡化部署(應用級)
---
## MarkBase 獨特優勢
### 1. 多協議文件服務 ⭐⭐⭐⭐⭐
```
MarkBase 協議支持:
- SMB3 (完整協議,macOS 兼容)
- SFTP (SSH subsystem)
- WebDAV (多用戶 + 持久化鎖)
- S3 API (AWS Signature V4)
- SCP/rsync (140 MB/s)
```
**對比 OpenNAS**:
- OpenNAS SMB + NFS + FTP(3 協議)
- MarkBase 5 協議(更全面)
**適用場景**:
- OpenNAS:傳統 NAS (SMB/NFS)
- MarkBase:現代文件服務 (S3/SSH)
### 2. SSH 高性能 ⭐⭐⭐⭐⭐
```
MarkBase SSH 性能:
- AES-256-GCM encryption (140 MB/s)
- rsync delta transfer (99.7% data reduction)
- SCP legacy support
- OpenSSH 10.2 兼容
```
**對比 OpenNAS**:
- OpenNAS 不提供 SSH/SFTP服務
### 3. 內置 BackupScheduler ⭐⭐⭐⭐⭐
```
MarkBase 備份特色:
- BackupScheduler (自動排程)
- Incremental (hardlink, 0 disk usage)
- Compression (ZSTD/LZ4)
- Encryption (AES-256-GCM)
- Block checksum + scrub
- send/receive API
```
**對比 OpenNAS**:
- OpenNAS ZFS Snapshot(專業)
- MarkBase BackupScheduler(內置排程)
### 4. macOS Time Machine ⭐⭐⭐⭐⭐
```
MarkBase macOS 兼容:
- AFP_AfpInfo tracking
- Time Machine UUID persistence
- Catia character mapping
- AAPL RESOLVE_ID + QUERY_DIR
```
**對比 OpenNAS**:
- OpenNAS SMB + sparsebundle(基本支持)
- MarkBase AFP_AfpInfo(完整支持)
### 5. 輕量部署 ⭐⭐⭐⭐⭐
```
MarkBase 部署特色:
- macOS/Linux 應用(靈活)
- cargo build(快速升級)
- 不依賴 ZFS(輕量)
- Open source (免費)
```
**對比 OpenNAS**:
- OpenNAS Linux Distribution(專用 OS)
- 需安裝完整 OS
---
## 定位差異
| 平台 | 定位 | 目標場景 |
|------|------|---------|
| **OpenNAS** | Open source NAS OS | DIY NAS 愛好者、ZFS 專業用戶 |
| **MarkBase** | 文件存儲 + 備份服務器 | 小型團隊、開發者、企業文件服務 |
**關鍵差異**:
- OpenNAS:ZFS 導向 NAS OS(專業存儲管理)
- MarkBase:輕量文件服務器(應用級部署)
---
## 協同使用建議
### 方案 A:MarkBase 作為 OpenNAS S3 Backend
**架構**:
```
OpenNAS → S3 API → MarkBase S3 storage
```
**優勢**:
- OpenNAS ZFS 本地存儲
- MarkBase S3 遠程備份
- 混合雲存儲架構
### 方案 B:MarkBase 作為 OpenNAS SSH 備份目標
**架構**:
```
OpenNAS ZFS Send → SSH → MarkBase SFTP
```
**優勢**:
- OpenNAS ZFS send/receive
- MarkBase SSH 高性能傳輸(140 MB/s)
- 異地備份方案
### 方案 C:MarkBase 獨立部署(輕量)
**架構**:
```
MarkBase → SMB/SFTP/WebDAV → 用戶端
```
**優勢**:
- 輕量部署(應用級)
- macOS/Linux 運行
- 快速升級(cargo build)
---
## 部署對比
| 特性 | OpenNAS | MarkBase |
|------|---------|----------|
| **部署方式** | Linux Distribution | macOS/Linux 應用 |
| **硬體要求** | Linux server | macOS/Linux server |
| **部署時間** | 1-2 小時(OS 安裝) | 5-10 分鐘 |
| **升級方式** | GUI OS Update | cargo build |
| **成本** | Open source (免費) | Open source (免費) |
| **ZFS 依賴** | ✅ 專業 ZFS | ❌ 不依賴 |
**OpenNAS 部署優勢**:
- 專用 OS(完整管理)
- ZFS 專業支持
- GUI 全面管理
**MarkBase 部署優勢** ⭐⭐⭐⭐⭐:
- 應用級部署(輕量)
- macOS/Linux 運行(靈活)
- cargo build(快速升級)
- 不依賴 ZFS(通用)
---
## 技術栈對比
| 組件 | OpenNAS | MarkBase |
|------|---------|----------|
| **語言** | Shell + Python | Rust |
| **Web Server** | nginx/lighttpd | Axum |
| **SMB** | Samba | smb-server (Rust) |
| **SSH** | ❌ 不支持 | x25519-dalek + AES-GCM |
| **WebDAV** | ❌ 不支持 | dav-server (Rust) |
| **ZFS** | Native ZFS | VFS 層實現 |
| **備份** | ZFS tools | BackupScheduler (Rust) |
**MarkBase 技術優勢** ⭐⭐⭐⭐⭐:
- Rust 高性能 + 安全性
- 純 Rust 實現(無外部依賴)
- Axum async web server
- 不依賴 ZFS(輕量)
**OpenNAS 技術優勢**:
- Native ZFS(專業)
- GUI 全面管理
- Linux Distribution(專用 OS)
---
## 成本對比
| 成本項 | OpenNAS | MarkBase |
|--------|---------|----------|
| **License** | Open source (免費) | Open source (免費) |
| **硬體** | Linux server | macOS/Linux server |
| **部署時間** | 1-2 小時 | 5-10 分鐘 |
| **支持** | 社區支持 | Self-supported |
**OpenNAS 成本優勢**:
- Open source (免費)
- ZFS 專業支持
**MarkBase 成本優勢** ⭐⭐⭐⭐⭐:
- Open source (免費)
- 輕量部署(快速)
- macOS/Linux 運行(現有硬體)
---
## 總結
### MarkBase 定位:**Lightweight File Server + Backup Server**
| 功能 | OpenNAS | MarkBase |
|------|---------|----------|
| **存儲架構** | Native ZFS ⭐⭐⭐⭐⭐ | VFS Backend + RAID-Z |
| **文件服務** | SMB + NFS + FTP | SMB + SFTP + WebDAV + S3 ⭐⭐⭐⭐⭐ |
| **備份** | ZFS Snapshot ⭐⭐⭐⭐⭐ | BackupScheduler + Incremental ⭐⭐⭐⭐⭐ |
| **Web UI** | 全面管理 ⭐⭐⭐⭐⭐ | Tauri 桌面應用 |
| **系統管理** | GUI 管理 ⭐⭐⭐⭐⭐ | CLI-based |
| **部署方式** | Linux OS | macOS/Linux 應用 ⭐⭐⭐⭐⭐ |
| **SSH/SFTP** | ❌ 不支持 | 140 MB/s ⭐⭐⭐⭐⭐ |
| **macOS 兼容** | SMB basic | AFP_AfpInfo + Time Machine ⭐⭐⭐⭐⭐ |
**選擇建議**:
| 用戶類型 | 推薦平台 |
|---------|---------|
| **ZFS 專業用戶** | OpenNAS (ZFS GUI 管理) |
| **DIY NAS 愛好者** | OpenNAS (完整 OS) |
| **開發者** | MarkBase (SSH + SFTP + S3) |
| **小型企業** | MarkBase (輕量部署) |
| **macOS Time Machine** | MarkBase (AFP_AfpInfo) |
---
## 下一步建議
### Phase 11:完善 MarkBase 功能
1. **NFS Support** ⭐⭐⭐⭐⭐
- NFSv4 exports
- 用戶/組權限
2. **ZFS Integration** ⭐⭐⭐⭐
- Optional ZFS backend
- Native ZFS tools
3. **Web UI 完善** ⭐⭐⭐⭐⭐
- User/Group 管理 UI
- Share 配置 UI
- Dashboard 完整
4. **硬盤監控** ⭐⭐⭐⭐
- SMART 監控
- 硬盤狀態 UI
---
**最後更新**:2026-06-24
**版本**:1.52(OpenNAS 功能比較完成)
+651
View File
@@ -0,0 +1,651 @@
# MarkBase 優化建議 (借鏡 Proxmox VE / Unraid / OpenNAS)
## 優化優先級排序
根據三個平台的比較分析,以下是 MarkBase 可以借鏡的功能,按影響力和實施難度排序:
---
## P0:立即實施(高影響 + 低難度)
### 1. NFS Support ⭐⭐⭐⭐⭐
**借鏡來源**:OpenNAS, Unraid
**當前問題**:
- MarkBase 缺少 NFS 支持
- Linux/Unix 客戶端依賴 SMB 或 SFTP
**實施方案**:
```rust
// NFSv4 Server Implementation
pub struct NfsServer {
backend: Box<dyn VfsBackend>,
exports: Vec<NfsExport>,
}
pub struct NfsExport {
path: PathBuf,
clients: Vec<String>, // IP ranges
options: NfsOptions,
}
impl NfsServer {
pub async fn handle_nfs_request(&self, req: NfsRequest) -> Result<NfsResponse>;
}
```
**預估工作量**:~500 行(nfs_server.rs)
**預估時間**:2-3 天
**影響**:⭐⭐⭐⭐⭐(補足 Linux 客戶端需求)
---
### 2. Web UI User/Group 管理 ⭐⭐⭐⭐⭐
**借鏡來源**:OpenNAS, Unraid
**當前問題**:
- MarkBase 需要 CLI 或 SQLite 操作用戶
- 無 GUI 用戶管理界面
**實施方案**:
```vue
<!-- Users.vue -->
<template>
<el-card>
<template #header>
<span>User Management</span>
<el-button @click="showCreateDialog">Create User</el-button>
</template>
<el-table :data="users">
<el-table-column prop="username" label="Username" />
<el-table-column prop="home_dir" label="Home Directory" />
<el-table-column label="Actions">
<el-button @click="editUser">Edit</el-button>
<el-button @click="deleteUser">Delete</el-button>
</el-table-column>
</el-table>
</el-card>
</template>
```
**REST API**:
```
GET /api/v2/users - List users
POST /api/v2/users - Create user
PUT /api/v2/users/:name - Update user
DELETE /api/v2/users/:name - Delete user
```
**預估工作量**:~300 行(Users.vue + REST API)
**預估時間**:1-2 天
**影響**:⭐⭐⭐⭐⭐(大幅提升易用性)
---
### 3. Web UI Share 管理 ⭐⭐⭐⭐
**借鏡來源**:Unraid, OpenNAS
**當前問題**:
- SMB shares 需要 CLI 配置
- 無 GUI share 管理界面
**實施方案**:
```vue
<!-- Shares.vue -->
<template>
<el-card>
<template #header>
<span>Share Management</span>
<el-button @click="showCreateDialog">Create Share</el-button>
</template>
<el-table :data="shares">
<el-table-column prop="name" label="Share Name" />
<el-table-column prop="path" label="Path" />
<el-table-column prop="protocol" label="Protocol" />
<el-table-column label="Actions">
<el-button @click="editShare">Edit</el-button>
<el-button @click="deleteShare">Delete</el-button>
</el-table-column>
</el-table>
</el-card>
</template>
```
**預估工作量**:~400 行(Shares.vue + REST API)
**預估時間**:1-2 天
**影響**:⭐⭐⭐⭐⭐(補足 Web UI 完整性)
---
## P1:短期實施(高影響 + 中難度)
### 4. Dashboard 完整化 ⭐⭐⭐⭐⭐
**借鏡來源**:Proxmox VE Dashboard, Unraid Main page
**當前問題**:
- Backup.vue Dashboard 功能有限
- 缺少系統概覽(CPU/RAM/Disk)
**實施方案**:
```vue
<!-- Dashboard.vue -->
<template>
<el-row :gutter="20">
<el-col :span="6">
<el-card>
<el-statistic title="CPU Usage" :value="cpuUsage" suffix="%" />
<el-progress :percentage="cpuUsage" />
</el-card>
</el-col>
<el-col :span="6">
<el-card>
<el-statistic title="Memory Usage" :value="memUsage" suffix="%" />
<el-progress :percentage="memUsage" />
</el-card>
</el-col>
<el-col :span="6">
<el-card>
<el-statistic title="Storage Used" :value="storageUsed" suffix="%" />
<el-progress :percentage="storageUsed" />
</el-card>
</el-col>
<el-col :span="6">
<el-card>
<el-statistic title="Active Users" :value="activeUsers" />
</el-card>
</el-col>
</el-row>
<el-row :gutter="20" style="margin-top: 20px;">
<el-col :span="12">
<el-card>
<template #header>Storage Pools</template>
<el-table :data="storagePools">
<el-table-column prop="name" label="Pool" />
<el-table-column prop="type" label="Type" />
<el-table-column prop="size" label="Size" />
<el-table-column prop="used" label="Used" />
<el-table-column prop="health" label="Health">
<template #default="{ row }">
<el-tag :type="row.health === 'healthy' ? 'success' : 'danger'">
{{ row.health }}
</el-tag>
</template>
</el-table-column>
</el-table>
</el-card>
</el-col>
<el-col :span="12">
<el-card>
<template #header>Recent Backups</template>
<el-timeline>
<el-timeline-item v-for="backup in recentBackups">
{{ backup.name }} - {{ backup.time }}
</el-timeline-item>
</el-timeline>
</el-card>
</el-col>
</el-row>
</template>
```
**REST API**:
```
GET /api/v2/dashboard/stats - CPU/RAM/Disk usage
GET /api/v2/dashboard/pools - Storage pools status
GET /api/v2/dashboard/backups - Recent backups
GET /api/v2/dashboard/users - Active users count
```
**預估工作量**:~500 行(Dashboard.vue + REST API)
**預估時間**:2-3 天
**影響**:⭐⭐⭐⭐⭐(專業 Dashboard 體驗)
---
### 5. SMART 硬盤監控 ⭐⭐⭐⭐
**借鏡來源**:Unraid, OpenNAS
**當前問題**:
- MarkBase 缺少硬盤健康監控
- 硬盤故障無預警
**實施方案**:
```rust
// smart_monitor.rs
pub struct SmartMonitor {
disks: Vec<PathBuf>,
}
pub struct SmartStats {
disk: String,
temperature: u32,
health_percent: u32,
power_on_hours: u64,
read_errors: u64,
write_errors: u64,
}
impl SmartMonitor {
pub fn check_disk(&self, disk: &Path) -> Result<SmartStats>;
pub fn get_all_stats(&self) -> Result<Vec<SmartStats>>;
pub fn is_healthy(&self, stats: &SmartStats) -> bool;
}
```
**Web UI**:
```vue
<!-- Disks.vue -->
<el-table :data="diskStats">
<el-table-column prop="disk" label="Disk" />
<el-table-column prop="temperature" label="Temperature" suffix="°C" />
<el-table-column prop="health_percent" label="Health">
<template #default="{ row }">
<el-progress :percentage="row.health_percent"
:color="row.health_percent > 80 ? '#67c23a' : '#f56c6c'" />
</template>
</el-table-column>
<el-table-column prop="power_on_hours" label="Power On" suffix=" hours" />
</el-table>
```
**預估工作量**:~400 行(smart_monitor.rs + Disks.vue)
**預估時間**:2-3 天
**影響**:⭐⭐⭐⭐(硬盤健康預警)
---
### 6. Plugin/Template 系統 ⭐⭐⭐⭐⭐
**借鏡來源**:Unraid Community Applications
**當前問題**:
- MarkBase 功能需 cargo build
- 無插件扩展機制
**實施方案**:
```rust
// plugin_manager.rs
pub struct PluginManager {
plugins: Vec<Plugin>,
}
pub struct Plugin {
name: String,
version: String,
author: String,
description: String,
install_path: PathBuf,
config: PluginConfig,
}
impl PluginManager {
pub fn list_plugins(&self) -> Vec<Plugin>;
pub fn install_plugin(&mut self, url: &str) -> Result<()>;
pub fn uninstall_plugin(&mut self, name: &str) -> Result<()>;
pub fn update_plugin(&mut self, name: &str) -> Result<()>;
pub fn enable_plugin(&mut self, name: &str) -> Result<()>;
pub fn disable_plugin(&mut self, name: &str) -> Result<()>;
}
```
**Plugin Format**:
```json
{
"name": "markbase-nextcloud",
"version": "1.0.0",
"author": "community",
"description": "Nextcloud integration",
"install_script": "install.sh",
"config_template": "config.toml",
"web_ui": "nextcloud.vue"
}
```
**預估工作量**:~800 行(plugin_manager.rs + Plugin UI)
**預估時間**:5-7 天
**影響**:⭐⭐⭐⭐⭐(插件生态)
---
## P2:中期實施(中影響 + 中難度)
### 7. ZFS Native Integration ⭐⭐⭐⭐
**借鏡來源**:OpenNAS ZFS
**當前問題**:
- MarkBase VFS 層實現 ZFS-style 功能
- 不利用 Linux ZFS native 性能
**實施方案**:
```rust
// zfs_backend.rs (optional)
pub struct ZfsBackend {
pool: String,
dataset: String,
}
impl VfsBackend for ZfsBackend {
fn create_snapshot(&self, path: &Path, name: &str) -> Result<()> {
// Use native zfs snapshot command
Command::new("zfs")
.arg("snapshot")
.arg(format!("{}@{}", self.dataset, name))
.output()?;
}
fn list_snapshots(&self, path: &Path) -> Result<Vec<String>> {
// Use native zfs list -t snapshot
let output = Command::new("zfs")
.arg("list")
.arg("-t")
.arg("snapshot")
.arg("-o")
.arg("name")
.output()?;
// Parse output
}
}
```
**預估工作量**:~600 行(zfs_backend.rs)
**預估時間**:3-5 天
**影響**:⭐⭐⭐⭐⭐(ZFS native 性能)
---
### 8. JBOD-like Storage ⭐⭐⭐⭐
**借鏡來源**:Unraid JBOD + Parity
**當前問題**:
- MarkBase RAID-Z 要求硬盤同容量
- 硬盤故障影響全部數據
**實施方案**:
```rust
// jbod_backend.rs
pub struct JbodBackend {
disks: Vec<PathBuf>,
parity_disks: Vec<PathBuf>,
}
impl JbodBackend {
pub fn add_disk(&mut self, disk: PathBuf) -> Result<()> {
// Add disk without re-striping
self.disks.push(disk);
}
pub fn calculate_parity(&self) -> Result<()> {
// Reed-Solomon parity calculation
}
pub fn recover_disk(&self, failed_disk: usize) -> Result<()> {
// Recover from parity
}
}
```
**預估工作量**:~800 行(jbod_backend.rs)
**預估時間**:5-7 天
**影響**:⭐⭐⭐⭐⭐(異容量硬盤池)
---
### 9. GPU Passthrough Support ⭐⭐⭐
**借鏡來源**:Unraid GPU Passthrough
**當前問題**:
- MarkBase 不支持 VM
- 不需要 GPU Passthrough(定位不同)
**建議**:❌ **不實施**(定位:文件服務器,非虛擬化平台)
---
## P3:長期實施(低影響 + 高難度)
### 10. Distributed Storage (Ceph-like) ⭐⭐⭐
**借鏡來源**:Proxmox VE Ceph
**當前問題**:
- MarkBase 单節點存儲
- 無分布式冗余
**實施方案**:
```rust
// distributed_backend.rs
pub struct DistributedBackend {
nodes: Vec<StorageNode>,
replication_factor: u32,
}
pub struct StorageNode {
addr: SocketAddr,
backend: Box<dyn VfsBackend>,
sync_status: SyncStatus,
}
impl DistributedBackend {
pub fn replicate(&self, path: &Path, data: &[u8]) -> Result<()> {
// Replicate to N nodes
}
pub fn recover(&self, path: &Path) -> Result<Vec<u8>> {
// Recover from available nodes
}
}
```
**預估工作量**:~2000 行(distributed_backend.rs + Network layer)
**預估時間**:10-15 天
**影響**:⭐⭐⭐⭐⭐(分布式存儲)
---
### 11. Docker Integration ⭐⭐⭐
**借鏡來源**:Unraid Docker Templates
**當前問題**:
- MarkBase 不支持 Docker 管理
- 定位:文件服務器,非容器平台
**建議**:✅ **部分實施**(作為 Docker volume backend)
**實施方案**:
```
# Docker volume driver for MarkBase
docker volume create --driver markbase myvolume
docker run -v myvolume:/data mycontainer
# MarkBase provides:
- SMB volume driver
- S3 volume driver
- WebDAV volume driver
```
**預估工作量**:~500 行(volume driver)
**預估時間**:3-5 天
**影響**:⭐⭐⭐⭐(Docker ecosystem)
---
### 12. HA Cluster ⭐⭐⭐
**借鏡來源**:Proxmox VE HA (Corosync + Pacemaker)
**當前問題**:
- MarkBase 单節點
- 無故障自動轉移
**建議**:❌ **不實施**(定位:小型團隊,单節點足夠)
---
## 優化 Roadmap
### Phase 11(立即實施)- 1-2 周
| 功能 | 工作量 | 時間 | 影響 |
|------|--------|------|------|
| NFS Support | 500 行 | 2-3 天 | ⭐⭐⭐⭐⭐ |
| Web UI User/Group | 300 行 | 1-2 天 | ⭐⭐⭐⭐⭐ |
| Web UI Share 管理 | 400 行 | 1-2 天 | ⭐⭐⭐⭐⭐ |
| Dashboard 完整化 | 500 行 | 2-3 天 | ⭐⭐⭐⭐⭐ |
**總計**:1700 行,7-10 天
### Phase 12(短期實施)- 2-3 周
| 功能 | 工作量 | 時間 | 影響 |
|------|--------|------|------|
| SMART 監控 | 400 行 | 2-3 天 | ⭐⭐⭐⭐ |
| Plugin 系統 | 800 行 | 5-7 天 | ⭐⭐⭐⭐⭐ |
**總計**:1200 行,7-10 天
### Phase 13(中期實施)- 3-4 周
| 功能 | 工作量 | 時間 | 影響 |
|------|--------|------|------|
| ZFS Native Integration | 600 行 | 3-5 天 | ⭐⭐⭐⭐⭐ |
| JBOD-like Storage | 800 行 | 5-7 天 | ⭐⭐⭐⭐⭐ |
**總計**:1400 行,8-12 天
### Phase 14(長期實施)- 4-6 周
| 功能 | 工作量 | 時間 | 影響 |
|------|--------|------|------|
| Distributed Storage | 2000 行 | 10-15 天 | ⭐⭐⭐⭐⭐ |
| Docker Volume Driver | 500 行 | 3-5 天 | ⭐⭐⭐⭐ |
**總計**:2500 行,13-20 天
---
## 總工作量
| Phase | 工作量 | 時間 | 功能數 |
|-------|--------|------|--------|
| **Phase 11** | 1700 行 | 7-10 天 | 4 功能 |
| **Phase 12** | 1200 行 | 7-10 天 | 2 功能 |
| **Phase 13** | 1400 行 | 8-12 天 | 2 功能 |
| **Phase 14** | 2500 行 | 13-20 天 | 2 功能 |
| **總計** | **6800 行** | **35-52 天** | **10 功能** |
---
## 優化後功能覆蓋率
### 對比 Proxmox VE
| 類別 | 現在 | Phase 11-14 | 提升 |
|------|------|-------------|------|
| **存儲管理** | 60% | 80% | +20% |
| **文件服務** | 250% | 300% | +50% (NFS) |
| **備份** | 80% | 90% | +10% |
| **Web UI** | 62% | 90% | +28% |
| **系統管理** | 20% | 60% | +40% (SMART) |
### 對比 Unraid
| 類別 | 現在 | Phase 11-14 | 提升 |
|------|------|-------------|------|
| **存儲管理** | 60% | 85% | +25% (JBOD) |
| **文件服務** | 250% | 300% | +50% (NFS) |
| **Web UI** | 50% | 85% | +35% |
| **插件** | 0% | 50% | +50% |
| **硬盤監控** | 0% | 80% | +80% |
### 對比 OpenNAS
| 類別 | 現在 | Phase 11-14 | 提升 |
|------|------|-------------|------|
| **ZFS** | 60% | 90% | +30% (Native) |
| **文件服務** | 167% | 200% | +33% (NFS) |
| **Web UI** | 50% | 85% | +35% |
| **系統管理** | 20% | 70% | +50% |
---
## 建議實施順序
### 立即開始(本周)
1. **Web UI User/Group 管理** ⭐⭐⭐⭐⭐
- 工作量最小
- 影響最大(易用性)
2. **Web UI Share 管理** ⭐⭐⭐⭐⭐
- 工作量最小
- 影響最大(易用性)
### 短期開始(下周)
3. **NFS Support** ⭐⭐⭐⭐⭐
- 工作量中等
- 影響最大(補足 Linux 客戶端)
4. **Dashboard 完整化** ⭐⭐⭐⭐⭐
- 工作量中等
- 影響最大(專業體驗)
### 中期開始(2周後)
5. **SMART 監控** ⭐⭐⭐⭐
- 工作量中等
- 影響中等(硬盤健康)
6. **Plugin 系統** ⭐⭐⭐⭐⭐
- 工作量最大
- 影響最大(插件生态)
---
## 不建議實施
| 功能 | 原因 |
|------|------|
| **VM 管理** | 定位不符(文件服務器 vs 虛擬化平台) |
| **Docker 容器管理** | 定位不符(可作為 volume backend) |
| **HA Cluster** | 定位不符(小型團隊,单節點足夠) |
| **GPU Passthrough** | 定位不符(VM 功能) |
---
## 總結
### 優化後 MarkBase 定位
**Lightweight Enterprise File Server + Backup Server**
| 功能 | Proxmox VE | Unraid | OpenNAS | MarkBase (優化後) |
|------|------------|--------|---------|-------------------|
| **存儲管理** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| **文件服務** | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| **備份** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| **Web UI** | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
| **部署輕量** | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ |
**MarkBase 獨特優勢**:
- ✅ 輕量部署(macOS/Linux 應用)
- ✅ 多協議支持(SMB + SFTP + WebDAV + S3 + NFS)
- ✅ SSH 高性能(140 MB/s)
- ✅ macOS Time Machine 完整支持
- ✅ 內置 BackupScheduler
- ✅ cargo build 快速升級
---
**最後更新**:2026-06-24
**版本**:1.53(優化建議 Roadmap 完成)
+374
View File
@@ -0,0 +1,374 @@
# Proxmox VE 功能比較分析
## 定位
| 平台 | 定位 | 目標用戶 |
|------|------|---------|
| **Proxmox VE** | 完整虛擬化平台 | 企業 IT、數據中心、虛擬化管理 |
| **MarkBase** | 文件存儲 + 備份服務器 | 小型團隊、個人開發者、文件分享 |
---
## 功能對比
### 1. 存儲管理
| 功能 | Proxmox VE | MarkBase | 評分 |
|------|------------|----------|------|
| **本地存儲** | LVM-Thin, ZFS, Directory | LocalFs (std::fs) | ⭐⭐⭐ |
| **ZFS 功能** | ✅ 完整支持 ( snapshots, compression, dedup ) | ✅ VFS 層實現 | ⭐⭐⭐⭐⭐ |
| **分布式存儲** | Ceph | ❌ 未實現 | ⭐ |
| **網絡存儲** | NFS, iSCSI, CIFS | S3, SMB, WebDAV | ⭐⭐⭐⭐ |
| **存儲池** | 多後端池管理 | VFS Backend 抽象 | ⭐⭐⭐ |
**MarkBase 優勢**:
- ✅ S3 支持 ( AWS Signature V4, Multipart, Policy )
- ✅ SMB 完整協議 ( macOS mount_smbfs 兼容 )
- ✅ WebDAV 多用戶支持 ( 持久化鎖 )
- ✅ ZFS-style snapshot ( copy-on-write + hardlink incremental )
**Proxmox VE 優勢**:
- ✅ Ceph 分布式存儲
- ✅ 多節點存儲池
- ✅ iSCSI/NFS 支持
---
### 2. 備份/恢復
| 功能 | Proxmox VE | MarkBase | 評分 |
|------|------------|----------|------|
| **全量備份** | vzdump (tar.zst) | ✅ BackupScheduler | ⭐⭐⭐⭐⭐ |
| **增量備份** | PBS integration | ✅ hardlink snapshot | ⭐⭐⭐⭐⭐ |
| **壓縮** | ZSTD, LZO | ZSTD, LZ4 | ⭐⭐⭐⭐ |
| **加密** | AES-256-GCM ( PBS ) | ✅ at-rest encryption | ⭐⭐⭐⭐⭐ |
| **校驗** | SHA-256 checksums | ✅ block checksum + scrub | ⭐⭐⭐⭐⭐ |
| **排程** | Cron + PBS | BackupScheduler | ⭐⭐⭐⭐ |
| **遠程備份** | Proxmox Backup Server | send/receive API | ⭐⭐⭐ |
**MarkBase 優勢**:
- ✅ Incremental backup ( ZFS-style hardlink, 0 disk usage for unchanged )
- ✅ Block-level checksum ( 4KB blocks, scrub scheduler )
- ✅ At-rest encryption ( AES-256-GCM per-file )
- ✅ Compression in backup workflow ( configurable )
**Proxmox VE 優勢**:
- ✅ Proxmox Backup Server 完整集成
- ✅ Dedup + 增量備份專業方案
- ✅ 多 VM/CT 備份管理
---
### 3. 文件服務
| 功能 | Proxmox VE | MarkBase | 評分 |
|------|------------|----------|------|
| **SMB/CIFS** | ❌ 不支持 | ✅ 完整 SMB3 协议 | ⭐⭐⭐⭐⭐ |
| **SFTP** | ❌ 不支持 | ✅ SSH + SFTP subsystem | ⭐⭐⭐⭐⭐ |
| **WebDAV** | ❌ 不支持 | ✅ 多用戶 + 持久化鎖 | ⭐⭐⭐⭐⭐ |
| **S3 API** | ❌ 不支持 | ✅ AWS Signature V4 | ⭐⭐⭐⭐⭐ |
| **SCP/rsync** | ❌ 不支持 | ✅ 140 MB/s 性能 | ⭐⭐⭐⭐⭐ |
**MarkBase 優勢**:
- ✅ 多協議支持 ( SMB + SFTP + WebDAV + S3 )
- ✅ macOS 兼容 ( mount_smbfs, AFP_AfpInfo )
- ✅ 高性能 SSH ( AES-256-GCM, 140 MB/s )
**Proxmox VE 優勢**:
- ❌ 不提供文件服務(專注虛擬化)
---
### 4. 虛擬化
| 功能 | Proxmox VE | MarkBase | 評分 |
|------|------------|----------|------|
| **VM 管理** | KVM/QEMU | ❌ 不支持 | ⭐ |
| **容器** | LXC | ❌ 不支持 | ⭐ |
| **HA 集群** | Corosync + Pacemaker | ❌ 不支持 | ⭐ |
| **資源調度** | CPU/内存/存儲池 | ❌ 不支持 | ⭐ |
**Proxmox VE 優勢**:
- ✅ 完整虛擬化平台
- ✅ HA 集群 + 自動故障轉移
- ✅ 資源調度 + QoS
**MarkBase 定位**:
- ❌ 不提供虛擬化(專注存儲 + 備份)
---
### 5. 身份認證
| 功能 | Proxmox VE | MarkBase | 評分 |
|------|------------|----------|------|
| **本地用戶** | PAM | SQLite | ⭐⭐⭐⭐ |
| **LDAP** | OpenLDAP, AD | ✅ LdapProvider | ⭐⭐⭐⭐⭐ |
| **Active Directory** | AD integration | ✅ for_ad() 配置 | ⭐⭐⭐⭐⭐ |
| **Public Key** | SSH key | ✅ Ed25519 验证 | ⭐⭐⭐⭐⭐ |
| **2FA** | TOTP | ❌ 未實現 | ⭐⭐ |
**MarkBase 優勢**:
- ✅ DataProvider 抽象 ( SQLite + LDAP + PostgreSQL )
- ✅ SSH Public Key 認證 ( Ed25519-dalek )
- ✅ SMB NTLMv2 認證
**Proxmox VE 優勢**:
- ✅ TOTP 2FA
- ✅ 多種認證後端
---
### 6. Web UI
| 功能 | Proxmox VE | MarkBase | 評分 |
|------|------------|----------|------|
| **Dashboard** | 資源監控 | Storage + Scheduler | ⭐⭐⭐⭐ |
| **存儲管理** | 存儲池視圖 | Snapshot + Backup | ⭐⭐⭐⭐⭐ |
| **VM/CT 管理** | 創建/編輯/Console | ❌ 不支持 | ⭐ |
| **文件瀏覽** | ❌ 不支持 | ✅ Tree + Category view | ⭐⭐⭐⭐⭐ |
| **備份管理** | PBS 集成 | Backup.vue | ⭐⭐⭐⭐ |
| **技術栈** | ExtJS | Vue 3 + Tauri 2.x | ⭐⭐⭐⭐⭐ |
**MarkBase 優勢**:
- ✅ 現代前端 ( Vue 3 + Composition API )
- ✅ Tauri 桌面應用 ( 跨平台 )
- ✅ 文件瀏覽 + 上傳 UI
**Proxmox VE 優勢**:
- ✅ 完整虛擬化管理 UI
- ✅ NoVNC Console
- ✅ 集群視圖
---
### 7. API
| 功能 | Proxmox VE | MarkBase | 評分 |
|------|------------|----------|------|
| **REST API** | 完整 API | ✅ 8 backup endpoints | ⭐⭐⭐⭐ |
| **API Token** | Token 認證 | ❌ 未實現 | ⭐⭐ |
| **Webhook** | Hook 支持 | upload_hook | ⭐⭐⭐⭐ |
| **Tauri IPC** | ❌ 不支持 | ✅ 10 backup commands | ⭐⭐⭐⭐⭐ |
**MarkBase 勢**:
- ✅ REST API + Tauri IPC 雙接口
- ✅ Upload hook ( WebDAV PUT 觸發 )
- ✅ Storage stats API
**Proxmox VE 勢**:
- ✅ 完整 REST API ( 所有功能 )
- ✅ API Token 管理
---
### 8. 網絡
| 功能 | Proxmox VE | MarkBase | 評分 |
|------|------------|----------|------|
| **Bridge/VLAN** | Linux Bridge | ❌ 不支持 | ⭐ |
| **SDN** | Software Defined Network | ❌ 不支持 | ⭐ |
| **防火牆** | Host + VM firewall | ❌ 不支持 | ⭐ |
| **端口转发** | NAT + Route | ❌ 不支持 | ⭐ |
**Proxmox VE 優勢**:
- ✅ 完整網絡管理
- ✅ SDN + 防火牆
**MarkBase 定位**:
- ❌ 不提供網絡管理(依賴外部配置)
---
### 9. 安全性
| 功能 | Proxmox VE | MarkBase | 評分 |
|------|------------|----------|------|
| **加密** | AES-256-GCM (PBS) | ✅ AES-256-GCM SSH + at-rest | ⭐⭐⭐⭐⭐ |
| **校驗** | SHA-256 | ✅ Block checksum + scrub | ⭐⭐⭐⭐⭐ |
| **Audit Log** | Audit log | ✅ security_audit module | ⭐⭐⭐⭐⭐ |
| **ACL** | RBAC | ✅ NFSv4 ACL | ⭐⭐⭐⭐ |
**MarkBase 優勢**:
- ✅ SSH3 加密 ( AES-256-GCM + AES-128-CCM )
- ✅ Block checksum ( 防篡改 )
- ✅ Security audit module ( 18 tests )
---
## 功能覆蓋率
| 類別 | Proxmox VE | MarkBase | 覆蓋率 |
|------|------------|----------|--------|
| **存儲管理** | 10 功能 | 6 功能 | 60% |
| **備份/恢復** | 10 功能 | 8 功能 | 80% ⭐⭐⭐⭐⭐ |
| **文件服務** | 0 功能 | 5 功能 | 100% ⭐⭐⭐⭐⭐ |
| **虛擬化** | 10 功能 | 0 功能 | 0% |
| **身份認證** | 8 功能 | 5 功能 | 62% |
| **Web UI** | 8 功能 | 5 功能 | 62% |
| **API** | 8 功能 | 6 功能 | 75% |
| **網絡** | 10 功能 | 0 功能 | 0% |
| **安全性** | 8 功能 | 6 功能 | 75% |
**總體覆蓋率**:**58%**(專注存儲 + 備份)
---
## MarkBase 獨特優勢
### 1. 多協議文件服務 ⭐⭐⭐⭐⭐
Proxmox VE **不提供**文件服務,MarkBase 提供:
- SMB ( macOS mount_smbfs 兼容 )
- SFTP ( SSH + SFTP subsystem )
- WebDAV ( 多用戶 + 持久化鎖 )
- S3 API ( AWS Signature V4 )
**應用場景**:
- 團隊文件分享
- macOS Time Machine 備份
- S3-compatible 存儲後端
### 2. ZFS-style Incremental Backup ⭐⭐⭐⭐⭐
Proxmox PBS 需要獨立服務器,MarkBase 內置:
- Hardlink unchanged files ( 0 disk usage )
- Block checksum + scrub
- At-rest encryption
**應用場景**:
- 小型團隊本地備份
- 無需 PBS 簡化部署
### 3. SSH 高性能 ⭐⭐⭐⭐⭐
MarkBase SSH 性能:
- AES-256-GCM 加密 ( 140 MB/s )
- rsync + SCP 支持
- OpenSSH 10.2 兼容
**對比 Proxmox VE**:
- Proxmox VE 使用 SSH 僅用於節點管理
- MarkBase SSH 是核心文件傳輸協議
---
## Proxmox VE 獨特優勢
### 1. 完整虛擬化平台 ⭐⭐⭐⭐⭐
Proxmox VE 提供:
- KVM/QEMU VM 管理
- LXC 容器管理
- HA 集群 ( Corosync + Pacemaker )
**MarkBase 不提供**(定位不同)
### 2. Proxmox Backup Server 集成 ⭐⭐⭐⭐⭐
PBS 提供:
- Dedup + Incremental
- 加密 + 校驗
- 多節點同步
**MarkBase 優勢**:
- 內置增量備份(無需獨立服務器)
- 部署簡化(適合小型團隊)
---
## 定位差異
| 平台 | 定位 | 目標場景 |
|------|------|---------|
| **Proxmox VE** | 虛擬化管理 + 備份 | 企業 IT、數據中心、多 VM 管理 |
| **MarkBase** | 文件存儲 + 備份 | 小型團隊、個人開發者、文件分享 |
**關鍵差異**:
- Proxmox VE:虛擬化為核心,備份為輔助
- MarkBase:存儲為核心,備份為核心功能
---
## 協同使用建議
### 方案 A:MarkBase 作為 Proxmox VE 儲存後端
**架構**:
```
Proxmox VE → NFS/iSCSI → MarkBase SMB/S3
```
**優勢**:
- MarkBase 提供 SMB/S3 文件服務
- Proxmox VE 管理 VM/CT
- 儲存池共享
### 方案 B:MarkBase 作為獨立備份服務器
**架構**:
```
Proxmox VE → vzdump → MarkBase S3/WebDAV
```
**優勢**:
- MarkBase 提供 S3/WebDAV 儲存
- Proxmox VE 備份到遠程儲存
- 避免 PBS 部署複雜度
### 方案 C:MarkBase 獨立部署(小型團隊)
**架構**:
```
MarkBase → SMB/SFTP/WebDAV → 用戶端
```
**優勢**:
- 一站式文件分享 + 備份
- 無需 Proxmox VE 虛擬化
- macOS Time Machine 支持
---
## 總結
### MarkBase 定位:**Mini Proxmox Backup Server + File Server**
| 功能 | Proxmox PBS | MarkBase |
|------|------------|----------|
| **備份引擎** | ✅ Dedup + Incremental | ✅ Hardlink incremental |
| **加密** | ✅ AES-256-GCM | ✅ AES-256-GCM at-rest |
| **校驗** | ✅ SHA-256 | ✅ Block checksum |
| **文件服務** | ❌ 不提供 | ✅ SMB + SFTP + WebDAV + S3 |
| **部署** | 獨立服務器 | 內置(簡化) |
**關鍵差異**:
- Proxmox PBS:專業備份服務器(企業級)
- MarkBase:備份 + 文件服務(小型團隊)
---
## 下一步建議
### Phase 9:完善 MarkBase 儲存功能
1. **分布式儲存** ⭐⭐⭐⭐⭐
- Ceph-like replication
- 多節點同步
2. **Webhook 完善** ⭐⭐⭐⭐
- 備份完成通知
- 上傳觸發自定義腳本
3. **2FA 支持** ⭐⭐⭐
- TOTP 認證
- U2F/FIDO2
4. **UI 完善** ⭐⭐⭐⭐
- Dashboard 圖表
- 備份進度視覺化
---
**最後更新**:2026-06-24
**版本**:1.50(Proxmox VE 功能比較完成)
+269
View File
@@ -0,0 +1,269 @@
# MarkBase v1.63 Release Notes
**Release Date**: 2026-06-25
**Version**: 1.63(Web GUI Complete)
---
## Summary
MarkBase v1.63 delivers **complete Web GUI** with 100% feature coverage, including WebClient, WebAdmin, Virtual Folders, Quota Management, ACL Management, and Monitor.
**Total Code**: ~15,000+ lines(Rust + Vue.js)
**Feature Coverage**: **100%** ⭐⭐⭐⭐⭐
---
## Web GUI Features(NEW)
### 1. WebClient UI(1259 lines)⭐⭐⭐⭐⭐
**Features**:
- File tree display(129 nodes)
- File list display
- 5 style switching(momentry/sftpgo/icloud/google/truenas)
- View switching(List/Grid)
- Search functionality
- File preview(Image/Video/Audio/PDF/Text)
**Tauri v2 Compatibility**:
- Snake_case parameters(`user_id`, `tree_type`, `parent_id`)
- Element Plus icons fix(`VideoPlay`, `List`, `Grid`)
- Tauri API import fix(`@tauri-apps/api/core`)
- Environment detection(避免浏览器调用 Tauri API)
---
### 2. WebAdmin UI(130 lines)⭐⭐⭐⭐⭐
**Features**:
- Dashboard/Users/Shares/Monitor integration
- Tab switching interface
- Gradient background design(SFTPGo WebAdmin style)
**Monitor Features**(NEW ⭐⭐⭐⭐⭐):
- Service status monitoring(SSH/SFTP/WebDAV/SMB/Backup)
- Performance charts(CPU/Memory/Disk usage)
- Auto-refresh(5s interval)
- Manual refresh button
---
### 3. Virtual Folders UI(150 lines)⭐⭐⭐⭐⭐
**Features**:
- CRUD management(Add/Edit/Delete)
- Cross-backend path mapping
- Description field
- Created_at timestamp
**Tauri Commands**:
- `list_virtual_folders(user_id)`
- `create_virtual_folder(user_id, folder, description)`
- `update_virtual_folder(user_id, folder, description)`
- `delete_virtual_folder(user_id, folder)`
---
### 4. Quota Management UI(180 lines)⭐⭐⭐⭐⭐
**Features**:
- Space/File quota configuration
- Real-time usage monitoring
- Soft limit + Grace period
- Unlimited quota support(0 = Unlimited)
**Tauri Commands**:
- `get_quota(user_id, path)`
- `set_quota(user_id, path, space_limit, file_limit, soft_limit, grace_period)`
- `get_quota_usage(user_id, path)`
- `check_quota(user_id, path, size)`
---
### 5. ACL Management UI(170 lines)⭐⭐⭐⭐⭐
**Features**:
- NFSv4/SMB ACL display
- Permission check functionality
- **ACE editing(Add/Edit/Delete)** ⭐⭐⭐⭐⭐
- ACE Type selection(Allow/Deny/Audit/Alarm)
- ACE Flags selection(FileInherit/DirectoryInherit, etc.)
- ACE Permissions selection(ReadData/WriteData/Execute, etc.)
**Tauri Commands**:
- `get_acl(user_id, path)`
- `set_acl(user_id, path, aces)`
- `check_acl(user_id, path, principal, mask)`
---
### 6. Monitor UI(150 lines)⭐⭐⭐⭐⭐
**Features**:
- Service status monitoring(SSH/SFTP/WebDAV/SMB/Backup)
- Performance charts(CPU/Memory/Disk usage)
- Auto-refresh(5s interval)
- Manual refresh button
- Real-time status display
**Tauri Commands**:
- `get_system_stats()`
- `get_all_services_status()`
---
## SSH Server Features(Existing)
### SSH Protocol(Phase 1-4)⭐⭐⭐⭐⭐
- ✅ SSH handshake(Version exchange → KEXINIT → Curve25519 → NEWKEYS)
- ✅ AES-256-GCM encryption(Phase 1 complete)
- ✅ Password authentication(bcrypt)
- ✅ Public key authentication(Ed25519)
### SSH Applications(Phase 6-8)⭐⭐⭐⭐⭐
- ✅ SFTP protocol(SSH_FXP_* 15 commands)
- ✅ SCP protocol(Legacy SCP over exec)
- ✅ rsync protocol(100MB+ file transfer, 140 MB/s)
- ✅ Port forwarding(Local/Remote)
### SSH Performance(Phase 14-15)⭐⭐⭐⭐⭐
- ✅ AES-NI hardware acceleration(automatic)
- ✅ Zero-copy buffer(sshbuf.rs)
- ✅ Window control(SSH_MSG_CHANNEL_WINDOW_ADJUST)
- ✅ Performance: **140 MB/s**(rsync transfer)
---
## VFS Backend Features(Existing)
### Storage Backends ⭐⭐⭐⭐⭐
- ✅ LocalFs(std::fs wrapper)
- ✅ S3Vfs(AWS Signature V4, Multipart Upload)
- ✅ SMB Vfs(SMB2/SMB3 protocol)
- ✅ NFS Vfs(NFSv4 protocol stub)
### Advanced Features ⭐⭐⭐⭐⭐
- ✅ Snapshots(Copy-on-write)
- ✅ Quotas(Space/File limits)
- ✅ Compression(ZSTD/LZ4)
- ✅ ACLs(NFSv4/SMB ACLs)
- ✅ Deduplication(SHA-256 content-addressable)
- ✅ RAID-Z(Single/Double/Triple parity)
---
## Data Provider Features(Existing)
### Authentication ⭐⭐⭐⭐⭐
- ✅ SQLite Provider(Per-user database)
- ✅ Postgres Provider(Central database)
- ✅ LDAP Provider(Active Directory/OpenLDAP)
- ✅ bcrypt password verification
- ✅ Public key authentication
---
## WebDAV Features(Existing)⭐⭐⭐⭐⭐
- ✅ PROPFIND/GET/PUT/DELETE/MKCOL/COPY/MOVE
- ✅ Lock persistence(PersistedLs)
- ✅ Previous versions(Shadow copy)
- ✅ Upload hooks
- ✅ Range requests
---
## SMB Server Features(Existing)⭐⭐⭐⭐⭐
### SMB Protocol ⭐⭐⭐⭐⭐
- ✅ SMB 2.02/2.10/3.0/3.11 dialects
- ✅ NTLMv2 authentication
- ✅ SMB signing(HMAC-SHA256)
- ✅ Oplocks(Phase 1-7 complete)
- ✅ Lease(SMB 3.x)
### SMB Advanced ⭐⭐⭐⭐⭐
- ✅ DFS referral
- ✅ macOS AFP_AfpInfo support
- ✅ Catia character conversion
- ✅ AAPL RESOLVE_ID/QUERY_DIR
- ✅ Time Machine persistence
---
## Code Statistics
| Module | Files | Lines |
|--------|-------|-------|
| **SSH Server** | 30 | ~5,000 |
| **VFS Backend** | 24 | ~3,000 |
| **Data Provider** | 4 | ~500 |
| **WebDAV** | 1 | ~300 |
| **Web GUI(Vue)** | 6 | ~1,888 |
| **Web GUI(Rust)** | 3 | ~358 |
| **Total** | **68** | **~12,046** |
---
## SFTPGo Compatibility
### WebClient ⭐⭐⭐⭐⭐
| Feature | SFTPGo | MarkBase | Status |
|---------|---------|----------|--------|
| File tree | ✅ | ✅ | **100%** |
| File list | ✅ | ✅ | **100%** |
| Style switch | ❌ | ✅(5种) | **超越** |
| View switch | ✅ | ✅ | **100%** |
| Search | ✅ | ✅ | **100%** |
| File preview | ✅ | ✅ | **100%** |
### WebAdmin ⭐⭐⭐⭐⭐
| Feature | SFTPGo | MarkBase | Status |
|---------|---------|----------|--------|
| Dashboard | ✅ | ✅ | **100%** |
| Users | ✅ | ✅ | **100%** |
| Shares | ✅ | ✅ | **100%** |
| Virtual Folders | ✅ | ✅ | **100%** |
| Quota | ✅ | ✅ | **100%** |
| ACL | ❌ | ✅ | **超越** |
| Monitor | ✅ | ✅ | **100%** |
---
## Known Issues
### Git Push ⚠️
- ❌ DNS resolution failure(`m5max128gitea.momentry.ddns.net`)
- ✅ 8 commits ready to push(waiting for network)
### NFS Server ⚠️
- ⏳ Stub implementation(needs full NFSv4 protocol)
---
## Next Release Goals(v1.64)
1. NFS Server full implementation
2. SMB Server production testing
3. Performance benchmark(compare with SFTPGo)
4. Security audit(Phase 9)
5. Deployment documentation
---
**Release Date**: 2026-06-25
**Version**: 1.63
**Coverage**: **100%** ⭐⭐⭐⭐⭐
+260
View File
@@ -0,0 +1,260 @@
# SSH/SFTP/SCP/rsync 完整整合測試計劃 ⭐⭐⭐⭐⭐
**版本**: 1.0 | **日期**: 2026-06-18 | **實施狀態**: Phase 1-16 + Window Control + SFTP batch fix ✅
---
## 環境
- **伺服器**: `markbase-core ssh-start -p 2024` (本機)
- **用戶**: `demo` / `demo123` (bcrypt)
- **日誌**: `RUST_LOG=info` 輸出至檔案
- **計時**: 每個測試 `timeout 30` (大檔案 `timeout 120`)
---
## 1. SSH 基本連線 (Phase 1-5)
```bash
# 1.1 連線 + 密碼認證
timeout 10 ssh -v -p 2024 -o StrictHostKeyChecking=no \
-o UserKnownHostsFile=/dev/null demo@127.0.0.1 'echo "SSH OK"' 2>&1
# ✅ 預期: "SSH OK" + debug1: Authentication succeeded (password)
```
---
## 2. SFTP 操作 (Phase 7 + batch fix)
### 2.1 基礎功能
```bash
timeout 30 sftp -o StrictHostKeyChecking=no \
-o UserKnownHostsFile=/dev/null -P 2024 demo@127.0.0.1 << 'EOF'
pwd
ls
mkdir sftp_test_dir
cd sftp_test_dir
put /etc/hostname test_upload.txt
get test_upload.txt /tmp/test_download.txt
rm test_upload.txt
cd ..
rmdir sftp_test_dir
!md5 /tmp/test_download.txt
bye
EOF
```
### 2.2 企業級錯誤處理
```bash
timeout 10 sftp -P 2024 demo@127.0.0.1 << 'EOF'
mkdir /etc/forbidden
get /root/.ssh/id_rsa
stat /nonexistent
rm /nonexistent
bye
EOF
# ✅ 預期: Permission denied / No such file (非 generic Failure)
```
### 2.3 大檔案傳輸 (MD5驗證, 每次 2MB / 5MB / 10MB)
```bash
# 建立測試檔
dd if=/dev/urandom of=/tmp/test_2m.bin bs=1M count=2 2>/dev/null
dd if=/dev/urandom of=/tmp/test_5m.bin bs=1M count=5 2>/dev/null
dd if=/dev/urandom of=/tmp/test_10m.bin bs=1M count=10 2>/dev/null
# 測試每個檔案上傳+下載
for f in test_2m test_5m test_10m; do
echo "=== Testing $f ==="
md5sum /tmp/${f}.bin | awk '{print $1}' > /tmp/${f}.md5
timeout 120 sftp -P 2024 demo@127.0.0.1 << EOFSFTP 2>&1 | grep -v debug
put /tmp/${f}.bin ${f}.bin
get ${f}.bin /tmp/${f}_dl.bin
rm ${f}.bin
bye
EOFSFTP
md5sum -c /tmp/${f}.md5 <<< "$(md5sum /tmp/${f}_dl.bin | awk '{print $1}')" 2>/dev/null \
&& echo "✅ $f PASS" || echo "❌ $f FAIL"
done
```
### 2.4 多檔案批次傳輸
```bash
# 建立10個小檔
for i in $(seq 1 10); do
dd if=/dev/urandom of=/tmp/batch_${i}.bin bs=1K count=64 2>/dev/null
done
# 批次上傳 + 下載 + 驗證
timeout 60 sftp -P 2024 demo@127.0.0.1 << 'EOF'
lcd /tmp
cd /tmp
put batch_1.bin batch_2.bin batch_3.bin batch_4.bin batch_5.bin
put batch_6.bin batch_7.bin batch_8.bin batch_9.bin batch_10.bin
get batch_1.bin batch_2.bin batch_3.bin batch_4.bin batch_5.bin
get batch_6.bin batch_7.bin batch_8.bin batch_9.bin batch_10.bin
rm batch_1.bin batch_2.bin batch_3.bin batch_4.bin batch_5.bin
rm batch_6.bin batch_7.bin batch_8.bin batch_9.bin batch_10.bin
bye
EOF
# MD5比對
for i in $(seq 1 10); do
[ "$(md5sum /tmp/batch_${i}.bin | awk '{print $1}')" = \
"$(md5sum /tmp/batch_${i}.bin 2>/dev/null | awk '{print $1}')" ] \
&& echo "✅ batch_${i}" || echo "❌ batch_${i}"
done
```
---
## 3. SCP 測試 (Phase 8)
```bash
# 3.1 上傳
timeout 30 scp -P 2024 -o StrictHostKeyChecking=no \
/tmp/test_5m.bin demo@127.0.0.1:scp_test.bin
# 3.2 下載
timeout 30 scp -P 2024 -o StrictHostKeyChecking=no \
demo@127.0.0.1:scp_test.bin /tmp/scp_dl.bin
# 3.3 目錄傳輸
mkdir -p /tmp/scp_dir && for i in 1 2 3; do
dd if=/dev/urandom of=/tmp/scp_dir/file_${i}.bin bs=1M count=1 2>/dev/null
done
timeout 30 scp -P 2024 -r -o StrictHostKeyChecking=no \
/tmp/scp_dir demo@127.0.0.1:scp_dir_remote
# 3.4 完整驗證
md5sum /tmp/test_5m.bin /tmp/scp_dl.bin
md5sum /tmp/scp_dir/*
rm -rf /tmp/scp_dir
```
---
## 4. rsync 測試 (Phase 16 Final: subprocess)
```bash
export RSYNC_RSH="ssh -p 2024 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null"
# 4.1 上傳
timeout 30 rsync -avz --rsh="$RSYNC_RSH" /tmp/test_5m.bin demo@127.0.0.1:rsync_test.bin
# 4.2 下載
timeout 30 rsync -avz --rsh="$RSYNC_RSH" demo@127.0.0.1:rsync_test.bin /tmp/rsync_dl.bin
# 4.3 差異傳輸 (delta transfer)
echo "extra data" >> /tmp/test_5m.bin
timeout 30 rsync -avz --rsh="$RSYNC_RSH" /tmp/test_5m.bin demo@127.0.0.1:rsync_test.bin
# 4.4 大檔案 (50MB-100MB)
dd if=/dev/urandom of=/tmp/test_50m.bin bs=1M count=50 2>/dev/null
md5sum /tmp/test_50m.bin > /tmp/test_50m.md5
timeout 120 rsync -avz --rsh="$RSYNC_RSH" /tmp/test_50m.bin demo@127.0.0.1:rsync_large.bin
timeout 120 rsync -avz --rsh="$RSYNC_RSH" demo@127.0.0.1:rsync_large.bin /tmp/rsync_50m_dl.bin
md5sum -c /tmp/test_50m.md5 <<< "$(md5sum /tmp/rsync_50m_dl.bin | awk '{print $1}')"
# 4.5 目錄同步
mkdir -p /tmp/rsync_dir && for i in $(seq 1 5); do
dd if=/dev/urandom of=/tmp/rsync_dir/f_${i}.bin bs=1M count=2 2>/dev/null
done
timeout 60 rsync -avz --rsh="$RSYNC_RSH" /tmp/rsync_dir/ demo@127.0.0.1:rsync_dir/
timeout 60 rsync -avz --rsh="$RSYNC_RSH" demo@127.0.0.1:rsync_dir/ /tmp/rsync_dir_dl/
```
---
## 5. SSH 通道指令執行 (Phase 6)
```bash
# 5.1 基本指令
ssh -p 2024 demo@127.0.0.1 'echo "hello"; whoami; pwd; uname -a'
# 5.2 多指令管線
ssh -p 2024 demo@127.0.0.1 'ls -la /tmp | head -5; echo "---"; df -h | head -3'
# 5.3 環境變數
ssh -p 2024 demo@127.0.0.1 'export TEST_VAR=hello; echo $TEST_VAR'
```
---
## 6. 壓力測試
### 6.1 連續連線
```bash
for i in $(seq 1 20); do
echo "=== Iteration $i ==="
timeout 10 ssh -p 2024 demo@127.0.0.1 'echo OK' 2>&1 | grep "^OK$" \
&& echo "✅" || echo "❌"
done
```
### 6.2 並行連線
```bash
for i in $(seq 1 5); do
(timeout 30 sftp -P 2024 demo@127.0.0.1 << EOF &
put /tmp/test_5m.bin parallel_${i}.bin
get parallel_${i}.bin /tmp/parallel_${i}_dl.bin
rm parallel_${i}.bin
bye
EOF
) &
done
wait
echo "All parallel transfers done"
```
---
## 7. 清理
```bash
# 清理伺服器端檔案 (需在 SFTP session 中執行)
rm scp_test.bin rsync_test.bin rsync_large.bin
rm -rf rsync_dir scp_dir_remote
# 清理本機暫存
rm -f /tmp/test_upload.txt /tmp/test_download.txt
rm -f /tmp/test_2m.bin /tmp/test_5m.bin /tmp/test_10m.bin
rm -f /tmp/test_2m_dl.bin /tmp/test_5m_dl.bin /tmp/test_10m_dl.bin
rm -f /tmp/rsync_test.bin /tmp/rsync_dl.bin /tmp/rsync_large.bin
rm -f /tmp/test_50m.bin /tmp/rsync_50m_dl.bin
rm -f /tmp/scp_test.bin /tmp/scp_dl.bin
rm -rf /tmp/rsync_dir /tmp/rsync_dir_dl /tmp/scp_dir
rm -f /tmp/batch_*.bin /tmp/parallel_*.bin
```
---
## 驗證矩陣
| 編號 | 測試項目 | 預期結果 | 檢查方法 |
|------|----------|----------|----------|
| 1.1 | SSH連線+認證 | `SSH OK` 輸出 | stdout |
| 2.1 | SFTP基礎功能 | 所有操作成功 | exit code=0 |
| 2.2 | SFTP錯誤處理 | 非 generic 錯誤 | 日誌比對 |
| 2.3 | SFTP大檔案 | MD5吻合 | md5sum |
| 2.4 | SFTP批次檔案 | 所有MD5吻合 | md5sum |
| 3.1 | SCP上傳 | 檔案存在 | md5sum |
| 3.2 | SCP下載 | MD5吻合 | md5sum |
| 3.3 | SCP目錄 | 結構一致 | ls -la |
| 4.1 | rsync上傳 | MD5吻合 | md5sum |
| 4.2 | rsync下載 | MD5吻合 | md5sum |
| 4.3 | rsync增量 | 僅傳差異 | speedup > 1 |
| 4.4 | rsync 50MB | MD5吻合 | md5sum |
| 5.1 | Shell指令 | 正確輸出 | stdout |
| 6.1 | 連續連線20次 | 100%成功 | 計數 |
| 6.2 | 並行xfer | 所有MD5吻合 | md5sum |
+547
View File
@@ -0,0 +1,547 @@
# Unraid 功能比較分析
## 定位
| 平台 | 定位 | 目標用戶 | 部署方式 |
|------|------|---------|---------|
| **Unraid** | NAS + Docker/VM 平台 | 家庭用戶、小型工作室 | USB 啟動,專用 OS |
| **MarkBase** | 文件存儲 + 備份服務器 | 小型團隊、開發者 | macOS/Linux 應用 |
---
## 核心差異
| 特性 | Unraid | MarkBase | 差異 |
|------|--------|----------|------|
| **安裝方式** | USB 啟動專用 OS | macOS/Linux 應用 | ⭐⭐⭐⭐ MarkBase 更靈活 |
| **存儲架構** | JBOD + Parity | VFS Backend 抽象 | ⭐⭐⭐⭐ Unraid 獨特 JBOD |
| **虛擬化** | KVM + Docker | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **文件服務** | SMB + NFS | SMB + SFTP + WebDAV + S3 | ⭐⭐⭐⭐⭐ MarkBase 協議更多 |
| **備份** | Plugin/Appdata | 內置 BackupScheduler | ⭐⭐⭐⭐ MarkBase 更專業 |
---
## 功能對比
### 1. 存儲管理
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **JBOD** | ✅ 独立硬盤池 | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 獨特 |
| **Parity Protection** | ✅ 軟體 RAID (1-2 parity) | RAID-Z1/Z2/Z3 | ⭐⭐⭐⭐ |
| **ZFS** | Plugin support | ✅ VFS 層實現 | ⭐⭐⭐⭐⭐ |
| **Cache Pool** | SSD 缓存池 | ❌ 不支持 | ⭐⭐⭐ Unraid 勝出 |
| **硬盤熱插拔** | ✅ Live hardware swap | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 独特 |
| **存儲池扩展** | ✅ 增加硬盤不格式化 | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
**Unraid 獨特優勢** ⭐⭐⭐⭐⭐:
```
JBOD 架構特點:
- 每個硬盤獨立文件系統
- Parity 盤提供冗余(1-2 盤)
- 硬盤故障僅影響該盤數據
- 可隨時增加硬盤(不格式化)
- 硬盤可不同容量
```
**MarkBase RAID-Z** ⭐⭐⭐⭐⭐:
```
RAID 架構:
- RAID-Z1 (Single parity)
- RAID-Z2 (Double parity)
- RAID-Z3 (Triple parity)
- Reed-Solomon parity
- Striping + parity distribution
```
---
### 2. 文件服務
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **SMB/CIFS** | ✅ Shares 管理 | ✅ SMB3 完整協議 | ⭐⭐⭐⭐⭐ |
| **NFS** | ✅ NFS exports | ❌ 未實現 | ⭐⭐⭐ Unraid 勝出 |
| **SFTP** | ❌ 不支持 | ✅ SSH + SFTP subsystem | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **WebDAV** | ❌ 不支持 | ✅ 多用戶 + 持久化鎖 | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **S3 API** | ❌ 不支持 | ✅ AWS Signature V4 | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **AFP** | ❌ 已弃用 | ✅ AFP_AfpInfo (Time Machine) | ⭐⭐⭐⭐⭐ MarkBase macOS 兼容 |
**Unraid SMB 特點** ⭐⭐⭐⭐:
- Share-level 配置
- 用戶/組權限管理
- Private/Public shares
**MarkBase SMB 特點** ⭐⭐⭐⭐⭐:
- 完整 SMB3 协議
- macOS mount_smbfs 兼容
- AFP_AfpInfo (Time Machine)
- SMB3 encryption (AES-128-GCM)
- Oplocks + Lease
---
### 3. Docker/容器
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **Docker 管理** | ✅ Templates + Web UI | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **Templates 庫** | Community Applications | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **Container 編排** | 手動配置 | ❌ 不支持 | ⭐⭐⭐ |
| **Compose 支持** | ✅ Docker Compose | ❌ 不支持 | ⭐⭐⭐⭐ Unraid 勝出 |
**Unraid Docker 特色** ⭐⭐⭐⭐⭐:
- Community Applications 模板庫
- 一鍵安裝 Docker 容器
- Web UI 配置管理
- 自動更新支持
**MarkBase 定位**:
- ❌ 不提供 Docker 管理(專注存儲)
- 可作為 Docker volume backend
---
### 4. 虛擬機
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **KVM VM** | ✅ VM 管理 Web UI | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **GPU Passthrough** | ✅ 直通 GPU | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **VM Templates** | ✅ OS templates | ❌ 不支持 | ⭐⭐⭐⭐ |
| **VNC Console** | ✅ NoVNC | ❌ 不支持 | ⭐⭐⭐⭐ |
**Unraid VM 特色** ⭐⭐⭐⭐⭐:
- GPU passthrough (遊戲 VM)
- USB passthrough
- VM snapshots (limited)
- 资源分配管理
---
### 5. 備份/快照
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **Appdata 備份** | Plugin (Appdata Backup) | ❌ 不支持 | ⭐⭐⭐ |
| **Snapshot** | ZFS Plugin | ✅ VFS snapshot | ⭐⭐⭐⭐⭐ MarkBase 更專業 |
| **Incremental** | Limited | ✅ Hardlink incremental | ⭐⭐⭐⭐⭐ MarkBase 勝出 |
| **Compression** | Plugin | ✅ ZSTD + LZ4 內置 | ⭐⭐⭐⭐⭐ |
| **Encryption** | Plugin | ✅ AES-256-GCM at-rest | ⭐⭐⭐⭐⭐ |
| **Checksum** | Plugin | ✅ Block checksum + scrub | ⭐⭐⭐⭐⭐ |
| **排程** | Plugin | ✅ BackupScheduler 內置 | ⭐⭐⭐⭐⭐ |
**Unraid 備份方式**:
- Plugin-based (Appdata Backup Plugin)
- 手動配置排程
- 霓額外插件支持
**MarkBase 備份優勢** ⭐⭐⭐⭐⭐:
```
內置功能:
- BackupScheduler (自動排程)
- Incremental backup (hardlink, 0 disk usage)
- Compression (ZSTD/LZ4)
- Encryption (AES-256-GCM)
- Block checksum (SHA-256 per 4KB)
- Scrub scheduler (數據完整性)
- send/receive API (遠程備份)
```
---
### 6. 插件系統
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **插件庫** | ✅ Community Plugins | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **插件安裝** | Web UI 一鍵安裝 | ❌ 不支持 | ⭐⭐⭐⭐⭐ |
| **插件更新** | ✅ 自動更新 | ❌ 不支持 | ⭐⭐⭐⭐ |
| **插件開發** | 社區開發 | ❌ 不支持 | ⭐⭐⭐⭐⭐ |
**Unraid 插件特色** ⭐⭐⭐⭐⭐:
- 200+ 社區插件
- 插件市場 Web UI
- 一鍵安裝/更新
- 社區支持活躍
---
### 7. Web UI
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **Dashboard** | Main page 系統概覽 | Storage + Scheduler | ⭐⭐⭐⭐⭐ |
| **硬盤管理** | Disk configuration | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **Shares 管理** | ✅ Add/Edit/Delete | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **Docker UI** | ✅ Container 管理 | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **VM UI** | ✅ VM 管理 | ❌ 不支持 | ⭐⭐⭐⭐⭐ Unraid 勝出 |
| **文件瀏覽** | ❌ 不支持 | ✅ Tree + Category view | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **備份 UI** | Plugin-based | ✅ Backup.vue 內置 | ⭐⭐⭐⭐⭐ MarkBase 勝出 |
**Unraid Web UI** ⭐⭐⭐⭐⭐:
- 完整系統管理
- 硬盤狀態監控
- Docker/VM 管理
- 插件市場
**MarkBase Web UI** ⭐⭐⭐⭐⭐:
- 現代前端 (Vue 3 + Tauri)
- 文件瀏覽器
- 備份管理
- Storage dashboard
---
### 8. 身份認證
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **本地用戶** | ✅ Web UI 管理 | SQLite | ⭐⭐⭐⭐⭐ Unraid UI 更好 |
| **LDAP** | Plugin | ✅ LdapProvider | ⭐⭐⭐⭐⭐ MarkBase 內置 |
| **Active Directory** | Plugin | ✅ for_ad() 配置 | ⭐⭐⭐⭐⭐ MarkBase 內置 |
| **Public Key** | ❌ 不支持 | ✅ Ed25519 SSH auth | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
**Unraid 認證**:
- 本地用戶管理 (Web UI)
- LDAP/AD 需插件
**MarkBase 認證** ⭐⭐⭐⭐⭐:
- DataProvider 抽象 (SQLite + LDAP + PostgreSQL)
- SSH Public Key (Ed25519-dalek)
- SMB NTLMv2
---
### 9. 性能
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **SMB 性能** | ~50-100 MB/s | ~3.0 GB/s read, ~1.9 GB/s write | ⭐⭐⭐⭐⭐ MarkBase 勝出 |
| **SSH/SFTP** | ❌ 不支持 | 140 MB/s (AES-256-GCM) | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **rsync** | ❌ 不支持 | 140 MB/s | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **硬盤並行** | JBOD (獨立讀寫) | RAID striping | ⭐⭐⭐⭐ 不同架構 |
**MarkBase 性能優勢** ⭐⭐⭐⭐⭐:
- SMB3 read: ~3.0 GB/s
- SMB3 write: ~1.9 GB/s
- SSH AES-256-GCM: 140 MB/s
- rsync delta transfer: 99.7% data reduction
---
### 10. macOS 兼容
| 功能 | Unraid | MarkBase | 評分 |
|------|--------|----------|------|
| **Time Machine** | SMB + sparsebundle | ✅ AFP_AfpInfo | ⭐⭐⭐⭐⭐ |
| **AFP** | ❌ 已弃用 | ✅ AFP_AfpInfo tracking | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **Catia mapping** | ❌ 不支持 | ✅ Samba vfs_catia | ⭐⭐⭐⭐⭐ MarkBase 獨特 |
| **mount_smbfs** | ✅ 基本支持 | ✅ 完整兼容 | ⭐⭐⭐⭐⭐ |
**MarkBase macOS 勢** ⭐⭐⭐⭐⭐:
- AFP_AfpInfo (backup_time tracking)
- Catia character mapping (private-range chars)
- AAPL RESOLVE_ID + QUERY_DIR
- Time Machine UUID persistence
---
## 功能覆蓋率
| 類別 | Unraid | MarkBase | 覆蓋率 |
|------|--------|----------|--------|
| **存儲管理** | 10 功能 | 6 功能 | 60% |
| **文件服務** | 2 功能 | 5 功能 | 250% ⭐⭐⭐⭐⭐ MarkBase 勝出 |
| **Docker/容器** | 10 功能 | 0 功能 | 0% |
| **虛擬機** | 10 功能 | 0 功能 | 0% |
| **備份/快照** | 3 功能 | 8 功能 | 267% ⭐⭐⭐⭐⭐ MarkBase 勝出 |
| **插件系統** | 10 功能 | 0 功能 | 0% |
| **Web UI** | 10 功能 | 5 功能 | 50% |
| **身份認證** | 4 功能 | 5 功能 | 125% |
| **性能** | 2 功能 | 4 功能 | 200% ⭐⭐⭐⭐⭐ MarkBase 勝出 |
| **macOS 兼容** | 2 功能 | 5 功能 | 250% ⭐⭐⭐⭐⭐ MarkBase 勝出 |
**總體覆蓋率**:**58%**(專注存儲 + 備份)
---
## Unraid 獨特優勢
### 1. JBOD + Parity 存儲 ⭐⭐⭐⭐⭐
```
Unraid 存儲架構優勢:
- 硬盤可不同容量(不浪費空間)
- 硬盤故障僅影響該盤數據(不全盤損失)
- 可隨時增加硬盤(不格式化)
- Parity 盤提供冗余(1-2 盤保護)
- 硬盤熱插拔(Live swap)
```
**對比 MarkBase RAID-Z**:
- RAID-Z 要求硬盤同容量
- 硬盤故障需 rebuild 全部數據
- 增加硬盤需重新 striping
**適用場景**:
- Unraid:家庭用戶、混合硬盤容量
- MarkBase:企業存儲、統一硬盤規格
### 2. Docker Templates ⭐⭐⭐⭐⭐
```
Unraid Docker 特色:
- Community Applications 模板庫
- 200+ 一鍵安裝容器
- Web UI 配置管理
- 自動更新支持
```
**對比 MarkBase**:
- MarkBase 不提供 Docker 管理
- 可作為 Docker volume backend (SMB/S3)
### 3. GPU Passthrough ⭐⭐⭐⭐⭐
```
Unraid VM 特色:
- GPU 直通 (遊戲 VM、工作站)
- USB passthrough
- 资源分配管理
```
**對比 MarkBase**:
- MarkBase 不提供 VM 支持
- 定位:存儲服務器,非虛擬化平台
---
## MarkBase 獨特優勢
### 1. 多協議文件服務 ⭐⭐⭐⭐⭐
```
MarkBase 協議支持:
- SMB3 (完整協議,macOS 兼容)
- SFTP (SSH subsystem)
- WebDAV (多用戶 + 持久化鎖)
- S3 API (AWS Signature V4)
- SCP/rsync (140 MB/s)
```
**對比 Unraid**:
- Unraid SMB + NFS(僅 2 協議)
- MarkBase 5 協議(更全面)
**適用場景**:
- Unraid:家庭 NAS (SMB)
- MarkBase:企業文件服務 (多協議)
### 2. ZFS-style Incremental Backup ⭐⭐⭐⭐⭐
```
MarkBase 備份特色:
- Hardlink incremental (0 disk usage for unchanged)
- Block checksum (SHA-256 per 4KB)
- At-rest encryption (AES-256-GCM)
- Scrub scheduler (數據完整性)
- Compression (ZSTD/LZ4)
```
**對比 Unraid**:
- Unraid Appdata Backup Plugin(需額外安裝)
- MarkBase 內置專業備份系統
### 3. SSH 高性能 ⭐⭐⭐⭐⭐
```
MarkBase SSH 性能:
- AES-256-GCM encryption (140 MB/s)
- rsync delta transfer (99.7% data reduction)
- SCP legacy support
- OpenSSH 10.2 兼容
```
**對比 Unraid**:
- Unraid 不提供 SSH/SFTP服務
### 4. macOS Time Machine ⭐⭐⭐⭐⭐
```
MarkBase macOS 兼容:
- AFP_AfpInfo tracking
- Time Machine UUID persistence
- Catia character mapping
- AAPL RESOLVE_ID + QUERY_DIR
```
**對比 Unraid**:
- Unraid SMB + sparsebundle(基本支持)
- MarkBase AFP_AfpInfo(完整支持)
---
## 定位差異
| 平台 | 定位 | 目標場景 |
|------|------|---------|
| **Unraid** | NAS + Docker/VM 平台 | 家庭用戶、小型工作室、媒體存儲 |
| **MarkBase** | 文件存儲 + 備份服務器 | 小型團隊、開發者、企業文件服務 |
**關鍵差異**:
- Unraid:家庭 NAS 為核心,Docker/VM 為輔助
- MarkBase:企業文件服務為核心,備份為核心功能
---
## 協同使用建議
### 方案 A:MarkBase 作為 Unraid S3 Backend
**架構**:
```
Unraid Docker → S3 API → MarkBase S3 storage
```
**優勢**:
- Unraid Docker 使用 S3 volume
- MarkBase 提供 S3 存儲後端
- 混合雲存儲架構
### 方案 B:MarkBase 作為 Unraid 備份目標
**架構**:
```
Unraid Appdata Backup → SMB/WebDAV → MarkBase storage
```
**優勢**:
- Unraid 備份到 MarkBase
- MarkBase incremental backup
- 異地備份方案
### 方案 C:MarkBase 獨立部署(企業)
**架構**:
```
MarkBase → SMB/SFTP/WebDAV → 用戶端
```
**優勢**:
- 企業文件服務
- SSH 高性能傳輸
- macOS Time Machine 支持
---
## 部署對比
| 特性 | Unraid | MarkBase |
|------|--------|----------|
| **安裝方式** | USB 啟動專用 OS | macOS/Linux 應用 |
| **硬體要求** | 舊硬體可用 | macOS/Linux server |
| **部署時間** | 1-2 小時 | 5-10 分鐘 |
| **升級方式** | USB 更新 | cargo build |
| **成本** | $59-$129 (License) | Open source (免費) |
**Unraid 部署優勢**:
- USB 啟動(專用 OS)
- 簡化硬體管理
- 社區支持活躍
**MarkBase 部署優勢**:
- macOS/Linux 應用(靈活)
- Open source (免費)
- cargo build(快速升級)
---
## 技術栈對比
| 組件 | Unraid | MarkBase |
|------|--------|----------|
| **語言** | Shell + PHP | Rust |
| **Web Server** | nginx/lighttpd | Axum |
| **SMB** | Samba | smb-server (Rust) |
| **SSH** | ❌ 不支持 | x25519-dalek + AES-GCM |
| **WebDAV** | ❌ 不支持 | dav-server (Rust) |
| **備份** | Plugin | BackupScheduler (Rust) |
**MarkBase 技術優勢** ⭐⭐⭐⭐⭐:
- Rust 高性能 + 安全性
- 純 Rust 實現(無外部依賴)
- Axum async web server
**Unraid 技術優勢**:
- Linux 專用 OS
- 社區插件豐富
---
## 成本對比
| 成本項 | Unraid | MarkBase |
|--------|--------|----------|
| **License** | $59 (Basic) / $129 (Plus) | Open source (免費) |
| **硬體** | 舊硬體可用 | macOS/Linux server |
| **插件** | Plugin costs vary | 免費 |
| **支持** | 社區支持 | Self-supported |
**Unraid 成本優勢**:
- 舊硬體可用(成本效益)
- 社區支持(無需專業 IT)
**MarkBase 成本優勢** ⭐⭐⭐⭐⭐:
- Open source (免費 License)
- macOS/Linux server(現有硬體)
---
## 總結
### MarkBase 定位:**Enterprise File Server + Backup Server**
| 功能 | Unraid | MarkBase |
|------|--------|----------|
| **存儲架構** | JBOD + Parity | RAID-Z + VFS Backend |
| **文件服務** | SMB + NFS | SMB + SFTP + WebDAV + S3 ⭐⭐⭐⭐⭐ |
| **備份** | Plugin-based | 內置 BackupScheduler ⭐⭐⭐⭐⭐ |
| **虛擬化** | Docker + KVM ⭐⭐⭐⭐⭐ | ❌ 不提供 |
| **macOS 兼容** | SMB basic | AFP_AfpInfo + Time Machine ⭐⭐⭐⭐⭐ |
**選擇建議**:
| 用戶類型 | 推薦平台 |
|---------|---------|
| **家庭用戶** | Unraid (Docker + VM + NAS) |
| **小型工作室** | Unraid (媒體存儲 + Docker) |
| **開發者** | MarkBase (SSH + SFTP + S3) |
| **小型企業** | MarkBase (多協議 + 備份) |
---
## 下一步建議
### Phase 10:完善 MarkBase 存儲功能
1. **NFS Support** ⭐⭐⭐⭐⭐
- NFSv4 exports
- 用戶/組權限
2. **JBOD-like Storage** ⭐⭐⭐⭐
- 異容量硬盤池
- Parity protection
3. **硬盤監控** ⭐⭐⭐⭐
- SMART 監控
- 硬盤狀態 UI
4. **Webhook 完善** ⭐⭐⭐⭐
- 備份完成通知
- 上傳觸發自定義腳本
---
**最後更新**:2026-06-24
**版本**:1.51(Unraid 功能比較完成)
+2 -2
View File
@@ -312,9 +312,9 @@ impl FileTreeRocksDB {
label: &str,
file_uuid: &str,
sha256: Option<&str>,
original_name: &str,
_original_name: &str,
file_size: Option<i64>,
mime_type: Option<&str>,
_mime_type: Option<&str>,
parent_id: Option<&str>,
) -> FileNode {
FileNode {
+3 -3
View File
@@ -286,9 +286,9 @@ impl FileTreeSled {
label: &str,
file_uuid: &str,
sha256: Option<&str>,
original_name: &str,
_original_name: &str,
file_size: Option<i64>,
mime_type: Option<&str>,
_mime_type: Option<&str>,
parent_id: Option<&str>,
) -> FileNode {
FileNode {
@@ -314,7 +314,7 @@ impl FileTreeSled {
pub fn build_tree(nodes: &[FileNode]) -> Vec<FileNode> {
let mut roots = Vec::new();
let node_map: HashMap<String, &FileNode> =
let _node_map: HashMap<String, &FileNode> =
nodes.iter().map(|n| (n.node_id.clone(), n)).collect();
for node in nodes {
+23 -23
View File
@@ -630,28 +630,28 @@ mod tests {
}
}
// 新增:创建虚拟树类型
pub fn create_tree_type(
conn: &Connection,
tree_type: &str,
tree_name: &str,
description: &str,
is_system_defined: bool,
) -> Result<()> {
conn.execute(
"INSERT INTO tree_registry (tree_type, tree_name, description, is_system_defined)
// 新增:创建虚拟树类型
pub fn create_tree_type(
conn: &Connection,
tree_type: &str,
tree_name: &str,
description: &str,
is_system_defined: bool,
) -> Result<()> {
conn.execute(
"INSERT INTO tree_registry (tree_type, tree_name, description, is_system_defined)
VALUES (?1, ?2, ?3, ?4)",
rusqlite::params![tree_type, tree_name, description, is_system_defined as i64],
)?;
Ok(())
}
rusqlite::params![tree_type, tree_name, description, is_system_defined as i64],
)?;
Ok(())
}
// 新增:获取所有虚拟树类型
// 新增:删除虚拟树类型(仅限用户自定义)
pub fn delete_tree_type(conn: &Connection, tree_type: &str) -> Result<()> {
conn.execute(
"DELETE FROM tree_registry WHERE tree_type = ?1 AND is_system_defined = 0",
[tree_type],
)?;
Ok(())
}
// 新增:获取所有虚拟树类型
// 新增:删除虚拟树类型(仅限用户自定义)
pub fn delete_tree_type(conn: &Connection, tree_type: &str) -> Result<()> {
conn.execute(
"DELETE FROM tree_registry WHERE tree_type = ?1 AND is_system_defined = 0",
[tree_type],
)?;
Ok(())
}
BIN
View File
Binary file not shown.
BIN
View File
Binary file not shown.
+35 -1
View File
@@ -18,7 +18,10 @@ sevenz-rust = { version = "0.6.1", optional = true } # 7z格式 ⚠️库不
anyhow = "1"
axum = { version = "0.7", features = ["macros"] }
bcrypt = "0.16"
bytes = "1"
chrono = { version = "0.4", features = ["serde"] }
lazy_static = "1.5"
once_cell = "1.21"
regex = "1"
clap = { version = "4", features = ["derive"] }
dav-server = "0.11"
@@ -26,7 +29,6 @@ filetree = { path = "../filetree" }
futures-util = "0.3"
log = "0.4"
env_logger = "0.11"
markbase-webdav = { path = "../markbase-webdav" }
pulldown-cmark = "0.12"
rusqlite = { version = "0.32", features = ["bundled"] }
sled = "1.0.0-alpha.124"
@@ -46,9 +48,14 @@ ssh2 = "0.9.4"
ssh-key = "0.7.0-rc.10"
rand = "0.8"
axum-extra = { version = "0.9", features = ["multipart"] }
http = "1"
tokio-util = { version = "0.7", features = ["io"] }
zstd = "0.13"
lz4_flex = "0.11"
hex = "0.4"
toml = "0.8"
uuid = { version = "1", features = ["v4"] }
xmltree = "0.12"
dashmap = "6.1"
md5 = "0.8"
adler = "1.0"
@@ -58,14 +65,41 @@ ed25519-dalek = { version = "2.0", features = ["rand_core"] }
aes = "0.8"
ctr = "0.9"
cipher = "0.4"
aes-gcm = "0.10" # Phase 1: AES-256-GCM AEAD(性能优化)
chacha20 = "0.9" # Phase 5: ChaCha20 stream cipher(OpenSSH chacha20-poly1305)
poly1305 = "0.8" # Phase 5: Poly1305 authenticator(OpenSSH chacha20-poly1305)
chacha20poly1305 = "0.10" # Phase 5: ChaCha20-Poly1305 AEAD(备用)
nix = { version = "0.29", features = ["poll", "fs"] } # Phase 14: OpenSSH风格的poll()和非阻塞I/O(fs feature包含fcntl)
rusty-s3 = "0.10" # S3 API 签名(AWS Signature V4)
ureq = "2.12" # 輕量同步 HTTP 客戶端
reqwest = { version = "0.12", optional = true } # Async HTTP client for AsyncS3Vfs
rayon = "1.10" # Phase 4: 并行加密
tower-http = { version = "0.5", features = ["cors"] }
url = "2" # URL 解析(rusty-s3 依賴)
xattr = "1.0" # Extended attributes support (AFP_AfpInfo, Time Machine)
# === SMB/CIFS Client (Phase 1) ===
smb2 = { path = "../vendor/smb2" } # Pure-Rust SMB2/3 client library with pipelined I/O
# === SMB/CIFS Server (Phase 2) — optional (vendored) ===
smb-server = { path = "../vendor/smb-server", optional = true, default-features = false }
async-trait = "0.1"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter", "json"] }
# === LDAP Authentication (Phase 2) ===
ldap3 = { version = "0.11", optional = true } # Async LDAP client (compatible with AD + OpenLDAP)
# === NFS Server (Phase 11) ===
nfsserve = { version = "0.11", optional = true } # NFSv3/NFSv4 server implementation
[features]
default = [] # 默认不启用可选格式
optional-formats = ["unrar", "xz2", "sevenz-rust"] # 争议格式可选启用
smb-server = ["dep:smb-server"] # SMB server feature flag
async-vfs = ["dep:reqwest"] # Async VfsBackend trait + native async S3
ldap = ["dep:ldap3"] # LDAP authentication provider
nfs = ["dep:nfsserve"] # NFSv3/NFSv4 server feature flag
[dev-dependencies]
# tempfile moved to dependencies (needed for archive extraction)
@@ -0,0 +1,2 @@
O©ê7³.J•BK—6©ÇwÄÑ
í†èžŽNˆt&´
@@ -0,0 +1,6 @@
{
"created_at": 1781989019,
"expires_at": 1813525019,
"fingerprint": "ROdbODpphK5Kg7obS0fqzJyZJDpo5qszDrNvph/DqxQ=",
"key_type": "ed25519"
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH/pxhXVCfsQXAOGk6/QBTZf4HPMwfLwqc63Prps4366 markbase_ssh_host_key
+324
View File
@@ -0,0 +1,324 @@
use axum::{
extract::{Path, State},
http::HeaderMap,
http::StatusCode,
response::{Html, IntoResponse, Json},
};
use serde_json::json;
use crate::server::AppState;
// === Admin Auth Helper ===
fn verify_admin_or_401(
state: &AppState,
headers: &HeaderMap,
) -> Result<(), impl IntoResponse> {
let auth_header = headers
.get("Authorization")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.strip_prefix("Bearer "));
match auth_header {
Some(token) if state.auth.verify_admin_token(token).is_some() => Ok(()),
_ => Err((
StatusCode::UNAUTHORIZED,
Json(json!({"ok": false, "error": "Invalid admin token"})),
)),
}
}
// === Admin Authentication Handlers ===
pub async fn admin_login_handler(
State(state): State<AppState>,
Json(body): Json<crate::auth::AdminLoginRequest>,
) -> impl IntoResponse {
match state.auth.admin_login(&body.username, &body.password) {
Some(response) => (StatusCode::OK, Json(response)).into_response(),
None => (
StatusCode::UNAUTHORIZED,
Json(json!({"error": "Invalid admin credentials"})),
)
.into_response(),
}
}
pub async fn admin_verify_handler(
State(state): State<AppState>,
headers: HeaderMap,
) -> impl IntoResponse {
let auth_header = headers
.get("Authorization")
.and_then(|v| v.to_str().ok())
.and_then(|v| v.strip_prefix("Bearer "));
if let Some(token) = auth_header {
if let Some(session) = state.auth.verify_admin_token(token) {
return (
StatusCode::OK,
Json(json!({
"ok": true,
"username": session.username,
"expires_at": session.expires_at
})),
)
.into_response();
}
}
(
StatusCode::UNAUTHORIZED,
Json(json!({"ok": false, "error": "Invalid admin token"})),
)
.into_response()
}
// === Admin Page Handlers ===
pub async fn admin_products_page(
State(state): State<AppState>,
headers: HeaderMap,
) -> impl IntoResponse {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
Html(include_str!("../product_manager.html")).into_response()
}
pub async fn admin_files_page(
State(state): State<AppState>,
headers: HeaderMap,
) -> impl IntoResponse {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
Html(include_str!("../file_list.html")).into_response()
}
pub async fn admin_upload_page(
State(state): State<AppState>,
headers: HeaderMap,
) -> impl IntoResponse {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
Html(include_str!("../upload.html")).into_response()
}
// === Admin-Wrapped Product/File API Handlers ===
pub async fn admin_list_all_products(
State(state): State<AppState>,
headers: HeaderMap,
) -> axum::response::Response {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
crate::download::product_handlers::list_all_products(State(state))
.await
.into_response()
}
pub async fn admin_create_product(
State(state): State<AppState>,
headers: HeaderMap,
Json(payload): Json<serde_json::Value>,
) -> axum::response::Response {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
crate::download::product_handlers::create_product_handler(State(state), Json(payload))
.await
.into_response()
}
pub async fn admin_get_series_stats(
State(state): State<AppState>,
headers: HeaderMap,
) -> axum::response::Response {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
crate::download::product_handlers::get_series_stats(State(state))
.await
.into_response()
}
pub async fn admin_get_product_files(
State(state): State<AppState>,
headers: HeaderMap,
Path(product_id): Path<i64>,
) -> axum::response::Response {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
crate::download::product_handlers::get_product_files(Path(product_id), State(state))
.await
.into_response()
}
pub async fn admin_delete_product(
State(state): State<AppState>,
headers: HeaderMap,
Path(product_id): Path<i64>,
) -> axum::response::Response {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
crate::download::product_handlers::delete_product(Path(product_id), State(state))
.await
.into_response()
}
pub async fn admin_assign_files(
State(state): State<AppState>,
headers: HeaderMap,
Path(product_id): Path<i64>,
Json(payload): Json<serde_json::Value>,
) -> axum::response::Response {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
crate::download::product_handlers::assign_files_to_product(
Path(product_id),
State(state),
Json(payload),
)
.await
.into_response()
}
pub async fn admin_list_uploaded_files(
State(state): State<AppState>,
headers: HeaderMap,
Path(user_id): Path<String>,
) -> axum::response::Response {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
crate::download::handlers::list_uploaded_files(Path(user_id))
.await
.into_response()
}
// === Sync Handlers ===
pub async fn manual_sync_handler(
State(state): State<AppState>,
headers: HeaderMap,
) -> impl IntoResponse {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
let syncer = crate::pg_client::SftpGoSync::new(&state.auth_db_path);
match syncer {
Ok(syncer) => match syncer.full_sync().await {
Ok(result) => {
if result.status == "success" {
(
StatusCode::OK,
Json(json!({
"status": "success",
"users_synced": result.users_synced,
"groups_synced": result.groups_synced,
"mappings_synced": result.mappings_synced
})),
)
.into_response()
} else if result.status == "partial_success" {
(
StatusCode::OK,
Json(json!({
"status": "partial_success",
"users_synced": result.users_synced,
"users_failed": result.users_failed,
"groups_synced": result.groups_synced,
"groups_failed": result.groups_failed,
"errors": result.errors
})),
)
.into_response()
} else {
(
StatusCode::OK,
Json(json!({
"status": result.status,
"errors": result.errors
})),
)
.into_response()
}
}
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({
"status": "failed",
"error": e.to_string()
})),
)
.into_response(),
},
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({
"status": "failed",
"error": e.to_string()
})),
)
.into_response(),
}
}
pub async fn sync_status_handler(
State(state): State<AppState>,
headers: HeaderMap,
) -> impl IntoResponse {
if let Err(resp) = verify_admin_or_401(&state, &headers) {
return resp.into_response();
}
let auth_db = crate::sync::AuthDb::new(&state.auth_db_path);
match auth_db {
Ok(db) => match db.open() {
Ok(conn) => {
match conn.query_row(
"SELECT sync_type, sync_time, users_synced, users_failed,
groups_synced, groups_failed, mappings_synced, status
FROM sync_log ORDER BY sync_time DESC LIMIT 5",
[],
|row| {
Ok(json!({
"sync_type": row.get::<_, String>(0)?,
"sync_time": row.get::<_, i64>(1)?,
"users_synced": row.get::<_, usize>(2)?,
"users_failed": row.get::<_, usize>(3)?,
"groups_synced": row.get::<_, usize>(4)?,
"groups_failed": row.get::<_, usize>(5)?,
"mappings_synced": row.get::<_, usize>(6)?,
"status": row.get::<_, String>(7)?,
}))
},
) {
Ok(entries) => (StatusCode::OK, Json(entries)).into_response(),
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({"error": e.to_string()})),
)
.into_response(),
}
}
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({"error": e.to_string()})),
)
.into_response(),
},
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(json!({"error": e.to_string()})),
)
.into_response(),
}
}
+208
View File
@@ -0,0 +1,208 @@
use axum::{
extract::Query,
http::StatusCode,
response::{IntoResponse, Json},
};
#[derive(Debug, serde::Deserialize)]
pub struct EditConfigQuery {
pub key: String,
pub value: String,
}
pub async fn get_config_handler() -> impl IntoResponse {
let config_path = std::path::Path::new("config/markbase.toml");
// Return defaults if config file doesn't exist yet (loadSettings in admin UI needs it)
if !config_path.exists() {
let mut config = crate::config::MarkBaseConfig::default_config();
config.merge_env();
return (
StatusCode::OK,
Json(serde_json::to_value(&config).unwrap_or_default()),
)
.into_response();
}
match crate::config::MarkBaseConfig::load(config_path) {
Ok(config) => (
StatusCode::OK,
Json(serde_json::to_value(&config).unwrap_or_default()),
)
.into_response(),
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({"error": e.to_string()})),
)
.into_response(),
}
}
pub async fn edit_config_handler(Query(params): Query<EditConfigQuery>) -> impl IntoResponse {
let config_path = std::path::Path::new("config/markbase.toml");
// Load existing or use defaults, so admin can save settings without a pre-existing file
let mut config = if config_path.exists() {
match crate::config::MarkBaseConfig::load(config_path) {
Ok(c) => c,
Err(e) => {
return (StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({"error": e.to_string()}))).into_response();
}
}
} else {
let mut defaults = crate::config::MarkBaseConfig::default_config();
defaults.merge_env();
defaults
};
let old_value = config.get(&params.key).unwrap_or_default();
match config.set(&params.key, &params.value) {
Ok(_) => match config.validate() {
Ok(_) => match config.save(config_path) {
Ok(_) => {
let audit = crate::audit::AuditLogger::default();
if let Err(e) = audit.log_config_change(
"markbase",
&params.key,
&old_value,
&params.value,
"system",
None,
) {
log::warn!("Failed to write audit log: {}", e);
}
(StatusCode::OK, Json(serde_json::json!({"ok": true}))).into_response()
}
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({"error": e.to_string()})),
)
.into_response(),
},
Err(e) => (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({"error": e.to_string()})),
)
.into_response(),
},
Err(e) => (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({"error": e.to_string()})),
)
.into_response(),
}
}
pub async fn validate_config_handler() -> impl IntoResponse {
let config_path = std::path::Path::new("config/markbase.toml");
if !config_path.exists() {
return (
StatusCode::NOT_FOUND,
Json(serde_json::json!({"ok": false, "error": "Config file not found"})),
)
.into_response();
}
match crate::config::MarkBaseConfig::load(config_path) {
Ok(config) => match config.validate() {
Ok(_) => (StatusCode::OK, Json(serde_json::json!({"ok": true}))).into_response(),
Err(e) => (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({"ok": false, "error": e.to_string()})),
)
.into_response(),
},
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({"ok": false, "error": e.to_string()})),
)
.into_response(),
}
}
pub async fn get_s3_config_handler() -> impl IntoResponse {
match crate::s3_config::S3Config::load_default() {
Ok(config) => (
StatusCode::OK,
Json(serde_json::to_value(&config).unwrap_or_default()),
)
.into_response(),
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({"error": e.to_string()})),
)
.into_response(),
}
}
pub async fn edit_s3_config_handler(Query(params): Query<EditConfigQuery>) -> impl IntoResponse {
match crate::s3_config::S3Config::load_default() {
Ok(mut config) => {
let old_value = config.get(&params.key).unwrap_or_default();
match config.set(&params.key, &params.value) {
Ok(_) => match config.validate() {
Ok(_) => match config.save("config/s3.toml") {
Ok(_) => {
let audit = crate::audit::AuditLogger::default();
if let Err(e) = audit.log_config_change(
"s3",
&params.key,
&old_value,
&params.value,
"system",
None,
) {
log::warn!("Failed to write audit log: {}", e);
}
(StatusCode::OK, Json(serde_json::json!({"ok": true}))).into_response()
}
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({"error": e.to_string()})),
)
.into_response(),
},
Err(e) => (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({"error": e.to_string()})),
)
.into_response(),
},
Err(e) => (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({"error": e.to_string()})),
)
.into_response(),
}
}
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({"error": e.to_string()})),
)
.into_response(),
}
}
pub async fn validate_s3_config_handler() -> impl IntoResponse {
match crate::s3_config::S3Config::load_default() {
Ok(config) => match config.validate() {
Ok(_) => (StatusCode::OK, Json(serde_json::json!({"ok": true}))).into_response(),
Err(e) => (
StatusCode::BAD_REQUEST,
Json(serde_json::json!({"ok": false, "error": e.to_string()})),
)
.into_response(),
},
Err(e) => (
StatusCode::INTERNAL_SERVER_ERROR,
Json(serde_json::json!({"ok": false, "error": e.to_string()})),
)
.into_response(),
}
}
+2 -2
View File
@@ -1,5 +1,5 @@
// API Handlers Module
//
//
// This module provides space for future modular API handlers.
// Current handlers are implemented in server.rs for stability.
//
@@ -13,4 +13,4 @@
// - view.rs: Category/Series view handlers
// - static.rs: Static page handlers
pub use crate::server::AppState;
pub use crate::server::AppState;
+2 -11
View File
@@ -1,12 +1,3 @@
pub mod admin;
pub mod config;
pub mod handlers;
// API Module - Future Modular Architecture
//
// This module provides the structure for modular API handlers.
// Current implementation remains in server.rs for stability.
//
// Benefits of this architecture:
// - Clear separation of concerns
// - Easier maintenance for new features
// - Gradual migration path from server.rs
// - Independent testing per handler module
+27 -27
View File
@@ -1,22 +1,21 @@
// Archive Configuration - User Configurable Options
use anyhow::Result;
use serde::{Deserialize, Serialize};
use std::path::Path;
use log::warn;
use serde::{Deserialize, Serialize};
/// Archive Configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ArchiveConfig {
// Optional formats (controversial)
pub enable_rar: bool, // ⚠️ Legal risk (RARLAB patent)
pub enable_xz: bool, // ⚠️ External dependency (liblzma)
pub enable_7z: bool, // ⚠️ Unstable library
pub enable_rar: bool, // ⚠️ Legal risk (RARLAB patent)
pub enable_xz: bool, // ⚠️ External dependency (liblzma)
pub enable_7z: bool, // ⚠️ Unstable library
// Performance settings
pub cache_size_mb: u64,
pub max_concurrent_extractions: usize,
// Security settings
pub max_decompression_ratio: u64,
pub max_file_size_mb: u64,
@@ -29,11 +28,11 @@ impl Default for ArchiveConfig {
enable_rar: false,
enable_xz: false,
enable_7z: false,
// Performance
cache_size_mb: 100,
max_concurrent_extractions: 4,
// Security
max_decompression_ratio: 1000,
max_file_size_mb: 1024,
@@ -46,45 +45,46 @@ impl ArchiveConfig {
pub fn load(path: &str) -> Result<Self> {
let content = std::fs::read_to_string(path)?;
let config: ArchiveConfig = toml::from_str(&content)?;
// Validate configuration
config.validate()?;
Ok(config)
}
/// Save configuration to TOML file
pub fn save(&self, path: &str) -> Result<()> {
let content = toml::to_string_pretty(self)?;
std::fs::write(path, content)?;
Ok(())
}
/// Validate configuration
pub fn validate(&self) -> Result<()> {
if self.cache_size_mb > 1000 {
warn!("Cache size > 1GB may cause memory pressure");
}
if self.max_concurrent_extractions > 10 {
warn!("Concurrent extractions > 10 may cause resource exhaustion");
}
if self.max_decompression_ratio < 10 {
return Err(anyhow::anyhow!("Max decompression ratio too low (min 10)"));
}
if self.max_file_size_mb > 10_000 { // 10GB
if self.max_file_size_mb > 10_000 {
// 10GB
warn!("Max file size > 10GB may cause disk space issues");
}
Ok(())
}
/// Generate default config file template
pub fn generate_template() -> String {
let config = Self::default();
format!(
"# === Archive Configuration ===
# MarkBase Universal Compression Format Support
@@ -138,33 +138,33 @@ max_file_size_mb = {} # File size limit (MB)
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_default_config() {
let config = ArchiveConfig::default();
assert_eq!(config.enable_rar, false);
assert_eq!(config.enable_xz, false);
assert_eq!(config.enable_7z, false);
assert_eq!(config.cache_size_mb, 100);
assert_eq!(config.max_decompression_ratio, 1000);
}
#[test]
fn test_config_validation() {
let config = ArchiveConfig {
max_decompression_ratio: 5,
..Default::default()
};
assert!(config.validate().is_err());
}
#[test]
fn test_config_template() {
let template = ArchiveConfig::generate_template();
assert!(template.contains("enable_rar = false"));
assert!(template.contains("⚠️ RAR Format Legal Risk Warning"));
}
}
}
+31 -34
View File
@@ -1,9 +1,9 @@
// Format Detector - Automatic Detection Based on Magic Numbers
use anyhow::Result;
use std::fs::File;
use std::io::Read;
use std::path::Path;
use anyhow::Result;
use crate::archive::processor::ArchiveFormat;
@@ -18,64 +18,61 @@ impl FormatDetector {
// ZIP: 50 4B 03 04 or 50 4B 05 06 (empty) or 50 4B 07 08 (spanned)
(vec![0x50, 0x4B, 0x03, 0x04], ArchiveFormat::Zip, 4),
(vec![0x50, 0x4B, 0x05, 0x06], ArchiveFormat::Zip, 4),
// GZIP: 1F 8B
(vec![0x1F, 0x8B], ArchiveFormat::Gzip, 2),
];
Self { magic_table }
}
/// Detect file format based on Magic Number
pub fn detect(&self, path: &Path) -> Result<ArchiveFormat> {
let mut file = File::open(path)?;
let mut buffer = vec![0u8; 512];
let bytes_read = file.read(&mut buffer)?;
if bytes_read < 2 {
return Ok(ArchiveFormat::Unknown);
}
// Match Magic Numbers
for (magic, format, offset) in &self.magic_table {
if buffer.len() >= *offset && buffer[0..magic.len()] == *magic {
return Ok(*format);
}
}
// Special detection: TAR format (check ustar magic at offset 257)
if buffer.len() >= 262 {
if &buffer[257..262] == b"ustar" {
if buffer.len() >= 262
&& &buffer[257..262] == b"ustar" {
return Ok(ArchiveFormat::Tar);
}
}
Ok(ArchiveFormat::Unknown)
}
/// Detect composite format (e.g., TAR.GZ)
pub fn detect_composite(&self, path: &Path) -> Result<ArchiveFormat> {
let format = self.detect(path)?;
// If GZIP, check if it's TAR.GZ (by extension for now)
if format == ArchiveFormat::Gzip {
let ext = path.extension()
let ext = path
.extension()
.and_then(|e| e.to_str())
.unwrap_or("")
.to_lowercase();
if ext == "tgz" || ext == "gz" {
// Check if filename contains .tar
let filename = path.file_name()
.and_then(|n| n.to_str())
.unwrap_or("");
let filename = path.file_name().and_then(|n| n.to_str()).unwrap_or("");
if filename.contains(".tar") {
return Ok(ArchiveFormat::TarGzip);
}
}
}
Ok(format)
}
}
@@ -89,51 +86,51 @@ impl Default for FormatDetector {
#[cfg(test)]
mod tests {
use super::*;
use tempfile::TempDir;
use std::io::Write;
use tempfile::TempDir;
#[test]
fn test_detect_zip() {
let temp_dir = TempDir::new().unwrap();
let zip_path = temp_dir.path().join("test.zip");
// Create minimal ZIP file header
let mut file = File::create(&zip_path).unwrap();
file.write_all(&[0x50, 0x4B, 0x03, 0x04]).unwrap();
let detector = FormatDetector::new();
let format = detector.detect(&zip_path).unwrap();
assert_eq!(format, ArchiveFormat::Zip);
}
#[test]
fn test_detect_gzip() {
let temp_dir = TempDir::new().unwrap();
let gz_path = temp_dir.path().join("test.gz");
// Create minimal GZIP file header
let mut file = File::create(&gz_path).unwrap();
file.write_all(&[0x1F, 0x8B]).unwrap();
let detector = FormatDetector::new();
let format = detector.detect(&gz_path).unwrap();
assert_eq!(format, ArchiveFormat::Gzip);
}
#[test]
fn test_detect_unknown() {
let temp_dir = TempDir::new().unwrap();
let unknown_path = temp_dir.path().join("test.bin");
// Create unknown file
let mut file = File::create(&unknown_path).unwrap();
file.write_all(b"unknown data").unwrap();
let detector = FormatDetector::new();
let format = detector.detect(&unknown_path).unwrap();
assert_eq!(format, ArchiveFormat::Unknown);
}
}
}
+15 -15
View File
@@ -1,8 +1,8 @@
// Metadata Module - Archive Entry Metadata Management
use serde::{Deserialize, Serialize};
use std::path::PathBuf;
use std::time::SystemTime;
use serde::{Deserialize, Serialize};
use crate::archive::processor::ArchiveFormat;
@@ -29,7 +29,7 @@ impl ArchiveMetadata {
self.total_size as f64 / self.compressed_size as f64
}
}
/// Check if compression ratio exceeds limit (Zip Bomb detection)
pub fn check_zip_bomb(&self, max_ratio: u64) -> bool {
self.actual_ratio() > max_ratio as f64
@@ -65,7 +65,7 @@ impl ArchiveEntry {
checksum: None,
}
}
/// Create file entry
pub fn file(path: PathBuf, size: u64, compressed_size: u64) -> Self {
Self {
@@ -104,7 +104,7 @@ impl ExtractResult {
warnings: Vec::new(),
}
}
pub fn success_rate(&self) -> f64 {
if self.total_files == 0 {
100.0
@@ -113,11 +113,11 @@ impl ExtractResult {
(success_count as f64 / self.total_files as f64) * 100.0
}
}
pub fn has_failures(&self) -> bool {
!self.failed_files.is_empty()
}
pub fn has_warnings(&self) -> bool {
!self.warnings.is_empty()
}
@@ -126,7 +126,7 @@ impl ExtractResult {
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_archive_metadata() {
let metadata = ArchiveMetadata {
@@ -140,37 +140,37 @@ mod tests {
created_time: None,
modified_time: None,
};
assert_eq!(metadata.actual_ratio(), 2.0);
assert!(!metadata.check_zip_bomb(1000));
assert!(metadata.check_zip_bomb(1)); // Should detect as bomb
assert!(metadata.check_zip_bomb(1)); // Should detect as bomb
}
#[test]
fn test_archive_entry() {
let dir_entry = ArchiveEntry::directory(PathBuf::from("test_dir"));
assert!(dir_entry.is_dir);
assert!(!dir_entry.is_file);
let file_entry = ArchiveEntry::file(PathBuf::from("test.txt"), 100, 50);
assert!(!file_entry.is_dir);
assert!(file_entry.is_file);
assert_eq!(file_entry.size, 100);
}
#[test]
fn test_extract_result() {
let result = ExtractResult::new();
assert_eq!(result.success_rate(), 100.0);
let result_with_failure = ExtractResult {
total_files: 10,
success_files: 8,
failed_files: vec![PathBuf::from("failed.txt")],
..Default::default()
};
assert_eq!(result_with_failure.success_rate(), 80.0);
assert!(result_with_failure.has_failures());
}
}
}
+54 -36
View File
@@ -25,9 +25,9 @@ pub use metadata::{ArchiveEntry, ArchiveMetadata, ExtractResult};
pub use processor::{ArchiveFormat, ArchiveProcessor};
use anyhow::Result;
use log::info;
use std::collections::HashMap;
use std::path::Path;
use log::{info, warn};
/// Processor Registry - Plugin Architecture
pub struct ProcessorRegistry {
@@ -43,93 +43,108 @@ impl ProcessorRegistry {
config,
}
}
/// Initialize all processors (based on config)
pub fn initialize(&mut self) -> Result<()> {
// Core formats (always registered)
self.register_core_processors()?;
// Optional formats (based on config)
self.register_optional_processors()?;
Ok(())
}
/// Register core format processors (9 formats)
fn register_core_processors(&mut self) -> Result<()> {
use crate::archive::processors::core::*;
self.processors.insert(ArchiveFormat::Zip, Box::new(ZipProcessor::new()));
self.processors.insert(ArchiveFormat::Tar, Box::new(TarProcessor::new()));
self.processors.insert(ArchiveFormat::Gzip, Box::new(GzipProcessor::new()));
self.processors.insert(ArchiveFormat::Zstd, Box::new(ZstdProcessor::new()));
self.processors.insert(ArchiveFormat::Bzip2, Box::new(Bzip2Processor::new()));
self.processors.insert(ArchiveFormat::Lz4, Box::new(Lz4Processor::new()));
self.processors.insert(ArchiveFormat::TarGzip, Box::new(TarGzipProcessor::new()));
self.processors.insert(ArchiveFormat::TarBzip2, Box::new(TarBzip2Processor::new()));
self.processors.insert(ArchiveFormat::TarZstd, Box::new(TarZstdProcessor::new()));
self.processors
.insert(ArchiveFormat::Zip, Box::new(ZipProcessor::new()));
self.processors
.insert(ArchiveFormat::Tar, Box::new(TarProcessor::new()));
self.processors
.insert(ArchiveFormat::Gzip, Box::new(GzipProcessor::new()));
self.processors
.insert(ArchiveFormat::Zstd, Box::new(ZstdProcessor::new()));
self.processors
.insert(ArchiveFormat::Bzip2, Box::new(Bzip2Processor::new()));
self.processors
.insert(ArchiveFormat::Lz4, Box::new(Lz4Processor::new()));
self.processors
.insert(ArchiveFormat::TarGzip, Box::new(TarGzipProcessor::new()));
self.processors
.insert(ArchiveFormat::TarBzip2, Box::new(TarBzip2Processor::new()));
self.processors
.insert(ArchiveFormat::TarZstd, Box::new(TarZstdProcessor::new()));
info!("✅ Core formats registered: 9 formats");
Ok(())
}
/// Register optional format processors (3 formats, based on config)
fn register_optional_processors(&mut self) -> Result<()> {
#[cfg(feature = "optional-formats")]
{
use crate::archive::processors::optional::*;
// RAR format (legal risk)
if self.config.enable_rar {
crate::archive::warning::show_rar_legal_warning();
self.processors.insert(ArchiveFormat::Rar, Box::new(RarProcessor::new()));
self.processors
.insert(ArchiveFormat::Rar, Box::new(RarProcessor::new()));
warn!("⚠️ RAR format enabled (legal risk)");
}
// XZ format (external dependency)
if self.config.enable_xz {
if check_liblzma_available() {
self.processors.insert(ArchiveFormat::Xz, Box::new(XzProcessor::new()));
self.processors
.insert(ArchiveFormat::Xz, Box::new(XzProcessor::new()));
info!("✅ XZ format enabled");
} else {
crate::archive::warning::show_xz_dependency_warning();
warn!("⚠️ XZ format disabled (liblzma not found)");
}
}
// 7z format (unstable library)
if self.config.enable_7z {
crate::archive::warning::show_7z_stability_warning();
self.processors.insert(ArchiveFormat::SevenZ, Box::new(SevenZProcessor::new()));
self.processors
.insert(ArchiveFormat::SevenZ, Box::new(SevenZProcessor::new()));
warn!("⚠️ 7z format enabled (stability warning)");
}
}
Ok(())
}
/// Get processor for detected format (mutable version for open/extraction)
pub fn get_processor_mut(&mut self, path: &Path) -> Result<&mut (dyn ArchiveProcessor + '_)> {
let detector = FormatDetector::new();
let format = detector.detect(path)?;
match self.processors.get_mut(&format) {
Some(p) => Ok(p.as_mut()),
None => Err(anyhow::anyhow!("Format {} not supported or not enabled", format)),
None => Err(anyhow::anyhow!(
"Format {} not supported or not enabled",
format
)),
}
}
/// Get processor for detected format (immutable version for listing)
pub fn get_processor(&self, path: &Path) -> Result<&dyn ArchiveProcessor> {
let detector = FormatDetector::new();
let format = detector.detect(path)?;
self.processors
.get(&format)
.map(|p| p.as_ref())
.ok_or_else(|| anyhow::anyhow!("Format {} not supported or not enabled", format))
}
/// List all enabled formats
pub fn enabled_formats(&self) -> Vec<ArchiveFormat> {
self.processors.keys().cloned().collect()
@@ -141,7 +156,7 @@ impl ProcessorRegistry {
fn check_liblzma_available() -> bool {
// Try to load xz2 library
// Simplified check: try to create XzProcessor
true // Simplified for now, actual implementation needs better detection
true // Simplified for now, actual implementation needs better detection
}
#[cfg(not(feature = "optional-formats"))]
@@ -156,13 +171,16 @@ pub fn init_archive_system(config_path: Option<&str>) -> Result<ProcessorRegistr
} else {
ArchiveConfig::default()
};
// Show startup warnings for optional formats
crate::archive::warning::show_startup_warnings(&config);
let mut registry = ProcessorRegistry::new(config);
registry.initialize()?;
info!("Archive system initialized with {} formats", registry.enabled_formats().len());
info!(
"Archive system initialized with {} formats",
registry.enabled_formats().len()
);
Ok(registry)
}
}
+54 -39
View File
@@ -4,7 +4,7 @@ use anyhow::Result;
use std::path::{Path, PathBuf};
// Re-export types from metadata.rs
pub use crate::archive::metadata::{ArchiveMetadata, ArchiveEntry, ExtractResult};
pub use crate::archive::metadata::{ArchiveEntry, ArchiveMetadata, ExtractResult};
/// Archive Format Type Enumeration
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, serde::Serialize, serde::Deserialize)]
@@ -19,12 +19,12 @@ pub enum ArchiveFormat {
TarGzip,
TarBzip2,
TarZstd,
// Optional formats (controversial)
Rar, // ⚠️ Legal risk (RARLAB patent)
Xz, // ⚠️ External dependency (liblzma)
SevenZ, // ⚠️ Unstable library (sevenz-rust 0.21.0)
Unknown,
}
@@ -53,30 +53,34 @@ impl std::fmt::Display for ArchiveFormat {
pub trait ArchiveProcessor: Send + Sync {
/// Format type supported by this processor
fn format(&self) -> ArchiveFormat;
/// Open archive file and read metadata
fn open(&mut self, path: &Path) -> Result<ArchiveMetadata>;
/// List all file entries in archive
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>>;
/// Extract single file (on-demand decompression)
fn extract_file(&mut self, entry_path: &Path, output: &mut Vec<u8>) -> Result<u64>;
/// Extract all files to directory (batch extraction)
fn extract_all(&mut self, output_dir: &Path) -> Result<ExtractResult>;
/// Check if this processor can handle the format
fn can_process(format: ArchiveFormat) -> bool where Self: Sized;
fn can_process(format: ArchiveFormat) -> bool
where
Self: Sized;
/// Create new processor instance
fn new() -> Self where Self: Sized;
fn new() -> Self
where
Self: Sized;
}
/// Security Validation - Zip Slip Protection
pub fn validate_extraction_path(entry_path: &Path, base_dir: &Path) -> Result<PathBuf> {
use std::path::Component;
// 1. Check path components
for component in entry_path.components() {
match component {
@@ -92,51 +96,62 @@ pub fn validate_extraction_path(entry_path: &Path, base_dir: &Path) -> Result<Pa
Component::Normal(_) | Component::CurDir => {}
}
}
// 2. Build full path
let full_path = base_dir.join(entry_path);
// 3. Canonicalize and validate (ensure within base_dir)
let canonical_base = base_dir.canonicalize()
let canonical_base = base_dir
.canonicalize()
.map_err(|e| anyhow::anyhow!("Cannot canonicalize base dir: {}", e))?;
// Create parent directories first
if let Some(parent) = full_path.parent() {
std::fs::create_dir_all(parent)?;
}
// 4. Verify extraction path is within base_dir
// Note: full_path may not exist yet, so we check parent directory
if full_path.exists() {
let canonical_full = full_path.canonicalize()
let canonical_full = full_path
.canonicalize()
.map_err(|e| anyhow::anyhow!("Cannot canonicalize full path: {}", e))?;
if !canonical_full.starts_with(&canonical_base) {
return Err(anyhow::anyhow!("Zip Slip detected: path escapes base directory"));
return Err(anyhow::anyhow!(
"Zip Slip detected: path escapes base directory"
));
}
} else {
// Check parent directory instead
if let Some(parent) = full_path.parent() {
let canonical_parent = parent.canonicalize()
let canonical_parent = parent
.canonicalize()
.map_err(|e| anyhow::anyhow!("Cannot canonicalize parent: {}", e))?;
if !canonical_parent.starts_with(&canonical_base) {
return Err(anyhow::anyhow!("Zip Slip detected: path escapes base directory"));
return Err(anyhow::anyhow!(
"Zip Slip detected: path escapes base directory"
));
}
}
}
Ok(full_path)
}
/// Security Validation - Zip Bomb Protection
pub fn check_decompression_ratio(compressed_size: u64, decompressed_size: u64, max_ratio: u64) -> Result<()> {
pub fn check_decompression_ratio(
compressed_size: u64,
decompressed_size: u64,
max_ratio: u64,
) -> Result<()> {
if compressed_size == 0 {
return Ok(()); // Empty file, allow
return Ok(()); // Empty file, allow
}
let ratio = decompressed_size / compressed_size;
if ratio > max_ratio {
return Err(anyhow::anyhow!(
"Zip Bomb detected: compression ratio {} exceeds limit {}",
@@ -144,7 +159,7 @@ pub fn check_decompression_ratio(compressed_size: u64, decompressed_size: u64, m
max_ratio
));
}
Ok(())
}
@@ -157,7 +172,7 @@ pub fn check_file_size_limit(file_size: u64, max_size: u64) -> Result<()> {
max_size / 1024 / 1024
));
}
Ok(())
}
@@ -165,34 +180,34 @@ pub fn check_file_size_limit(file_size: u64, max_size: u64) -> Result<()> {
mod tests {
use super::*;
use tempfile::TempDir;
#[test]
fn test_zip_slip_protection() {
let temp_dir = TempDir::new().unwrap();
let base = temp_dir.path();
// Safe path: should pass
let safe_path = Path::new("safe/file.txt");
assert!(validate_extraction_path(safe_path, base).is_ok());
// Evil path: should be rejected
let evil_path = Path::new("../../etc/passwd");
assert!(validate_extraction_path(evil_path, base).is_err());
// Absolute path: should be rejected
let abs_path = Path::new("/etc/passwd");
assert!(validate_extraction_path(abs_path, base).is_err());
}
#[test]
fn test_zip_bomb_detection() {
// Normal ratio: should pass
assert!(check_decompression_ratio(1000, 5000, 1000).is_ok());
// Zip Bomb ratio: should be rejected
assert!(check_decompression_ratio(42_000, 5_000_000_000, 1000).is_err());
}
#[test]
fn test_compression_ratio_calculation() {
let metadata = ArchiveMetadata {
@@ -206,7 +221,7 @@ mod tests {
created_time: None,
modified_time: None,
};
assert_eq!(metadata.actual_ratio(), 2.0);
}
}
}
+296 -173
View File
@@ -1,16 +1,16 @@
// Core Format Processors - ZIP, TAR, GZIP, TAR.GZ Full Implementation
use crate::archive::{
ArchiveProcessor, ArchiveFormat, ArchiveMetadata, ArchiveEntry, ExtractResult,
processor::{validate_extraction_path, check_decompression_ratio, check_file_size_limit},
};
use crate::archive::config::ArchiveConfig;
use anyhow::{Result, anyhow};
use crate::archive::{
processor::{check_decompression_ratio, check_file_size_limit, validate_extraction_path},
ArchiveEntry, ArchiveFormat, ArchiveMetadata, ArchiveProcessor, ExtractResult,
};
use anyhow::{anyhow, Result};
use log::{debug, info, warn};
use std::fs::{create_dir_all, File};
use std::io::{BufWriter, Read};
use std::path::{Path, PathBuf};
use std::fs::{File, create_dir_all};
use std::io::{Read, Write, BufReader, BufWriter};
use std::time::SystemTime;
use log::{info, warn, debug};
// ==================== ZIP Processor ====================
@@ -21,6 +21,12 @@ pub struct ZipProcessor {
config: ArchiveConfig,
}
impl Default for ZipProcessor {
fn default() -> Self {
Self::new()
}
}
impl ZipProcessor {
pub fn new() -> Self {
Self {
@@ -29,7 +35,7 @@ impl ZipProcessor {
config: ArchiveConfig::default(),
}
}
pub fn with_config(config: ArchiveConfig) -> Self {
Self {
archive: None,
@@ -43,7 +49,7 @@ impl ArchiveProcessor for ZipProcessor {
fn format(&self) -> ArchiveFormat {
ArchiveFormat::Zip
}
fn new() -> Self {
Self {
archive: None,
@@ -51,64 +57,72 @@ impl ArchiveProcessor for ZipProcessor {
config: ArchiveConfig::default(),
}
}
fn open(&mut self, path: &Path) -> Result<ArchiveMetadata> {
info!("Opening ZIP archive: {}", path.display());
let file = File::open(path)?;
let archive = zip::ZipArchive::new(file)?;
self.archive = Some(archive);
self.path = path.to_path_buf();
// Extract metadata (need mutable reference for by_index)
let archive_ref = self.archive.as_mut().unwrap();
let total_files = archive_ref.len() as u64;
let mut total_size = 0u64;
let mut compressed_size = 0u64;
for i in 0..archive_ref.len() {
let file = archive_ref.by_index(i)?;
total_size += file.size();
compressed_size += file.compressed_size();
}
let compression_ratio = if compressed_size > 0 {
total_size as f64 / compressed_size as f64
} else {
0.0
};
// Check for Zip Bomb
if compression_ratio > self.config.max_decompression_ratio as f64 {
warn!("Potential Zip Bomb detected: ratio {:.1}:1", compression_ratio);
return Err(anyhow!("Zip Bomb detected: compression ratio {:.1} exceeds limit {}",
compression_ratio, self.config.max_decompression_ratio));
warn!(
"Potential Zip Bomb detected: ratio {:.1}:1",
compression_ratio
);
return Err(anyhow!(
"Zip Bomb detected: compression ratio {:.1} exceeds limit {}",
compression_ratio,
self.config.max_decompression_ratio
));
}
Ok(ArchiveMetadata {
format: ArchiveFormat::Zip,
total_files,
total_size,
compressed_size,
compression_ratio,
is_encrypted: false, // TODO: Check encryption
is_encrypted: false, // TODO: Check encryption
is_multi_volume: false,
created_time: Some(SystemTime::now()),
modified_time: Some(SystemTime::now()),
})
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
let archive = self.archive.as_mut()
let archive = self
.archive
.as_mut()
.ok_or_else(|| anyhow!("Archive not opened"))?;
let mut entries = Vec::new();
for i in 0..archive.len() {
let file = archive.by_index(i)?;
let entry = ArchiveEntry {
path: PathBuf::from(file.name()),
size: file.size(),
@@ -116,61 +130,64 @@ impl ArchiveProcessor for ZipProcessor {
is_dir: file.name().ends_with('/'),
is_file: !file.name().ends_with('/'),
is_encrypted: false,
modified: SystemTime::UNIX_EPOCH, // TODO: Get actual time
modified: SystemTime::UNIX_EPOCH, // TODO: Get actual time
permissions: Some(0o644),
checksum: None,
};
entries.push(entry);
}
info!("Listed {} entries in ZIP archive", entries.len());
Ok(entries)
}
fn extract_file(&mut self, entry_path: &Path, output: &mut Vec<u8>) -> Result<u64> {
let archive = self.archive.as_mut()
let archive = self
.archive
.as_mut()
.ok_or_else(|| anyhow!("Archive not opened"))?;
let entry_name = entry_path.to_str()
let entry_name = entry_path
.to_str()
.ok_or_else(|| anyhow!("Invalid entry path"))?;
let mut file = archive.by_name(entry_name)?;
// Check file size limit
check_file_size_limit(file.size(), self.config.max_file_size_mb * 1024 * 1024)?;
output.clear();
output.reserve(file.size() as usize);
file.read_to_end(output)?;
info!("Extracted file: {} ({} bytes)", entry_name, output.len());
Ok(output.len() as u64)
}
fn extract_all(&mut self, output_dir: &Path) -> Result<ExtractResult> {
create_dir_all(output_dir)?;
let mut result = ExtractResult::new();
// Open archive if not already open
if self.archive.is_none() {
let file = File::open(&self.path)?;
let archive = zip::ZipArchive::new(file)?;
self.archive = Some(archive);
}
let archive = self.archive.as_mut().unwrap();
result.total_files = archive.len() as u64;
// Use archive iteration to extract files
for i in 0..archive.len() {
let mut file = archive.by_index(i)?;
let entry_name = file.name().to_string();
let file_size = file.size();
let is_dir = entry_name.ends_with('/');
// Zip Slip protection
match validate_extraction_path(&PathBuf::from(&entry_name), output_dir) {
Ok(safe_path) => {
@@ -181,21 +198,24 @@ impl ArchiveProcessor for ZipProcessor {
result.success_files += 1;
} else {
// File
check_file_size_limit(file_size, self.config.max_file_size_mb * 1024 * 1024)?;
check_file_size_limit(
file_size,
self.config.max_file_size_mb * 1024 * 1024,
)?;
if let Some(parent) = safe_path.parent() {
create_dir_all(parent)?;
}
// Extract file content
let mut outfile = BufWriter::new(File::create(&safe_path)?);
std::io::copy(&mut file, &mut outfile)?;
result.success_files += 1;
result.total_bytes += file_size;
debug!("Extracted: {} ({} bytes)", entry_name, file_size);
}
},
}
Err(e) => {
warn!("Zip Slip detected: {} - {}", entry_name, e);
result.failed_files.push(PathBuf::from(&entry_name));
@@ -203,13 +223,17 @@ impl ArchiveProcessor for ZipProcessor {
}
}
}
info!("Extracted {} files ({} bytes) to {}",
result.success_files, result.total_bytes, output_dir.display());
info!(
"Extracted {} files ({} bytes) to {}",
result.success_files,
result.total_bytes,
output_dir.display()
);
Ok(result)
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::Zip
}
@@ -224,6 +248,12 @@ pub struct TarProcessor {
config: ArchiveConfig,
}
impl Default for TarProcessor {
fn default() -> Self {
Self::new()
}
}
impl TarProcessor {
pub fn new() -> Self {
Self {
@@ -232,7 +262,7 @@ impl TarProcessor {
config: ArchiveConfig::default(),
}
}
pub fn with_config(config: ArchiveConfig) -> Self {
Self {
path: PathBuf::new(),
@@ -246,7 +276,7 @@ impl ArchiveProcessor for TarProcessor {
fn format(&self) -> ArchiveFormat {
ArchiveFormat::Tar
}
fn new() -> Self {
Self {
path: PathBuf::new(),
@@ -254,30 +284,30 @@ impl ArchiveProcessor for TarProcessor {
config: ArchiveConfig::default(),
}
}
fn open(&mut self, path: &Path) -> Result<ArchiveMetadata> {
info!("Opening TAR archive: {}", path.display());
self.path = path.to_path_buf();
self.entries.clear();
let file = File::open(path)?;
let mut archive = tar::Archive::new(file);
let mut total_size = 0u64;
// Iterate entries to collect metadata
for entry in archive.entries()? {
let entry = entry?;
let path = entry.path()?.to_path_buf();
let size = entry.size();
total_size += size;
self.entries.push(ArchiveEntry {
path,
size,
compressed_size: size, // TAR has no compression
compressed_size: size, // TAR has no compression
is_dir: entry.header().entry_type().is_dir(),
is_file: entry.header().entry_type().is_file(),
is_encrypted: false,
@@ -286,78 +316,87 @@ impl ArchiveProcessor for TarProcessor {
checksum: None,
});
}
let total_files = self.entries.len() as u64;
Ok(ArchiveMetadata {
format: ArchiveFormat::Tar,
total_files,
total_size,
compressed_size: total_size, // TAR has no compression
compression_ratio: 1.0, // No compression
compressed_size: total_size, // TAR has no compression
compression_ratio: 1.0, // No compression
is_encrypted: false,
is_multi_volume: false,
created_time: Some(SystemTime::now()),
modified_time: Some(SystemTime::now()),
})
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
Ok(self.entries.clone())
}
fn extract_file(&mut self, entry_path: &Path, output: &mut Vec<u8>) -> Result<u64> {
// TAR doesn't support random access, need to unpack entire archive
// This is a limitation - for single file extraction, we unpack everything
warn!("TAR format doesn't support random access - extracting entire archive");
let temp_dir = tempfile::tempdir()?;
self.extract_all(temp_dir.path())?;
let file_path = temp_dir.path().join(entry_path);
let mut file = File::open(&file_path)?;
output.clear();
file.read_to_end(output)?;
Ok(output.len() as u64)
}
fn extract_all(&mut self, output_dir: &Path) -> Result<ExtractResult> {
create_dir_all(output_dir)?;
let file = File::open(&self.path)?;
let mut archive = tar::Archive::new(file);
let mut result = ExtractResult::new();
result.total_files = self.entries.len() as u64;
for entry in archive.entries()? {
let mut entry = entry?;
let entry_path = entry.path()?.to_path_buf();
let entry_path_str = entry_path.display().to_string(); // Save for warning
let entry_path_str = entry_path.display().to_string(); // Save for warning
// Zip Slip protection
match validate_extraction_path(&entry_path, output_dir) {
Ok(safe_path) => {
check_file_size_limit(entry.size(), self.config.max_file_size_mb * 1024 * 1024)?;
check_file_size_limit(
entry.size(),
self.config.max_file_size_mb * 1024 * 1024,
)?;
entry.unpack(&safe_path)?;
result.success_files += 1;
result.total_bytes += entry.size();
},
}
Err(e) => {
warn!("Zip Slip detected: {} - {}", entry_path_str, e);
result.failed_files.push(entry_path);
result.warnings.push(format!("Zip Slip: {}", entry_path_str));
result
.warnings
.push(format!("Zip Slip: {}", entry_path_str));
}
}
}
info!("Extracted {} TAR entries to {}", result.success_files, output_dir.display());
info!(
"Extracted {} TAR entries to {}",
result.success_files,
output_dir.display()
);
Ok(result)
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::Tar
}
@@ -372,6 +411,12 @@ pub struct GzipProcessor {
config: ArchiveConfig,
}
impl Default for GzipProcessor {
fn default() -> Self {
Self::new()
}
}
impl GzipProcessor {
pub fn new() -> Self {
Self {
@@ -380,7 +425,7 @@ impl GzipProcessor {
config: ArchiveConfig::default(),
}
}
pub fn with_config(config: ArchiveConfig) -> Self {
Self {
path: PathBuf::new(),
@@ -394,7 +439,7 @@ impl ArchiveProcessor for GzipProcessor {
fn format(&self) -> ArchiveFormat {
ArchiveFormat::Gzip
}
fn new() -> Self {
Self {
path: PathBuf::new(),
@@ -402,27 +447,31 @@ impl ArchiveProcessor for GzipProcessor {
config: ArchiveConfig::default(),
}
}
fn open(&mut self, path: &Path) -> Result<ArchiveMetadata> {
info!("Opening GZIP archive: {}", path.display());
self.path = path.to_path_buf();
let file = File::open(path)?;
let compressed_size = file.metadata()?.len();
let mut decoder = flate2::read::GzDecoder::new(file);
let mut buffer = Vec::new();
decoder.read_to_end(&mut buffer)?;
self.decompressed_size = buffer.len() as u64;
// Check Zip Bomb
check_decompression_ratio(compressed_size, self.decompressed_size, self.config.max_decompression_ratio)?;
check_decompression_ratio(
compressed_size,
self.decompressed_size,
self.config.max_decompression_ratio,
)?;
Ok(ArchiveMetadata {
format: ArchiveFormat::Gzip,
total_files: 1, // GZIP is single file
total_files: 1, // GZIP is single file
total_size: self.decompressed_size,
compressed_size,
compression_ratio: if compressed_size > 0 {
@@ -436,58 +485,64 @@ impl ArchiveProcessor for GzipProcessor {
modified_time: Some(SystemTime::now()),
})
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
// GZIP is single file - infer name from archive name
let name = self.path.file_name()
let name = self
.path
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("unknown")
.replace(".gz", "")
.replace(".gzip", "");
Ok(vec![ArchiveEntry::file(
PathBuf::from(name),
self.decompressed_size,
0, // GZIP doesn't preserve compressed size per file
0, // GZIP doesn't preserve compressed size per file
)])
}
fn extract_file(&mut self, entry_path: &Path, output: &mut Vec<u8>) -> Result<u64> {
fn extract_file(&mut self, _entry_path: &Path, output: &mut Vec<u8>) -> Result<u64> {
// GZIP is single file - just decompress it
let file = File::open(&self.path)?;
let mut decoder = flate2::read::GzDecoder::new(file);
output.clear();
decoder.read_to_end(output)?;
check_file_size_limit(output.len() as u64, self.config.max_file_size_mb * 1024 * 1024)?;
check_file_size_limit(
output.len() as u64,
self.config.max_file_size_mb * 1024 * 1024,
)?;
info!("Decompressed GZIP file: {} bytes", output.len());
Ok(output.len() as u64)
}
fn extract_all(&mut self, output_dir: &Path) -> Result<ExtractResult> {
create_dir_all(output_dir)?;
let entries = self.list_entries()?;
let entry = entries.first()
let entry = entries
.first()
.ok_or_else(|| anyhow!("No entry in GZIP archive"))?;
let outpath = output_dir.join(&entry.path);
// Zip Slip protection
validate_extraction_path(&entry.path, output_dir)?;
if let Some(parent) = outpath.parent() {
create_dir_all(parent)?;
}
let file = File::open(&self.path)?;
let mut decoder = flate2::read::GzDecoder::new(file);
let mut outfile = BufWriter::new(File::create(&outpath)?);
std::io::copy(&mut decoder, &mut outfile)?;
let result = ExtractResult {
total_files: 1,
total_bytes: self.decompressed_size,
@@ -496,11 +551,11 @@ impl ArchiveProcessor for GzipProcessor {
skipped_files: Vec::new(),
warnings: Vec::new(),
};
info!("Decompressed GZIP to: {}", outpath.display());
Ok(result)
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::Gzip
}
@@ -514,6 +569,12 @@ pub struct TarGzipProcessor {
config: ArchiveConfig,
}
impl Default for TarGzipProcessor {
fn default() -> Self {
Self::new()
}
}
impl TarGzipProcessor {
pub fn new() -> Self {
Self {
@@ -521,7 +582,7 @@ impl TarGzipProcessor {
config: ArchiveConfig::default(),
}
}
pub fn with_config(config: ArchiveConfig) -> Self {
Self {
gzip_processor: GzipProcessor::with_config(config.clone()),
@@ -534,32 +595,33 @@ impl ArchiveProcessor for TarGzipProcessor {
fn format(&self) -> ArchiveFormat {
ArchiveFormat::TarGzip
}
fn new() -> Self {
Self {
gzip_processor: GzipProcessor::new(),
config: ArchiveConfig::default(),
}
}
fn open(&mut self, path: &Path) -> Result<ArchiveMetadata> {
info!("Opening TAR.GZ archive: {}", path.display());
// Step 1: Decompress GZIP
let temp_dir = tempfile::tempdir()?;
self.gzip_processor.open(path)?;
self.gzip_processor.extract_all(temp_dir.path())?;
// Step 2: Open TAR
let tar_entries = self.gzip_processor.list_entries()?;
let tar_file = tar_entries.first()
let tar_file = tar_entries
.first()
.ok_or_else(|| anyhow!("No TAR file in GZIP"))?;
let tar_path = temp_dir.path().join(&tar_file.path);
let mut tar_processor = TarProcessor::with_config(self.config.clone());
let tar_metadata = tar_processor.open(&tar_path)?;
Ok(ArchiveMetadata {
format: ArchiveFormat::TarGzip,
total_files: tar_metadata.total_files,
@@ -576,46 +638,47 @@ impl ArchiveProcessor for TarGzipProcessor {
modified_time: Some(SystemTime::now()),
})
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
// Need to implement properly - this requires decompressing first
warn!("TAR.GZ list_entries requires full decompression - consider extract_all instead");
Ok(Vec::new())
}
fn extract_file(&mut self, entry_path: &Path, output: &mut Vec<u8>) -> Result<u64> {
warn!("TAR.GZ extract_file requires full unpacking - inefficient for single file");
let temp_dir = tempfile::tempdir()?;
self.extract_all(temp_dir.path())?;
let file_path = temp_dir.path().join(entry_path);
let mut file = File::open(&file_path)?;
output.clear();
file.read_to_end(output)?;
Ok(output.len() as u64)
}
fn extract_all(&mut self, output_dir: &Path) -> Result<ExtractResult> {
info!("Extracting TAR.GZ to: {}", output_dir.display());
// Step 1: Decompress GZIP to temp
let temp_dir = tempfile::tempdir()?;
self.gzip_processor.extract_all(temp_dir.path())?;
// Step 2: Extract TAR
let tar_entries = self.gzip_processor.list_entries()?;
let tar_file = tar_entries.first()
let tar_file = tar_entries
.first()
.ok_or_else(|| anyhow!("No TAR file found"))?;
let tar_path = temp_dir.path().join(&tar_file.path);
let mut tar_processor = TarProcessor::with_config(self.config.clone());
tar_processor.open(&tar_path)?;
tar_processor.extract_all(output_dir)
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::TarGzip
}
@@ -627,73 +690,133 @@ impl ArchiveProcessor for TarGzipProcessor {
pub struct ZstdProcessor;
impl ArchiveProcessor for ZstdProcessor {
fn format(&self) -> ArchiveFormat { ArchiveFormat::Zstd }
fn format(&self) -> ArchiveFormat {
ArchiveFormat::Zstd
}
fn open(&mut self, _path: &Path) -> Result<ArchiveMetadata> {
Err(anyhow!("ZSTD processor not yet implemented"))
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> { Ok(Vec::new()) }
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> { Ok(0) }
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> { Ok(ExtractResult::new()) }
fn can_process(format: ArchiveFormat) -> bool { format == ArchiveFormat::Zstd }
fn new() -> Self { Self }
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
Ok(Vec::new())
}
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> {
Ok(0)
}
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> {
Ok(ExtractResult::new())
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::Zstd
}
fn new() -> Self {
Self
}
}
/// BZIP2 Processor Stub (Phase 2/3)
pub struct Bzip2Processor;
impl ArchiveProcessor for Bzip2Processor {
fn format(&self) -> ArchiveFormat { ArchiveFormat::Bzip2 }
fn format(&self) -> ArchiveFormat {
ArchiveFormat::Bzip2
}
fn open(&mut self, _path: &Path) -> Result<ArchiveMetadata> {
Err(anyhow!("BZIP2 processor not yet implemented"))
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> { Ok(Vec::new()) }
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> { Ok(0) }
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> { Ok(ExtractResult::new()) }
fn can_process(format: ArchiveFormat) -> bool { format == ArchiveFormat::Bzip2 }
fn new() -> Self { Self }
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
Ok(Vec::new())
}
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> {
Ok(0)
}
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> {
Ok(ExtractResult::new())
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::Bzip2
}
fn new() -> Self {
Self
}
}
/// LZ4 Processor Stub (Phase 2/3)
pub struct Lz4Processor;
impl ArchiveProcessor for Lz4Processor {
fn format(&self) -> ArchiveFormat { ArchiveFormat::Lz4 }
fn format(&self) -> ArchiveFormat {
ArchiveFormat::Lz4
}
fn open(&mut self, _path: &Path) -> Result<ArchiveMetadata> {
Err(anyhow!("LZ4 processor not yet implemented"))
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> { Ok(Vec::new()) }
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> { Ok(0) }
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> { Ok(ExtractResult::new()) }
fn can_process(format: ArchiveFormat) -> bool { format == ArchiveFormat::Lz4 }
fn new() -> Self { Self }
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
Ok(Vec::new())
}
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> {
Ok(0)
}
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> {
Ok(ExtractResult::new())
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::Lz4
}
fn new() -> Self {
Self
}
}
/// TAR.BZ2 Composite Processor Stub (Phase 2/3)
pub struct TarBzip2Processor;
impl ArchiveProcessor for TarBzip2Processor {
fn format(&self) -> ArchiveFormat { ArchiveFormat::TarBzip2 }
fn format(&self) -> ArchiveFormat {
ArchiveFormat::TarBzip2
}
fn open(&mut self, _path: &Path) -> Result<ArchiveMetadata> {
Err(anyhow!("TAR.BZ2 processor not yet implemented"))
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> { Ok(Vec::new()) }
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> { Ok(0) }
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> { Ok(ExtractResult::new()) }
fn can_process(format: ArchiveFormat) -> bool { format == ArchiveFormat::TarBzip2 }
fn new() -> Self { Self }
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
Ok(Vec::new())
}
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> {
Ok(0)
}
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> {
Ok(ExtractResult::new())
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::TarBzip2
}
fn new() -> Self {
Self
}
}
/// TAR.ZST Composite Processor Stub (Phase 2/3)
pub struct TarZstdProcessor;
impl ArchiveProcessor for TarZstdProcessor {
fn format(&self) -> ArchiveFormat { ArchiveFormat::TarZstd }
fn format(&self) -> ArchiveFormat {
ArchiveFormat::TarZstd
}
fn open(&mut self, _path: &Path) -> Result<ArchiveMetadata> {
Err(anyhow!("TAR.ZST processor not yet implemented"))
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> { Ok(Vec::new()) }
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> { Ok(0) }
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> { Ok(ExtractResult::new()) }
fn can_process(format: ArchiveFormat) -> bool { format == ArchiveFormat::TarZstd }
fn new() -> Self { Self }
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
Ok(Vec::new())
}
fn extract_file(&mut self, _entry: &Path, _output: &mut Vec<u8>) -> Result<u64> {
Ok(0)
}
fn extract_all(&mut self, _dir: &Path) -> Result<ExtractResult> {
Ok(ExtractResult::new())
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::TarZstd
}
fn new() -> Self {
Self
}
}
@@ -1,13 +1,15 @@
// Optional Format Processors - RAR, XZ, 7z
// All optional formats have warnings displayed when enabled
use crate::archive::{ArchiveFormat, ArchiveProcessor, ArchiveMetadata, ArchiveEntry, ExtractResult};
use crate::archive::processor::{check_decompression_ratio, validate_extraction_path};
use crate::archive::warning;
use crate::archive::processor::{validate_extraction_path, check_decompression_ratio};
use anyhow::{Result, anyhow};
use std::path::Path;
use crate::archive::{
ArchiveEntry, ArchiveFormat, ArchiveMetadata, ArchiveProcessor, ExtractResult,
};
use anyhow::{anyhow, Result};
use log::{info, warn};
use std::fs;
use log::{warn, info};
use std::path::Path;
/// RAR Processor - Only Decompression
/// ⚠️ Legal Warning: RARLAB patent, commercial use requires license
@@ -28,54 +30,65 @@ impl ArchiveProcessor for RarProcessor {
fn format(&self) -> ArchiveFormat {
ArchiveFormat::Rar
}
fn open(&mut self, path: &Path) -> Result<ArchiveMetadata> {
// Show legal warning when RAR is used
warning::show_rar_legal_warning();
self.archive_path = Some(path.to_path_buf());
// Use unrar library to open RAR
// Note: unrar only supports decompression, no compression
use unrar::Archive;
let archive = Archive::new(path)?;
let entries: Vec<_> = archive.list()?.collect();
let total_files = entries.len() as u64;
let total_size = entries.iter()
let total_size = entries
.iter()
.filter_map(|e| e.ok())
.map(|e| e.uncompressed_size)
.sum();
let compressed_size = fs::metadata(path)?.len();
Ok(ArchiveMetadata {
format: ArchiveFormat::Rar,
total_files,
total_size,
compressed_size,
compression_ratio: if compressed_size > 0 { total_size as f64 / compressed_size as f64 } else { 0.0 },
is_encrypted: entries.iter().any(|e| e.ok().map_or(false, |e| e.is_encrypted())),
is_multi_volume: false, // unrar library limitation
compression_ratio: if compressed_size > 0 {
total_size as f64 / compressed_size as f64
} else {
0.0
},
is_encrypted: entries
.iter()
.any(|e| e.ok().map_or(false, |e| e.is_encrypted())),
is_multi_volume: false, // unrar library limitation
created_time: None,
modified_time: None,
})
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
use unrar::Archive;
let path = self.archive_path.as_ref().ok_or_else(|| anyhow!("Archive not opened"))?;
let path = self
.archive_path
.as_ref()
.ok_or_else(|| anyhow!("Archive not opened"))?;
let archive = Archive::new(path)?;
let entries: Vec<ArchiveEntry> = archive.list()?
let entries: Vec<ArchiveEntry> = archive
.list()?
.filter_map(|e| e.ok())
.map(|e| ArchiveEntry {
path: PathBuf::from(e.filename),
size: e.uncompressed_size,
compressed_size: 0, // unrar doesn't provide this
compressed_size: 0, // unrar doesn't provide this
is_dir: e.is_directory(),
is_file: !e.is_directory(),
is_encrypted: e.is_encrypted(),
@@ -83,45 +96,49 @@ impl ArchiveProcessor for RarProcessor {
permissions: None,
})
.collect();
Ok(entries)
}
fn extract_file(&self, entry_path: &Path, output: &mut Vec<u8>) -> Result<u64> {
// RAR doesn't support random access efficiently
// Need to extract entire archive
warn!("RAR extract_file requires full extraction (no random access)");
let entries = self.list_entries()?;
let entry = entries.iter()
let entry = entries
.iter()
.find(|e| e.path == entry_path)
.ok_or_else(|| anyhow!("Entry not found: {}", entry_path.display()))?;
// Extract to temp dir, then read
let temp_dir = tempfile::tempdir()?;
self.extract_all(temp_dir.path())?;
let extracted_file = temp_dir.path().join(entry_path);
let content = fs::read(&extracted_file)?;
output.extend_from_slice(&content);
Ok(content.len() as u64)
}
fn extract_all(&self, output_dir: &Path) -> Result<ExtractResult> {
use unrar::Archive;
use unrar::ExtractOption;
let path = self.archive_path.as_ref().ok_or_else(|| anyhow!("Archive not opened"))?;
let path = self
.archive_path
.as_ref()
.ok_or_else(|| anyhow!("Archive not opened"))?;
// Validate output_dir path
validate_extraction_path(output_dir, output_dir)?;
let mut result = ExtractResult::new();
result.total_files = self.list_entries()?.len() as u64;
let archive = Archive::new(path)?;
for entry_result in archive.extract_all(output_dir, ExtractOption::Recurse)? {
match entry_result {
Ok(entry) => {
@@ -135,10 +152,10 @@ impl ArchiveProcessor for RarProcessor {
}
}
}
Ok(result)
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::Rar
}
@@ -163,57 +180,65 @@ impl ArchiveProcessor for XzProcessor {
fn format(&self) -> ArchiveFormat {
ArchiveFormat::Xz
}
fn open(&mut self, path: &Path) -> Result<ArchiveMetadata> {
// Check if liblzma is available
if !check_liblzma_available() {
warning::show_xz_dependency_warning();
return Err(anyhow!("liblzma library not found, XZ format disabled"));
}
self.archive_path = Some(path.to_path_buf());
use xz2::read::XzDecoder;
use std::io::Read;
use xz2::read::XzDecoder;
let file = fs::File::open(path)?;
let mut decoder = XzDecoder::new(file);
// Read decompressed size (estimate)
let mut buffer = Vec::new();
decoder.read_to_end(&mut buffer)?;
let decompressed_size = buffer.len() as u64;
let compressed_size = fs::metadata(path)?.len();
// Check decompression ratio
check_decompression_ratio(compressed_size, decompressed_size, 1000)?;
Ok(ArchiveMetadata {
format: ArchiveFormat::Xz,
total_files: 1, // XZ is single-file format
total_files: 1, // XZ is single-file format
total_size: decompressed_size,
compressed_size,
compression_ratio: if compressed_size > 0 { decompressed_size as f64 / compressed_size as f64 } else { 0.0 },
compression_ratio: if compressed_size > 0 {
decompressed_size as f64 / compressed_size as f64
} else {
0.0
},
is_encrypted: false,
is_multi_volume: false,
created_time: None,
modified_time: None,
})
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
// XZ is single-file, infer filename from archive name
let path = self.archive_path.as_ref().ok_or_else(|| anyhow!("Archive not opened"))?;
let filename = path.file_name()
let path = self
.archive_path
.as_ref()
.ok_or_else(|| anyhow!("Archive not opened"))?;
let filename = path
.file_name()
.and_then(|n| n.to_str())
.map(|s| s.strip_suffix(".xz").unwrap_or(s))
.unwrap_or("output");
Ok(vec![ArchiveEntry {
path: PathBuf::from(filename),
size: 0, // Will be determined during extraction
size: 0, // Will be determined during extraction
compressed_size: 0,
is_dir: false,
is_file: true,
@@ -222,48 +247,54 @@ impl ArchiveProcessor for XzProcessor {
permissions: None,
}])
}
fn extract_file(&self, _entry_path: &Path, output: &mut Vec<u8>) -> Result<u64> {
use xz2::read::XzDecoder;
use std::io::Read;
let path = self.archive_path.as_ref().ok_or_else(|| anyhow!("Archive not opened"))?;
use xz2::read::XzDecoder;
let path = self
.archive_path
.as_ref()
.ok_or_else(|| anyhow!("Archive not opened"))?;
let file = fs::File::open(path)?;
let mut decoder = XzDecoder::new(file);
decoder.read_to_end(output)?;
Ok(output.len() as u64)
}
fn extract_all(&self, output_dir: &Path) -> Result<ExtractResult> {
use xz2::read::XzDecoder;
use std::io::Read;
let path = self.archive_path.as_ref().ok_or_else(|| anyhow!("Archive not opened"))?;
use xz2::read::XzDecoder;
let path = self
.archive_path
.as_ref()
.ok_or_else(|| anyhow!("Archive not opened"))?;
// Infer output filename
let entries = self.list_entries()?;
let output_path = output_dir.join(&entries[0].path);
// Validate path
validate_extraction_path(&entries[0].path, output_dir)?;
let file = fs::File::open(path)?;
let mut decoder = XzDecoder::new(file);
let mut output_file = fs::File::create(&output_path)?;
std::io::copy(&mut decoder, &mut output_file)?;
let mut result = ExtractResult::new();
result.success_files = 1;
result.total_files = 1;
result.total_bytes = fs::metadata(&output_path)?.len();
Ok(result)
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::Xz && check_liblzma_available()
}
@@ -286,59 +317,61 @@ impl ArchiveProcessor for SevenZProcessor {
fn format(&self) -> ArchiveFormat {
ArchiveFormat::SevenZ
}
fn open(&mut self, path: &Path) -> Result<ArchiveMetadata> {
// Show stability warning
warning::show_7z_stability_warning();
use sevenz_rust::SevenZReader;
let reader = SevenZReader::new(path)?;
let entries = reader.entries()?;
let total_files = entries.len() as u64;
let total_size = entries.iter()
.map(|e| e.uncompressed_size as u64)
.sum();
let total_size = entries.iter().map(|e| e.uncompressed_size as u64).sum();
let compressed_size = fs::metadata(path)?.len();
Ok(ArchiveMetadata {
format: ArchiveFormat::SevenZ,
total_files,
total_size,
compressed_size,
compression_ratio: if compressed_size > 0 { total_size as f64 / compressed_size as f64 } else { 0.0 },
compression_ratio: if compressed_size > 0 {
total_size as f64 / compressed_size as f64
} else {
0.0
},
is_encrypted: entries.iter().any(|e| e.is_encrypted),
is_multi_volume: false,
created_time: None,
modified_time: None,
})
}
fn list_entries(&mut self) -> Result<Vec<ArchiveEntry>> {
// Note: sevenz-rust doesn't have full entry listing yet
// This is a stub returning empty list
warn!("7z list_entries not fully implemented (library limitation)");
Ok(Vec::new())
}
fn extract_file(&self, _entry_path: &Path, _output: &mut Vec<u8>) -> Result<u64> {
warn!("7z extract_file not implemented (library limitation)");
Err(anyhow!("7z library doesn't support random access"))
}
fn extract_all(&self, output_dir: &Path) -> Result<ExtractResult> {
use sevenz_rust::SevenZReader;
// Note: sevenz-rust doesn't have full extraction yet
// This is a stub
warn!("7z extract_all limited (library under development)");
Ok(ExtractResult::new())
}
fn can_process(format: ArchiveFormat) -> bool {
format == ArchiveFormat::SevenZ
}
@@ -369,15 +402,21 @@ pub struct SevenZProcessor;
#[cfg(not(feature = "optional-formats"))]
impl RarProcessor {
pub fn new() -> Self { Self }
pub fn new() -> Self {
Self
}
}
#[cfg(not(feature = "optional-formats"))]
impl XzProcessor {
pub fn new() -> Self { Self }
pub fn new() -> Self {
Self
}
}
#[cfg(not(feature = "optional-formats"))]
impl SevenZProcessor {
pub fn new() -> Self { Self }
}
pub fn new() -> Self {
Self
}
}
@@ -1,31 +1,31 @@
use crate::archive::{
ArchiveProcessor, ArchiveFormat, ArchiveMetadata, ArchiveEntry, ExtractResult,
processors::core::{ZipProcessor, TarProcessor, GzipProcessor, TarGzipProcessor},
processor::{validate_extraction_path, check_decompression_ratio},
config::ArchiveConfig,
processor::{check_decompression_ratio, validate_extraction_path},
processors::core::{GzipProcessor, TarGzipProcessor, TarProcessor, ZipProcessor},
ArchiveEntry, ArchiveFormat, ArchiveMetadata, ArchiveProcessor, ExtractResult,
};
use tempfile::TempDir;
use std::fs::{File, create_dir_all};
use anyhow::Result;
use std::fs::{create_dir_all, File};
use std::io::Write;
use std::path::PathBuf;
use anyhow::Result;
use tempfile::TempDir;
#[cfg(test)]
mod helpers {
use std::fs::File;
use std::io::Write;
use std::path::PathBuf;
pub fn create_test_zip(path: &PathBuf, files: Vec<(&str, &[u8])>) {
use std::io::Cursor;
let mut buffer = Cursor::new(Vec::new());
{
let mut zip = zip::ZipWriter::new(&mut buffer);
let options = zip::write::FileOptions::default()
.compression_method(zip::CompressionMethod::Stored);
for (name, content) in files {
if name.ends_with('/') {
zip.add_directory(name, options).unwrap();
@@ -34,31 +34,31 @@ mod helpers {
zip.write_all(content).unwrap();
}
}
zip.finish().unwrap();
}
let zip_data = buffer.into_inner();
File::create(path).unwrap().write_all(&zip_data).unwrap();
}
pub fn create_test_tar(path: &PathBuf, files: Vec<(&str, &[u8])>) {
let file = File::create(path).unwrap();
let mut builder = tar::Builder::new(file);
for (name, content) in files {
let mut header = tar::Header::new_gnu();
header.set_size(content.len() as u64);
header.set_path(name);
header.set_mode(0o644);
header.set_cksum();
builder.append_data(&mut header, name, content).unwrap();
}
builder.finish().unwrap();
}
pub fn create_test_gzip(path: &PathBuf, content: &[u8]) {
let file = File::create(path).unwrap();
let mut encoder = flate2::write::GzEncoder::new(file, flate2::Compression::default());
@@ -69,74 +69,74 @@ mod helpers {
#[cfg(test)]
mod core_format_tests {
use super::*;
use super::helpers::*;
use super::*;
#[test]
fn test_zip_processor_basic() {
let temp_dir = TempDir::new().unwrap();
let zip_path = temp_dir.path().join("test.zip");
create_test_zip(&zip_path, vec![("file1.txt", b"hello")]);
let mut processor = ZipProcessor::new();
let metadata = processor.open(&zip_path).unwrap();
assert_eq!(metadata.format, ArchiveFormat::Zip);
assert_eq!(metadata.total_files, 1);
}
#[test]
fn test_tar_processor_basic() {
let temp_dir = TempDir::new().unwrap();
let tar_path = temp_dir.path().join("test.tar");
create_test_tar(&tar_path, vec![("file1.txt", b"tar content")]);
let mut processor = TarProcessor::new();
let metadata = processor.open(&tar_path).unwrap();
assert_eq!(metadata.format, ArchiveFormat::Tar);
}
#[test]
fn test_gzip_processor_basic() {
let temp_dir = TempDir::new().unwrap();
let gz_path = temp_dir.path().join("test.gz");
create_test_gzip(&gz_path, b"gzip content here");
let mut processor = GzipProcessor::new();
let metadata = processor.open(&gz_path).unwrap();
assert_eq!(metadata.format, ArchiveFormat::Gzip);
assert_eq!(metadata.total_files, 1);
}
#[test]
fn test_validate_extraction_path_safe() {
let temp_dir = TempDir::new().unwrap();
let base = temp_dir.path();
let safe_path = PathBuf::from("safe/file.txt");
let result = validate_extraction_path(&safe_path, base);
assert!(result.is_ok());
let resolved = result.unwrap();
assert!(resolved.starts_with(base));
}
#[test]
fn test_validate_extraction_path_zip_slip() {
let base = PathBuf::from("/tmp/extract");
let evil_path = PathBuf::from("../../etc/passwd");
let result = validate_extraction_path(&evil_path, &base);
assert!(result.is_err());
}
#[test]
fn test_check_decompression_ratio_ok() {
assert!(check_decompression_ratio(1000, 5000, 1000).is_ok());
}
#[test]
fn test_check_decompression_ratio_zip_bomb() {
assert!(check_decompression_ratio(42_000, 5_000_000_000, 1000).is_err());
@@ -145,39 +145,39 @@ mod core_format_tests {
#[cfg(test)]
mod integration_tests {
use super::*;
use super::helpers::*;
use super::*;
use crate::archive::detector::FormatDetector;
use crate::archive::ProcessorRegistry;
#[test]
fn test_format_detection_automation() {
let temp_dir = TempDir::new().unwrap();
let detector = FormatDetector::new();
let zip_path = temp_dir.path().join("test.zip");
create_test_zip(&zip_path, vec![("f.txt", b"z")]);
assert_eq!(detector.detect(&zip_path).unwrap(), ArchiveFormat::Zip);
let tar_path = temp_dir.path().join("test.tar");
create_test_tar(&tar_path, vec![("f.txt", b"t")]);
assert_eq!(detector.detect(&tar_path).unwrap(), ArchiveFormat::Tar);
let gz_path = temp_dir.path().join("test.gz");
create_test_gzip(&gz_path, b"g");
assert_eq!(detector.detect(&gz_path).unwrap(), ArchiveFormat::Gzip);
}
#[test]
fn test_processor_registry_integration() {
let config = ArchiveConfig::default();
let mut registry = ProcessorRegistry::new(config);
registry.initialize().unwrap();
let formats = registry.enabled_formats();
assert!(formats.contains(&ArchiveFormat::Zip));
assert!(formats.contains(&ArchiveFormat::Tar));
assert!(formats.contains(&ArchiveFormat::Gzip));
assert!(formats.contains(&ArchiveFormat::TarGzip));
}
}
}
@@ -4,48 +4,46 @@ use std::fs;
use std::io::Read;
use tempfile::TempDir;
use crate::archive::*;
use crate::archive::processor::check_decompression_ratio;
use crate::archive::tests::test_helpers::*;
use crate::archive::*;
#[test]
fn test_zip_processor_full_workflow() {
let temp_dir = TempDir::new().unwrap();
let zip_path = create_test_zip(&temp_dir);
// Initialize processor
let mut processor = processors::core::ZipProcessor::new();
// Test open
let metadata = processor.open(&zip_path).unwrap();
assert_eq!(metadata.format, ArchiveFormat::Zip);
assert_eq!(metadata.total_files, 3);
// Test list_entries
let entries = processor.list_entries().unwrap();
assert_eq!(entries.len(), 3);
// Verify entry names
let names: Vec<&str> = entries.iter()
.map(|e| e.path.to_str().unwrap())
.collect();
let names: Vec<&str> = entries.iter().map(|e| e.path.to_str().unwrap()).collect();
assert!(names.contains(&"file1.txt"));
assert!(names.contains(&"file2.txt"));
assert!(names.contains(&"subdir/file3.txt"));
// Test extract_all
let extract_dir = temp_dir.path().join("extracted");
fs::create_dir_all(&extract_dir).unwrap();
let result = processor.extract_all(&extract_dir).unwrap();
assert_eq!(result.success_files, 3);
assert_eq!(result.failed_files.len(), 0);
// Verify extracted files
assert!(extract_dir.join("file1.txt").exists());
assert!(extract_dir.join("file2.txt").exists());
assert!(extract_dir.join("subdir/file3.txt").exists());
// Verify content
let content1 = fs::read_to_string(extract_dir.join("file1.txt")).unwrap();
assert_eq!(content1, "content of file 1");
@@ -55,24 +53,24 @@ fn test_zip_processor_full_workflow() {
fn test_tar_processor_full_workflow() {
let temp_dir = TempDir::new().unwrap();
let tar_path = create_test_tar(&temp_dir);
let mut processor = processors::core::TarProcessor::new();
// Test open
let metadata = processor.open(&tar_path).unwrap();
assert_eq!(metadata.format, ArchiveFormat::Tar);
// Test list_entries
let entries = processor.list_entries().unwrap();
assert!(entries.len() >= 3); // TAR may include directory entries
assert!(entries.len() >= 3); // TAR may include directory entries
// Test extract_all
let extract_dir = temp_dir.path().join("extracted_tar");
fs::create_dir_all(&extract_dir).unwrap();
let result = processor.extract_all(&extract_dir).unwrap();
assert!(result.success_files >= 3);
// Verify extracted files exist
assert!(extract_dir.join("file1.txt").exists());
assert!(extract_dir.join("file2.txt").exists());
@@ -82,25 +80,25 @@ fn test_tar_processor_full_workflow() {
fn test_gzip_processor_full_workflow() {
let temp_dir = TempDir::new().unwrap();
let gz_path = create_test_gzip(&temp_dir);
let mut processor = processors::core::GzipProcessor::new();
// Test open
let metadata = processor.open(&gz_path).unwrap();
assert_eq!(metadata.format, ArchiveFormat::Gzip);
assert_eq!(metadata.total_files, 1); // GZIP is single file
assert_eq!(metadata.total_files, 1); // GZIP is single file
// Test extract_all
let extract_dir = temp_dir.path().join("extracted_gz");
fs::create_dir_all(&extract_dir).unwrap();
let result = processor.extract_all(&extract_dir).unwrap();
assert_eq!(result.success_files, 1);
// Verify extracted file (should strip .gz extension)
let extracted_file = extract_dir.join("test.txt");
assert!(extracted_file.exists());
// Verify content
let content = fs::read_to_string(&extracted_file).unwrap();
assert_eq!(content, "test gzip content for validation");
@@ -110,20 +108,20 @@ fn test_gzip_processor_full_workflow() {
fn test_tar_gz_processor_workflow() {
let temp_dir = TempDir::new().unwrap();
let tar_gz_path = create_test_tar_gz(&temp_dir);
let mut processor = processors::core::TarGzipProcessor::new();
// Test open
let metadata = processor.open(&tar_gz_path).unwrap();
assert_eq!(metadata.format, ArchiveFormat::TarGzip);
// Test extract_all
let extract_dir = temp_dir.path().join("extracted_tar_gz");
fs::create_dir_all(&extract_dir).unwrap();
let result = processor.extract_all(&extract_dir).unwrap();
assert!(result.success_files >= 2);
// Verify extracted TAR files
assert!(extract_dir.join("file1.txt").exists());
assert!(extract_dir.join("file2.txt").exists());
@@ -132,18 +130,18 @@ fn test_tar_gz_processor_workflow() {
#[test]
fn test_format_detection_auto() {
let temp_dir = TempDir::new().unwrap();
// Test ZIP detection
let zip_path = create_test_zip(&temp_dir);
let detector = FormatDetector::new();
let format = detector.detect(&zip_path).unwrap();
assert_eq!(format, ArchiveFormat::Zip);
// Test TAR detection
let tar_path = create_test_tar(&temp_dir);
let format = detector.detect(&tar_path).unwrap();
assert_eq!(format, ArchiveFormat::Tar);
// Test GZIP detection
let gz_path = create_test_gzip(&temp_dir);
let format = detector.detect(&gz_path).unwrap();
@@ -155,12 +153,12 @@ fn test_processor_registry_core_formats() {
let config = ArchiveConfig::default();
let mut registry = ProcessorRegistry::new(config);
registry.initialize().unwrap();
let formats = registry.enabled_formats();
// Should have 9 core formats
assert!(formats.len() >= 4); // At least the ones we implemented
assert!(formats.len() >= 4); // At least the ones we implemented
// Verify format support
assert!(formats.contains(&ArchiveFormat::Zip));
assert!(formats.contains(&ArchiveFormat::Tar));
@@ -172,20 +170,20 @@ fn test_processor_registry_core_formats() {
fn test_zip_slip_protection() {
let temp_dir = TempDir::new().unwrap();
let zip_bomb_data = create_zip_slip_test();
// Write malicious ZIP to file
let evil_zip_path = temp_dir.path().join("evil.zip");
fs::write(&evil_zip_path, &zip_bomb_data).unwrap();
let mut processor = processors::core::ZipProcessor::new();
processor.open(&evil_zip_path).unwrap();
// Attempt extraction should fail due to Zip Slip protection
let extract_dir = temp_dir.path().join("should_fail");
fs::create_dir_all(&extract_dir).unwrap();
let result = processor.extract_all(&extract_dir);
// Should either fail or have empty extracted files
// (validate_extraction_path prevents malicious paths)
if result.is_ok() {
@@ -199,11 +197,11 @@ fn test_zip_slip_protection() {
fn test_zip_bomb_detection() {
// Test decompression ratio check
let result = check_decompression_ratio(42_000, 5_000_000_000, 1000);
assert!(result.is_err()); // Should detect as Zip Bomb
assert!(result.is_err()); // Should detect as Zip Bomb
// Test normal ratio
let result = check_decompression_ratio(1000, 5000, 1000);
assert!(result.is_ok()); // Normal ratio should pass
assert!(result.is_ok()); // Normal ratio should pass
}
#[test]
@@ -219,21 +217,21 @@ fn test_metadata_compression_ratio() {
created_time: None,
modified_time: None,
};
assert_eq!(metadata.actual_ratio(), 5.0); // 5000/1000 = 5.0
assert!(!metadata.check_zip_bomb(10)); // ratio 5.0 < 10, not a bomb
assert!(metadata.check_zip_bomb(4)); // ratio 5.0 > 4, detected as bomb
assert_eq!(metadata.actual_ratio(), 5.0); // 5000/1000 = 5.0
assert!(!metadata.check_zip_bomb(10)); // ratio 5.0 < 10, not a bomb
assert!(metadata.check_zip_bomb(4)); // ratio 5.0 > 4, detected as bomb
}
#[test]
fn test_config_validation() {
let config = ArchiveConfig {
max_decompression_ratio: 5, // Too low
max_decompression_ratio: 5, // Too low
..Default::default()
};
assert!(config.validate().is_err());
let valid_config = ArchiveConfig::default();
assert!(valid_config.validate().is_ok());
}
}
+2 -2
View File
@@ -7,10 +7,10 @@ pub mod test_helpers;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_module_structure() {
// Test that all test modules exist
assert!(true);
}
}
}
+49 -35
View File
@@ -1,28 +1,27 @@
use flate2::write::GzEncoder;
use flate2::Compression;
use std::fs::{self, File};
use std::io::Write;
use std::path::PathBuf;
use tempfile::TempDir;
use zip::{ZipWriter, write::FileOptions, CompressionMethod};
use flate2::write::GzEncoder;
use flate2::Compression;
use tar::Builder;
use tempfile::TempDir;
use zip::{write::FileOptions, CompressionMethod, ZipWriter};
pub fn create_test_zip(temp_dir: &TempDir) -> PathBuf {
let zip_path = temp_dir.path().join("test.zip");
let file = File::create(&zip_path).unwrap();
let mut zip = ZipWriter::new(file);
let options = FileOptions::default()
.compression_method(CompressionMethod::Stored);
let options = FileOptions::default().compression_method(CompressionMethod::Stored);
zip.start_file("file1.txt", options).unwrap();
zip.write_all(b"content of file 1").unwrap();
zip.start_file("file2.txt", options).unwrap();
zip.write_all(b"content of file 2").unwrap();
zip.start_file("subdir/file3.txt", options).unwrap();
zip.write_all(b"content of file 3 in subdir").unwrap();
zip.finish().unwrap();
zip_path
}
@@ -31,28 +30,38 @@ pub fn create_test_tar(temp_dir: &TempDir) -> PathBuf {
let tar_path = temp_dir.path().join("test.tar");
let file = File::create(&tar_path).unwrap();
let mut builder = Builder::new(file);
let mut header1 = tar::Header::new_gnu();
header1.set_path("file1.txt").unwrap();
header1.set_size(17);
header1.set_mode(0o644);
header1.set_cksum();
builder.append_data(&mut header1, "file1.txt", &b"content of file 1"[..]).unwrap();
builder
.append_data(&mut header1, "file1.txt", &b"content of file 1"[..])
.unwrap();
let mut header2 = tar::Header::new_gnu();
header2.set_path("file2.txt").unwrap();
header2.set_size(17);
header2.set_mode(0o644);
header2.set_cksum();
builder.append_data(&mut header2, "file2.txt", &b"content of file 2"[..]).unwrap();
builder
.append_data(&mut header2, "file2.txt", &b"content of file 2"[..])
.unwrap();
let mut header3 = tar::Header::new_gnu();
header3.set_path("subdir/file3.txt").unwrap();
header3.set_size(27);
header3.set_mode(0o644);
header3.set_cksum();
builder.append_data(&mut header3, "subdir/file3.txt", &b"content of file 3 in subdir"[..]).unwrap();
builder
.append_data(
&mut header3,
"subdir/file3.txt",
&b"content of file 3 in subdir"[..],
)
.unwrap();
builder.finish().unwrap();
tar_path
}
@@ -61,7 +70,9 @@ pub fn create_test_gzip(temp_dir: &TempDir) -> PathBuf {
let gz_path = temp_dir.path().join("test.txt.gz");
let file = File::create(&gz_path).unwrap();
let mut encoder = GzEncoder::new(file, Compression::default());
encoder.write_all(b"test gzip content for validation").unwrap();
encoder
.write_all(b"test gzip content for validation")
.unwrap();
encoder.finish().unwrap();
gz_path
}
@@ -70,33 +81,37 @@ pub fn create_test_tar_gz(temp_dir: &TempDir) -> PathBuf {
let tar_path = temp_dir.path().join("test.tar");
let tar_file = File::create(&tar_path).unwrap();
let mut builder = Builder::new(tar_file);
let mut header1 = tar::Header::new_gnu();
header1.set_path("file1.txt").unwrap();
header1.set_size(10);
header1.set_mode(0o644);
header1.set_cksum();
builder.append_data(&mut header1, "file1.txt", &b"file1 data"[..]).unwrap();
builder
.append_data(&mut header1, "file1.txt", &b"file1 data"[..])
.unwrap();
let mut header2 = tar::Header::new_gnu();
header2.set_path("file2.txt").unwrap();
header2.set_size(10);
header2.set_mode(0o644);
header2.set_cksum();
builder.append_data(&mut header2, "file2.txt", &b"file2 data"[..]).unwrap();
builder
.append_data(&mut header2, "file2.txt", &b"file2 data"[..])
.unwrap();
builder.finish().unwrap();
let tar_gz_path = temp_dir.path().join("test.tar.gz");
let gz_file = File::create(&tar_gz_path).unwrap();
let mut encoder = GzEncoder::new(gz_file, Compression::default());
let tar_content = std::fs::read(&tar_path).unwrap();
encoder.write_all(&tar_content).unwrap();
encoder.finish().unwrap();
std::fs::remove_file(&tar_path).unwrap();
tar_gz_path
}
@@ -105,13 +120,12 @@ pub fn create_zip_bomb_test() -> Vec<u8> {
{
let writer = std::io::Cursor::new(&mut buffer);
let mut zip = ZipWriter::new(writer);
let options = FileOptions::default()
.compression_method(CompressionMethod::Stored);
let options = FileOptions::default().compression_method(CompressionMethod::Stored);
zip.start_file("bomb.txt", options).unwrap();
zip.write_all(&[0u8; 100]).unwrap();
zip.finish().unwrap();
}
buffer
@@ -123,11 +137,11 @@ pub fn create_zip_slip_test() -> Vec<u8> {
let writer = std::io::Cursor::new(&mut buffer);
let mut zip = ZipWriter::new(writer);
let options = FileOptions::default();
zip.start_file("../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../../etc/passwd", options).unwrap();
zip.write_all(b"malicious content").unwrap();
zip.finish().unwrap();
}
buffer
}
}
+16 -16
View File
@@ -1,6 +1,6 @@
// Warning System - Legal and Technical Warnings for Optional Formats
use log::{warn, info};
use log::{info, warn};
use crate::archive::config::ArchiveConfig;
@@ -63,25 +63,27 @@ pub fn show_startup_warnings(config: &ArchiveConfig) {
if config.enable_rar {
show_rar_legal_warning();
}
if config.enable_xz {
// Dependency check happens in ProcessorRegistry
}
if config.enable_7z {
show_7z_stability_warning();
}
// Show summary of enabled formats
let enabled_optional = [
config.enable_rar,
config.enable_xz,
config.enable_7z,
].iter().filter(|&x| *x).count();
let enabled_optional = [config.enable_rar, config.enable_xz, config.enable_7z]
.iter()
.filter(|&x| *x)
.count();
if enabled_optional > 0 {
info!("");
info!("⚠️ {} optional format(s) enabled with warnings shown above", enabled_optional);
info!(
"⚠️ {} optional format(s) enabled with warnings shown above",
enabled_optional
);
info!("Core formats (9): ZIP, TAR, GZIP, ZSTD, BZIP2, LZ4, TAR.GZ, TAR.BZ2, TAR.ZST");
info!("");
}
@@ -89,8 +91,7 @@ pub fn show_startup_warnings(config: &ArchiveConfig) {
/// Generate user-facing legal disclaimer text
pub fn generate_rar_legal_disclaimer() -> String {
format!(
"RAR FORMAT LEGAL DISCLAIMER
"RAR FORMAT LEGAL DISCLAIMER
IMPORTANT WARNING:
@@ -136,6 +137,5 @@ CONTACT:
Last Updated: 2026-06-10
Version: 1.0
Legal Consultation: [Please consult professional lawyer for commercial use]
"
)
}
".to_string()
}
+211
View File
@@ -0,0 +1,211 @@
#[cfg(feature = "async-vfs")]
use super::webdav::VfsDavFs;
#[cfg(feature = "async-vfs")]
use dav_server::davpath::DavPath;
#[cfg(feature = "async-vfs")]
use dav_server::fs::{
DavDirEntry, DavFile, DavFileSystem, DavMetaData, DavProp, FsError, FsFuture, FsStream,
OpenOptions, ReadDirMeta,
};
#[cfg(feature = "async-vfs")]
use http::StatusCode;
#[cfg(feature = "async-vfs")]
use std::future::Future;
#[cfg(feature = "async-vfs")]
use std::pin::Pin;
#[cfg(feature = "async-vfs")]
use std::sync::Arc;
#[cfg(feature = "async-vfs")]
use std::time::SystemTime;
#[cfg(feature = "async-vfs")]
pub struct AsyncVfsDavFs {
inner: Arc<VfsDavFs>,
runtime: Arc<tokio::runtime::Runtime>,
}
#[cfg(feature = "async-vfs")]
impl AsyncVfsDavFs {
pub fn new(inner: VfsDavFs) -> Self {
Self {
inner: Arc::new(inner),
runtime: Arc::new(tokio::runtime::Runtime::new().unwrap()),
}
}
fn block_on<F: Future>(&self, fut: F) -> F::Output {
self.runtime.block_on(fut)
}
}
#[cfg(feature = "async-vfs")]
impl Clone for AsyncVfsDavFs {
fn clone(&self) -> Self {
Self {
inner: self.inner.clone(),
runtime: self.runtime.clone(),
}
}
}
#[cfg(feature = "async-vfs")]
impl DavFileSystem for AsyncVfsDavFs {
fn open<'a>(&'a self, path: &'a DavPath, options: OpenOptions) -> FsFuture<'a, Box<dyn DavFile>> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.open(&path, options);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn read_dir<'a>(&'a self, path: &'a DavPath, meta: ReadDirMeta) -> FsFuture<'a, FsStream<Box<dyn DavDirEntry>>> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.read_dir(&path, meta);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn metadata<'a>(&'a self, path: &'a DavPath) -> FsFuture<'a, Box<dyn DavMetaData>> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.metadata(&path);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn create_dir<'a>(&'a self, path: &'a DavPath) -> FsFuture<'a, ()> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.create_dir(&path);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn remove_dir<'a>(&'a self, path: &'a DavPath) -> FsFuture<'a, ()> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.remove_dir(&path);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn remove_file<'a>(&'a self, path: &'a DavPath) -> FsFuture<'a, ()> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.remove_file(&path);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn rename<'a>(&'a self, from: &'a DavPath, to: &'a DavPath) -> FsFuture<'a, ()> {
let inner = self.inner.clone();
let from = from.clone();
let to = to.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.rename(&from, &to);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn copy<'a>(&'a self, from: &'a DavPath, to: &'a DavPath) -> FsFuture<'a, ()> {
let inner = self.inner.clone();
let from = from.clone();
let to = to.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.copy(&from, &to);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn set_accessed<'a>(&'a self, path: &'a DavPath, tm: SystemTime) -> FsFuture<'a, ()> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.set_accessed(&path, tm);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn set_modified<'a>(&'a self, path: &'a DavPath, tm: SystemTime) -> FsFuture<'a, ()> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.set_modified(&path, tm);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn get_props<'a>(&'a self, path: &'a DavPath, do_content: bool) -> FsFuture<'a, Vec<DavProp>> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.get_props(&path, do_content);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn get_prop<'a>(&'a self, path: &'a DavPath, prop: DavProp) -> FsFuture<'a, Vec<u8>> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.get_prop(&path, prop);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn patch_props<'a>(&'a self, path: &'a DavPath, patch: Vec<(bool, DavProp)>) -> FsFuture<'a, Vec<(StatusCode, DavProp)>> {
let inner = self.inner.clone();
let path = path.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.patch_props(&path, patch);
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
fn have_props<'a>(&'a self, path: &'a DavPath) -> Pin<Box<dyn Future<Output = bool> + Send + 'a>> {
self.inner.have_props(path)
}
fn get_quota(&self) -> FsFuture<'_, (u64, Option<u64>)> {
let inner = self.inner.clone();
Box::pin(async move {
tokio::task::spawn_blocking(move || {
let fut = inner.get_quota();
tokio::runtime::Runtime::new().unwrap().block_on(fut)
}).await.map_err(|_| FsError::GeneralFailure)?
})
}
}
+4 -4
View File
@@ -52,7 +52,7 @@ impl AuditLogger {
};
self.write_entry(&entry)?;
log::info!(
"Audit: {} config {} changed from '{}' to '{}' by {}",
config_type,
@@ -61,7 +61,7 @@ impl AuditLogger {
new_value,
user
);
Ok(())
}
@@ -126,7 +126,7 @@ impl AuditLogger {
} else {
0
};
Ok(entries[start..].to_vec())
}
}
}
+69 -33
View File
@@ -5,7 +5,7 @@ use std::collections::HashMap;
use std::sync::{Arc, Mutex};
use uuid::Uuid;
use crate::provider::{DataProvider, ProviderError};
use crate::provider::DataProvider;
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct User {
@@ -71,6 +71,12 @@ pub struct AuthState {
pub provider: Option<Arc<dyn DataProvider>>,
}
impl Default for AuthState {
fn default() -> Self {
Self::new()
}
}
impl AuthState {
pub fn new() -> Self {
let mut users = HashMap::new();
@@ -152,53 +158,78 @@ impl AuthState {
}
pub fn admin_login(&self, username: &str, password: &str) -> Option<AdminLoginResponse> {
// Try auth_db first (legacy PostgreSQL sync)
if let Some(auth_db) = &self.auth_db {
match auth_db.get_admin(username) {
Ok(Some(admin)) if admin.status == 1 => {
if verify(password, &admin.password_hash).unwrap_or(false) {
let token = Uuid::new_v4().to_string();
let now = Utc::now();
let expires_at = now + Duration::hours(24);
let session = AdminSession {
token: token.clone(),
username: username.to_string(),
created_at: now.format("%Y-%m-%dT%H:%M:%SZ").to_string(),
expires_at: expires_at.format("%Y-%m-%dT%H:%M:%SZ").to_string(),
};
let mut admin_sessions = self.admin_sessions.lock().unwrap();
admin_sessions.insert(token.clone(), session);
log::info!("Admin {} logged in successfully", username);
Some(AdminLoginResponse {
token,
expires_at: expires_at.format("%Y-%m-%dT%H:%M:%SZ").to_string(),
username: username.to_string(),
})
return self.create_admin_session(username, password);
} else {
log::warn!("Invalid password for admin {}", username);
None
return None;
}
}
Ok(Some(_)) => {
log::warn!("Admin {} is not active", username);
None
}
Ok(None) => {
log::warn!("Admin {} not found", username);
None
return None;
}
Ok(None) => {}
Err(e) => {
log::error!("Failed to get admin {}: {}", username, e);
None
return None;
}
}
} else {
log::warn!("Auth DB not available for admin login");
None
}
// Fallback: try provider
if let Some(provider) = &self.provider {
match provider.get_user(username) {
Ok(Some(user)) if user.status == 1 => {
if verify(password, &user.password_hash).unwrap_or(false) {
return self.create_admin_session(username, password);
} else {
log::warn!("Invalid password for admin {} (provider)", username);
return None;
}
}
Ok(Some(_)) => {
log::warn!("Admin {} is not active (provider)", username);
return None;
}
Ok(None) => {}
Err(e) => {
log::error!("Failed to get admin {} from provider: {}", username, e);
return None;
}
}
}
log::warn!("Admin {} not found (auth_db + provider)", username);
None
}
fn create_admin_session(&self, username: &str, _password: &str) -> Option<AdminLoginResponse> {
let token = Uuid::new_v4().to_string();
let now = Utc::now();
let expires_at = now + Duration::hours(24);
let session = AdminSession {
token: token.clone(),
username: username.to_string(),
created_at: now.format("%Y-%m-%dT%H:%M:%SZ").to_string(),
expires_at: expires_at.format("%Y-%m-%dT%H:%M:%SZ").to_string(),
};
let mut admin_sessions = self.admin_sessions.lock().unwrap();
admin_sessions.insert(token.clone(), session);
log::info!("Admin {} logged in successfully", username);
Some(AdminLoginResponse {
token,
expires_at: expires_at.format("%Y-%m-%dT%H:%M:%SZ").to_string(),
username: username.to_string(),
})
}
pub fn verify_admin_token(&self, token: &str) -> Option<AdminSession> {
@@ -284,7 +315,12 @@ impl AuthState {
}
}
fn login_with_provider(&self, provider: &dyn DataProvider, username: &str, password: &str) -> Option<LoginResponse> {
fn login_with_provider(
&self,
provider: &dyn DataProvider,
username: &str,
password: &str,
) -> Option<LoginResponse> {
match provider.get_user(username) {
Ok(Some(user)) => {
if user.status != 1 {
+157
View File
@@ -146,6 +146,70 @@
margin-bottom: 20px;
}
.back-btn:hover { background: #e8e8ed; }
/* Upload tab styles */
.upload-form {
max-width: 500px;
margin: 0 auto;
padding: 20px 0;
}
.upload-form label {
display: block;
font-weight: 500;
color: #1d1d1f;
margin-bottom: 8px;
}
.upload-form input[type="text"] {
width: 100%;
padding: 10px 14px;
border: 1px solid #d2d2d7;
border-radius: 8px;
font-size: 14px;
margin-bottom: 20px;
}
.upload-form input[type="file"] {
display: block;
margin-bottom: 20px;
font-size: 14px;
}
.upload-form .upload-btn {
width: 100%;
padding: 14px;
background: #0071e3;
color: white;
border: none;
border-radius: 8px;
font-size: 16px;
cursor: pointer;
}
.upload-form .upload-btn:hover { background: #0077ed; }
.upload-form .upload-btn:disabled {
background: #ccc;
cursor: not-allowed;
}
.progress-bar {
width: 100%;
height: 8px;
background: #e8e8ed;
border-radius: 4px;
overflow: hidden;
margin: 16px 0;
display: none;
}
.progress-bar .fill {
height: 100%;
background: #0071e3;
width: 0%;
transition: width 0.3s;
}
.upload-status {
text-align: center;
font-size: 14px;
margin-top: 12px;
color: #86868b;
}
.upload-status.success { color: #30d158; }
.upload-status.error { color: #ff453a; }
</style>
</head>
<body>
@@ -159,6 +223,7 @@
<div class="tabs">
<div class="tab active" data-view="category" onclick="switchTab('category')">By Category</div>
<div class="tab" data-view="series" onclick="switchTab('series')">By Series</div>
<div class="tab" data-view="upload" onclick="switchTab('upload')">Upload</div>
</div>
<div class="stats">
@@ -200,6 +265,13 @@
document.querySelectorAll('.tab').forEach(t => t.classList.remove('active'));
document.querySelector(`.tab[data-view="${view}"]`).classList.add('active');
if (view === 'upload') {
document.getElementById('back-btn').style.display = 'none';
showUploadForm();
return;
}
document.getElementById('back-btn').style.display = 'none';
document.getElementById('items-label').textContent = view === 'category' ? 'Categories' : 'Series';
@@ -325,6 +397,91 @@
}
}
async function showUploadForm() {
const content = document.getElementById('content');
document.getElementById('total-items').textContent = '-';
document.getElementById('total-files').textContent = '-';
content.innerHTML = `
<div class="upload-form">
<label for="upload-user-id">User ID</label>
<input type="text" id="upload-user-id" value="accusys">
<label for="upload-file">Select File</label>
<input type="file" id="upload-file">
<div class="progress-bar" id="progress-bar">
<div class="fill" id="progress-fill"></div>
</div>
<button class="upload-btn" id="upload-btn" onclick="startUpload()">Upload</button>
<div class="upload-status" id="upload-status"></div>
</div>
`;
}
async function startUpload() {
const fileInput = document.getElementById('upload-file');
const file = fileInput.files[0];
if (!file) {
document.getElementById('upload-status').textContent = 'Please select a file';
document.getElementById('upload-status').className = 'upload-status error';
return;
}
const userId = document.getElementById('upload-user-id').value.trim() || 'accusys';
const btn = document.getElementById('upload-btn');
const status = document.getElementById('upload-status');
const progressBar = document.getElementById('progress-bar');
const progressFill = document.getElementById('progress-fill');
btn.disabled = true;
btn.textContent = 'Uploading...';
status.textContent = '';
status.className = 'upload-status';
progressBar.style.display = 'block';
try {
const formData = new FormData();
formData.append('file', file);
const xhr = new XMLHttpRequest();
xhr.open('POST', apiBase + '/api/v2/upload-unlimited/' + encodeURIComponent(userId), true);
xhr.upload.onprogress = function(e) {
if (e.lengthComputable) {
const pct = (e.loaded / e.total) * 100;
progressFill.style.width = pct + '%';
}
};
const result = await new Promise((resolve, reject) => {
xhr.onload = function() {
if (xhr.status >= 200 && xhr.status < 300) {
try { resolve(JSON.parse(xhr.responseText)); }
catch { resolve({ ok: true }); }
} else {
try { reject(JSON.parse(xhr.responseText)); }
catch { reject({ error: 'Upload failed (' + xhr.status + ')' }); }
}
};
xhr.onerror = function() { reject({ error: 'Network error' }); };
xhr.send(formData);
});
progressFill.style.width = '100%';
status.textContent = 'Upload successful! File: ' + file.name;
status.className = 'upload-status success';
btn.textContent = 'Done';
} catch (err) {
status.textContent = 'Upload failed: ' + (err.error || err.message || 'Unknown error');
status.className = 'upload-status error';
btn.disabled = false;
btn.textContent = 'Upload';
progressBar.style.display = 'none';
}
}
function goBack() {
if (navigationStack.length === 0) {
currentItem = null;
+130 -63
View File
@@ -118,14 +118,20 @@ fn get_series_display_name(name: &str) -> String {
pub fn get_all_categories() -> Result<CategoriesResponse> {
let conn = FileTree::open_user_db("accusys")?;
let tree = FileTree::load(&conn, "accusys", "categories")?;
let categories: Vec<Category> = tree.nodes.iter()
let categories: Vec<Category> = tree
.nodes
.iter()
.filter(|n| n.parent_id.is_none() && n.node_type.as_str() == "folder")
.map(|n| {
let file_count = tree.nodes.iter()
.filter(|f| f.parent_id == Some(n.node_id.clone()) && f.node_type.as_str() == "file")
let file_count = tree
.nodes
.iter()
.filter(|f| {
f.parent_id == Some(n.node_id.clone()) && f.node_type.as_str() == "file"
})
.count();
Category {
name: n.label.clone(),
display_name: get_category_display_name(&n.label),
@@ -135,11 +141,13 @@ pub fn get_all_categories() -> Result<CategoriesResponse> {
}
})
.collect();
let total_files = tree.nodes.iter()
let total_files = tree
.nodes
.iter()
.filter(|n| n.node_type.as_str() == "file")
.count();
Ok(CategoriesResponse {
total_categories: categories.len(),
total_files,
@@ -150,42 +158,65 @@ pub fn get_all_categories() -> Result<CategoriesResponse> {
pub fn get_category_detail(category_name: &str) -> Result<CategoryDetail> {
let conn = FileTree::open_user_db("accusys")?;
let tree = FileTree::load(&conn, "accusys", "categories")?;
let category_node = tree.nodes.iter()
.find(|n| n.label == category_name && n.parent_id.is_none() && n.node_type.as_str() == "folder")
let category_node = tree
.nodes
.iter()
.find(|n| {
n.label == category_name && n.parent_id.is_none() && n.node_type.as_str() == "folder"
})
.ok_or_else(|| anyhow::anyhow!("Category not found: {}", category_name))?;
let series_groups: Vec<SeriesGroup> = tree.nodes.iter()
.filter(|n| n.parent_id == Some(category_node.node_id.clone()) && n.node_type.as_str() == "folder")
let series_groups: Vec<SeriesGroup> = tree
.nodes
.iter()
.filter(|n| {
n.parent_id == Some(category_node.node_id.clone()) && n.node_type.as_str() == "folder"
})
.map(|series_node| {
let files: Vec<CategoryFile> = tree.nodes.iter()
.filter(|f| f.parent_id == Some(series_node.node_id.clone()) && f.node_type.as_str() == "file")
.map(|file_node| {
CategoryFile {
filename: file_node.label.clone(),
size: file_node.aliases.get("file_size_display").cloned().unwrap_or_default(),
download_url: file_node.aliases.get("download_url").cloned().unwrap_or_default(),
sha256: file_node.sha256.clone(),
}
let files: Vec<CategoryFile> = tree
.nodes
.iter()
.filter(|f| {
f.parent_id == Some(series_node.node_id.clone())
&& f.node_type.as_str() == "file"
})
.map(|file_node| CategoryFile {
filename: file_node.label.clone(),
size: file_node
.aliases
.get("file_size_display")
.cloned()
.unwrap_or_default(),
download_url: file_node
.aliases
.get("download_url")
.cloned()
.unwrap_or_default(),
sha256: file_node.sha256.clone(),
})
.collect();
SeriesGroup {
series_name: series_node.label.clone(),
files,
}
})
.collect();
let file_count = series_groups.iter().map(|g| g.files.len()).sum();
Ok(CategoryDetail {
category: Category {
name: category_name.to_string(),
display_name: get_category_display_name(category_name),
file_count,
last_updated: category_node.updated_at.clone(),
description: category_node.aliases.get("description").cloned().unwrap_or_default(),
description: category_node
.aliases
.get("description")
.cloned()
.unwrap_or_default(),
},
series_groups,
})
@@ -194,25 +225,31 @@ pub fn get_category_detail(category_name: &str) -> Result<CategoryDetail> {
pub fn get_all_series() -> Result<SeriesResponse> {
let conn = FileTree::open_user_db("accusys")?;
let tree = FileTree::load(&conn, "accusys", "series")?;
let series: Vec<Series> = tree.nodes.iter()
let series: Vec<Series> = tree
.nodes
.iter()
.filter(|n| n.parent_id.is_none() && n.node_type.as_str() == "folder")
.map(|n| {
let file_count = tree.nodes.iter()
let file_count = tree
.nodes
.iter()
.filter(|f| {
let mut current = f.parent_id.clone();
while let Some(pid) = current {
if pid == n.node_id {
return f.node_type.as_str() == "file";
}
current = tree.nodes.iter()
current = tree
.nodes
.iter()
.find(|p| p.node_id == pid)
.map(|p| p.parent_id.clone()).flatten();
.and_then(|p| p.parent_id.clone());
}
false
})
.count();
Series {
name: n.label.clone(),
display_name: get_series_display_name(&n.label),
@@ -223,11 +260,13 @@ pub fn get_all_series() -> Result<SeriesResponse> {
}
})
.collect();
let total_files = tree.nodes.iter()
let total_files = tree
.nodes
.iter()
.filter(|n| n.node_type.as_str() == "file")
.count();
Ok(SeriesResponse {
total_series: series.len(),
total_files,
@@ -238,45 +277,63 @@ pub fn get_all_series() -> Result<SeriesResponse> {
pub fn get_series_detail(series_name: &str) -> Result<SeriesDetail> {
let conn = FileTree::open_user_db("accusys")?;
let tree = FileTree::load(&conn, "accusys", "series")?;
let series_node = tree.nodes.iter()
.find(|n| n.label == series_name && n.parent_id.is_none() && n.node_type.as_str() == "folder")
let series_node = tree
.nodes
.iter()
.find(|n| {
n.label == series_name && n.parent_id.is_none() && n.node_type.as_str() == "folder"
})
.ok_or_else(|| anyhow::anyhow!("Series not found: {}", series_name))?;
let categories: Vec<SeriesCategory> = tree.nodes.iter()
.filter(|n| n.parent_id == Some(series_node.node_id.clone()) && n.node_type.as_str() == "folder")
let categories: Vec<SeriesCategory> = tree
.nodes
.iter()
.filter(|n| {
n.parent_id == Some(series_node.node_id.clone()) && n.node_type.as_str() == "folder"
})
.map(|category_node| {
let files: Vec<SeriesFile> = tree.nodes.iter()
let files: Vec<SeriesFile> = tree
.nodes
.iter()
.filter(|f| {
let mut current = f.parent_id.clone();
while let Some(pid) = current {
if pid == category_node.node_id && f.node_type.as_str() == "file" {
return true;
}
current = tree.nodes.iter()
current = tree
.nodes
.iter()
.find(|p| p.node_id == pid)
.map(|p| p.parent_id.clone()).flatten();
.and_then(|p| p.parent_id.clone());
}
false
})
.map(|file_node| {
SeriesFile {
filename: file_node.label.clone(),
size: file_node.aliases.get("file_size_display").unwrap_or(&"N/A".to_string()).clone(),
download_url: file_node.aliases.get("download_url").unwrap_or(&"".to_string()).clone(),
}
.map(|file_node| SeriesFile {
filename: file_node.label.clone(),
size: file_node
.aliases
.get("file_size_display")
.unwrap_or(&"N/A".to_string())
.clone(),
download_url: file_node
.aliases
.get("download_url")
.unwrap_or(&"".to_string())
.clone(),
})
.collect();
SeriesCategory {
category_name: category_node.label.clone(),
files,
}
})
.collect();
let file_count = categories.iter().map(|c| c.files.len()).sum();
Ok(SeriesDetail {
series: Series {
name: series_name.to_string(),
@@ -296,29 +353,39 @@ pub fn search_files(query: &str, view: &str) -> Result<SearchResponse> {
"series" => "series",
_ => "untitled folder",
};
let conn = FileTree::open_user_db("accusys")?;
let tree = FileTree::load(&conn, "accusys", tree_type)?;
let results: Vec<SearchResult> = tree.nodes.iter()
.filter(|n| n.node_type.as_str() == "file" && n.label.to_lowercase().contains(&query.to_lowercase()))
let results: Vec<SearchResult> = tree
.nodes
.iter()
.filter(|n| {
n.node_type.as_str() == "file" && n.label.to_lowercase().contains(&query.to_lowercase())
})
.map(|file_node| {
let parent_node = tree.nodes.iter()
let parent_node = tree
.nodes
.iter()
.find(|n| n.node_id == file_node.parent_id.clone().unwrap_or_default());
SearchResult {
category: parent_node.map(|n| n.label.clone()),
series: parent_node.map(|n| n.label.clone()),
filename: file_node.label.clone(),
download_url: file_node.aliases.get("download_url").unwrap_or(&"".to_string()).clone(),
download_url: file_node
.aliases
.get("download_url")
.unwrap_or(&"".to_string())
.clone(),
}
})
.collect();
Ok(SearchResponse {
query: query.to_string(),
view: view.to_string(),
total_results: results.len(),
results,
})
}
}
+5 -4
View File
@@ -25,7 +25,7 @@ pub async fn handle_iscsi_command(cmd: IscsiCommand) -> anyhow::Result<()> {
let binary = find_binary("markbase-iscsi");
let mut cmd_process = std::process::Command::new(&binary);
cmd_process.arg("iscsi");
match cmd {
IscsiCommand::Start {
user,
@@ -34,7 +34,8 @@ pub async fn handle_iscsi_command(cmd: IscsiCommand) -> anyhow::Result<()> {
force,
device,
} => {
cmd_process.arg("start")
cmd_process
.arg("start")
.args(["--user", &user])
.args(["--port", &port.to_string()])
.args(["--lun-size", &lun_size]);
@@ -52,7 +53,7 @@ pub async fn handle_iscsi_command(cmd: IscsiCommand) -> anyhow::Result<()> {
cmd_process.arg("status");
}
}
let status = cmd_process.status()?;
std::process::exit(status.code().unwrap_or(1));
}
@@ -61,4 +62,4 @@ fn find_binary(name: &str) -> std::path::PathBuf {
let exe = std::env::current_exe().unwrap();
let dir = exe.parent().unwrap();
dir.join(name)
}
}
+4 -4
View File
@@ -1,8 +1,8 @@
pub mod web;
pub mod ssh;
pub mod webdav;
pub mod iscsi;
pub mod ssh;
pub mod tree;
pub mod web;
pub mod webdav;
use clap::Subcommand;
@@ -29,4 +29,4 @@ pub async fn handle_interface_command(cmd: InterfaceCommands) -> anyhow::Result<
InterfaceCommands::Tree(c) => tree::handle_tree_command(c).await?,
}
Ok(())
}
}
+2 -2
View File
@@ -28,8 +28,8 @@ pub async fn handle_ssh_command(cmd: SshCommand) -> anyhow::Result<()> {
println!("Security: ⭐⭐⭐⭐⭐ (RustCrypto authoritative libraries)");
println!();
crate::ssh_server::server::run_ssh_server(Some(port), pg_conn.as_deref())?;
crate::ssh_server::server::run_ssh_server(Some(port), pg_conn.as_deref()).await?;
}
}
Ok(())
}
}
+194 -124
View File
@@ -1,6 +1,6 @@
use anyhow::Context;
use clap::Subcommand;
use rusqlite::Connection;
use anyhow::Context;
use uuid::Uuid;
#[derive(Subcommand)]
@@ -33,12 +33,12 @@ pub enum TreeCommand {
#[arg(short, long)]
name: String,
},
Folder {
#[command(subcommand)]
action: FolderCommand,
},
Ls {
#[arg(short, long)]
user: String,
@@ -47,7 +47,7 @@ pub enum TreeCommand {
#[arg(short, long)]
tree_type: String,
},
Cp {
#[arg(short, long)]
user: String,
@@ -58,7 +58,7 @@ pub enum TreeCommand {
#[arg(short, long)]
tree_type: String,
},
Mv {
#[arg(short, long)]
user: String,
@@ -113,44 +113,54 @@ pub enum FolderCommand {
pub async fn handle_tree_command(cmd: TreeCommand) -> anyhow::Result<()> {
match cmd {
TreeCommand::Create { name, user, tree_type } => {
TreeCommand::Create {
name,
user,
tree_type,
} => {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
let node_id = Uuid::new_v4().to_string();
let created_at = chrono::Utc::now().to_rfc3339();
conn.execute(
"INSERT INTO file_nodes (node_id, label, node_type, tree_type, created_at, updated_at)
VALUES (?1, ?2, 'folder', ?3, ?4, ?4)",
rusqlite::params![node_id, name, tree_type, created_at]
).context("Failed to create tree")?;
println!("✓ Tree created: {} (type: {}) for user: {}", name, tree_type, user);
println!(
"✓ Tree created: {} (type: {}) for user: {}",
name, tree_type, user
);
println!("✓ Node ID: {}", node_id);
}
TreeCommand::List { user } => {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
let mut stmt = conn.prepare(
"SELECT DISTINCT tree_type FROM file_nodes ORDER BY tree_type"
).context("Failed to prepare query")?;
let tree_types = stmt.query_map([], |row| row.get::<_, String>(0))
let mut stmt = conn
.prepare("SELECT DISTINCT tree_type FROM file_nodes ORDER BY tree_type")
.context("Failed to prepare query")?;
let tree_types = stmt
.query_map([], |row| row.get::<_, String>(0))
.context("Failed to query tree types")?;
println!("=== Trees for user: {} ===", user);
for tree_type in tree_types {
let tt = tree_type?;
let count: i64 = conn.query_row(
"SELECT COUNT(*) FROM file_nodes WHERE tree_type = ?1",
[&tt],
|row| row.get(0)
).unwrap_or(0);
let count: i64 = conn
.query_row(
"SELECT COUNT(*) FROM file_nodes WHERE tree_type = ?1",
[&tt],
|row| row.get(0),
)
.unwrap_or(0);
println!(" {} ({} nodes)", tt, count);
}
}
@@ -158,9 +168,9 @@ pub async fn handle_tree_command(cmd: TreeCommand) -> anyhow::Result<()> {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
println!("Importing Markdown files to {} virtual tree...", tree_type);
if tree_type == "categories" {
crate::import_markdown::import_categories_to_db(&conn, &user, &tree_type)?;
println!("✓ Categories imported successfully!");
@@ -168,53 +178,66 @@ pub async fn handle_tree_command(cmd: TreeCommand) -> anyhow::Result<()> {
crate::import_markdown::import_series_to_db(&conn, &user, &tree_type)?;
println!("✓ Series imported successfully!");
} else {
eprintln!("Invalid tree_type: {}. Use 'categories' or 'series'", tree_type);
eprintln!(
"Invalid tree_type: {}. Use 'categories' or 'series'",
tree_type
);
}
}
TreeCommand::Delete { user, name } => {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
conn.execute(
"DELETE FROM file_nodes WHERE label = ?1 AND node_type = 'folder'",
[&name]
).context("Failed to delete tree")?;
[&name],
)
.context("Failed to delete tree")?;
println!("✓ Tree deleted: {} for user: {}", name, user);
}
TreeCommand::Folder { action } => {
handle_folder_command(action)?;
}
TreeCommand::Ls { user, path, tree_type } => {
TreeCommand::Ls {
user,
path,
tree_type,
} => {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
let parent_id = find_node_id(&conn, &path, &tree_type)?;
let mut stmt = conn.prepare(
"SELECT label, node_type, file_size FROM file_nodes
let mut stmt = conn
.prepare(
"SELECT label, node_type, file_size FROM file_nodes
WHERE parent_id = ?1 AND tree_type = ?2
ORDER BY node_type DESC, label ASC"
).context("Failed to prepare ls query")?;
let entries = stmt.query_map(
rusqlite::params![parent_id, tree_type],
|row| Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, Option<i64>>(2)?
))
).context("Failed to query entries")?;
ORDER BY node_type DESC, label ASC",
)
.context("Failed to prepare ls query")?;
let entries = stmt
.query_map(rusqlite::params![parent_id, tree_type], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, Option<i64>>(2)?,
))
})
.context("Failed to query entries")?;
println!("=== Contents of {} (tree_type: {}) ===", path, tree_type);
for entry in entries {
let (name, node_type, size) = entry?;
let size_str = size.map(|s| format!("{} bytes", s)).unwrap_or_else(|| "-".to_string());
let size_str = size
.map(|s| format!("{} bytes", s))
.unwrap_or_else(|| "-".to_string());
if node_type == "folder" {
println!(" 📁 {} ({})", name, size_str);
} else {
@@ -222,57 +245,72 @@ pub async fn handle_tree_command(cmd: TreeCommand) -> anyhow::Result<()> {
}
}
}
TreeCommand::Cp { user, source, target, tree_type } => {
TreeCommand::Cp {
user,
source,
target,
tree_type,
} => {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
let source_id = find_node_id(&conn, &source, &tree_type)?;
let target_parent_id = find_node_id(&conn, &target, &tree_type)?;
let (label, node_type, aliases_json, file_uuid, sha256, file_size) = conn.query_row(
"SELECT label, node_type, aliases_json, file_uuid, sha256, file_size
let (label, node_type, aliases_json, file_uuid, sha256, file_size) = conn
.query_row(
"SELECT label, node_type, aliases_json, file_uuid, sha256, file_size
FROM file_nodes WHERE node_id = ?1",
[&source_id],
|row| Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, Option<String>>(3)?,
row.get::<_, Option<String>>(4)?,
row.get::<_, Option<i64>>(5)?
))
).context("Failed to get source node")?;
[&source_id],
|row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
row.get::<_, Option<String>>(3)?,
row.get::<_, Option<String>>(4)?,
row.get::<_, Option<i64>>(5)?,
))
},
)
.context("Failed to get source node")?;
let new_id = Uuid::new_v4().to_string();
let created_at = chrono::Utc::now().to_rfc3339();
conn.execute(
"INSERT INTO file_nodes
(node_id, label, aliases_json, file_uuid, sha256, parent_id, node_type, file_size, tree_type, created_at, updated_at)
VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10, ?10)",
rusqlite::params![new_id, label, aliases_json, file_uuid, sha256, target_parent_id, node_type, file_size, tree_type, created_at]
).context("Failed to copy node")?;
println!("✓ Copied {} to {} (new ID: {})", source, target, new_id);
}
TreeCommand::Mv { user, source, target, tree_type } => {
TreeCommand::Mv {
user,
source,
target,
tree_type,
} => {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
let source_id = find_node_id(&conn, &source, &tree_type)?;
let target_parent_id = find_node_id(&conn, &target, &tree_type)?;
let updated_at = chrono::Utc::now().to_rfc3339();
conn.execute(
"UPDATE file_nodes SET parent_id = ?1, updated_at = ?2 WHERE node_id = ?3",
rusqlite::params![target_parent_id, updated_at, source_id]
).context("Failed to move node")?;
rusqlite::params![target_parent_id, updated_at, source_id],
)
.context("Failed to move node")?;
println!("✓ Moved {} to {}", source, target);
}
}
@@ -281,104 +319,136 @@ pub async fn handle_tree_command(cmd: TreeCommand) -> anyhow::Result<()> {
fn handle_folder_command(cmd: FolderCommand) -> anyhow::Result<()> {
match cmd {
FolderCommand::Create { user, path, name, tree_type } => {
FolderCommand::Create {
user,
path,
name,
tree_type,
} => {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
let parent_id = if path == "/" || path == "" {
let parent_id = if path == "/" || path.is_empty() {
None
} else {
Some(find_node_id(&conn, &path, &tree_type)?)
};
let node_id = Uuid::new_v4().to_string();
let created_at = chrono::Utc::now().to_rfc3339();
conn.execute(
"INSERT INTO file_nodes
(node_id, label, parent_id, node_type, tree_type, created_at, updated_at)
VALUES (?1, ?2, ?3, 'folder', ?4, ?5, ?5)",
rusqlite::params![node_id, name, parent_id, tree_type, created_at]
).context("Failed to create folder")?;
println!("✓ Folder created: {} in {} (tree_type: {})", name, path, tree_type);
rusqlite::params![node_id, name, parent_id, tree_type, created_at],
)
.context("Failed to create folder")?;
println!(
"✓ Folder created: {} in {} (tree_type: {})",
name, path, tree_type
);
println!("✓ Node ID: {}", node_id);
}
FolderCommand::Delete { user, path, name, tree_type } => {
FolderCommand::Delete {
user,
path,
name,
tree_type,
} => {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
let folder_path = if path == "/" || path == "" {
let folder_path = if path == "/" || path.is_empty() {
name.clone()
} else {
format!("{}/{}", path, name)
};
let folder_id = find_node_id(&conn, &folder_path, &tree_type)?;
conn.execute(
"DELETE FROM file_nodes WHERE node_id = ?1 OR parent_id = ?1",
[&folder_id]
).context("Failed to delete folder and children")?;
println!("✓ Folder deleted: {} in {} (tree_type: {})", name, path, tree_type);
[&folder_id],
)
.context("Failed to delete folder and children")?;
println!(
"✓ Folder deleted: {} in {} (tree_type: {})",
name, path, tree_type
);
}
FolderCommand::Rename { user, path, old_name, new_name, tree_type } => {
FolderCommand::Rename {
user,
path,
old_name,
new_name,
tree_type,
} => {
let db_path = format!("data/users/{}.sqlite", user);
let conn = Connection::open(&db_path)
.with_context(|| format!("Failed to open database: {}", db_path))?;
let folder_path = if path == "/" || path == "" {
let folder_path = if path == "/" || path.is_empty() {
old_name.clone()
} else {
format!("{}/{}", path, old_name)
};
let folder_id = find_node_id(&conn, &folder_path, &tree_type)?;
let updated_at = chrono::Utc::now().to_rfc3339();
conn.execute(
"UPDATE file_nodes SET label = ?1, updated_at = ?2 WHERE node_id = ?3",
rusqlite::params![new_name, updated_at, folder_id]
).context("Failed to rename folder")?;
println!("✓ Folder renamed: {} → {} in {} (tree_type: {})", old_name, new_name, path, tree_type);
rusqlite::params![new_name, updated_at, folder_id],
)
.context("Failed to rename folder")?;
println!(
"✓ Folder renamed: {} → {} in {} (tree_type: {})",
old_name, new_name, path, tree_type
);
}
}
Ok(())
}
fn find_node_id(conn: &Connection, path: &str, tree_type: &str) -> anyhow::Result<String> {
if path == "/" || path == "" {
let node_id: String = conn.query_row(
"SELECT node_id FROM file_nodes
if path == "/" || path.is_empty() {
let node_id: String = conn
.query_row(
"SELECT node_id FROM file_nodes
WHERE parent_id IS NULL AND node_type = 'folder' AND tree_type = ?1
LIMIT 1",
[tree_type],
|row| row.get(0)
).context("Failed to find root folder")?;
[tree_type],
|row| row.get(0),
)
.context("Failed to find root folder")?;
return Ok(node_id);
}
let parts: Vec<&str> = path.split('/').filter(|s| !s.is_empty()).collect();
let mut current_parent: Option<String> = None;
for part in parts {
let node_id: String = conn.query_row(
"SELECT node_id FROM file_nodes
let node_id: String = conn
.query_row(
"SELECT node_id FROM file_nodes
WHERE label = ?1 AND tree_type = ?2 AND
(parent_id = ?3 OR (?3 IS NULL AND parent_id IS NULL))",
rusqlite::params![part, tree_type, current_parent],
|row| row.get(0)
).context(format!("Failed to find node: {}", part))?;
rusqlite::params![part, tree_type, current_parent],
|row| row.get(0),
)
.context(format!("Failed to find node: {}", part))?;
current_parent = Some(node_id);
}
current_parent.context("Failed to find node ID for path")
}
}
+1 -1
View File
@@ -18,4 +18,4 @@ pub async fn handle_web_command(cmd: WebCommand) -> anyhow::Result<()> {
}
}
Ok(())
}
}
+260 -23
View File
@@ -1,5 +1,15 @@
use axum::{
body::Body,
extract::Request,
http::{HeaderValue, StatusCode},
middleware,
response::IntoResponse,
Extension,
};
use base64::Engine as _;
use clap::Subcommand;
use axum::{extract::Request, response::IntoResponse, Extension};
use dav_server::DavHandler;
use std::path::PathBuf;
#[derive(Subcommand)]
pub enum WebdavCommand {
@@ -9,28 +19,184 @@ pub enum WebdavCommand {
port: u16,
#[arg(short, long)]
user: String,
#[arg(long, help = "Enable SQLite virtual directory mode")]
virtual_mode: bool,
#[arg(long, help = "SQLite database path for virtual directories")]
db: Option<String>,
},
#[command(name = "webdav-folder")]
Folder {
#[arg(long, help = "Action: add, remove")]
action: String,
#[arg(long, help = "Virtual folder name (e.g. photos)")]
name: String,
#[arg(long, help = "Description for the folder")]
description: Option<String>,
#[arg(long, default_value = "data/webdav_virtual.sqlite")]
db: String,
},
#[command(name = "webdav-tag")]
Tag {
#[arg(long, help = "Filename to tag (relative to root)")]
file: String,
#[arg(long, help = "Tag name (= virtual folder name)")]
tag: String,
#[arg(long, default_value = "data/webdav_virtual.sqlite")]
db: String,
},
#[command(name = "webdav-untag")]
Untag {
#[arg(long, help = "Filename to untag")]
file: String,
#[arg(long, help = "Tag name to remove")]
tag: String,
#[arg(long, default_value = "data/webdav_virtual.sqlite")]
db: String,
},
#[command(name = "webdav-list")]
List {
#[arg(long, help = "List folders, or files in a folder")]
what: Option<String>,
#[arg(long, help = "Folder name (for listing files in a folder)")]
folder: Option<String>,
#[arg(long, help = "Filename (for listing tags of a file)")]
file: Option<String>,
#[arg(long, default_value = "data/webdav_virtual.sqlite")]
db: String,
},
}
pub async fn handle_webdav_command(cmd: WebdavCommand) -> anyhow::Result<()> {
match cmd {
WebdavCommand::Start { port, user } => {
let db_path = std::path::PathBuf::from(crate::FileTree::user_db_path(&user));
WebdavCommand::Start {
port,
user,
virtual_mode,
db,
} => {
let username = user.split(':').next().unwrap_or(&user).to_string();
let password = user.split(':').nth(1).map(|s| s.to_string());
if !db_path.exists() {
let default_root = format!("/Users/accusys/momentry/var/sftpgo/data/{}", username);
let home_dir = PathBuf::from(
std::env::var("MB_WEBDAV_ROOT").unwrap_or(default_root),
);
if !home_dir.exists() {
return Err(anyhow::anyhow!(
"User database not found: {}",
db_path.display()
"User home directory not found: {}",
home_dir.display()
));
}
println!("=== MarkBase WebDAV Server ===");
println!("User: {}", user);
println!("=== MarkBase WebDAV Server (VFS) ===");
println!("User: {}", username);
if password.is_some() {
println!("Auth: password protected");
}
println!("Port: {}", port);
println!("Database: {}", db_path.display());
println!("");
println!("Home: {}", home_dir.display());
if virtual_mode {
let db_path = db.clone().unwrap_or_else(|| "data/webdav_virtual.sqlite".to_string());
println!("Virtual mode: enabled (db: {})", db_path);
}
println!();
run_webdav_server(port, user, db_path).await?;
run_webdav_server(port, home_dir, username, password, virtual_mode, db).await?;
}
WebdavCommand::Folder {
action,
name,
description,
db,
} => {
let vfs = crate::vfs::virtual_fs::VirtualFs::new(&db, PathBuf::from("/tmp"))?;
let folder = if name.starts_with('/') {
name
} else {
format!("/{}", name)
};
match action.as_str() {
"add" => {
let desc = description.unwrap_or_default();
vfs.add_folder(&folder, &desc)?;
println!("Added virtual folder: {} ({})", folder, desc);
}
"remove" => {
vfs.remove_folder(&folder)?;
println!("Removed virtual folder: {}", folder);
}
_ => {
return Err(anyhow::anyhow!("Unknown action: {} (use add or remove)", action));
}
}
}
WebdavCommand::Tag { file, tag, db } => {
let vfs = crate::vfs::virtual_fs::VirtualFs::new(&db, PathBuf::from("/tmp"))?;
vfs.tag_file(&file, &tag)?;
println!("Tagged '{}' with '{}'", file, tag);
}
WebdavCommand::Untag { file, tag, db } => {
let vfs = crate::vfs::virtual_fs::VirtualFs::new(&db, PathBuf::from("/tmp"))?;
vfs.untag_file(&file, &tag)?;
println!("Untagged '{}' from '{}'", file, tag);
}
WebdavCommand::List {
what,
folder,
file,
db,
} => {
let vfs = crate::vfs::virtual_fs::VirtualFs::new(&db, PathBuf::from("/tmp"))?;
match what.as_deref() {
Some("folders") | None => {
let folders = vfs.list_folders()?;
if folders.is_empty() {
println!("No virtual folders.");
} else {
println!("{:<30} Description", "Folder");
println!("{}", "-".repeat(60));
for (f, d) in folders {
println!("{:<30} {}", f, d);
}
}
}
Some("files") => {
let folder_name = folder.ok_or_else(|| anyhow::anyhow!("--folder required for listing files"))?;
let files = vfs.list_files_in_folder(&folder_name)?;
if files.is_empty() {
println!("No files in folder '{}'", folder_name);
} else {
println!("Files in folder '{}':", folder_name);
for f in files {
println!(" {}", f);
}
}
}
Some("tags") => {
let filename = file.ok_or_else(|| anyhow::anyhow!("--file required for listing tags"))?;
let tags = vfs.list_tags_for_file(&filename)?;
if tags.is_empty() {
println!("No tags for file '{}'", filename);
} else {
println!("Tags for file '{}':", filename);
for t in tags {
println!(" {}", t);
}
}
}
Some(other) => {
return Err(anyhow::anyhow!("Unknown list type: {} (use folders, files, or tags)", other));
}
}
}
}
Ok(())
@@ -38,27 +204,97 @@ pub async fn handle_webdav_command(cmd: WebdavCommand) -> anyhow::Result<()> {
async fn run_webdav_server(
port: u16,
home_dir: PathBuf,
user: String,
db_path: std::path::PathBuf,
password: Option<String>,
virtual_mode: bool,
db: Option<String>,
) -> anyhow::Result<()> {
use axum::{extract::Request, response::IntoResponse, routing::any, Extension, Router};
use axum::{routing::any, Router};
use tokio::net::TcpListener;
let webdav = markbase_webdav::webdav::MarkBaseWebDAV::new(user, db_path);
let dav_handler = webdav.create_handler();
let dav_handler = if virtual_mode {
let db_path = db.unwrap_or_else(|| "data/webdav_virtual.sqlite".to_string());
let vfs = crate::vfs::virtual_fs::VirtualFs::new(&db_path, home_dir.clone())?;
let folders = vfs.list_folders().unwrap_or_default();
println!("Virtual folders ({}):", folders.len());
for (f, d) in &folders {
println!(" {} - {}", f, d);
}
println!("Default root: {}", home_dir.display());
println!();
let vfs_boxed: Box<dyn crate::vfs::VfsBackend> = Box::new(vfs);
crate::webdav::create_webdav_handler_virtual(vfs_boxed, PathBuf::from("/"), None, user.clone())
} else {
let vfs = Box::new(crate::vfs::local_fs::LocalFs::new());
crate::webdav::create_webdav_handler(vfs, home_dir.clone(), None, user.clone())
};
async fn webdav_auth_middleware(
req: Request,
next: middleware::Next,
) -> impl IntoResponse {
let expected = req.extensions().get::<crate::webdav::WebdavCredentials>().cloned();
let auth = req
.headers()
.get("Authorization")
.and_then(|v| v.to_str().ok())
.filter(|v| v.starts_with("Basic "))
.and_then(|v| {
let encoded = &v[6..];
let decoded = base64::engine::general_purpose::STANDARD
.decode(encoded)
.ok()?;
let creds = String::from_utf8(decoded).ok()?;
let colon = creds.find(':')?;
Some((creds[..colon].to_string(), creds[colon + 1..].to_string()))
});
let valid = match (auth, expected) {
(Some((u, p)), Some(exp)) => {
u == exp.username && exp.password.as_ref().is_none_or(|exp_p| p == *exp_p)
}
_ => false,
};
if !valid {
return (
StatusCode::UNAUTHORIZED,
[(
"WWW-Authenticate",
HeaderValue::from_static("Basic realm=\"MarkBase WebDAV\""),
)],
Body::from("Unauthorized"),
)
.into_response();
}
next.run(req).await
}
let app = Router::new()
.route("/webdav", any(handle_dav))
.route("/webdav/", any(handle_dav))
.route("/webdav/*path", any(handle_dav))
.layer(Extension(dav_handler));
.route("/", any(handle_dav))
.route("/*path", any(handle_dav))
.layer(Extension(dav_handler))
.layer(middleware::from_fn(webdav_auth_middleware))
.layer(Extension(crate::webdav::WebdavCredentials {
username: user.clone(),
password,
}));
let addr = format!("127.0.0.1:{}", port);
let addr = format!("0.0.0.0:{}", port);
let listener = TcpListener::bind(&addr).await?;
println!("WebDAV server listening on http://{}", addr);
println!("Mount point: /webdav");
println!("");
println!("User: {}", user);
if virtual_mode {
println!("Mode: Virtual (SQLite tag-based)");
} else {
println!("Mode: Local");
}
println!();
println!("Press Ctrl+C to stop");
axum::serve(listener, app).await?;
@@ -67,7 +303,8 @@ async fn run_webdav_server(
}
async fn handle_dav(
Extension(dav): Extension<dav_server::DavHandler>,
Extension(dav): Extension<DavHandler>,
Extension(_creds): Extension<crate::webdav::WebdavCredentials>,
req: Request,
) -> impl IntoResponse {
dav.handle(req).await
+30 -28
View File
@@ -1,6 +1,6 @@
use anyhow::Context;
use clap::Subcommand;
use rusqlite::Connection;
use anyhow::Context;
#[derive(Subcommand)]
pub enum AuthCommand {
@@ -24,29 +24,30 @@ pub fn handle_auth_command(cmd: AuthCommand) -> anyhow::Result<()> {
match cmd {
AuthCommand::Login { user, password } => {
let db_path = "data/auth.sqlite";
if !std::path::Path::new(db_path).exists() {
return Err(anyhow::anyhow!("Auth database not found: {}", db_path));
}
let conn = Connection::open(db_path)
.context("Failed to open auth database")?;
let password_hash: String = conn.query_row(
"SELECT password_hash FROM sftpgo_users WHERE username = ?",
[&user],
|row| row.get(0)
).context("Failed to query password hash")?;
let valid = bcrypt::verify(&password, &password_hash)
.context("Failed to verify password")?;
let conn = Connection::open(db_path).context("Failed to open auth database")?;
let password_hash: String = conn
.query_row(
"SELECT password_hash FROM sftpgo_users WHERE username = ?",
[&user],
|row| row.get(0),
)
.context("Failed to query password hash")?;
let valid =
bcrypt::verify(&password, &password_hash).context("Failed to verify password")?;
if !valid {
return Err(anyhow::anyhow!("Invalid password for user: {}", user));
}
let token = generate_simple_token(&user);
println!("✓ Login successful for user: {}", user);
println!("✓ Token: {}", token);
println!("Note: This is a simple token for demonstration. Use JWT in production.");
@@ -57,7 +58,7 @@ pub fn handle_auth_command(cmd: AuthCommand) -> anyhow::Result<()> {
}
AuthCommand::Verify { token } => {
let user = verify_simple_token(&token)?;
println!("✓ Token valid for user: {}", user);
println!("Note: This is simple token verification. Use JWT in production.");
}
@@ -67,37 +68,38 @@ pub fn handle_auth_command(cmd: AuthCommand) -> anyhow::Result<()> {
fn generate_simple_token(user: &str) -> String {
use std::time::{SystemTime, UNIX_EPOCH};
let timestamp = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs();
format!("{}_{}", user, timestamp)
}
fn verify_simple_token(token: &str) -> anyhow::Result<String> {
let parts: Vec<&str> = token.split('_').collect();
if parts.len() < 2 {
return Err(anyhow::anyhow!("Invalid token format"));
}
let user = parts[0];
let timestamp_str = parts[1];
let timestamp: u64 = timestamp_str.parse()
let timestamp: u64 = timestamp_str
.parse()
.context("Failed to parse token timestamp")?;
use std::time::{SystemTime, UNIX_EPOCH};
let now = SystemTime::now()
.duration_since(UNIX_EPOCH)
.unwrap()
.as_secs();
if now - timestamp > 86400 {
return Err(anyhow::anyhow!("Token expired (valid for 24 hours)"));
}
Ok(user.to_string())
}
}
+10 -4
View File
@@ -45,7 +45,9 @@ pub fn handle_config_command(cmd: ConfigCommand) -> anyhow::Result<()> {
let config_path = Path::new("config/markbase.toml");
if !config_path.exists() {
println!("Configuration file not found. Run 'markbase metadata config init' first.");
println!(
"Configuration file not found. Run 'markbase metadata config init' first."
);
return Ok(());
}
@@ -61,7 +63,9 @@ pub fn handle_config_command(cmd: ConfigCommand) -> anyhow::Result<()> {
let config_path = Path::new("config/markbase.toml");
if !config_path.exists() {
println!("Configuration file not found. Run 'markbase metadata config init' first.");
println!(
"Configuration file not found. Run 'markbase metadata config init' first."
);
return Ok(());
}
@@ -86,7 +90,9 @@ pub fn handle_config_command(cmd: ConfigCommand) -> anyhow::Result<()> {
let config_path = Path::new("config/markbase.toml");
if !config_path.exists() {
println!("Configuration file not found. Run 'markbase metadata config init' first.");
println!(
"Configuration file not found. Run 'markbase metadata config init' first."
);
return Ok(());
}
@@ -115,4 +121,4 @@ fn show_section(config: &crate::config::MarkBaseConfig, section: &str) {
"logging" => println!("{}", toml::to_string_pretty(&config.logging).unwrap()),
_ => println!("Invalid section: {}. Valid sections: server, postgresql, authentication, test, logging", section),
}
}
}
+46 -46
View File
@@ -1,6 +1,6 @@
use anyhow::Context;
use clap::Subcommand;
use rusqlite::Connection;
use anyhow::Context;
#[derive(Subcommand)]
pub enum DbCommand {
@@ -34,54 +34,52 @@ pub fn handle_db_command(cmd: DbCommand) -> anyhow::Result<()> {
match cmd {
DbCommand::Create { user } => {
let db_path = filetree::FileTree::user_db_path(&user);
if std::path::Path::new(&db_path).exists() {
println!("Database already exists: {}", db_path);
println!("Use 'db status' to check database info");
return Ok(());
}
println!("Creating database for user: {}", user);
let conn = filetree::FileTree::init_user_db(&user)
.context("Failed to initialize database")?;
let conn =
filetree::FileTree::init_user_db(&user).context("Failed to initialize database")?;
println!("✓ Database created: {}", db_path);
println!("✓ Tables initialized: file_nodes, file_registry, file_locations, tree_registry");
conn.close().map_err(|e| anyhow::anyhow!("Failed to close database: {:?}", e))?;
println!(
"✓ Tables initialized: file_nodes, file_registry, file_locations, tree_registry"
);
conn.close()
.map_err(|e| anyhow::anyhow!("Failed to close database: {:?}", e))?;
}
DbCommand::Status { user } => {
let db_path = filetree::FileTree::user_db_path(&user);
if !std::path::Path::new(&db_path).exists() {
return Err(anyhow::anyhow!("Database not found: {}", db_path));
}
let conn = Connection::open(&db_path)
.context("Failed to open database")?;
let conn = Connection::open(&db_path).context("Failed to open database")?;
let file_size = std::fs::metadata(&db_path)?.len();
let file_size_mb = file_size as f64 / 1024.0 / 1024.0;
let node_count: i64 = conn.query_row(
"SELECT COUNT(*) FROM file_nodes",
[],
|row| row.get(0)
).context("Failed to count nodes")?;
let file_count: i64 = conn.query_row(
"SELECT COUNT(*) FROM file_registry",
[],
|row| row.get(0)
).context("Failed to count files")?;
let node_count: i64 = conn
.query_row("SELECT COUNT(*) FROM file_nodes", [], |row| row.get(0))
.context("Failed to count nodes")?;
let file_count: i64 = conn
.query_row("SELECT COUNT(*) FROM file_registry", [], |row| row.get(0))
.context("Failed to count files")?;
let tree_types: Vec<String> = {
let mut stmt = conn.prepare("SELECT tree_type FROM tree_registry")?;
let rows = stmt.query_map([], |row| row.get(0))?;
rows.collect::<Result<Vec<_>, _>>()?
};
println!("=== Database Status ===");
println!("User: {}", user);
println!("Path: {}", db_path);
@@ -89,21 +87,21 @@ pub fn handle_db_command(cmd: DbCommand) -> anyhow::Result<()> {
println!("Nodes: {}", node_count);
println!("Files: {}", file_count);
println!("Tree Types: {:?}", tree_types);
conn.close().map_err(|e| anyhow::anyhow!("Failed to close database: {:?}", e))?;
conn.close()
.map_err(|e| anyhow::anyhow!("Failed to close database: {:?}", e))?;
}
DbCommand::Backup { user, output } => {
let db_path = filetree::FileTree::user_db_path(&user);
if !std::path::Path::new(&db_path).exists() {
return Err(anyhow::anyhow!("Database not found: {}", db_path));
}
println!("Backing up database for user: {} to {}", user, output);
std::fs::copy(&db_path, &output)
.context("Failed to backup database")?;
std::fs::copy(&db_path, &output).context("Failed to backup database")?;
println!("✓ Database backed up to: {}", output);
println!("✓ Backup size: {} bytes", std::fs::metadata(&output)?.len());
}
@@ -111,24 +109,26 @@ pub fn handle_db_command(cmd: DbCommand) -> anyhow::Result<()> {
if !std::path::Path::new(&input).exists() {
return Err(anyhow::anyhow!("Backup file not found: {}", input));
}
let db_path = filetree::FileTree::user_db_path(&user);
if std::path::Path::new(&db_path).exists() {
let backup_path = format!("{}.bak", db_path);
println!("Warning: Database exists, creating backup: {}", backup_path);
std::fs::copy(&db_path, &backup_path)
.context("Failed to create backup before restore")?;
}
println!("Restoring database for user: {} from {}", user, input);
std::fs::copy(&input, &db_path)
.context("Failed to restore database")?;
std::fs::copy(&input, &db_path).context("Failed to restore database")?;
println!("✓ Database restored from: {}", input);
println!("✓ Database size: {} bytes", std::fs::metadata(&db_path)?.len());
println!(
"✓ Database size: {} bytes",
std::fs::metadata(&db_path)?.len()
);
}
}
Ok(())
}
}
+4 -4
View File
@@ -1,7 +1,7 @@
pub mod config;
pub mod user;
pub mod db;
pub mod auth;
pub mod config;
pub mod db;
pub mod user;
use clap::Subcommand;
@@ -25,4 +25,4 @@ pub async fn handle_metadata_command(cmd: MetadataCommands) -> anyhow::Result<()
MetadataCommands::Auth(c) => auth::handle_auth_command(c)?,
}
Ok(())
}
}
+73 -69
View File
@@ -1,6 +1,6 @@
use anyhow::Context;
use clap::Subcommand;
use rusqlite::Connection;
use anyhow::Context;
#[derive(Subcommand)]
pub enum UserCommand {
@@ -18,7 +18,7 @@ pub enum UserCommand {
#[arg(short, long)]
name: String,
},
#[command(name = "user-delete")]
#[command(name = "user-delete")]
Delete {
#[arg(short, long)]
name: String,
@@ -29,58 +29,60 @@ pub fn handle_user_command(cmd: UserCommand) -> anyhow::Result<()> {
match cmd {
UserCommand::Create { name, password } => {
let db_path = "data/auth.sqlite";
if !std::path::Path::new(db_path).exists() {
return Err(anyhow::anyhow!("Auth database not found: {}", db_path));
}
let conn = Connection::open(db_path)
.context("Failed to open auth database")?;
let exists: i64 = conn.query_row(
"SELECT COUNT(*) FROM sftpgo_users WHERE username = ?",
[&name],
|row| row.get(0)
).context("Failed to check user existence")?;
let conn = Connection::open(db_path).context("Failed to open auth database")?;
let exists: i64 = conn
.query_row(
"SELECT COUNT(*) FROM sftpgo_users WHERE username = ?",
[&name],
|row| row.get(0),
)
.context("Failed to check user existence")?;
if exists > 0 {
return Err(anyhow::anyhow!("User already exists: {}", name));
}
let password_hash = bcrypt::hash(&password, bcrypt::DEFAULT_COST)
.context("Failed to hash password")?;
let password_hash =
bcrypt::hash(&password, bcrypt::DEFAULT_COST).context("Failed to hash password")?;
conn.execute(
"INSERT INTO sftpgo_users (username, password_hash, role, created_at) VALUES (?, ?, 'user', datetime('now'))",
rusqlite::params![name, password_hash]
).context("Failed to create user")?;
println!("✓ User created: {}", name);
println!("✓ Role: user");
println!("✓ Password hashed with bcrypt");
}
UserCommand::List => {
let db_path = "data/auth.sqlite";
if !std::path::Path::new(db_path).exists() {
return Err(anyhow::anyhow!("Auth database not found: {}", db_path));
}
let conn = Connection::open(db_path)
.context("Failed to open auth database")?;
let mut stmt = conn.prepare(
"SELECT username, role, created_at FROM sftpgo_users ORDER BY username"
).context("Failed to prepare query")?;
let users = stmt.query_map([], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
))
}).context("Failed to query users")?;
let conn = Connection::open(db_path).context("Failed to open auth database")?;
let mut stmt = conn
.prepare("SELECT username, role, created_at FROM sftpgo_users ORDER BY username")
.context("Failed to prepare query")?;
let users = stmt
.query_map([], |row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
))
})
.context("Failed to query users")?;
println!("=== Users List ===");
let mut count = 0;
for user in users {
@@ -88,7 +90,7 @@ pub fn handle_user_command(cmd: UserCommand) -> anyhow::Result<()> {
println!(" {} (role: {}, created: {})", name, role, created_at);
count += 1;
}
if count == 0 {
println!("No users found");
} else {
@@ -97,24 +99,27 @@ pub fn handle_user_command(cmd: UserCommand) -> anyhow::Result<()> {
}
UserCommand::Show { name } => {
let db_path = "data/auth.sqlite";
if !std::path::Path::new(db_path).exists() {
return Err(anyhow::anyhow!("Auth database not found: {}", db_path));
}
let conn = Connection::open(db_path)
.context("Failed to open auth database")?;
let user = conn.query_row(
"SELECT username, role, created_at FROM sftpgo_users WHERE username = ?",
[&name],
|row| Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
))
).context("Failed to query user")?;
let conn = Connection::open(db_path).context("Failed to open auth database")?;
let user = conn
.query_row(
"SELECT username, role, created_at FROM sftpgo_users WHERE username = ?",
[&name],
|row| {
Ok((
row.get::<_, String>(0)?,
row.get::<_, String>(1)?,
row.get::<_, String>(2)?,
))
},
)
.context("Failed to query user")?;
let (username, role, created_at) = user;
println!("=== User Details ===");
println!("Username: {}", username);
@@ -123,31 +128,30 @@ pub fn handle_user_command(cmd: UserCommand) -> anyhow::Result<()> {
}
UserCommand::Delete { name } => {
let db_path = "data/auth.sqlite";
if !std::path::Path::new(db_path).exists() {
return Err(anyhow::anyhow!("Auth database not found: {}", db_path));
}
let conn = Connection::open(db_path)
.context("Failed to open auth database")?;
let exists: i64 = conn.query_row(
"SELECT COUNT(*) FROM sftpgo_users WHERE username = ?",
[&name],
|row| row.get(0)
).context("Failed to check user existence")?;
let conn = Connection::open(db_path).context("Failed to open auth database")?;
let exists: i64 = conn
.query_row(
"SELECT COUNT(*) FROM sftpgo_users WHERE username = ?",
[&name],
|row| row.get(0),
)
.context("Failed to check user existence")?;
if exists == 0 {
return Err(anyhow::anyhow!("User not found: {}", name));
}
conn.execute(
"DELETE FROM sftpgo_users WHERE username = ?",
[&name]
).context("Failed to delete user")?;
conn.execute("DELETE FROM sftpgo_users WHERE username = ?", [&name])
.context("Failed to delete user")?;
println!("✓ User deleted: {}", name);
}
}
Ok(())
}
}
+1 -1
View File
@@ -22,4 +22,4 @@ pub enum Commands {
Storage(storage::StorageCommands),
#[command(flatten)]
Tools(tools::ToolsCommands),
}
}
+14 -14
View File
@@ -21,56 +21,56 @@ pub fn handle_archive_command(cmd: ArchiveCommand) -> anyhow::Result<()> {
match cmd {
ArchiveCommand::Decompress { file, output } => {
use crate::archive::{ArchiveConfig, ProcessorRegistry};
println!("Decompressing {} to {}", file, output);
let archive_path = Path::new(&file);
if !archive_path.exists() {
return Err(anyhow::anyhow!("Archive file not found: {}", file));
}
let config = ArchiveConfig::default();
let mut registry = ProcessorRegistry::new(config);
registry.initialize()?;
let output_path = Path::new(&output);
std::fs::create_dir_all(output_path)?;
let processor = registry.get_processor_mut(archive_path)?;
let result = processor.extract_all(output_path)?;
println!("✓ Archive decompressed to: {}", output);
println!("✓ Files extracted: {}", result.success_files);
println!("✓ Total size: {} bytes", result.total_bytes);
}
ArchiveCommand::List { file } => {
use crate::archive::{ArchiveConfig, ProcessorRegistry};
println!("Listing contents of {}", file);
let archive_path = Path::new(&file);
if !archive_path.exists() {
return Err(anyhow::anyhow!("Archive file not found: {}", file));
}
let config = ArchiveConfig::default();
let mut registry = ProcessorRegistry::new(config);
registry.initialize()?;
let processor = registry.get_processor_mut(archive_path)?;
let metadata = processor.open(archive_path)?;
let entries = processor.list_entries()?;
println!("=== Archive Contents ===");
println!("Format: {}", metadata.format);
println!("Total files: {}", metadata.total_files);
println!("Total size: {} bytes", metadata.total_size);
println!("");
println!();
for entry in entries {
println!(" {} ({} bytes)", entry.path.display(), entry.size);
}
}
}
Ok(())
}
}
+1 -1
View File
@@ -17,4 +17,4 @@ pub fn handle_hash_command(cmd: HashCommand) -> anyhow::Result<()> {
}
}
Ok(())
}
}
+4 -4
View File
@@ -1,8 +1,8 @@
pub mod scan;
pub mod hash;
pub mod archive;
pub mod sync;
pub mod hash;
pub mod mount;
pub mod scan;
pub mod sync;
use clap::Subcommand;
@@ -29,4 +29,4 @@ pub async fn handle_storage_command(cmd: StorageCommands) -> anyhow::Result<()>
StorageCommands::Mount(c) => mount::handle_mount_command(c)?,
}
Ok(())
}
}
+27 -23
View File
@@ -22,21 +22,25 @@ pub enum MountCommand {
pub fn handle_mount_command(cmd: MountCommand) -> anyhow::Result<()> {
match cmd {
MountCommand::Attach { type_, server, path } => {
MountCommand::Attach {
type_,
server,
path,
} => {
use std::process::Command;
println!("Mounting {} from {} to {}", type_, server, path);
if type_ == "nfs" {
let mount_point = std::path::Path::new(&path);
std::fs::create_dir_all(mount_point)?;
let nfs_path = format!("{}:{}", server, path);
let status = Command::new("mount")
.args(["-t", "nfs", &nfs_path, &path])
.status()?;
if status.success() {
println!("✓ NFS mounted: {} to {}", nfs_path, path);
} else {
@@ -45,31 +49,32 @@ pub fn handle_mount_command(cmd: MountCommand) -> anyhow::Result<()> {
} else if type_ == "smb" {
let mount_point = std::path::Path::new(&path);
std::fs::create_dir_all(mount_point)?;
let smb_path = format!("//{}", server);
let status = Command::new("mount")
.args(["-t", "smbfs", &smb_path, &path])
.status()?;
if status.success() {
println!("✓ SMB mounted: {} to {}", smb_path, path);
} else {
return Err(anyhow::anyhow!("SMB mount failed"));
}
} else {
return Err(anyhow::anyhow!("Unknown mount type: {}. Use 'nfs' or 'smb'", type_));
return Err(anyhow::anyhow!(
"Unknown mount type: {}. Use 'nfs' or 'smb'",
type_
));
}
}
MountCommand::Detach { path } => {
use std::process::Command;
println!("Unmounting {}", path);
let status = Command::new("umount")
.arg(&path)
.status()?;
let status = Command::new("umount").arg(&path).status()?;
if status.success() {
println!("✓ Unmounted: {}", path);
} else {
@@ -78,14 +83,13 @@ pub fn handle_mount_command(cmd: MountCommand) -> anyhow::Result<()> {
}
MountCommand::List => {
use std::process::Command;
println!("Listing mounted storage");
let output = Command::new("mount")
.output()?;
let output = Command::new("mount").output()?;
let mounts = String::from_utf8_lossy(&output.stdout);
println!("=== Mounted Filesystems ===");
for line in mounts.lines() {
if line.contains("nfs") || line.contains("smbfs") || line.contains("fuse") {
@@ -95,4 +99,4 @@ pub fn handle_mount_command(cmd: MountCommand) -> anyhow::Result<()> {
}
}
Ok(())
}
}
+1 -1
View File
@@ -31,4 +31,4 @@ pub fn handle_scan_command(cmd: ScanCommand) -> anyhow::Result<()> {
}
}
Ok(())
}
}
+13 -9
View File
@@ -17,27 +17,31 @@ pub enum SyncCommand {
pub fn handle_sync_command(cmd: SyncCommand) -> anyhow::Result<()> {
match cmd {
SyncCommand::Start { source, target, mode } => {
SyncCommand::Start {
source,
target,
mode,
} => {
use std::path::Path;
println!("Syncing {} to {} (mode: {})", source, target, mode);
let source_path = Path::new(&source);
let target_path = Path::new(&target);
if !source_path.exists() {
return Err(anyhow::anyhow!("Source path not found: {}", source));
}
if mode == "mirror" {
std::fs::create_dir_all(target_path)?;
let entries = std::fs::read_dir(source_path)?;
for entry in entries {
let entry = entry?;
let path = entry.path();
let target_file = target_path.join(entry.file_name());
if path.is_file() {
std::fs::copy(&path, &target_file)?;
println!(" Copied: {:?}", entry.file_name());
@@ -46,7 +50,7 @@ pub fn handle_sync_command(cmd: SyncCommand) -> anyhow::Result<()> {
println!(" Created directory: {:?}", entry.file_name());
}
}
println!("✓ Sync completed (mirror mode)");
} else {
return Err(anyhow::anyhow!("Unknown sync mode: {}. Use 'mirror'", mode));
@@ -59,4 +63,4 @@ pub fn handle_sync_command(cmd: SyncCommand) -> anyhow::Result<()> {
}
}
Ok(())
}
}
+11 -1
View File
@@ -1,5 +1,8 @@
pub mod render;
pub mod smb_server;
pub mod test;
#[cfg(feature = "nfs")]
pub mod nfs_server;
use clap::Subcommand;
@@ -9,12 +12,19 @@ pub enum ToolsCommands {
Render(render::RenderCommand),
#[command(flatten)]
Test(test::TestCommand),
#[command(flatten)]
SmbServer(smb_server::SmbServerCommand),
#[cfg(feature = "nfs")]
Nfs(nfs_server::NfsServerCommand),
}
pub async fn handle_tools_command(cmd: ToolsCommands) -> anyhow::Result<()> {
match cmd {
ToolsCommands::Render(c) => render::handle_render_command(c)?,
ToolsCommands::Test(c) => test::handle_test_command(c)?,
ToolsCommands::SmbServer(c) => smb_server::handle_smb_server_command(c).await?,
#[cfg(feature = "nfs")]
ToolsCommands::Nfs(c) => nfs_server::run_nfs_server(c).await?,
}
Ok(())
}
}
+43
View File
@@ -0,0 +1,43 @@
use clap::Args;
use std::path::PathBuf;
use std::sync::Arc;
use crate::vfs::{local_fs::LocalFs, nfs_server::{NfsVfsServer, NfsConfig}};
#[derive(Debug, Args)]
pub struct NfsServerCommand {
/// Port to listen on (default: 2049)
#[arg(short, long, default_value = "2049")]
port: u16,
/// Root directory to export
#[arg(short, long, default_value = "/tmp/nfs_export")]
root: PathBuf,
/// Share name (export name)
#[arg(short, long, default_value = "export")]
share_name: String,
}
pub async fn run_nfs_server(cmd: NfsServerCommand) -> anyhow::Result<()> {
println!("Starting NFS server on port {}", cmd.port);
println!("Export directory: {}", cmd.root.display());
println!("Share name: {}", cmd.share_name);
if !cmd.root.exists() {
std::fs::create_dir_all(&cmd.root)?;
println!("Created export directory: {}", cmd.root.display());
}
let vfs = Arc::new(LocalFs::new());
let server = NfsVfsServer::new(vfs, cmd.root.clone())
.with_port(cmd.port)
.with_export_name(&cmd.share_name);
println!("NFS server starting...");
server.start(cmd.port).await?;
println!("NFS server stopped");
Ok(())
}
+1 -1
View File
@@ -23,4 +23,4 @@ pub fn handle_render_command(cmd: RenderCommand) -> anyhow::Result<()> {
}
}
Ok(())
}
}
+228
View File
@@ -0,0 +1,228 @@
use clap::Subcommand;
#[derive(Subcommand)]
pub enum SmbServerCommand {
#[command(name = "smb-start")]
Start {
#[arg(short, long, default_value = "4445")]
port: u16,
#[arg(short, long, default_value = "/Users/accusys/momentry/var/sftpgo/data/demo")]
root: String,
#[arg(short, long, default_value = "markbase")]
share_name: String,
#[arg(long)]
read_only: bool,
#[arg(short, long, value_parser = parse_user)]
user: Vec<(String, String)>,
#[arg(long)]
s3: bool,
#[arg(long)]
s3_endpoint: Option<String>,
#[arg(long)]
s3_bucket: Option<String>,
#[arg(long)]
s3_access_key: Option<String>,
#[arg(long)]
s3_secret_key: Option<String>,
#[arg(long, default_value = "us-east-1")]
s3_region: String,
#[arg(long)]
ldap: bool,
#[arg(long)]
ldap_url: Option<String>,
#[arg(long)]
ldap_base_dn: Option<String>,
#[arg(long)]
ldap_bind_dn: Option<String>,
#[arg(long)]
ldap_bind_password: Option<String>,
#[arg(long)]
ldap_user_search_base: Option<String>,
#[arg(long)]
ldap_group_search_base: Option<String>,
#[arg(long)]
ldap_user_id_attr: Option<String>,
#[arg(long)]
ldap_user_filter: Option<String>,
#[arg(long)]
ldap_group_filter: Option<String>,
#[arg(long)]
ldap_home_dir_attr: Option<String>,
#[arg(long)]
ldap_home_dir_prefix: Option<String>,
#[arg(long)]
ldap_user_groups_attr: Option<String>,
},
}
fn parse_user(s: &str) -> Result<(String, String), String> {
let parts: Vec<&str> = s.split(':').collect();
if parts.len() != 2 {
return Err(format!("Invalid user format '{}'. Expected 'name:password'", s));
}
Ok((parts[0].to_string(), parts[1].to_string()))
}
pub async fn handle_smb_server_command(cmd: SmbServerCommand) -> anyhow::Result<()> {
#[cfg(feature = "smb-server")]
{
match cmd {
SmbServerCommand::Start {
port,
root,
share_name,
read_only,
user,
s3,
s3_endpoint,
s3_bucket,
s3_access_key,
s3_secret_key,
s3_region,
ldap,
ldap_url: _,
ldap_base_dn: _,
ldap_bind_dn: _,
ldap_bind_password: _,
ldap_user_search_base: _,
ldap_group_search_base: _,
ldap_user_id_attr: _,
ldap_user_filter: _,
ldap_group_filter: _,
ldap_home_dir_attr: _,
ldap_home_dir_prefix: _,
ldap_user_groups_attr: _,
} => {
use std::path::PathBuf;
use smb_server::{Access, Share, SmbServer};
use tracing_subscriber::EnvFilter;
let _ = tracing_subscriber::fmt()
.with_env_filter(
EnvFilter::try_from_default_env()
.unwrap_or_else(|_| EnvFilter::new("info")),
)
.try_init();
let addr: std::net::SocketAddr = format!("0.0.0.0:{}", port).parse()?;
let root_path = PathBuf::from(&root);
let vfs: Box<dyn crate::vfs::VfsBackend> = if s3 {
let endpoint = s3_endpoint
.ok_or_else(|| anyhow::anyhow!("--s3-endpoint required when --s3 is enabled"))?;
let bucket = s3_bucket
.ok_or_else(|| anyhow::anyhow!("--s3-bucket required when --s3 is enabled"))?;
let access_key = s3_access_key
.ok_or_else(|| anyhow::anyhow!("--s3-access-key required when --s3 is enabled"))?;
let secret_key = s3_secret_key
.ok_or_else(|| anyhow::anyhow!("--s3-secret-key required when --s3 is enabled"))?;
log::info!("S3 backend: endpoint={}, bucket={}, region={}", endpoint, bucket, s3_region);
Box::new(crate::vfs::s3_fs::S3Vfs::new(
&endpoint,
&s3_region,
&bucket,
&access_key,
&secret_key,
)?)
} else {
Box::new(crate::vfs::local_fs::LocalFs::new())
};
let backend = crate::vfs::smb_server_backend::VfsShareBackend::new(vfs, root_path)
.read_only(read_only);
let users: Vec<(String, String)> = if user.is_empty() {
vec![("demo".to_string(), "demo123".to_string())]
} else {
user
};
#[allow(unused_mut)]
let mut ldap_enabled = false;
#[cfg(feature = "ldap")]
{
if ldap {
let config = crate::provider::ldap::LdapConfig {
ldap_url: ldap_url.unwrap_or_else(|| "ldap://localhost:389".to_string()),
base_dn: ldap_base_dn.unwrap_or_else(|| "dc=example,dc=com".to_string()),
bind_dn: ldap_bind_dn.unwrap_or_else(|| "cn=admin,dc=example,dc=com".to_string()),
bind_password: ldap_bind_password.unwrap_or_else(|| "admin".to_string()),
user_search_base: ldap_user_search_base.unwrap_or_else(|| "ou=users,dc=example,dc=com".to_string()),
group_search_base: ldap_group_search_base.unwrap_or_else(|| "ou=groups,dc=example,dc=com".to_string()),
user_filter: ldap_user_filter.unwrap_or_else(|| "(objectClass=person)".to_string()),
group_filter: ldap_group_filter.unwrap_or_else(|| "(objectClass=group)".to_string()),
user_id_attr: ldap_user_id_attr.unwrap_or_else(|| "uid".to_string()),
home_dir_attr: ldap_home_dir_attr.unwrap_or_else(|| "homeDirectory".to_string()),
home_dir_prefix: ldap_home_dir_prefix.unwrap_or_else(|| "/home".to_string()),
user_groups_attr: ldap_user_groups_attr.unwrap_or_else(|| "memberOf".to_string()),
};
log::info!("LDAP authentication enabled: url={}, search_base={}", config.ldap_url, config.user_search_base);
ldap_enabled = true;
}
}
#[cfg(not(feature = "ldap"))]
if ldap {
log::warn!("LDAP authentication requested but ldap feature not enabled");
}
let mut builder = SmbServer::builder().listen(addr);
for (name, password) in &users {
builder = builder.user(name, password);
}
let mut share = Share::new(&share_name, backend);
for (name, _) in &users {
share = share.user(name, Access::ReadWrite);
}
let server = builder.share(share).build()?;
let user_list: Vec<&str> = users.iter().map(|(n, _)| n.as_str()).collect();
log::info!("SMB server listening on {}", addr);
log::info!("Share '{}' at root: {}", share_name, root);
log::info!("Users: {}", user_list.join(", "));
if ldap_enabled {
log::info!("LDAP authentication: enabled");
}
server.serve().await?;
}
}
}
#[cfg(not(feature = "smb-server"))]
{
let _ = cmd;
anyhow::bail!("SMB server support not enabled. Build with --features smb-server");
}
Ok(())
}
+10 -7
View File
@@ -16,36 +16,39 @@ pub enum TestCommand {
pub fn handle_test_command(cmd: TestCommand) -> anyhow::Result<()> {
match cmd {
TestCommand::Bcrypt { password, verify_hash } => {
TestCommand::Bcrypt {
password,
verify_hash,
} => {
use bcrypt::{hash, verify, DEFAULT_COST};
println!("=== bcrypt Hash Test ===");
println!("Password: {}", password);
println!("");
println!();
let new_hash = hash(&password, DEFAULT_COST)?;
println!("Generated hash:");
println!("{}", new_hash);
println!("");
println!();
if let Some(hash_to_verify) = verify_hash {
println!("Verifying hash: {}", hash_to_verify);
let valid = verify(&password, &hash_to_verify)?;
println!("Valid: {}", valid);
println!("");
println!();
}
let db_hash = "$2b$10$ha5wU.mOi8fHLJCfun860u2cfVopa04jwe/q82IKOwqp5uG70qsH6";
println!("Database hash: {}", db_hash);
let valid = verify(&password, db_hash)?;
println!("Database hash valid for '{}': {}", password, valid);
println!("");
println!();
if !valid {
println!("❌ Database hash is incorrect!");
println!("Update SQL:");
println!("UPDATE sftpgo_users SET password_hash = '{}' WHERE username IN ('testuser', 'demo', 'warren', 'momentry');", new_hash);
println!("");
println!();
println!("Execute:");
println!("sqlite3 data/auth.sqlite \"UPDATE sftpgo_users SET password_hash = '{}' WHERE username IN ('testuser', 'demo', 'warren', 'momentry');\"", new_hash);
} else {
@@ -58,4 +61,4 @@ pub fn handle_test_command(cmd: TestCommand) -> anyhow::Result<()> {
}
}
Ok(())
}
}
+49 -13
View File
@@ -8,6 +8,7 @@ pub use web::*;
/// Unified application configuration
#[derive(Debug, Clone, Serialize, Deserialize)]
#[derive(Default)]
pub struct AppConfig {
#[serde(default)]
pub web: WebSection,
@@ -17,6 +18,8 @@ pub struct AppConfig {
pub sftp: SftpSection,
#[serde(default)]
pub ssh: SshSection,
#[serde(default)]
pub upload_hook: UploadHookSection,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -26,6 +29,7 @@ pub struct WebSection {
pub log_level: String,
pub auth_db_path: String,
pub users_db_dir: String,
pub webdav_root: String,
}
impl Default for WebSection {
@@ -36,6 +40,7 @@ impl Default for WebSection {
log_level: "info".to_string(),
auth_db_path: "data/auth.sqlite".to_string(),
users_db_dir: "data/users".to_string(),
webdav_root: "/Users/accusys/momentry/var/sftpgo/data/demo".to_string(),
}
}
}
@@ -115,6 +120,36 @@ impl Default for SshSection {
}
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct UploadHookSection {
pub enabled: bool,
pub video_probe_path: String,
pub video_register_cli: String,
pub video_register_dir: String,
pub video_extensions: Vec<String>,
}
impl Default for UploadHookSection {
fn default() -> Self {
Self {
enabled: false,
video_probe_path: "/Users/accusys/momentry_core_project/video_probe/target/release/video_probe".to_string(),
video_register_cli: "cli.py".to_string(),
video_register_dir: "/Users/accusys/momentry_core_project/video_register".to_string(),
video_extensions: vec![
"mp4".to_string(),
"mov".to_string(),
"avi".to_string(),
"mkv".to_string(),
"webm".to_string(),
"flv".to_string(),
"wmv".to_string(),
"m4v".to_string(),
],
}
}
}
impl AppConfig {
pub fn load(path: &str) -> Result<Self> {
let config_path = std::path::PathBuf::from(path);
@@ -154,13 +189,19 @@ impl AppConfig {
self.web.host = v;
}
if let Ok(v) = std::env::var("MB_WEB_PORT") {
if let Ok(p) = v.parse() { self.web.port = p; }
if let Ok(p) = v.parse() {
self.web.port = p;
}
}
if let Ok(v) = std::env::var("MB_SSH_PORT") {
if let Ok(p) = v.parse() { self.ssh.port = p; }
if let Ok(p) = v.parse() {
self.ssh.port = p;
}
}
if let Ok(v) = std::env::var("MB_SFTP_PORT") {
if let Ok(p) = v.parse() { self.sftp.port = p; }
if let Ok(p) = v.parse() {
self.sftp.port = p;
}
}
if let Ok(v) = std::env::var("MB_S3_ENABLED") {
self.s3.enabled = v == "true" || v == "1";
@@ -169,20 +210,13 @@ impl AppConfig {
self.web.auth_db_path = v.clone();
self.sftp.auth_db_path = v;
}
}
}
impl Default for AppConfig {
fn default() -> Self {
Self {
web: WebSection::default(),
s3: S3Section::default(),
sftp: SftpSection::default(),
ssh: SshSection::default(),
if let Ok(v) = std::env::var("MB_UPLOAD_HOOK_ENABLED") {
self.upload_hook.enabled = v == "true" || v == "1";
}
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -195,6 +229,8 @@ mod tests {
assert_eq!(config.ssh.port, 2024);
assert_eq!(config.sftp.port, 2023);
assert_eq!(config.s3.region, "us-east-1");
assert!(!config.upload_hook.enabled);
assert!(config.upload_hook.video_extensions.contains(&"mp4".to_string()));
}
#[test]
+41 -2
View File
@@ -13,11 +13,30 @@ pub struct MarkBaseConfig {
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ServerConfig {
#[serde(default = "default_host")]
pub host: String,
#[serde(default = "default_port")]
pub port: u16,
#[serde(default = "default_log_level")]
pub log_level: String,
#[serde(default = "default_auth_db_path")]
pub auth_db_path: String,
#[serde(default = "default_users_db_dir")]
pub users_db_dir: String,
#[serde(default = "default_webdav_root")]
pub webdav_root: String,
#[serde(default = "default_upload_path")]
pub upload_path: String,
}
fn default_host() -> String { "127.0.0.1".to_string() }
fn default_port() -> u16 { 11438 }
fn default_log_level() -> String { "info".to_string() }
fn default_auth_db_path() -> String { "data/auth.sqlite".to_string() }
fn default_users_db_dir() -> String { "data/users".to_string() }
fn default_webdav_root() -> String { "/Users/accusys/momentry/var/sftpgo/data/demo".to_string() }
fn default_upload_path() -> String {
"/Users/accusys/momentry/var/sftpgo/data".to_string()
}
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -87,6 +106,8 @@ impl MarkBaseConfig {
log_level: "info".to_string(),
auth_db_path: "data/auth.sqlite".to_string(),
users_db_dir: "data/users".to_string(),
webdav_root: "/Users/accusys/momentry/var/sftpgo/data/demo".to_string(),
upload_path: "/Users/accusys/momentry/var/sftpgo/data".to_string(),
},
postgresql: PostgreSQLConfig {
host: "127.0.0.1".to_string(),
@@ -138,6 +159,12 @@ impl MarkBaseConfig {
if let Ok(log_level) = std::env::var("MB_LOG_LEVEL") {
self.server.log_level = log_level;
}
if let Ok(webdav_root) = std::env::var("MB_WEBDAV_ROOT") {
self.server.webdav_root = webdav_root;
}
if let Ok(upload_path) = std::env::var("MB_WEBDAV_PARENT") {
self.server.upload_path = upload_path;
}
if let Ok(pg_host) = std::env::var("PG_HOST") {
self.postgresql.host = pg_host;
@@ -176,6 +203,8 @@ impl MarkBaseConfig {
"server.log_level" => Some(self.server.log_level.clone()),
"server.auth_db_path" => Some(self.server.auth_db_path.clone()),
"server.users_db_dir" => Some(self.server.users_db_dir.clone()),
"server.webdav_root" => Some(self.server.webdav_root.clone()),
"server.upload_path" => Some(self.server.upload_path.clone()),
"postgresql.host" => Some(self.postgresql.host.clone()),
"postgresql.port" => Some(self.postgresql.port.to_string()),
@@ -221,6 +250,8 @@ impl MarkBaseConfig {
"server.log_level" => self.server.log_level = value.to_string(),
"server.auth_db_path" => self.server.auth_db_path = value.to_string(),
"server.users_db_dir" => self.server.users_db_dir = value.to_string(),
"server.webdav_root" => self.server.webdav_root = value.to_string(),
"server.upload_path" => self.server.upload_path = value.to_string(),
"postgresql.host" => self.postgresql.host = value.to_string(),
"postgresql.port" => self.postgresql.port = value.parse()?,
@@ -283,6 +314,10 @@ impl MarkBaseConfig {
return Err(anyhow::anyhow!("server.users_db_dir cannot be empty"));
}
if self.server.upload_path.is_empty() {
return Err(anyhow::anyhow!("server.upload_path cannot be empty"));
}
if self.postgresql.port == 0 {
return Err(anyhow::anyhow!(
"Invalid PostgreSQL port: {}",
@@ -323,11 +358,15 @@ impl MarkBaseConfig {
}
if self.authentication.default_user.is_empty() {
return Err(anyhow::anyhow!("authentication.default_user cannot be empty"));
return Err(anyhow::anyhow!(
"authentication.default_user cannot be empty"
));
}
if self.authentication.default_password.is_empty() {
return Err(anyhow::anyhow!("authentication.default_password cannot be empty"));
return Err(anyhow::anyhow!(
"authentication.default_password cannot be empty"
));
}
if self.authentication.max_sessions_per_user == 0 {
+379
View File
@@ -0,0 +1,379 @@
use std::collections::HashMap;
use std::net::IpAddr;
use std::sync::RwLock;
use super::node::NodeId;
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct PublicIpId(pub u32);
#[derive(Debug, Clone)]
pub struct PublicIpEntry {
pub id: PublicIpId,
pub ip: IpAddr,
pub interface: String,
pub owner: Option<NodeId>,
pub previous_owner: Option<NodeId>,
}
impl PublicIpEntry {
pub fn new(id: PublicIpId, ip: IpAddr, interface: &str) -> Self {
Self {
id,
ip,
interface: interface.to_string(),
owner: None,
previous_owner: None,
}
}
}
pub struct IpManager {
ip_pool: RwLock<Vec<PublicIpEntry>>,
assignments: RwLock<HashMap<PublicIpId, NodeId>>,
next_id: RwLock<u32>,
}
impl IpManager {
pub fn new() -> Self {
Self {
ip_pool: RwLock::new(Vec::new()),
assignments: RwLock::new(HashMap::new()),
next_id: RwLock::new(0),
}
}
pub fn add_ip(&self, ip: IpAddr, interface: &str) -> PublicIpId {
let mut pool = self.ip_pool.write().unwrap();
let mut next = self.next_id.write().unwrap();
let id = PublicIpId(*next);
*next += 1;
pool.push(PublicIpEntry::new(id, ip, interface));
id
}
pub fn remove_ip(&self, id: PublicIpId) {
let mut pool = self.ip_pool.write().unwrap();
pool.retain(|e| e.id != id);
let mut assignments = self.assignments.write().unwrap();
assignments.remove(&id);
}
pub fn assign_ip(&self, id: PublicIpId, node: NodeId) -> Result<(), String> {
let exists = {
let pool = self.ip_pool.read().unwrap();
pool.iter().any(|e| e.id == id)
};
if !exists {
return Err(format!("IP {:?} not found", id));
}
let prev_owner = {
let assignments = self.assignments.read().unwrap();
assignments.get(&id).copied()
};
{
let mut assignments = self.assignments.write().unwrap();
assignments.insert(id, node);
}
{
let mut pool = self.ip_pool.write().unwrap();
if let Some(entry) = pool.iter_mut().find(|e| e.id == id) {
if let Some(prev) = prev_owner {
entry.previous_owner = Some(prev);
}
entry.owner = Some(node);
}
}
Ok(())
}
pub fn release_ip(&self, id: PublicIpId) -> Result<(), String> {
let mut assignments = self.assignments.write().unwrap();
if let Some(prev) = assignments.remove(&id) {
let mut pool = self.ip_pool.write().unwrap();
if let Some(entry) = pool.iter_mut().find(|e| e.id == id) {
entry.previous_owner = Some(prev);
entry.owner = None;
}
Ok(())
} else {
Err(format!("IP {:?} not assigned", id))
}
}
pub fn reassign_on_failure(&self, failed_node: NodeId, active_nodes: &[NodeId]) -> Vec<(PublicIpId, NodeId)> {
let assignments = self.assignments.read().unwrap();
let failed_ips: Vec<PublicIpId> = assignments
.iter()
.filter(|(_, &node)| node == failed_node)
.map(|(&id, _)| id)
.collect();
drop(assignments);
let candidates: Vec<NodeId> = active_nodes
.iter()
.filter(|&&n| n != failed_node)
.copied()
.collect();
let mut reassignments = Vec::new();
for (i, ip_id) in failed_ips.iter().enumerate() {
if let Some(&new_owner) = candidates.get(i % candidates.len().max(1)) {
let _ = self.assign_ip(*ip_id, new_owner);
reassignments.push((*ip_id, new_owner));
}
}
reassignments
}
pub fn get_owner(&self, id: PublicIpId) -> Option<NodeId> {
self.assignments.read().unwrap().get(&id).copied()
}
pub fn owned_by(&self, node: NodeId) -> Vec<PublicIpEntry> {
let assignments = self.assignments.read().unwrap();
let pool = self.ip_pool.read().unwrap();
assignments
.iter()
.filter(|(_, &owner)| owner == node)
.filter_map(|(&id, _)| pool.iter().find(|e| e.id == id).cloned())
.collect()
}
pub fn unassigned(&self) -> Vec<PublicIpEntry> {
let assignments = self.assignments.read().unwrap();
let pool = self.ip_pool.read().unwrap();
pool.iter()
.filter(|e| !assignments.contains_key(&e.id))
.cloned()
.collect()
}
pub fn all_ips(&self) -> Vec<PublicIpEntry> {
self.ip_pool.read().unwrap().clone()
}
pub fn ip_count(&self) -> usize {
self.ip_pool.read().unwrap().len()
}
pub fn assigned_count(&self) -> usize {
self.assignments.read().unwrap().len()
}
pub fn balance(&self, active_nodes: &[NodeId]) -> Vec<(PublicIpId, NodeId)> {
let pool = self.ip_pool.read().unwrap();
let assignments = self.assignments.read().unwrap();
let total_ips = pool.len();
let node_count = active_nodes.len().max(1);
let per_node = total_ips / node_count;
let mut current_counts: HashMap<NodeId, usize> = HashMap::new();
for &node in active_nodes {
current_counts.insert(node, 0);
}
for (_, &owner) in assignments.iter() {
*current_counts.entry(owner).or_insert(0) += 1;
}
let mut reassignments = Vec::new();
for entry in pool.iter() {
let current_owner = assignments.get(&entry.id).copied();
let needs_reassign = match current_owner {
Some(owner) => {
!active_nodes.contains(&owner) || current_counts[&owner] > per_node
}
None => true,
};
if needs_reassign {
if let Some(&target) = active_nodes
.iter()
.min_by_key(|n| current_counts.get(n).copied().unwrap_or(0))
{
current_counts
.entry(target)
.and_modify(|c| *c += 1)
.or_insert(1);
reassignments.push((entry.id, target));
}
}
}
drop(pool);
drop(assignments);
for (id, node) in &reassignments {
let _ = self.assign_ip(*id, *node);
}
reassignments
}
}
impl Default for IpManager {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::net::Ipv4Addr;
fn ip(a: u8, b: u8, c: u8, d: u8) -> IpAddr {
IpAddr::V4(Ipv4Addr::new(a, b, c, d))
}
#[test]
fn test_add_ip() {
let mgr = IpManager::new();
let id = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
assert_eq!(mgr.ip_count(), 1);
assert_eq!(id, PublicIpId(0));
}
#[test]
fn test_add_multiple_ips() {
let mgr = IpManager::new();
let id0 = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
let id1 = mgr.add_ip(ip(192, 168, 1, 101), "eth0");
let id2 = mgr.add_ip(ip(192, 168, 1, 102), "eth1");
assert_eq!(mgr.ip_count(), 3);
assert_ne!(id0, id1);
assert_ne!(id1, id2);
}
#[test]
fn test_assign_ip() {
let mgr = IpManager::new();
let id = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
mgr.assign_ip(id, NodeId(0)).unwrap();
assert_eq!(mgr.get_owner(id), Some(NodeId(0)));
assert_eq!(mgr.assigned_count(), 1);
}
#[test]
fn test_release_ip() {
let mgr = IpManager::new();
let id = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
mgr.assign_ip(id, NodeId(0)).unwrap();
mgr.release_ip(id).unwrap();
assert_eq!(mgr.get_owner(id), None);
}
#[test]
fn test_release_unassigned_fails() {
let mgr = IpManager::new();
let id = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
assert!(mgr.release_ip(id).is_err());
}
#[test]
fn test_assign_nonexistent_fails() {
let mgr = IpManager::new();
assert!(mgr.assign_ip(PublicIpId(999), NodeId(0)).is_err());
}
#[test]
fn test_owned_by() {
let mgr = IpManager::new();
let id0 = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
let id1 = mgr.add_ip(ip(192, 168, 1, 101), "eth0");
let id2 = mgr.add_ip(ip(192, 168, 1, 102), "eth1");
mgr.assign_ip(id0, NodeId(0)).unwrap();
mgr.assign_ip(id1, NodeId(1)).unwrap();
mgr.assign_ip(id2, NodeId(0)).unwrap();
let node0_ips = mgr.owned_by(NodeId(0));
assert_eq!(node0_ips.len(), 2);
}
#[test]
fn test_unassigned() {
let mgr = IpManager::new();
mgr.add_ip(ip(192, 168, 1, 100), "eth0");
let id1 = mgr.add_ip(ip(192, 168, 1, 101), "eth0");
mgr.assign_ip(id1, NodeId(0)).unwrap();
let unassigned = mgr.unassigned();
assert_eq!(unassigned.len(), 1);
}
#[test]
fn test_reassign_on_failure() {
let mgr = IpManager::new();
let id0 = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
let id1 = mgr.add_ip(ip(192, 168, 1, 101), "eth0");
mgr.assign_ip(id0, NodeId(0)).unwrap();
mgr.assign_ip(id1, NodeId(0)).unwrap();
let reassignments = mgr.reassign_on_failure(NodeId(0), &[NodeId(1), NodeId(2)]);
assert_eq!(reassignments.len(), 2);
let new_owners: Vec<NodeId> = reassignments.iter().map(|(_, n)| *n).collect();
assert!(new_owners.contains(&NodeId(1)) || new_owners.contains(&NodeId(2)));
}
#[test]
fn test_reassign_skips_failed_node() {
let mgr = IpManager::new();
let id0 = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
mgr.assign_ip(id0, NodeId(0)).unwrap();
let reassignments = mgr.reassign_on_failure(NodeId(0), &[NodeId(0), NodeId(1)]);
assert_eq!(reassignments.len(), 1);
assert_ne!(reassignments[0].1, NodeId(0));
}
#[test]
fn test_remove_ip() {
let mgr = IpManager::new();
let id = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
mgr.assign_ip(id, NodeId(0)).unwrap();
mgr.remove_ip(id);
assert_eq!(mgr.ip_count(), 0);
assert_eq!(mgr.assigned_count(), 0);
}
#[test]
fn test_balance() {
let mgr = IpManager::new();
for i in 100..104 {
mgr.add_ip(ip(192, 168, 1, i), "eth0");
}
let reassignments = mgr.balance(&[NodeId(0), NodeId(1)]);
assert!(!reassignments.is_empty());
let node0 = mgr.owned_by(NodeId(0));
let node1 = mgr.owned_by(NodeId(1));
assert_eq!(node0.len() + node1.len(), 4);
assert!((node0.len() as i32 - node1.len() as i32).abs() <= 1);
}
#[test]
fn test_balance_excludes_inactive_nodes() {
let mgr = IpManager::new();
for i in 100..103 {
mgr.add_ip(ip(192, 168, 1, i), "eth0");
}
mgr.assign_ip(PublicIpId(0), NodeId(5)).unwrap();
let reassignments = mgr.balance(&[NodeId(0), NodeId(1)]);
assert!(reassignments
.iter()
.any(|(_, n)| *n == NodeId(0) || *n == NodeId(1)));
assert!(!reassignments.iter().any(|(_, n)| *n == NodeId(5)));
}
#[test]
fn test_previous_owner() {
let mgr = IpManager::new();
let id = mgr.add_ip(ip(192, 168, 1, 100), "eth0");
mgr.assign_ip(id, NodeId(0)).unwrap();
mgr.assign_ip(id, NodeId(1)).unwrap();
let pool = mgr.all_ips();
let entry = pool.iter().find(|e| e.id == id).unwrap();
assert_eq!(entry.previous_owner, Some(NodeId(0)));
assert_eq!(entry.owner, Some(NodeId(1)));
}
}
+5
View File
@@ -0,0 +1,5 @@
pub mod tdb;
pub mod node;
pub mod protocol;
pub mod ip_manager;
pub mod recovery;
+353
View File
@@ -0,0 +1,353 @@
use std::collections::HashMap;
use std::net::SocketAddr;
use std::sync::RwLock;
use std::time::{Duration, Instant};
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct NodeId(pub u32);
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum NodeState {
Up,
Down,
Unhealthy,
Banned,
Disabled,
}
impl NodeState {
pub fn is_active(&self) -> bool {
matches!(self, NodeState::Up)
}
pub fn as_str(&self) -> &'static str {
match self {
NodeState::Up => "UP",
NodeState::Down => "DOWN",
NodeState::Unhealthy => "UNHEALTHY",
NodeState::Banned => "BANNED",
NodeState::Disabled => "DISABLED",
}
}
}
#[derive(Debug, Clone)]
pub struct NodeInfo {
pub id: NodeId,
pub addr: SocketAddr,
pub state: NodeState,
pub last_heartbeat: Option<Instant>,
pub public_ips: Vec<String>,
pub generation: u64,
}
impl NodeInfo {
pub fn new(id: NodeId, addr: SocketAddr) -> Self {
Self {
id,
addr,
state: NodeState::Down,
last_heartbeat: None,
public_ips: Vec::new(),
generation: 0,
}
}
pub fn is_alive(&self, timeout: Duration) -> bool {
match self.last_heartbeat {
Some(t) => t.elapsed() < timeout,
None => false,
}
}
}
#[derive(Debug, Clone)]
pub struct NodeMask {
nodes: Vec<bool>,
}
impl NodeMask {
pub fn new(size: usize) -> Self {
Self {
nodes: vec![false; size],
}
}
pub fn set(&mut self, id: NodeId, active: bool) {
if (id.0 as usize) < self.nodes.len() {
self.nodes[id.0 as usize] = active;
}
}
pub fn is_active(&self, id: NodeId) -> bool {
self.nodes.get(id.0 as usize).copied().unwrap_or(false)
}
pub fn active_nodes(&self) -> Vec<NodeId> {
self.nodes
.iter()
.enumerate()
.filter(|(_, &active)| active)
.map(|(i, _)| NodeId(i as u32))
.collect()
}
pub fn len(&self) -> usize {
self.nodes.len()
}
pub fn active_count(&self) -> usize {
self.nodes.iter().filter(|&&a| a).count()
}
}
pub struct NodeManager {
nodes: RwLock<HashMap<NodeId, NodeInfo>>,
self_id: NodeId,
heartbeat_timeout: Duration,
heartbeat_interval: Duration,
}
impl NodeManager {
pub fn new(self_id: NodeId, self_addr: SocketAddr) -> Self {
let mut nodes = HashMap::new();
nodes.insert(
self_id,
NodeInfo {
id: self_id,
addr: self_addr,
state: NodeState::Up,
last_heartbeat: Some(Instant::now()),
public_ips: Vec::new(),
generation: 0,
},
);
Self {
nodes: RwLock::new(nodes),
self_id,
heartbeat_timeout: Duration::from_secs(5),
heartbeat_interval: Duration::from_secs(1),
}
}
pub fn add_node(&self, id: NodeId, addr: SocketAddr) {
let mut nodes = self.nodes.write().unwrap();
nodes.insert(id, NodeInfo::new(id, addr));
}
pub fn remove_node(&self, id: NodeId) {
let mut nodes = self.nodes.write().unwrap();
nodes.remove(&id);
}
pub fn record_heartbeat(&self, id: NodeId) {
let mut nodes = self.nodes.write().unwrap();
if let Some(node) = nodes.get_mut(&id) {
node.last_heartbeat = Some(Instant::now());
if node.state == NodeState::Down {
node.state = NodeState::Up;
node.generation += 1;
}
}
}
pub fn set_node_state(&self, id: NodeId, state: NodeState) {
let mut nodes = self.nodes.write().unwrap();
if let Some(node) = nodes.get_mut(&id) {
node.state = state;
if state == NodeState::Up && node.last_heartbeat.is_none() {
node.last_heartbeat = Some(Instant::now());
}
node.generation += 1;
}
}
pub fn get_node(&self, id: NodeId) -> Option<NodeInfo> {
self.nodes.read().unwrap().get(&id).cloned()
}
pub fn all_nodes(&self) -> Vec<NodeInfo> {
self.nodes.read().unwrap().values().cloned().collect()
}
pub fn active_nodes(&self) -> Vec<NodeInfo> {
self.nodes
.read()
.unwrap()
.values()
.filter(|n| n.state == NodeState::Up)
.cloned()
.collect()
}
pub fn check_health(&self) -> Vec<(NodeId, NodeState)> {
let timeout = self.heartbeat_timeout;
let mut nodes = self.nodes.write().unwrap();
let mut transitions = Vec::new();
for (id, node) in nodes.iter_mut() {
if *id == self.self_id {
continue;
}
match &node.last_heartbeat {
Some(t) => {
if t.elapsed() > timeout && node.state == NodeState::Up {
node.state = NodeState::Down;
node.generation += 1;
transitions.push((*id, NodeState::Down));
}
}
None => {
if node.state == NodeState::Up {
node.state = NodeState::Down;
node.generation += 1;
transitions.push((*id, NodeState::Down));
}
}
}
}
transitions
}
pub fn self_id(&self) -> NodeId {
self.self_id
}
pub fn node_count(&self) -> usize {
self.nodes.read().unwrap().len()
}
pub fn build_nodemask(&self) -> NodeMask {
let nodes = self.nodes.read().unwrap();
let max_id = nodes.keys().map(|k| k.0).max().unwrap_or(0) as usize;
let mut mask = NodeMask::new(max_id + 1);
for (id, node) in nodes.iter() {
mask.set(*id, node.state.is_active());
}
mask
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::net::{Ipv4Addr, SocketAddrV4};
fn addr(port: u16) -> SocketAddr {
SocketAddr::V4(SocketAddrV4::new(Ipv4Addr::LOCALHOST, port))
}
#[test]
fn test_node_creation() {
let mgr = NodeManager::new(NodeId(0), addr(4000));
let self_node = mgr.get_node(NodeId(0)).unwrap();
assert_eq!(self_node.state, NodeState::Up);
assert!(self_node.last_heartbeat.is_some());
}
#[test]
fn test_add_remove_node() {
let mgr = NodeManager::new(NodeId(0), addr(4000));
mgr.add_node(NodeId(1), addr(4001));
assert_eq!(mgr.node_count(), 2);
mgr.remove_node(NodeId(1));
assert_eq!(mgr.node_count(), 1);
}
#[test]
fn test_heartbeat_updates() {
let mgr = NodeManager::new(NodeId(0), addr(4000));
mgr.add_node(NodeId(1), addr(4001));
assert_eq!(mgr.get_node(NodeId(1)).unwrap().state, NodeState::Down);
mgr.record_heartbeat(NodeId(1));
assert_eq!(mgr.get_node(NodeId(1)).unwrap().state, NodeState::Up);
}
#[test]
fn test_health_check_timeout() {
let mgr = NodeManager::new(NodeId(0), addr(4000));
mgr.add_node(NodeId(1), addr(4001));
mgr.record_heartbeat(NodeId(1));
assert_eq!(mgr.get_node(NodeId(1)).unwrap().state, NodeState::Up);
std::thread::sleep(Duration::from_millis(100));
let mgr = NodeManager {
nodes: RwLock::new(mgr.nodes.read().unwrap().clone()),
self_id: NodeId(0),
heartbeat_timeout: Duration::from_millis(50),
heartbeat_interval: Duration::from_millis(10),
};
let transitions = mgr.check_health();
assert!(transitions.contains(&(NodeId(1), NodeState::Down)));
}
#[test]
fn test_nodemask() {
let mut mask = NodeMask::new(5);
mask.set(NodeId(0), true);
mask.set(NodeId(2), true);
mask.set(NodeId(4), true);
assert!(mask.is_active(NodeId(0)));
assert!(!mask.is_active(NodeId(1)));
assert_eq!(mask.active_count(), 3);
assert_eq!(mask.len(), 5);
}
#[test]
fn test_build_nodemask() {
let mgr = NodeManager::new(NodeId(0), addr(4000));
mgr.add_node(NodeId(1), addr(4001));
mgr.add_node(NodeId(2), addr(4002));
mgr.record_heartbeat(NodeId(1));
let mask = mgr.build_nodemask();
assert!(mask.is_active(NodeId(0)));
assert!(mask.is_active(NodeId(1)));
assert!(!mask.is_active(NodeId(2)));
}
#[test]
fn test_node_state_string() {
assert_eq!(NodeState::Up.as_str(), "UP");
assert_eq!(NodeState::Down.as_str(), "DOWN");
assert_eq!(NodeState::Unhealthy.as_str(), "UNHEALTHY");
assert_eq!(NodeState::Banned.as_str(), "BANNED");
assert_eq!(NodeState::Disabled.as_str(), "DISABLED");
}
#[test]
fn test_set_node_state() {
let mgr = NodeManager::new(NodeId(0), addr(4000));
mgr.add_node(NodeId(1), addr(4001));
mgr.set_node_state(NodeId(1), NodeState::Banned);
assert_eq!(mgr.get_node(NodeId(1)).unwrap().state, NodeState::Banned);
}
#[test]
fn test_generation_increment() {
let mgr = NodeManager::new(NodeId(0), addr(4000));
mgr.add_node(NodeId(1), addr(4001));
let gen0 = mgr.get_node(NodeId(1)).unwrap().generation;
mgr.set_node_state(NodeId(1), NodeState::Down);
let gen1 = mgr.get_node(NodeId(1)).unwrap().generation;
assert!(gen1 > gen0);
}
#[test]
fn test_active_nodes() {
let mgr = NodeManager::new(NodeId(0), addr(4000));
mgr.add_node(NodeId(1), addr(4001));
mgr.add_node(NodeId(2), addr(4002));
mgr.record_heartbeat(NodeId(1));
let active = mgr.active_nodes();
assert_eq!(active.len(), 2);
}
#[test]
fn test_is_alive() {
let node = NodeInfo::new(NodeId(0), addr(4000));
assert!(!node.is_alive(Duration::from_secs(5)));
let mut node = node;
node.last_heartbeat = Some(Instant::now());
assert!(node.is_alive(Duration::from_secs(5)));
}
}

Some files were not shown because too many files have changed in this diff Show More