328 lines
14 KiB
Markdown
328 lines
14 KiB
Markdown
# Ceph RADOS Integration Analysis for MarkBase
|
||
|
||
**Date**: 2026-06-25
|
||
**Status**: Shelved (不符合 macOS 跨平台定位)
|
||
**Library**: ceph-async (4.0.5)
|
||
**Constraint**: Linux-only (requires librados.so symlink)
|
||
|
||
---
|
||
|
||
## Executive Summary
|
||
|
||
### Goal
|
||
Add Ceph RADOS as a VfsBackend option for distributed, highly scalable storage.
|
||
|
||
### Key Findings
|
||
| Aspect | Finding |
|
||
|--------|---------|
|
||
| **Platform** | ❌ Linux-only (librados.so FFI, macOS needs Docker/VM) |
|
||
| **Deployment** | ⚠️ Requires full cluster (Monitor + OSD + MGR) |
|
||
| **Complexity** | ⚠️⚠️⚠️⚠️⚠️ High (超出 Lightweight 定位) |
|
||
| **Positioning** | ❌ 不符合 MarkBase macOS 跨平台定位 |
|
||
|
||
### Recommendation
|
||
**当前搁置**。优先考虑：
|
||
1. **MinIO** — S3-compatible，已有 S3Vfs 支持，跨平台
|
||
2. **内置分布式** — DedupFs + S3Vfs 组合，轻量级
|
||
|
||
---
|
||
|
||
## Architecture Overview
|
||
|
||
```
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ MarkBase Application Layer │
|
||
│ ├── SMB Server (Port 4445) │
|
||
│ ├── SFTP Server (Port 2024) │
|
||
│ ├── WebDAV Server (Port 11438) │
|
||
│ └───────────────────────────────────────────────────────────────────────┘
|
||
│ ↓ │
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ VFS Abstraction Layer (VfsBackend trait) │
|
||
│ ├── LocalFs — POSIX local filesystem │
|
||
│ ├── S3Vfs — S3-compatible storage (HTTP API) │
|
||
│ ├── SmbVfs — SMB client backend │
|
||
│ ├── CephVfs — Ceph RADOS backend (搁置) │
|
||
│ ├── EncryptedFs — Encryption layer │
|
||
│ ├── Compression — ZSTD/LZ4 compression layer │
|
||
│ ├── DedupFs — Block deduplication layer │
|
||
│ ├── RaidFs — RAID-Z emulation layer │
|
||
│ └─────────────────────────────────────────────────────────────────────┘
|
||
│ ↓ │
|
||
┌─────────────────────────────────────────────────────────────────────────┐
|
||
│ Ceph Storage Cluster (RADOS) │
|
||
│ ├── Monitor (MON) — Cluster map, authentication │
|
||
│ ├── OSD Daemons — Object storage (data replication) │
|
||
│ ├── Manager (MGR) — Dashboard, telemetry │
|
||
│ ├── MDS (optional) — CephFS metadata server │
|
||
│ ├── RGW (optional) — S3/Swift gateway │
|
||
│ └─────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## Library Analysis
|
||
|
||
### Rust Ceph Crates
|
||
|
||
| Crate | Version | Description | Platform |
|
||
|-------|---------|-------------|----------|
|
||
| `ceph` | 3.2.5 | Official librados FFI (sync) | Linux-only |
|
||
| `ceph-async` | 4.0.5 | Async librados FFI (futures 0.3) | Linux-only |
|
||
| `ceph-rbd` | 0.3.2 | RADOS Block Device bindings | Linux-only |
|
||
|
||
### ceph-async Module Structure
|
||
|
||
```
|
||
ceph_async::
|
||
├── CephClient — Admin operations (OSD/Pool/Mon commands)
|
||
├── rados:: — Low-level FFI bindings (100+ functions)
|
||
│ ├── rados_read/write/stat/remove — Object I/O
|
||
│ ├── rados_pool_create/delete/lookup — Pool management
|
||
│ ├── rados_ioctx_* — I/O context (pool handle)
|
||
│ ├── rados_snap_* — Snapshot management
|
||
│ ├── rados_lock_* — Distributed locking
|
||
│ ├── rados_aio_* — Async I/O
|
||
│ ├── rados_omap_* — Key-value store per object
|
||
│ └── rados_write_op_* / rados_read_op_* — Compound operations
|
||
├── completion:: — Async completion handling
|
||
├── read_stream:: — Async read stream
|
||
├── write_sink:: — Async write sink
|
||
└── list_stream:: — Async object listing
|
||
```
|
||
|
||
### CephClient API
|
||
|
||
```rust
|
||
let client = CephClient::new("admin", "/etc/ceph/ceph.conf")?;
|
||
|
||
// OSD operations
|
||
client.osd_tree()?; // Get OSD tree (CRUSH map)
|
||
client.osd_out(osd_id)?; // Mark OSD out
|
||
client.osd_crush_remove(osd_id)?; // Remove from CRUSH map
|
||
|
||
// Pool operations
|
||
client.osd_pool_get(pool, option)?; // Get pool config
|
||
client.osd_pool_set(pool, key, val)?; // Set pool config
|
||
client.osd_pool_quota_get(pool)?; // Get pool quota
|
||
|
||
// Cluster status
|
||
client.status()?; // Cluster health
|
||
client.mon_dump()?; // Monitor list
|
||
client.version()?; // Ceph version
|
||
```
|
||
|
||
---
|
||
|
||
## Implementation Phases
|
||
|
||
| Phase | Task | Code Lines | Priority | Risk | Dependencies |
|
||
|-------|------|------------|----------|------|--------------|
|
||
| **Phase 1** | CephVfs struct + basic I/O | ~400 | P0 | Medium ⚠️⚠️⚠️ | ceph-async crate |
|
||
| **Phase 2** | Pool management CLI | ~150 | P1 | Low ⚠️ | Phase 1 |
|
||
| **Phase 3** | Snapshot support | ~200 | P2 | Medium ⚠️⚠️⚠️ | librados snap API |
|
||
| **Phase 4** | Distributed locking | ~100 | P2 | Medium ⚠️⚠️⚠️ | librados lock API |
|
||
| **Phase 5** | OMAP key-value | ~150 | P3 | Low ⚠️ | librados omap API |
|
||
| **Phase 6** | Async integration | ~300 | P1 | High ⚠️⚠️⚠️⚠️ | async-vfs feature |
|
||
| **Phase 7** | Docker test environment | ~50 | P0 | Low ⚠️ | Docker compose |
|
||
| **Phase 8** | Performance benchmark | ~100 | P2 | Low ⚠️ | Benchmark scripts |
|
||
| **Total** | | **~1350** | | | |
|
||
|
||
---
|
||
|
||
## Phase 1: CephVfs Core Implementation
|
||
|
||
### Key Design Decisions
|
||
|
||
**1. Object vs File mapping**:
|
||
- RADOS is object storage (no directories)
|
||
- Path `/foo/bar.txt` → Object `foo/bar.txt` in pool
|
||
- Directories simulated via zero-byte objects with `/` suffix (like S3)
|
||
|
||
**2. Pool-per-share vs single pool**:
|
||
- Option A: Single pool + path prefix (simpler, less isolation)
|
||
- Option B: Pool-per-share (better isolation, quota per pool)
|
||
- **Recommend**: Option B (pool-per-share) for enterprise use
|
||
|
||
**3. I/O context caching**:
|
||
- Each pool requires separate `rados_ioctx_t`
|
||
- Cache ioctx per share to avoid recreation overhead
|
||
|
||
### CephVfs Struct (Draft)
|
||
|
||
```rust
|
||
pub struct CephVfs {
|
||
cluster: rados_t, // RADOS cluster handle
|
||
pool_name: String, // Pool name for this share
|
||
ioctx: rados_ioctx_t, // I/O context (cached)
|
||
root_prefix: String, // Path prefix within pool
|
||
}
|
||
|
||
pub struct CephVfsFile {
|
||
ioctx: rados_ioctx_t,
|
||
object_id: String, // Object name in pool
|
||
position: u64,
|
||
write_buffer: Vec<u8>, // Buffer for writes (flush on close)
|
||
size: u64,
|
||
}
|
||
```
|
||
|
||
### VfsBackend Method Mapping
|
||
|
||
| Method | RADOS equivalent | Complexity |
|
||
|--------|-----------------|------------|
|
||
| `read_dir()` | `rados_nobjects_list_*` | High (pagination) |
|
||
| `open_file()` | Custom (object ops) | Medium |
|
||
| `stat()` | `rados_stat()` | Low |
|
||
| `create_dir()` | `rados_write_full(0-byte)` | Low |
|
||
| `remove_dir()` | `rados_remove()` | Low |
|
||
| `remove_file()` | `rados_remove()` | Low |
|
||
| `rename()` | Custom (copy + delete) | Medium |
|
||
| `exists()` | `rados_stat()` | Low |
|
||
| `copy()` | `rados_clone_range()` | Low |
|
||
| `hard_link()` | `rados_clone_range()` | Low |
|
||
| `read_link()` | Unsupported | N/A |
|
||
| `create_symlink()` | Unsupported | N/A |
|
||
|
||
---
|
||
|
||
## Risk Assessment
|
||
|
||
| Risk | Level | Mitigation |
|
||
|------|-------|------------|
|
||
| **Linux-only** | ⚠️⚠️⚠️⚠️⚠️ Critical | Docker/VM for macOS; 不符合跨平台定位 |
|
||
| **librados.so symlink** | ⚠️⚠️⚠️ Medium | Document setup; CI check |
|
||
| **Pool-level snapshots** | ⚠️⚠️ Low | Document limitation; consider RGW |
|
||
| **Async overhead** | ⚠️⚠️⚠️ Medium | Benchmark; spawn_blocking wrapper |
|
||
| **Cluster complexity** | ⚠️⚠️⚠️⚠️⚠️ Critical | 超出 Lightweight 定位; Docker compose |
|
||
| **SMB Oplocks integration** | ⚠️⚠️⚠️ Medium | RADOS locking API; careful design |
|
||
|
||
---
|
||
|
||
## Alternatives (推荐方案)
|
||
|
||
### 方案对比
|
||
|
||
| 方案 | 跨平台 | 部署复杂度 | 定位匹配 | 状态 |
|
||
|------|--------|-----------|---------|------|
|
||
| **Ceph RADOS** | ❌ Linux-only | ⚠️⚠️⚠️⚠️⚠️ 极高 | ❌ 不匹配 | 搁置 |
|
||
| **Ceph RGW (S3)** | ✅ HTTP API | ⚠️⚠️⚠️⚠️ 高 | ⭐⭐⭐ 中等 | 已有 S3Vfs |
|
||
| **MinIO** | ✅ 全平台 | ⚠️⚠️ 低 | ⭐⭐⭐⭐⭐ 完全匹配 | 已有 S3Vfs |
|
||
| **GlusterFS** | ✅ POSIX | ⚠️⚠️⚠️ 中 | ⭐⭐⭐⭐ 高 | 待研究 |
|
||
| **内置分布式** | ✅ 全平台 | ⚠️⚠️ 低 | ⭐⭐⭐⭐⭐ 完全匹配 | 已有基础 |
|
||
|
||
### 方案 1: MinIO (推荐)
|
||
|
||
**优势**:
|
||
- ✅ S3-compatible API（已有 S3Vfs，无需新代码）
|
||
- ✅ 单节点部署（轻量级）
|
||
- ✅ 跨平台（macOS/Linux/Windows）
|
||
- ✅ 高性能（纠删码）
|
||
- ✅ 开源 + 企业版
|
||
|
||
**部署**:
|
||
```bash
|
||
# macOS 单节点
|
||
minio server /data --console-address ":9001"
|
||
|
||
# MarkBase 配置
|
||
MB_S3_ENDPOINT=http://localhost:9000
|
||
MB_S3_BUCKET=markbase
|
||
```
|
||
|
||
**集成**: 无需修改代码，S3Vfs 已支持。
|
||
|
||
---
|
||
|
||
### 方案 2: 内置分布式存储
|
||
|
||
**已有基础**:
|
||
| 功能 | 文件 | 分布式潜力 |
|
||
|------|------|-----------|
|
||
| DedupFs | dedup.rs | ✅ SHA-256 块存储可跨节点共享 |
|
||
| RaidFs | raid.rs | ⚠️ 单节点 RAID-Z |
|
||
| Send-Receive | send_receive.rs | ⚠️ 类似 ZFS send/receive |
|
||
| Checksum | checksum.rs | ✅ 数据完整性验证 |
|
||
| Compression | compression.rs | ✅ ZSTD 压缩 |
|
||
|
||
**扩展方向**:
|
||
1. DedupFs + S3Vfs: Dedup 块存储到 MinIO/S3（跨节点共享）
|
||
2. Checksum + Replication: 增加跨节点复制
|
||
3. Send-Receive + Remote: 增加远程 replication
|
||
|
||
---
|
||
|
||
## Technical Details
|
||
|
||
### librados API Functions
|
||
|
||
**Object I/O**:
|
||
- `rados_read(ioctx, oid, buf, len, offset)` — Read at offset
|
||
- `rados_write(ioctx, oid, buf, len, offset)` — Write at offset
|
||
- `rados_write_full(ioctx, oid, buf, len)` — Write entire object
|
||
- `rados_append(ioctx, oid, buf, len)` — Append to object
|
||
- `rados_stat(ioctx, oid, psize, pmtime)` — Get object size/mtime
|
||
- `rados_remove(ioctx, oid)` — Delete object
|
||
|
||
**Pool Operations**:
|
||
- `rados_pool_create(cluster, pool_name)` — Create pool
|
||
- `rados_pool_delete(cluster, pool_name)` — Delete pool
|
||
- `rados_pool_lookup(cluster, pool_name)` — Find pool ID
|
||
- `rados_ioctx_create(cluster, pool_name, ioctx)` — Create I/O context
|
||
|
||
**Snapshots**:
|
||
- `rados_ioctx_snap_create(ioctx, snap_name)` — Create pool snapshot
|
||
- `rados_ioctx_snap_list(ioctx, snaps)` — List snapshots
|
||
- `rados_ioctx_snap_remove(ioctx, snap_id)` — Delete snapshot
|
||
- `rados_ioctx_snap_rollback(ioctx, oid, snap_id)` — Rollback object
|
||
|
||
**Locking**:
|
||
- `rados_lock_exclusive(ioctx, oid, name, cookie, desc, duration, flags)` — Exclusive lock
|
||
- `rados_lock_shared(ioctx, oid, name, cookie, tag, desc, duration, flags)` — Shared lock
|
||
- `rados_unlock(ioctx, oid, name, cookie)` — Release lock
|
||
- `rados_list_lockers(ioctx, oid, name, ...)` — List lock holders
|
||
|
||
**OMAP (Key-Value)**:
|
||
- `rados_omap_set(ioctx, oid, map)` — Set key-value pairs
|
||
- `rados_omap_get(ioctx, oid, ...)` — Get values by keys
|
||
- `rados_omap_get_keys(ioctx, oid, ...)` — List keys
|
||
- `rados_omap_rm_keys(ioctx, oid, keys)` — Delete keys
|
||
|
||
**Async I/O**:
|
||
- `rados_aio_read(ioctx, oid, completion, buf, len, offset)` — Async read
|
||
- `rados_aio_write(ioctx, oid, completion, buf, len, offset)` — Async write
|
||
- `rados_aio_flush(ioctx)` — Flush pending async ops
|
||
- `rados_aio_wait_for_complete(completion)` — Wait for completion
|
||
|
||
---
|
||
|
||
## Open Questions
|
||
|
||
1. **部署目标**: Linux-only production vs macOS development?
|
||
2. **Backend choice**: RADOS (librados) vs RGW (S3 API)?
|
||
3. **Pool strategy**: Pool-per-share vs single pool + path prefix?
|
||
4. **SMB Oplocks**: Should CephVfs support SMB Oplocks via RADOS locking?
|
||
5. **Priority**: Start with basic I/O or full async integration first?
|
||
|
||
---
|
||
|
||
## Conclusion
|
||
|
||
**当前搁置 Ceph RADOS 集成**，原因：
|
||
1. ❌ Linux-only 约束不符合 macOS 跨平台定位
|
||
2. ⚠️ 部署复杂度超出 Lightweight 定位
|
||
3. ⚠️ 需要完整 Ceph 集群（Monitor + OSD + MGR）
|
||
|
||
**推荐替代方案**：
|
||
1. ⭐⭐⭐⭐⭐ **MinIO** — S3-compatible，已有 S3Vfs，轻量级
|
||
2. ⭐⭐⭐⭐⭐ **内置分布式** — DedupFs + S3Vfs 组合
|
||
|
||
**后续行动**：
|
||
- MinIO 集成文档（0 行代码）
|
||
- DedupFs + S3Vfs 组合研究（~100 行）
|
||
- 内置 Replication 功能（~400 行）
|
||
|
||
---
|
||
|
||
**文档创建**: 2026-06-25
|
||
**最后更新**: 2026-06-25 |