Distributed storage research: Ceph (shelved) + MinIO guide + DedupS3 design
This commit is contained in:
90
AGENTS.md
90
AGENTS.md
@@ -4861,3 +4861,93 @@ let signing_key = match dialect {
|
||||
**MS-SMB2 specification**:
|
||||
- §3.1.4.1: Signing key derivation per dialect
|
||||
- §3.1.4.2: SP 800-108 KDF for SMB 3.x
|
||||
|
||||
---
|
||||
|
||||
## Distributed Storage Research(2026-06-25)
|
||||
|
||||
### Ceph RADOS Integration(搁置)
|
||||
|
||||
**文档**: `docs/CEPH_INTEGRATION_ANALYSIS.md`(340 行)
|
||||
|
||||
**结论**: 不符合 MarkBase macOS 跨平台定位
|
||||
- ❌ Linux-only(librados.so FFI)
|
||||
- ⚠️ 需完整 Ceph 集群(Monitor + OSD + MGR)
|
||||
- ⚠️ 部署复杂度超出 Lightweight 定位
|
||||
|
||||
**推荐替代方案**:
|
||||
1. MinIO — S3-compatible,已有 S3Vfs 支持
|
||||
2. 内置分布式 — DedupFs + S3Vfs 组合
|
||||
|
||||
---
|
||||
|
||||
### MinIO Integration(推荐 ⭐⭐⭐⭐⭐)
|
||||
|
||||
**文档**: `docs/MINIO_INTEGRATION.md`(200 行)
|
||||
|
||||
**优势**:
|
||||
- ✅ S3-compatible API(已有 S3Vfs,无需修改代码)
|
||||
- ✅ 跨平台(macOS/Linux/Windows)
|
||||
- ✅ 轻量级部署(单节点即可)
|
||||
- ✅ 高性能(纠删码 + 分布式扩展)
|
||||
|
||||
**部署方式**:
|
||||
```bash
|
||||
# macOS 单节点
|
||||
brew install minio/stable/minio
|
||||
minio server /data --console-address ":9001"
|
||||
|
||||
# Docker
|
||||
docker run minio/minio server /data --console-address ":9001"
|
||||
```
|
||||
|
||||
**集成方式**: 环境变量 + CLI 参数
|
||||
```bash
|
||||
export MB_S3_ENDPOINT=http://localhost:9000
|
||||
export MB_S3_BUCKET=markbase
|
||||
export MB_S3_ACCESS_KEY=minioadmin
|
||||
export MB_S3_SECRET_KEY=minioadmin
|
||||
```
|
||||
|
||||
**支持功能**:
|
||||
- Versioning(替代 Snapshot)
|
||||
- Bucket Policy(ACL 管理)
|
||||
- Lifecycle Rules(Backup 清理)
|
||||
- Erasure Coding(自动容错)
|
||||
|
||||
---
|
||||
|
||||
### DedupFs + S3Vfs Combination(设计 ⭐⭐⭐⭐)
|
||||
|
||||
**文档**: `docs/DEDUP_S3_COMBINATION.md`(320 行)
|
||||
|
||||
**目标**: 分布式 dedup 存储(跨节点共享 dedup 块)
|
||||
|
||||
**架构**:
|
||||
```
|
||||
MarkBase Node A → DedupS3Store → MinIO Cluster
|
||||
↓
|
||||
MarkBase Node B → DedupS3Store → 共享 Node A 的块
|
||||
```
|
||||
|
||||
**核心设计**:
|
||||
- 块存储到 S3 对象(SHA-256 hash 作为 key)
|
||||
- 引用计数存储到 S3 object metadata
|
||||
- Manifest 存储到 S3 对象(JSON 格式)
|
||||
|
||||
**实现工作量**: ~1000 行(7 phases)
|
||||
|
||||
**关键技术挑战**:
|
||||
- 引用计数非原子操作(需 versioning 或分布式锁)
|
||||
- 网络延迟 overhead(~5-10ms per block vs ~0.1ms local)
|
||||
- Dedup ratio 因文件类型差异(VM images ~80%, photos ~5%)
|
||||
|
||||
**下一步**:
|
||||
1. Phase 1: DedupS3Store struct + basic I/O(~300 行)
|
||||
2. Phase 2: CLI integration(~100 行)
|
||||
3. Phase 3: Performance benchmark
|
||||
|
||||
---
|
||||
|
||||
**最后更新**: 2026-06-25
|
||||
**版本**: 1.61(分布式存储研究完成)
|
||||
|
||||
Reference in New Issue
Block a user