MarkBase Admin
ac75faa0cc
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
2.6 KiB
2.6 KiB
MarkBase-12B Swift Metal Inference Engine
快速开始
构建
cd /Users/accusys/MarkBase12B
swift build
测试
swift test
swift test --filter E4BSimpleInferenceTest.testTokenizerEncoding # Tokenizer测试
swift test --filter E4BSimpleInferenceTest.testMultimodalVisionInference # Multimodal测试
运行服务器（开发中）
swift run G12BServer /path/to/model 8080 markbase-e4b
API Endpoints
文本生成
POST /v1/chat/completions
{
"model": "markbase-e4b",
"messages": [{"role": "user", "content": "Hello"}],
"max_tokens": 100
}
多模态生成
POST /v1/multimodal/chat/completions
{
"model": "markbase-e4b",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Describe this image"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
]
}],
"max_tokens": 100
}
性能指标
| Metric | Value |
|---|---|
| RDMA带宽 | 5761 MB/s (Thunderbolt 5) |
| POC吞吐 | 658 tokens/s (分布式) |
| Embedding验证 | Swift = Python精确匹配 |
架构
Swift Metal Engine
├── MarkBaseEngine (Metal kernels)
├── E4BModel (42 layers)
├── BPETokenizer (sentencepiece)
├── MultimodalModel
│ ├── VisionTower (16 layers)
│ ├── AudioTower (12 layers)
│ ├── Vision preprocessing
│ ├── Pooling (196→1)
│ └── Normalization
└── MarkBaseServer (API handlers)
文件结构
Sources/
├── G12B/ # 核心库
│ ├── Metal/ # Metal kernels
│ ├── Tokenizer/ # Tokenizer
│ ├── Sampling/ # Sampling strategies
│ ├── Vision/ # Vision tower
│ ├── Audio/ # Audio tower
│ ├── Generator/ # Streaming generator
│ └── Model.swift # Main model
│
├── G12BServer/ # API服务器
│ ├── MarkBaseServer.swift # Main server
│ ├── MultimodalAPI.swift # Multimodal types
│ ├── ModelsAPI.swift # API models
│ └── Errors.swift # Error handling
│
└── Tests/
└── E4BSimpleInferenceTest.swift # 测试
限制说明
E4B-MarkBase 是 Gemma4ForConditionalGeneration (multimodal)
- 纯文本生成产生随机输出
- 需要 vision/audio conditioning
- 这是模型架构特性，不是bug
下一步
- HTTP服务器集成 (Hummingbird)
- Python参考验证
- Audio预处理
- 性能优化