MarkBase Admin
ac75faa0cc
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
112 lines
2.6 KiB
Markdown
112 lines
2.6 KiB
Markdown
# MarkBase-12B Swift Metal Inference Engine
|
|
|
|
## 快速开始
|
|
|
|
### 构建
|
|
```bash
|
|
cd /Users/accusys/MarkBase12B
|
|
swift build
|
|
```
|
|
|
|
### 测试
|
|
```bash
|
|
swift test
|
|
swift test --filter E4BSimpleInferenceTest.testTokenizerEncoding # Tokenizer测试
|
|
swift test --filter E4BSimpleInferenceTest.testMultimodalVisionInference # Multimodal测试
|
|
```
|
|
|
|
### 运行服务器（开发中）
|
|
```bash
|
|
swift run G12BServer /path/to/model 8080 markbase-e4b
|
|
```
|
|
|
|
## API Endpoints
|
|
|
|
### 文本生成
|
|
```
|
|
POST /v1/chat/completions
|
|
{
|
|
"model": "markbase-e4b",
|
|
"messages": [{"role": "user", "content": "Hello"}],
|
|
"max_tokens": 100
|
|
}
|
|
```
|
|
|
|
### 多模态生成
|
|
```
|
|
POST /v1/multimodal/chat/completions
|
|
{
|
|
"model": "markbase-e4b",
|
|
"messages": [{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "text", "text": "Describe this image"},
|
|
{"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
|
|
]
|
|
}],
|
|
"max_tokens": 100
|
|
}
|
|
```
|
|
|
|
## 性能指标
|
|
|
|
| Metric | Value |
|
|
|--------|-------|
|
|
| RDMA带宽 | 5761 MB/s (Thunderbolt 5) |
|
|
| POC吞吐 | 658 tokens/s (分布式) |
|
|
| Embedding验证 | Swift = Python精确匹配 |
|
|
|
|
## 架构
|
|
|
|
```
|
|
Swift Metal Engine
|
|
├── MarkBaseEngine (Metal kernels)
|
|
├── E4BModel (42 layers)
|
|
├── BPETokenizer (sentencepiece)
|
|
├── MultimodalModel
|
|
│ ├── VisionTower (16 layers)
|
|
│ ├── AudioTower (12 layers)
|
|
│ ├── Vision preprocessing
|
|
│ ├── Pooling (196→1)
|
|
│ └── Normalization
|
|
└── MarkBaseServer (API handlers)
|
|
```
|
|
|
|
## 文件结构
|
|
|
|
```
|
|
Sources/
|
|
├── G12B/ # 核心库
|
|
│ ├── Metal/ # Metal kernels
|
|
│ ├── Tokenizer/ # Tokenizer
|
|
│ ├── Sampling/ # Sampling strategies
|
|
│ ├── Vision/ # Vision tower
|
|
│ ├── Audio/ # Audio tower
|
|
│ ├── Generator/ # Streaming generator
|
|
│ └── Model.swift # Main model
|
|
│
|
|
├── G12BServer/ # API服务器
|
|
│ ├── MarkBaseServer.swift # Main server
|
|
│ ├── MultimodalAPI.swift # Multimodal types
|
|
│ ├── ModelsAPI.swift # API models
|
|
│ └── Errors.swift # Error handling
|
|
│
|
|
└── Tests/
|
|
└── E4BSimpleInferenceTest.swift # 测试
|
|
```
|
|
|
|
## 限制说明
|
|
|
|
**E4B-MarkBase 是 Gemma4ForConditionalGeneration (multimodal)**
|
|
- 纯文本生成产生随机输出
|
|
- 需要 vision/audio conditioning
|
|
- 这是模型架构特性，不是bug
|
|
|
|
## 下一步
|
|
|
|
1. HTTP服务器集成 (Hummingbird)
|
|
2. Python参考验证
|
|
3. Audio预处理
|
|
4. 性能优化
|
|
|