Initial commit: E4B-MarkBase model integration with passing tests
CI / build-and-test (push) Has been cancelled
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
This commit is contained in:
MarkBase Admin
@@ -0,0 +1,433 @@
|
||||
# MarkBase-12B 支持的模型列表
|
||||
|
||||
## 模型架构支持
|
||||
|
||||
### 当前支持的模型类型
|
||||
|
||||
**Gemma-4 系列**:
|
||||
- ✓ Gemma-4 E4B (Early Access 4B)
|
||||
- ✓ Gemma-4 12B
|
||||
- ✓ E4B-MarkBase (multimodal variant)
|
||||
- ✓ MarkBase-12B (multimodal variant)
|
||||
|
||||
**模型架构**:
|
||||
- Gemma4ForConditionalGeneration (multimodal)
|
||||
- 42层 Transformer
|
||||
- 262,144 vocabulary size
|
||||
- Vision Tower (16 layers)
|
||||
- Audio Tower (12 layers)
|
||||
|
||||
---
|
||||
|
||||
## 模型加载方式
|
||||
|
||||
### 1. 从本地目录加载
|
||||
|
||||
**基本用法**:
|
||||
```bash
|
||||
swift run G12BServer <model_dir> <port> <model_id>
|
||||
```
|
||||
|
||||
**示例**:
|
||||
```bash
|
||||
# 加载 E4B-MarkBase
|
||||
swift run G12BServer /Users/accusys/MarkBase12B/models/E4B-MarkBase 8080 markbase-12b
|
||||
|
||||
# 加载其他模型
|
||||
swift run G12BServer /path/to/your/model 8080 custom-model
|
||||
```
|
||||
|
||||
**所需文件**:
|
||||
```
|
||||
model_dir/
|
||||
model.safetensors - 模型权重（4-bit quantized）
|
||||
model.safetensors.index.json - 权重索引（如果分片）
|
||||
config.json - 模型配置
|
||||
tokenizer.json - Tokenizer
|
||||
tokenizer_config.json - Tokenizer配置
|
||||
generation_config.json - 生成配置
|
||||
```
|
||||
|
||||
### 2. Safetensors 格式要求
|
||||
|
||||
**权重格式**:
|
||||
- Safetensors binary format
|
||||
- 4-bit quantization (uint32 packed)
|
||||
- Group size: 64
|
||||
- BF16 scales/biases
|
||||
|
||||
**支持的数据类型**:
|
||||
- INT4 (4-bit quantized)
|
||||
- BF16 (scales/biases)
|
||||
- F32 (activations)
|
||||
|
||||
### 3. 配置文件格式
|
||||
|
||||
**config.json 示例**:
|
||||
```json
|
||||
{
|
||||
"model_type": "gemma4",
|
||||
"architectures": ["Gemma4ForConditionalGeneration"],
|
||||
"hidden_size": 2560,
|
||||
"num_hidden_layers": 42,
|
||||
"vocab_size": 262144,
|
||||
"num_attention_heads": 8,
|
||||
"num_key_value_heads": 2,
|
||||
"head_dim": 256,
|
||||
"intermediate_size": 10240,
|
||||
"max_position_embeddings": 131072,
|
||||
"quantization_config": {
|
||||
"bits": 4,
|
||||
"group_size": 64
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Vision 配置**:
|
||||
```json
|
||||
{
|
||||
"vision_config": {
|
||||
"hidden_size": 768,
|
||||
"num_hidden_layers": 16,
|
||||
"patch_size": 16,
|
||||
"image_size": 224
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Audio 配置**:
|
||||
```json
|
||||
{
|
||||
"audio_config": {
|
||||
"hidden_size": 640,
|
||||
"num_hidden_layers": 12,
|
||||
"num_mel_bins": 128
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 模型选择指南
|
||||
|
||||
### 根据 Use Case 选择
|
||||
|
||||
#### 1. 纯文本推理
|
||||
**推荐模型**: Gemma-4-4B-IT (Instruction Tuned)
|
||||
- 适用: 文本生成、对话、问答
|
||||
- 不需要: Vision/Audio components
|
||||
- 性能: 更快（无 multimodal overhead）
|
||||
|
||||
#### 2. 视觉理解
|
||||
**推荐模型**: E4B-MarkBase, MarkBase-12B
|
||||
- 适用: 图像描述、视觉问答、场景理解
|
||||
- 需要: Vision Tower (16 layers)
|
||||
- 输入: 224x224 RGB images
|
||||
|
||||
#### 3. 音频理解
|
||||
**推荐模型**: MarkBase-12B (with audio tower)
|
||||
- 适用: 音频描述、语音识别、音频问答
|
||||
- 需要: Audio Tower (12 layers)
|
||||
- 输入: Mel spectrograms (128 bands)
|
||||
|
||||
#### 4. 多模态推理
|
||||
**推荐模型**: E4B-MarkBase, MarkBase-12B
|
||||
- 适用: Vision + Audio + Text
|
||||
- 需要: Vision + Audio Towers
|
||||
- 输入: Images + Audio + Text
|
||||
|
||||
#### 5. 分布式推理
|
||||
**推荐模型**: 12B variants
|
||||
- 适用: 跨设备推理、高性能
|
||||
- 需要: RDMA setup (Thunderbolt 5)
|
||||
- 性能: 658 tok/s (distributed)
|
||||
|
||||
---
|
||||
|
||||
## 模型规格对比
|
||||
|
||||
### Gemma-4 模型系列
|
||||
|
||||
| 模型 | 参数量 | Layers | Hidden Size | Vocab | Vision | Audio |
|
||||
|------|--------|--------|-------------|-------|--------|-------|
|
||||
| Gemma-4-4B | 4B | 42 | 2560 | 262K | - | - |
|
||||
| Gemma-4-12B | 12B | 42 | 3072 | 262K | - | - |
|
||||
| E4B-MarkBase | ~12B | 42 | 2560 | 262K | ✓ (16) | ✓ (12) |
|
||||
| MarkBase-12B | ~12B | 42 | 3072? | 262K | ✓ (16) | ✓ (12) |
|
||||
|
||||
### 量化规格
|
||||
|
||||
| 量化类型 | Bits | Group Size | 压缩比 | 精度损失 |
|
||||
|----------|------|------------|--------|----------|
|
||||
| INT4 | 4 | 64 | 8x | Minimal |
|
||||
| BF16 | 16 | - | 2x | None |
|
||||
| F32 | 32 | - | 1x | None |
|
||||
|
||||
---
|
||||
|
||||
## 模型转换指南
|
||||
|
||||
### 从 HuggingFace 转换
|
||||
|
||||
**步骤**:
|
||||
1. Download original model
|
||||
2. Quantize to 4-bit (if needed)
|
||||
3. Convert to safetensors format
|
||||
4. Organize config files
|
||||
5. Load with MarkBase-12B
|
||||
|
||||
**工具**:
|
||||
- `safetensors` Python library
|
||||
- `transformers` (for config)
|
||||
- Custom quantization scripts
|
||||
|
||||
**示例转换脚本**:
|
||||
```python
|
||||
from safetensors.torch import save_file
|
||||
from transformers import AutoModelForCausalLM
|
||||
|
||||
# Load original model
|
||||
model = AutoModelForCausalLM.from_pretrained("model_name")
|
||||
|
||||
# Quantize (custom implementation)
|
||||
quantized_model = quantize_model(model, bits=4, group_size=64)
|
||||
|
||||
# Save as safetensors
|
||||
save_file(quantized_model.state_dict(), "model.safetensors")
|
||||
```
|
||||
|
||||
### 模型文件组织
|
||||
|
||||
**目录结构**:
|
||||
```
|
||||
model_dir/
|
||||
├── model.safetensors (or model-00001-of-00002.safetensors)
|
||||
├── model.safetensors.index.json (if sharded)
|
||||
├── config.json
|
||||
├── tokenizer.json
|
||||
├── tokenizer_config.json
|
||||
├── generation_config.json
|
||||
├── processor_config.json (for multimodal)
|
||||
└── chat_template.jinja (optional)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 自定义模型支持
|
||||
|
||||
### 添加新模型
|
||||
|
||||
**要求**:
|
||||
1. Gemma-4 architecture family
|
||||
2. 4-bit quantized weights
|
||||
3. Safetensors format
|
||||
4. Compatible config.json
|
||||
|
||||
**修改代码**:
|
||||
```swift
|
||||
// Sources/G12B/Model.swift
|
||||
// Adjust architecture parameters if needed
|
||||
|
||||
public init(modelDir: String, engine: MarkBaseEngine, maxContextLength: Int) throws {
|
||||
// Load custom config
|
||||
let config = try loadConfig(modelDir)
|
||||
|
||||
// Initialize based on config
|
||||
self.numHiddenLayers = config.num_hidden_layers
|
||||
self.hiddenSize = config.hidden_size
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
### 模型配置适配
|
||||
|
||||
**config.json 适配器**:
|
||||
```swift
|
||||
struct ModelConfig: Codable {
|
||||
let model_type: String
|
||||
let architectures: [String]
|
||||
let hidden_size: Int
|
||||
let num_hidden_layers: Int
|
||||
let vocab_size: Int
|
||||
|
||||
// Optional: Vision config
|
||||
let vision_config: VisionConfig?
|
||||
|
||||
// Optional: Audio config
|
||||
let audio_config: AudioConfig?
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 模型性能对比
|
||||
|
||||
### 单设备性能
|
||||
|
||||
| 模型 | 推理速度 | 内存占用 | 启动时间 |
|
||||
|------|----------|----------|----------|
|
||||
| 4B | ~50 tok/s | ~2GB | ~30s |
|
||||
| 12B | ~30 tok/s | ~4GB | ~90s |
|
||||
| E4B-MarkBase | ~25 tok/s | ~6GB | ~90s |
|
||||
|
||||
### 分布式性能
|
||||
|
||||
| 模型 | Distributed | Bandwidth | Latency |
|
||||
|------|-------------|-----------|---------|
|
||||
| 12B | 658 tok/s | 5761 MB/s | Low |
|
||||
|
||||
---
|
||||
|
||||
## 模型限制
|
||||
|
||||
### 当前限制
|
||||
|
||||
1. **架构限制**:
|
||||
- 仅支持 Gemma-4 family
|
||||
- 需要 4-bit quantization
|
||||
- Safetensors format only
|
||||
|
||||
2. **配置要求**:
|
||||
- 必须有完整的 config.json
|
||||
- Tokenizer 文件必需
|
||||
- Quantization config 需要
|
||||
|
||||
3. **Multimodal限制**:
|
||||
- Vision Tower 需要 safetensors 中的权重
|
||||
- Audio Tower 需要特定架构
|
||||
- 测试时 output quality 需验证
|
||||
|
||||
### 未来扩展
|
||||
|
||||
**计划支持**:
|
||||
- 其他架构（LLaMA, Mistral, etc）
|
||||
- 8-bit quantization
|
||||
- FP16 weights
|
||||
- 更多 tokenizer 格式
|
||||
|
||||
---
|
||||
|
||||
## 推荐模型来源
|
||||
|
||||
### HuggingFace Models
|
||||
|
||||
**Gemma-4 相关**:
|
||||
- `google/gemma-4-4b-it` (instruction tuned)
|
||||
- `google/gemma-4-12b-it`
|
||||
- Custom variants (MarkBase, etc)
|
||||
|
||||
**下载方法**:
|
||||
```bash
|
||||
# Using huggingface-cli
|
||||
huggingface-cli download model_name --local-dir ./model
|
||||
|
||||
# Using Python
|
||||
from huggingface_hub import snapshot_download
|
||||
snapshot_download("model_name", local_dir="./model")
|
||||
```
|
||||
|
||||
### 本地模型
|
||||
|
||||
**自定义训练模型**:
|
||||
- 训练后转换为 safetensors
|
||||
- 量化到 4-bit
|
||||
- 组织配置文件
|
||||
- 加载测试
|
||||
|
||||
---
|
||||
|
||||
## 使用示例
|
||||
|
||||
### 选择并加载模型
|
||||
|
||||
**E4B-MarkBase (Multimodal)**:
|
||||
```bash
|
||||
swift run G12BServer /models/E4B-MarkBase 8080 markbase
|
||||
|
||||
curl -X POST http://localhost:8080/v1/multimodal/chat/completions \
|
||||
-d '{"messages":[{"role":"user","content":[{"type":"text","text":"Describe"},{"type":"image_url","image_url":{"url":"data:image/png;base64,..."}}]}]}'
|
||||
```
|
||||
|
||||
**Gemma-4-4B-IT (Text-only)**:
|
||||
```bash
|
||||
swift run G12BServer /models/gemma-4-4b-it 8080 gemma-4b
|
||||
|
||||
curl -X POST http://localhost:8080/v1/chat/completions \
|
||||
-d '{"messages":[{"role":"user","content":"Hello"}]}'
|
||||
```
|
||||
|
||||
**Custom Model**:
|
||||
```bash
|
||||
swift run G12BServer /models/my-custom-model 8080 custom
|
||||
|
||||
# Test if architecture is compatible
|
||||
swift test --filter testModelLoading
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 故障排除
|
||||
|
||||
### 模型加载失败
|
||||
|
||||
**常见错误**:
|
||||
```
|
||||
Error: Model not found
|
||||
→ Check model_dir path is correct
|
||||
|
||||
Error: Config not found
|
||||
→ Ensure config.json exists
|
||||
|
||||
Error: Unsupported architecture
|
||||
→ Check model_type in config.json
|
||||
|
||||
Error: Quantization mismatch
|
||||
→ Verify bits=4, group_size=64
|
||||
```
|
||||
|
||||
### 配置检查
|
||||
|
||||
**验证配置**:
|
||||
```bash
|
||||
# Check config.json
|
||||
jq '.' model_dir/config.json
|
||||
|
||||
# Verify architecture
|
||||
jq '.model_type, .architectures' model_dir/config.json
|
||||
|
||||
# Check quantization
|
||||
jq '.quantization_config' model_dir/config.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 总结
|
||||
|
||||
**支持的模型类型**:
|
||||
- ✓ Gemma-4 family (E4B, 12B, MarkBase)
|
||||
- ✓ 4-bit quantized safetensors
|
||||
- ✓ Multimodal (Vision + Audio)
|
||||
|
||||
**选择建议**:
|
||||
- 纯文本: Gemma-4-4B-IT
|
||||
- 视觉理解: E4B-MarkBase
|
||||
- 音频理解: MarkBase-12B
|
||||
- 分布式: 12B variants
|
||||
|
||||
**加载方法**:
|
||||
```bash
|
||||
swift run G12BServer <model_dir> <port> <model_id>
|
||||
```
|
||||
|
||||
**要求**:
|
||||
- Safetensors weights
|
||||
- Complete config files
|
||||
- 4-bit quantization
|
||||
- Gemma-4 architecture
|
||||
|
||||
---
|
||||
|
||||
**文档生成**: June 19, 2026
|
||||
**支持模型**: Gemma-4 Family
|
||||
**格式要求**: Safetensors + Config
|
||||
|
||||
Reference in New Issue
Block a user