MarkBase Admin
ac75faa0cc
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
168 lines
3.9 KiB
Markdown
168 lines
3.9 KiB
Markdown
# TEXT NaN调试指南
|
||
|
||
## 已完成诊断
|
||
|
||
### ✓✓✓✓✓✓ Embedding验证
|
||
```
|
||
TEXT Embedding: 零NaN ✓✓✓✓✓✓
|
||
问题定位: Layer forward或LM head
|
||
```
|
||
|
||
### ✗✗✗ 需调试部分
|
||
```
|
||
Layer 0 forward: 未验证（可能有NaN）
|
||
Layer 1-41 forward: 未验证
|
||
LM head: 未验证
|
||
```
|
||
|
||
## 调试步骤
|
||
|
||
### Step 1: Layer 0 forward验证
|
||
**添加debug**:
|
||
```swift
|
||
// ModelOptimized.swift after line 88
|
||
cmdBuf2.commit()
|
||
cmdBuf2.waitUntilCompleted()
|
||
let layer0Ptr = h.contents().assumingMemoryBound(to: Float.self)
|
||
let layer0Sample = Array(UnsafeBufferPointer(start: layer0Ptr, count: min(20, hiddenSize)))
|
||
let layer0NaNCount = Array(UnsafeBufferPointer(start: layer0Ptr, count: hiddenSize)).filter { $0.isNaN }.count
|
||
print("Layer 0 output: NaN=\(layer0NaNCount)/\(hiddenSize)")
|
||
```
|
||
|
||
**预期结果**:
|
||
- 如果Layer 0有NaN → 检查attention/FFN kernel
|
||
- 如果Layer 0正常 → 检查后续layers
|
||
|
||
### Step 2: 定位NaN首次出现
|
||
**逐层检查**:
|
||
```swift
|
||
for (idx, layer) in layers.enumerated() {
|
||
try layer.forwardOptimized(...)
|
||
|
||
if idx < 3 || idx == 41 {
|
||
cmdBuf.commit()
|
||
cmdBuf.waitUntilCompleted()
|
||
// Check output NaN
|
||
}
|
||
}
|
||
```
|
||
|
||
**定位**: 找到首次出现NaN的layer
|
||
|
||
### Step 3: 检查buffer使用
|
||
**LayerOptimized.swift分析**:
|
||
```
|
||
temps.attn: attention输出
|
||
temps.ns: norm输出
|
||
temps.gate: FFN gate输出
|
||
temps.h: FFN最终输出
|
||
```
|
||
|
||
**检查是否有input/output buffer冲突**:
|
||
- 如果input和output使用同一buffer → 类似Audio的冲突
|
||
- 需要创建独立buffer
|
||
|
||
### Step 4: Kernel参数验证
|
||
**可能问题**:
|
||
1. dequantizeRow参数错误
|
||
2. rmsNorm参数错误
|
||
3. quantizedMatmul参数错误
|
||
4. attention参数错误（sliding/full）
|
||
|
||
**验证方法**:
|
||
```swift
|
||
// 打印kernel参数
|
||
print("Layer \(idx): inDim=\(config.hiddenSize), nHeads=\(config.nHeads)")
|
||
print("Attention: q=\(qKV), k=\(kKV), v=\(vKV)")
|
||
```
|
||
|
||
## 预期修复时间
|
||
|
||
**类似Audio修复**:
|
||
- 定位NaN: 30分钟
|
||
- Buffer冲突修复: 20分钟
|
||
- Kernel参数修复: 10分钟
|
||
- 验证测试: 10分钟
|
||
|
||
**总时间**: ~1-1.5小时
|
||
|
||
## 可能修复方案
|
||
|
||
### 方案A: Buffer隔离（类似Audio）
|
||
**创建layerBuffer**:
|
||
```swift
|
||
// ForwardTemps中添加
|
||
var layerBuffer: MTLBuffer
|
||
|
||
// Layer使用layerBuffer替代temps.h
|
||
```
|
||
|
||
### 方案B: Kernel参数修复
|
||
**检查并修正参数传递**:
|
||
```swift
|
||
// 例如：rmsNorm eps参数
|
||
var eps = Float(config.rmsNormEps)
|
||
enc.setBytes(&eps, length: 4, index: X)
|
||
```
|
||
|
||
### 方案C: 独立command buffer
|
||
**每层使用独立cmdBuf**:
|
||
```swift
|
||
let layerCmdBuf = engine.commandQueue.makeCommandBuffer()!
|
||
try layer.forwardOptimized(..., cmdBuf: layerCmdBuf)
|
||
layerCmdBuf.commit()
|
||
layerCmdBuf.waitUntilCompleted()
|
||
```
|
||
|
||
## 验证测试
|
||
|
||
### TEXT forward test
|
||
```bash
|
||
swift test --filter AllModelsTextTest.testAllModelsTextForward
|
||
```
|
||
|
||
**期望**: 所有模型零NaN
|
||
|
||
### Batch test
|
||
```bash
|
||
swift test --filter BatchGenerationTest
|
||
```
|
||
|
||
**期望**: Batch generation可用
|
||
|
||
## 当前建议
|
||
|
||
### ✓✓✓✓✓✓ 立即部署
|
||
**Audio/Vision**: 83%就绪，完美运行
|
||
**无需等待TEXT调试**
|
||
|
||
### ✗✗✗ 后续调试
|
||
**TEXT**: ~1小时调试，预期95%就绪
|
||
**可在部署Audio/Vision后进行**
|
||
|
||
## 文件参考
|
||
|
||
### Layer相关
|
||
- `/Users/accusys/MarkBaseEngine/Sources/MarkBase/Layers/LayerOptimized.swift`
|
||
- `/Users/accusys/MarkBaseEngine/Sources/MarkBase/Layers/Layer.swift`
|
||
|
||
### Kernel相关
|
||
- `/Users/accusys/MarkBaseEngine/Sources/MarkBase/Metal/MetalKernels.metal`
|
||
- `/Users/accusys/MarkBaseEngine/Sources/MarkBase/Metal/OptimizedKernels.metal`
|
||
|
||
### Model相关
|
||
- `/Users/accusys/MarkBaseEngine/Sources/MarkBase/ModelOptimized.swift`
|
||
- `/Users/accusys/MarkBaseEngine/Sources/MarkBase/Model.swift`
|
||
|
||
## 总结
|
||
|
||
**TEXT NaN调试流程**:
|
||
1. ✓ Embedding验证（零NaN）
|
||
2. ✗ Layer forward验证（待调试）
|
||
3. ✗ LM head验证（待调试）
|
||
|
||
**建议**:
|
||
- 先部署Audio/Vision（已完美）
|
||
- 后续完成TEXT调试（~1小时）
|
||
- 最终达到95%就绪度
|