MarkBase Admin
ac75faa0cc
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
130 lines
3.2 KiB
Markdown
130 lines
3.2 KiB
Markdown
# Batch Processing Analysis Report
|
||
|
||
## Current Status
|
||
|
||
**Test Results** (E4B-MarkBase):
|
||
|
||
```
|
||
Single token: 29.7 ms/token ✓✓✓
|
||
Batch(2): 270.6 ms/token (9.1x SLOWER!)
|
||
Batch(4): 140.6 ms/token (4.7x SLOWER)
|
||
Batch(8): 76.3 ms/token (2.6x SLOWER)
|
||
```
|
||
|
||
**Problem**: Batch processing is **significantly slower** than single token processing.
|
||
|
||
## Root Cause Analysis
|
||
|
||
### 1. Sequential Embedding Lookup
|
||
|
||
**Current implementation** (BatchGenerationTrue.swift:26-52):
|
||
|
||
```swift
|
||
for i in 0..<batchSize {
|
||
let embedCmdBuf = engine.commandQueue.makeCommandBuffer()!
|
||
try dequantizeRowOptimized(...)
|
||
embedCmdBuf.commit()
|
||
embedCmdBuf.waitUntilCompleted() // ← WAIT per token
|
||
memcpy(...)
|
||
}
|
||
```
|
||
|
||
**Bottleneck**: batchSize × waitUntilCompleted()
|
||
|
||
For batch(8): **8 waits** for embedding alone!
|
||
|
||
### 2. Batch Embedding Kernel Attempt
|
||
|
||
**Created kernel**: `dequantize_row_batch` (MetalKernels.metal:1988-2019)
|
||
|
||
**Status**: ❌ CRASH (SIGSEGV - segmentation fault)
|
||
|
||
**Reason**: Memory access violation, needs debugging
|
||
|
||
**Deferred**: Using sequential approach for stability
|
||
|
||
### 3. Layer Processing
|
||
|
||
**Current**: Uses batch kernels (LayerBatch.swift)
|
||
|
||
**Status**: ✓✓✓ Working correctly
|
||
|
||
**Performance**: Unknown ( overshadowed by embedding bottleneck)
|
||
|
||
## Performance Impact
|
||
|
||
**Embedding bottleneck dominates**:
|
||
|
||
```
|
||
Embedding: batchSize × ~5ms = 40ms for batch(8)
|
||
Layer processing: ~25ms
|
||
Total: 65ms+ → 76.3ms/token observed ✓
|
||
```
|
||
|
||
**Without optimization**: Batch is **slower** than single!
|
||
|
||
## Optimization Priority
|
||
|
||
### Phase 1: Fix Batch Embedding Kernel (CRITICAL)
|
||
|
||
**Goal**: Single GPU dispatch for entire batch
|
||
|
||
**Current**: 8 waits → Target: 1 wait
|
||
|
||
**Expected impact**:
|
||
- Embedding: 40ms → ~5ms (8x faster)
|
||
- Batch(8): 76ms → ~35ms (2x faster)
|
||
- Per-token: 35ms/8 = 4.4ms ✓✓✓
|
||
|
||
**Status**: ❌ Crash, needs debugging
|
||
|
||
### Phase 2: Optimize Batch Layer Processing
|
||
|
||
**Current**: Batch kernels exist but performance unknown
|
||
|
||
**Goal**: Verify and optimize batch layer kernels
|
||
|
||
**Expected**: Additional 2-3x speedup
|
||
|
||
### Phase 3: Model Loading Optimization
|
||
|
||
**31B loading**: 65 seconds
|
||
|
||
**Goal**: Parallel weight loading
|
||
|
||
**Expected**: 50% reduction (32s)
|
||
|
||
## Lessons Learned
|
||
|
||
1. **Batch processing ≠ automatic speedup**
|
||
- Sequential operations in batch code kill performance
|
||
- Need true parallel GPU dispatch for all phases
|
||
|
||
2. **Embedding is critical bottleneck**
|
||
- Small operation but high overhead (multiple waits)
|
||
- Must be batched for effective performance
|
||
|
||
3. **Kernel debugging is time-consuming**
|
||
- SIGSEGV requires careful memory bounds checking
|
||
- Better to defer and use stable approach first
|
||
|
||
## Next Steps
|
||
|
||
**Immediate**: Document findings, move to next optimization
|
||
|
||
**Short-term**:
|
||
1. Debug batch embedding kernel (when time permits)
|
||
2. Optimize model loading (higher ROI, easier)
|
||
|
||
**Long-term**:
|
||
1. Metal kernel fusion
|
||
2. SIMD expansion
|
||
3. Expert caching
|
||
|
||
## Conclusion
|
||
|
||
**Batch processing currently SLOWER** due to embedding bottleneck.
|
||
|
||
**Key insight**: Sequential waits in "batch" code defeat parallelism.
|
||
|
||
**Recommendation**: Focus on model loading optimization first (higher ROI, easier implementation), then revisit batch embedding kernel debugging. |