MarkBase Admin
ac75faa0cc
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
3.2 KiB
Batch Processing Analysis Report
Current Status
Test Results (E4B-MarkBase):
Single token: 29.7 ms/token ✓✓✓
Batch(2): 270.6 ms/token (9.1x SLOWER!)
Batch(4): 140.6 ms/token (4.7x SLOWER)
Batch(8): 76.3 ms/token (2.6x SLOWER)
Problem: Batch processing is significantly slower than single token processing.
Root Cause Analysis
1. Sequential Embedding Lookup
Current implementation (BatchGenerationTrue.swift:26-52):
for i in 0..<batchSize {
let embedCmdBuf = engine.commandQueue.makeCommandBuffer()!
try dequantizeRowOptimized(...)
embedCmdBuf.commit()
embedCmdBuf.waitUntilCompleted() // ← WAIT per token
memcpy(...)
}
Bottleneck: batchSize × waitUntilCompleted()
For batch(8): 8 waits for embedding alone!
2. Batch Embedding Kernel Attempt
Created kernel: dequantize_row_batch (MetalKernels.metal:1988-2019)
Status: ❌ CRASH (SIGSEGV - segmentation fault)
Reason: Memory access violation, needs debugging
Deferred: Using sequential approach for stability
3. Layer Processing
Current: Uses batch kernels (LayerBatch.swift)
Status: ✓✓✓ Working correctly
Performance: Unknown ( overshadowed by embedding bottleneck)
Performance Impact
Embedding bottleneck dominates:
Embedding: batchSize × ~5ms = 40ms for batch(8)
Layer processing: ~25ms
Total: 65ms+ → 76.3ms/token observed ✓
Without optimization: Batch is slower than single!
Optimization Priority
Phase 1: Fix Batch Embedding Kernel (CRITICAL)
Goal: Single GPU dispatch for entire batch
Current: 8 waits → Target: 1 wait
Expected impact:
- Embedding: 40ms → ~5ms (8x faster)
- Batch(8): 76ms → ~35ms (2x faster)
- Per-token: 35ms/8 = 4.4ms ✓✓✓
Status: ❌ Crash, needs debugging
Phase 2: Optimize Batch Layer Processing
Current: Batch kernels exist but performance unknown
Goal: Verify and optimize batch layer kernels
Expected: Additional 2-3x speedup
Phase 3: Model Loading Optimization
31B loading: 65 seconds
Goal: Parallel weight loading
Expected: 50% reduction (32s)
Lessons Learned
-
Batch processing ≠ automatic speedup
- Sequential operations in batch code kill performance
- Need true parallel GPU dispatch for all phases
-
Embedding is critical bottleneck
- Small operation but high overhead (multiple waits)
- Must be batched for effective performance
-
Kernel debugging is time-consuming
- SIGSEGV requires careful memory bounds checking
- Better to defer and use stable approach first
Next Steps
Immediate: Document findings, move to next optimization
Short-term:
- Debug batch embedding kernel (when time permits)
- Optimize model loading (higher ROI, easier)
Long-term:
- Metal kernel fusion
- SIMD expansion
- Expert caching
Conclusion
Batch processing currently SLOWER due to embedding bottleneck.
Key insight: Sequential waits in "batch" code defeat parallelism.
Recommendation: Focus on model loading optimization first (higher ROI, easier implementation), then revisit batch embedding kernel debugging.