# Batch Processing Analysis Report

## Current Status

**Test Results** (E4B-MarkBase):

```
Single token: 29.7 ms/token ✓✓✓
Batch(2):     270.6 ms/token (9.1x SLOWER!)
Batch(4):     140.6 ms/token (4.7x SLOWER)
Batch(8):     76.3 ms/token  (2.6x SLOWER)
```

**Problem**: Batch processing is **significantly slower** than single token processing.

## Root Cause Analysis

### 1. Sequential Embedding Lookup

**Current implementation** (BatchGenerationTrue.swift:26-52):

```swift
for i in 0..<batchSize {
    let embedCmdBuf = engine.commandQueue.makeCommandBuffer()!
    try dequantizeRowOptimized(...)
    embedCmdBuf.commit()
    embedCmdBuf.waitUntilCompleted()  // ← WAIT per token
    memcpy(...)
}
```

**Bottleneck**: batchSize × waitUntilCompleted()

For batch(8): **8 waits** for embedding alone!

### 2. Batch Embedding Kernel Attempt

**Created kernel**: `dequantize_row_batch` (MetalKernels.metal:1988-2019)

**Status**: ❌ CRASH (SIGSEGV - segmentation fault)

**Reason**: Memory access violation, needs debugging

**Deferred**: Using sequential approach for stability

### 3. Layer Processing

**Current**: Uses batch kernels (LayerBatch.swift)

**Status**: ✓✓✓ Working correctly

**Performance**: Unknown ( overshadowed by embedding bottleneck)

## Performance Impact

**Embedding bottleneck dominates**:

```
Embedding: batchSize × ~5ms = 40ms for batch(8)
Layer processing: ~25ms
Total: 65ms+ → 76.3ms/token observed ✓
```

**Without optimization**: Batch is **slower** than single!

## Optimization Priority

### Phase 1: Fix Batch Embedding Kernel (CRITICAL)

**Goal**: Single GPU dispatch for entire batch

**Current**: 8 waits → Target: 1 wait

**Expected impact**: 
- Embedding: 40ms → ~5ms (8x faster)
- Batch(8): 76ms → ~35ms (2x faster)
- Per-token: 35ms/8 = 4.4ms ✓✓✓

**Status**: ❌ Crash, needs debugging

### Phase 2: Optimize Batch Layer Processing

**Current**: Batch kernels exist but performance unknown

**Goal**: Verify and optimize batch layer kernels

**Expected**: Additional 2-3x speedup

### Phase 3: Model Loading Optimization

**31B loading**: 65 seconds

**Goal**: Parallel weight loading

**Expected**: 50% reduction (32s)

## Lessons Learned

1. **Batch processing ≠ automatic speedup**
   - Sequential operations in batch code kill performance
   - Need true parallel GPU dispatch for all phases

2. **Embedding is critical bottleneck**
   - Small operation but high overhead (multiple waits)
   - Must be batched for effective performance

3. **Kernel debugging is time-consuming**
   - SIGSEGV requires careful memory bounds checking
   - Better to defer and use stable approach first

## Next Steps

**Immediate**: Document findings, move to next optimization

**Short-term**: 
1. Debug batch embedding kernel (when time permits)
2. Optimize model loading (higher ROI, easier)

**Long-term**:
1. Metal kernel fusion
2. SIMD expansion
3. Expert caching

## Conclusion

**Batch processing currently SLOWER** due to embedding bottleneck.

**Key insight**: Sequential waits in "batch" code defeat parallelism.

**Recommendation**: Focus on model loading optimization first (higher ROI, easier implementation), then revisit batch embedding kernel debugging.