MarkBase Admin
ac75faa0cc
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
86 lines
2.2 KiB
Markdown
86 lines
2.2 KiB
Markdown
# Layer Construction Performance Analysis
|
||
|
||
## Current Observations
|
||
|
||
From test results:
|
||
```
|
||
31B Total Load: 64s
|
||
Shard Loading: 1.3ms ✓✓✓ (极快)
|
||
Layer Construction: 63s ← Bottleneck
|
||
|
||
Layer Breakdown:
|
||
- 60 layers
|
||
- Each layer ~1.05s
|
||
- MoE layers: 128 experts × ~1.05s = 134.4s (major bottleneck!)
|
||
|
||
## Analysis
|
||
|
||
The bottleneck is clearly in **layer construction**, not shard loading.
|
||
|
||
**Key Operations**:
|
||
1. **Weight Reading** - File IO operations
|
||
- Each weight requires reading from disk
|
||
- MoE: 128 experts × 3 files per expert
|
||
- Sequential reads are major bottleneck
|
||
|
||
2. **Buffer Creation** - Memory allocation
|
||
- MTLBuffer creation is relatively fast
|
||
- But needs to allocate large buffers
|
||
|
||
3. **Layer Initialization** - Object creation
|
||
- Creating E4BLayer objects
|
||
- Setting up quantization parameters
|
||
|
||
## Next Steps
|
||
|
||
**Priority 1: Parallel Weight Loading**
|
||
- Goal: Reduce weight loading from ~63s to ~20s
|
||
- Approach:
|
||
1. Pre-identify all weights needed for layer construction
|
||
2. Use DispatchGroup to load weights in parallel
|
||
3. Store weights in temporary arrays
|
||
4. Build layers after all weights loaded
|
||
|
||
**Expected Improvement**: 3x speedup (63s → 20s)
|
||
|
||
**Priority 2: MoE Expert Loading Optimization**
|
||
- Goal: Reduce MoE expert loading from 134s to 30s
|
||
- Approach:
|
||
1. Parallel expert loading
|
||
2. Batch expert creation
|
||
3. Optimize expert weight reading
|
||
|
||
**Expected Improvement**: 4.5x speedup (134s → 30s)
|
||
|
||
**Priority 3: Memory Allocation Optimization**
|
||
- Goal: Optimize MTLBuffer creation
|
||
- Approach:
|
||
1. Pre-allocate large buffers
|
||
2. Reuse buffers across layers
|
||
3. Minimize buffer copies
|
||
|
||
**Expected Improvement**: 10-15% speedup
|
||
|
||
## Implementation Priority
|
||
|
||
**Phase 1** (Immediate): Parallel Weight Loading
|
||
- Highest ROI (3x speedup)
|
||
- Easiest to implement
|
||
- Quick verification
|
||
|
||
**Phase 2** (Short-term): MoE Expert Loading
|
||
- Medium ROI (4.5x speedup)
|
||
- More complex
|
||
- Requires careful coordination
|
||
|
||
**Phase 3** (Long-term): Memory Optimization
|
||
- Lower ROI (10-15%)
|
||
- Most complex
|
||
- Requires architecture changes
|
||
|
||
## Decision
|
||
|
||
Starting with **Phase 1**: Parallel Weight Loading
|
||
- Quick wins
|
||
- Clear bottleneck
|
||
- Easy to measure and verify |