markbaseengine/26B_A4B_ROUTER_FIX_FAILED_ANALYSIS.md

# Router Scale Fix Result - Needs Further Investigation

## Test Date
2026-06-20 22:17-22:19

## ❌ Router Scale Normalization Fix Did NOT Solve Generation Hanging

### Fix Applied
```swift
// Model.swift:518
routerScale = rawRouterScale / Float(hiddenSize)
// Before: 31.25
// After: 31.25/2816 = 0.01105
```

### Test Result
**Generation test**: STILL HANGS (timeout after 120s)

**No improvement**: Router scale normalization alone did not fix the issue

## ⚠️ New Findings

### Issue Complexity
**Not just router scale**: Multiple normalization issues possible

**Potential additional problems**:
1. **Expert scales normalization**
   - Expert gate/up/down scales might need normalization
   - Similar to 26B-Standard scales fix
   
2. **Router proj weights normalization**
   - Router projection output might need scaling
   
3. **Expert intermediate computation**
   - Expert fusion computation might overflow
   
4. **Top-k expert selection**
   - Expert selection logic might hang

### Next Steps Required

**Immediate debugging**:
1. ✅ Add debug prints to MoE forward pass
2. ✅ Check router computation step by step
3. ✅ Check expert scales values
4. ✅ Check expert selection process

**Additional normalization fixes**:
1. ⏳ Expert scales normalization (divide by expertInDim?)
2. ⏳ Router proj output normalization
3. ⏳ Expert intermediate normalization

### Comparison: What Worked for 26B-Standard

**26B-Standard had multiple fixes**:
```
Fix 1: Scales normalization (divide by hiddenSize)
Fix 2: Logits scaling (multiply by 0.00486)
Fix 3: Remove softcapping
Fix 4: Sampler temperature fix
```

**26B-A4B might need similar multiple fixes**:
```
Fix 1: Router scale normalization (applied, but not enough)
Fix 2: Expert scales normalization (not yet applied)
Fix 3: Router output normalization (not yet applied)
Fix 4: Expert intermediate normalization (not yet applied)
```

## 🔍 Debugging Strategy

### Step 1: Add Debug Prints

**Add to Layer.swift moeForward**:
```swift
// After router computation
let routerData = engine.readFloats(from: temps.gate, count: numExperts)
print("Router logits: \(routerData[0..<10])")
print("Router max/min: \(routerData.max()), \(routerData.min())")

// After scaling
var scaled = routerData.map { $0 * routerScale }
print("Scaled logits: \(scaled[0..<10])")
print("Scaled max/min: \(scaled.max()), \(scaled.min())")

// After softmax
print("Softmax weights: \(scaled[0..<10])")
```

### Step 2: Check Expert Scales

**Add to Model.swift loadExpertGroup**:
```swift
// After loading expert scales
print("Expert scales first 10: \(scalesData[0..<10])")
let expertScalesMax = scalesData.max()
print("Expert scales max: \(expertScalesMax)")
// If large (>100), need normalization
```

### Step 3: Test Router Forward Pass

**Create minimal router test**:
- Test router computation only (no expert)
- Check if router works with normalized scale
- Verify softmax is stable

## 📊 Current Status

| Component | Status | Issue |
|-----------|--------|-------|
| Model loading | ✅ Works | All 30 layers, 3840 experts |
| Router structure | ✅ Works | All components present |
| Router scale fix | ⚠️ Applied | Normalized (31.25→0.01105) |
| Token generation | ❌ Hangs | Timeout 120s, no response |
| Expert computation | ⏳ Unknown | Needs testing |

## 💡 Revised Assessment

### Router Scale Fix Confidence

**Previous confidence**: ⭐⭐⭐⭐⭐ (5/5)
**Actual result**: ❌ Did not fix

**Lesson**: MoE models have more complex normalization requirements than Dense models

### New Hypothesis

**MoE normalization complexity**:
1. Router scale normalization (tried, not enough)
2. Expert scales normalization (not tried yet)
3. Multiple normalization steps needed

**Similar to 26B-Standard**: Multiple fixes required
**MoE adds**: More components need normalization (router + experts)

## 🎯 Next Action Plan

### Option A: Add Debug Prints (Recommended) ⭐⭐⭐⭐⭐

**Reason**: Need to see where it hangs
**Time**: 10-15 minutes
**Benefit**: Identify exact problem location

**Steps**:
1. Add debug prints to moeForward
2. Run test with prints
3. Identify where it hangs
4. Fix specific issue

### Option B: Try Expert Scales Fix ⭐⭐⭐⭐

**Reason**: Expert scales might be too large
**Time**: 5-10 minutes
**Benefit**: Additional normalization

**Steps**:
1. Add expert scales normalization
2. Divide by expertInDim (2816)
3. Test generation

### Option C: Multiple Fixes ⭐⭐⭐

**Reason**: Combine router + expert fixes
**Time**: 15-20 minutes
**Benefit**: Comprehensive fix

**Steps**:
1. Router scale fix (already applied)
2. Expert scales fix
3. Router output fix
4. Test generation

## 📈 Timeline Estimate

**Option A (Debug prints)**:
- Add prints: 10 minutes
- Run test: 2-5 minutes
- Analyze: 5-10 minutes
- Fix issue: 10-30 minutes
- **Total**: 30-60 minutes ⭐⭐⭐⭐⭐

**Option B (Expert fix)**:
- Apply fix: 5 minutes
- Test: 2-5 minutes
- **Total**: 7-10 minutes ⭐⭐⭐⭐

**Option C (Multiple fixes)**:
- Apply multiple fixes: 15-20 minutes
- Test: 2-5 minutes
- **Total**: 20-25 minutes ⭐⭐⭐

## Recommendation

**Use Option A (Debug prints)** ⭐⭐⭐⭐⭐

**Reasons**:
- Router scale fix didn't work → need to see where hangs
- Debug prints give visibility
- Identify exact problem
- Fix specific issue

**Alternative**: Combine A + B (add debug prints + expert scales fix)

---

## Files Updated

**Fix applied**:
- `/Users/accusys/MarkBase12B/Sources/G12B/Model.swift` (lines 516-519)

**Documentation**:
- `/Users/accusys/MarkBase12B/ROUTER_SCALE_FIX_APPLIED.md`
- `/Users/accusys/MarkBase12B/26B_A4B_ROUTER_FIX_FAILED_ANALYSIS.md`

---

## Summary

**✅ Router scale fix applied**: 31.25 → 0.01105 (normalized)

**❌ Generation still hangs**: Router fix not sufficient

**⏳ Next**: Add debug prints to identify exact hang location

**📊 Lesson**: MoE needs multiple normalization fixes, similar to 26B-Standard

**💡 Recommendation**: Add debug prints to moeForward, identify where it hangs