markbaseengine/MOE_DEBUG_ANALYSIS_FINAL.md

MoE Debug Analysis - Final Findings

Test Attempts

Time: 2026-06-20 22:20-22:30 (~10 minutes) Tests Run: 3 attempts with debug prints Results: ALL TIMEOUT, NO DEBUG OUTPUT

⚠️ Critical Finding

Debug prints added:

• Layer.swift:827-861 (router computation debug)
• Layer.swift:841-861 (softmax computation debug)

Expected output:

[MoE DEBUG] Layer 0: Starting router computation...
[MoE DEBUG] Layer 0: Router matmul completed
[MoE DEBUG] Layer 0: Router logits first 10: [...]
...

Actual output: NOTHING (no debug prints appear)

🔍 Diagnosis

Problem: Debug prints not appearing indicates:

Most likely ⭐⭐⭐⭐⭐:

• moeForward() is NEVER called
• Generation hangs BEFORE reaching MoE forward
• Issue is in earlier stage (embedding, tokenizer, or generator setup)

Less likely ⭐⭐⭐:

• stdout buffering (but we added fflush)
• Prints suppressed by test framework

Unlikely ⭐:

• MoE forward logic issue (would see prints before hang)

📊 Current Understanding

Generation Flow

1. Tokenizer.encode(prompt) → [token_ids]
2. Embedding lookup → input buffer
3. Forward pass for each layer → MoE forward called here
4. Logits computation → sampler
5. Decode token → output

Where It Hangs

Based on no debug prints: ⭐⭐⭐⭐⭐

• Hangs BEFORE step 3 (MoE forward)
• Possible hang points:
  • Step 1: Tokenizer.encode (unlikely)
  • Step 2: Embedding lookup (possible)
  • Generator initialization (likely)
  • First buffer allocation (possible)

🎯 Revised Next Steps

Option A: Add earlier debug prints ⭐⭐⭐⭐⭐ (RECOMMENDED)

Where to add:

// In StreamingGenerator.generateComplete()
print("[GEN DEBUG] Starting generation...")
print("[GEN DEBUG] Encoded prompt: \(tokens)")
print("[GEN DEBUG] Creating buffers...")
print("[GEN DEBUG] Calling forward...")

Reason: Find where EXACTLY it hangs before MoE forward

Time: 10-15 minutes

---

Option B: Test tokenizer separately ⭐⭐⭐⭐

Test:

let tokenizer = try TokenizerFactory.load(modelDir: modelDir)
let tokens = tokenizer.encode(text: "Hello")
print("Tokens: \(tokens)")

Reason: Verify tokenizer works

Time: 5 minutes

---

Option C: Test embedding lookup ⭐⭐⭐⭐

Test:

let embed = model.embedTokens
let embedData = engine.readFloats(from: embed.weight, offset: 2 * model.hiddenSize, count: model.hiddenSize)
print("Embedding data: \(embedData[0..<10])")

Reason: Verify embedding works

Time: 5 minutes

---

💡 Recommendation

Combine A + B + C ⭐⭐⭐⭐⭐

Reason: Systematically test each stage

Sequence:

1. Test tokenizer (5 min)
2. Test embedding (5 min)
3. Add earlier debug prints in generator (10 min)
4. Test generation (2-5 min)

Total: 20-30 minutes

Expected: Identify exact hang location

---

📈 Timeline

22:20 - Added debug prints to MoE forward
22:21-22:30 - Ran 3 tests, all timeout, NO DEBUG OUTPUT
22:30 - Diagnosis: moeForward never called
22:30 - Revised plan: add earlier debug prints

🎓 Lessons

1. Debug prints location matters

   • Prints in moeForward → no output → never called
   • Need prints earlier in pipeline

2. Systematic debugging

   • Test each stage separately
   • Identify exact hang point
   • Don't assume where issue is

3. MoE generation complexity

   • More stages than Dense
   • More potential hang points

---

📝 Files

Debug prints added:

• /Users/accusys/MarkBase12B/Sources/G12B/Layers/Layer.swift (lines 827-861)

Tests created:

• /Users/accusys/MarkBase12B/Tests/G12BTests/MoEDebugMinimalTest.swift

Logs:

• /Users/accusys/MarkBase12B/MOE_GENERATION_DEBUG_PRINTS.log (empty)
• /Users/accusys/MarkBase12B/MOE_MINIMAL_TEST.log (timeout)

---

✅ Progress Summary

Task	Status	Finding
Add MoE debug prints	✅ Done	Layer.swift:827-861
Run generation test	❌ Timeout	No debug output
Diagnose issue	✅ Done	moeForward never called
Revised plan	✅ Created	Add earlier debug prints

---

🔧 Immediate Action

Next: Add debug prints to StreamingGenerator before MoE forward

Files to edit:

• StreamingGenerator.swift (add early debug prints)

Expected: Identify exact hang location

---

Status: ⚠️ MoE forward never reached
Issue: Hangs before MoE computation
Next: Debug earlier in pipeline
Time: 20-30 minutes remaining work

---

Conclusion: Generation hangs BEFORE MoE forward pass. Need to add debug prints earlier in the pipeline (tokenizer, embedding, generator initialization).