PROBLEM CONFIRMED:
- 12B has 3 NaN in forward pass (new discovery)
- Root cause: Configuration mismatch between config.json and weights
CONFIGURATION MISMATCH:
- Config.json says: num_key_value_heads = 8
- Expected k_proj out_dim: 8 × 256 = 2048
- Actual weight file: k_proj out_dim = 512
- Effective num_kv_heads: 512 / 256 = 2 (NOT 8!)
- Mismatch factor: 4x difference
IMPACT ANALYSIS:
- Embedding: 0 NaN (perfect)
- Forward pass: 3 NaN (generated during forward)
- Problem location: Likely in attention computation
- Reason: Q and K dimension mismatch (4096 vs 512)
WHY PREVIOUS TESTS DIDN'T DETECT:
- Different test positions/tokens
- Different execution order
- Random uninitialized memory values
COMPARISON WITH OTHER MODELS:
- E4B: Config matches weights → 0 NaN
- 31B: Auto-correction works → 0 NaN
- E2B: Config matches weights → 0 NaN
- 12B: Auto-correction incomplete → 3 NaN
IMMEDIATE SOLUTIONS:
1. Update config.json: num_key_value_heads = 2
2. Re-quantize model with correct config
3. Use E4B/31B/E2B as alternatives
Recommendations:
- ⚠️ Do NOT use 12B in production until fixed
- ✅ Use E4B (0 NaN, KV sharing) or 31B (0 NaN, larger) instead
- ✅ Fix config or re-download/re-quantize model
MarkBase Admin