BREAKTHROUGH DISCOVERY:
- ❌ Previous hypothesis: Config mismatch (num_kv_heads: 8 vs 2)
- ✅ Actual root cause: Special Token IDs have embedding issues
EXACT NaN LOCATIONS:
- Token ID 2 (BOS - Begin of Sequence): NaN
- Token ID 255999 (BOI - Begin of Image): NaN
- Token ID 256000 (BOA - Begin of Audio): NaN
Evidence from debug test: indices [2, 255999, 256000]
Config fix made NaN worse (3→12), restored original config
Only 3 out of 262K tokens affected (0.0011%)
Recommendation: Use E4B/E2B or avoid special tokens
MarkBase Admin