MarkBase Admin
78257a947c
CI / build-and-test (push) Has been cancelled
BREAKTHROUGH DISCOVERY: - ❌ Previous hypothesis: Config mismatch (num_kv_heads: 8 vs 2) - ✅ Actual root cause: Special Token IDs have embedding issues EXACT NaN LOCATIONS: - Token ID 2 (BOS - Begin of Sequence): NaN - Token ID 255999 (BOI - Begin of Image): NaN - Token ID 256000 (BOA - Begin of Audio): NaN Evidence from debug test: indices [2, 255999, 256000] Config fix made NaN worse (3→12), restored original config Only 3 out of 262K tokens affected (0.0011%) Recommendation: Use E4B/E2B or avoid special tokens