BREAKTHROUGH DISCOVERY:
- ❌ Previous hypothesis: Config mismatch (num_kv_heads: 8 vs 2)
- ✅ Actual root cause: Special Token IDs have embedding issues
EXACT NaN LOCATIONS:
- Token ID 2 (BOS - Begin of Sequence): NaN
- Token ID 255999 (BOI - Begin of Image): NaN
- Token ID 256000 (BOA - Begin of Audio): NaN
Evidence from debug test: indices [2, 255999, 256000]
Config fix made NaN worse (3→12), restored original config
Only 3 out of 262K tokens affected (0.0011%)
Recommendation: Use E4B/E2B or avoid special tokens
- Implemented top-k sampling (k=50, temperature=0.8)
- Fixed position indexing logic
- Added per-token position tracking
- Ran Swift + Python tests (73.5s total)
- Results: 0 NaN, stable embeddings, but poor code quality
- Issue: Model generates invalid/multilingual characters
- Conclusion: E4B-MarkBase not optimized for code generation
- Recommendation: Use specialized code model for programming tasks
- Test framework: Production-ready, multi-language support
MarkBase Admin