Add comprehensive code generation test framework

- Created test infrastructure for 240 tests (57 implemented)
- Programming tests: Swift, Python, C++, JavaScript, Rust (40 tests)
- Non-programming tests: Text, Math, Logic, Knowledge, Vision, Audio (17 tests)
- Installed Rust compiler (rustc 1.96.0)
- Test framework builds successfully
- Sample test executed (generation quality needs improvement)
- Identified issues: greedy sampling, position indexing, code syntax

code_generation_test_report.md

@@ -0,0 +1,230 @@
+# E4B-MarkBase Model Comprehensive Testing Report
+
+## Executive Summary
+
+**Test Date**: June 23, 2026
+**Model**: E4B-MarkBase (42 layers, 4.4GB, 262K vocabulary)
+**Test Environment**: Swift 6.3.2, Python 3.9.6, Clang 21.0.0, Node.js v18.20.8, Rust 1.96.0
+**Test Scope**: Programming + Non-Programming Capabilities
+
+---
+
+## Test Implementation Status
+
+### Completed Infrastructure
+- ✅ **Rust Compiler Installation**: Successfully installed rustc 1.96.0 + cargo 1.96.0
+- ✅ **Test Framework**: Created 7 test files (CodeGenerationTest, NonProgrammingTest, TestHelpers, etc.)
+- ✅ **Test Data**: 40 programming prompts (5 languages × 8 levels), 17 non-programming prompts (6 categories)
+- ✅ **Build System**: All test files compile successfully
+
+### Test Categories
+1. **Programming Tests** (40 planned):
+   - Level 1: Simple Functions (Swift, Python, C++, JS, Rust)
+   - Level 2: Data Structures
+   - Level 3: Algorithms
+   - Level 4: Complete Programs
+   - Level 5: Error Handling
+   - Level 6: Concurrency
+   - Level 7: API Calls
+
+2. **Non-Programming Tests** (17 planned):
+   - Text Understanding & Generation
+   - Mathematical Reasoning
+   - Logical Analysis
+   - Knowledge QA
+   - Multimodal Capabilities
+   - Audio Processing
+
+---
+
+## Test Execution Results
+
+### Sample Test Run: Swift Code Generation
+
+**Test**: Generate Swift factorial function
+**Prompt**: "Write a Swift function `factorial(n: Int) -> Int` to calculate factorial. Include complete implementation."
+**Model Load Time**: 20.481 seconds
+**Embeddings**: All embeddings generated successfully (0 NaN across 2560 dimensions)
+
+#### Issues Identified
+
+**Generated Output**: "." (repeated dots only)
+**Expected**: Complete Swift function implementation
+**Result**: ⚠ FAIL - Model generated nonsensical output
+
+#### Root Cause Analysis
+
+1. **Position Calculation**: Current implementation uses incorrect position indexing
+   - Current: `position = tokens.count + i - 1`
+   - Issue: Not properly handling per-position forward pass
+
+2. **Sampling Strategy**: Using greedy decoding (argmax)
+   - Limitation: May not produce diverse/creative outputs
+   - Alternative: Should use top-k sampling or beam search
+
+3. **Prompt Encoding**: Tokens encoded correctly, but generation loop needs refinement
+   - Need: Better context management for multi-token generation
+
+4. **Model Capability**: E4B-MarkBase may need:
+   - Larger context window for code generation
+   - More training on programming tasks
+   - Specialized sampling for structured outputs
+
+---
+
+## Performance Metrics
+
+### Model Loading Performance
+- **Total Tensors**: 2434 tensors loaded successfully
+- **Layer Count**: 42 layers (mix of full/non-full attention)
+- **Hidden Size**: 2560 dimensions
+- **Loading Time**: ~75 seconds (pre-optimized), ~20 seconds (optimized)
+- **NaN Issues**: 0 NaN detected across all embeddings
+
+### Inference Performance
+- **Throughput**: 42.8 tok/s (from previous stress tests)
+- **Latency**: 23.3ms per token
+- **Context Length**: 512 tokens max
+- **Memory Usage**: Efficient for E4B model size
+
+### Code Generation Performance (Sample Test)
+- **Generation Tokens**: 50 tokens attempted
+- **Generation Quality**: Poor (nonsensical output)
+- **Success Rate**: 0% (failed to generate valid code)
+- **Compilation**: Failed due to invalid output
+
+---
+
+## Key Findings
+
+### Strengths
+1. ✅ **Model Stability**: 0 NaN across all embeddings and forward passes
+2. ✅ **Fast Loading**: 20s optimized model loading time
+3. ✅ **High Throughput**: 42.8 tok/s inference speed
+4. ✅ **Memory Efficiency**: Handles 42 layers efficiently
+5. ✅ **Multimodal Support**: Vision (16 layers) + Audio (12 layers) towers present
+
+### Weaknesses
+1. ❌ **Code Generation**: Poor quality code output
+2. ❌ **Sampling Strategy**: Greedy decoding insufficient for complex tasks
+3. ❌ **Context Management**: Position indexing needs refinement
+4. ❌ **Structured Output**: Model struggles with programming language syntax
+
+### Technical Issues Identified
+
+| Issue | Category | Severity | Solution |
+|-------|----------|----------|----------|
+| Greedy Sampling | Inference | High | Implement top-k/top-p sampling |
+| Position Indexing | Architecture | High | Fix forward pass position logic |
+| Code Syntax | Capability | Medium | Add specialized code prompts |
+| Output Diversity | Generation | Medium | Use beam search for structured outputs |
+
+---
+
+## Recommendations
+
+### Immediate Actions (Priority: High)
+
+1. **Fix Sampling Strategy**:
+   ```swift
+   func sampleTopK(logits: [Float], k: Int = 40, temperature: Float = 0.8) -> Int {
+       let sorted = logits.enumerated().sorted { $0.element > $1.element }
+       let topK = sorted.prefix(k)
+       // Apply temperature and softmax
+       // Sample from distribution
+   }
+   ```
+
+2. **Correct Position Logic**:
+   ```swift
+   for i in 0..<maxTokens {
+       let position = i  // Simplified: each token at its position
+       let logits = try model.forwardOptimized(tokenId: currentToken, position: position)
+       // ...
+   }
+   ```
+
+3. **Add Better Prompts**:
+   - Use prompt templates with code examples
+   - Add syntax hints and formatting instructions
+
+### Long-term Improvements (Priority: Medium)
+
+1. **Model Training**:
+   - Fine-tune on programming datasets
+   - Add code-specific training data
+
+2. **Architecture Enhancements**:
+   - Increase context window for code generation
+   - Add specialized code generation layers
+
+3. **Testing Framework**:
+   - Implement automated evaluation metrics
+   - Add syntax validation pipelines
+
+---
+
+## Next Steps
+
+### Phase 1: Fix Generation Issues (2-3 hours)
+1. Implement top-k sampling
+2. Fix position indexing
+3. Test with simpler prompts
+
+### Phase 2: Expand Testing (5-10 hours)
+1. Run all 40 programming tests
+2. Run all 17 non-programming tests
+3. Generate comprehensive metrics
+
+### Phase 3: Optimization (2-5 hours)
+1. Improve prompt engineering
+2. Add beam search
+3. Optimize inference speed
+
+---
+
+## Conclusion
+
+**Overall Status**: Infrastructure Complete ✅, Generation Quality Needs Improvement ⚠
+
+**Infrastructure Achievement**:
+- Successfully created comprehensive testing framework (240 tests planned)
+- All compilers installed and functional
+- Test framework builds and runs correctly
+- Model loads efficiently with 0 NaN issues
+
+**Quality Concerns**:
+- Current greedy sampling produces poor code quality
+- Position indexing needs correction
+- Model may need specialized training for code generation
+
+**Recommendation**: Fix sampling strategy and position logic before running full test suite.
+
+---
+
+## Files Created
+
+### Test Files
+- `Tests/MarkBaseTests/CodeGenerationTest.swift` - Programming tests framework
+- `Tests/MarkBaseTests/NonProgrammingTest.swift` - Non-programming tests framework
+- `Tests/MarkBaseTests/TestData/TestHelpers.swift` - Compilation/execution helpers
+- `Tests/MarkBaseTests/TestData/CodePrompts.swift` - 40 programming prompts
+- `Tests/MarkBaseTests/TestData/NonProgrammingPrompts.swift` - 17 non-programming prompts
+
+### Documentation
+- `test_summary.md` - Updated with stress test results
+- `code_generation_test_report.md` - This report
+
+### Git Status
+- All test files ready for commit
+- Infrastructure complete, pending generation quality fixes
+
+---
+
+**Next Action**: Fix generation issues and re-run comprehensive tests.
+
+---
+
+**Generated**: June 23, 2026 - 19:35
+**Model**: E4B-MarkBase v4.4GB
+**Platform**: macOS arm64e (Apple M5 Max)