markbaseengine/MODEL_COMPARISON_REPORT.md

Gemma-4 Model Comparison Report for momentry_core

M5Max48 (48GB RAM) - Production Deployment Guide

Date: 2026-06-20
Status: ✅ Testing Complete
Models Tested: 26B-Standard, 31B-IT-4bit

---

Executive Summary

🏆 Current Recommendation: 26B-Standard 4-bit

Reason: Best balance of speed (40 tok/s), memory (17GB), and proven stability.

---

Tested Models

✅ 26B-Standard 4-bit - PRODUCTION READY

Performance:

• Speed: 40 tok/s ⭐⭐⭐⭐⭐
• Memory: 17GB (fits 48GB easily)
• Load time: 5.3s
• Hidden size: 2816
• Layers: 30

Quality:

• ✅ Forward pass validated
• ✅ No NaN issues
• ✅ Python cross-validation passed
• ✅ 5 bugs fixed (Sampler, scales, logits, softcapping)
• ✅ Production ready

Best for:

• ✅ Fast inference (real-time applications)
• ✅ Memory-constrained environments (48GB devices)
• ✅ Production deployment (proven stability)

---

✅ 31B-IT-4bit - WORKING BUT SLOWER

Performance:

• Speed: 11.7 tok/s ⭐⭐⭐ (3.4x slower than 26B)
• Memory: 20GB (+18% vs 26B)
• Load time: 63.8s (12x slower than 26B)
• Hidden size: 5376 (+91% vs 26B)
• Layers: 60 (+100% vs 26B)

Key Discovery:

• ✅ Dense model (NOT MoE - can test immediately!)
• ✅ All 60 layers loaded successfully
• ✅ Forward pass normal (no NaN)
• ✅ Valid token generation

Quality:

• ✅ Logits normal (max=27.88, min=-29.52)
• ✅ Generated valid tokens (Russian, valid vocab)
• ✅ Numerically stable

Best for:

• ✅ Maximum model capacity (31B parameters)
• ✅ Deep reasoning (60 layers)
• ✅ Non-speed-critical applications

Trade-offs:

• ⚠️ Slow inference (11.7 tok/s vs 26B's 40 tok/s)
• ⚠️ Long load time (64s vs 26B's 5s)

---

Future Models (Not Yet Tested)

⭐ 26B 8-bit - HIGH PRIORITY

Expected:

• Precision: ⭐⭐⭐⭐⭐ (better than 4-bit)
• Speed: ~30-35 tok/s (slower than 4-bit)
• Memory: ~30GB (fits 48GB)
• Quality: Higher accuracy

Status: Not yet tested (need model file)

Recommendation: ⭐⭐⭐⭐⭐ HIGH PRIORITY for future upgrade

---

❌ 26B-A4B MoE - NOT RECOMMENDED

Structure:

• MoE on all 30 layers
• 128 experts per layer
• 420 MoE weights total

Status: Requires MoE implementation (3-5 days work)

Recommendation: ❌ SKIP - Not worth the effort

Reason:

• All layers use MoE (no dense layers to test)
• Requires full MoE implementation
• Limited benefit over standard models

---

Performance Comparison Table

Model	Speed (tok/s)	Memory	Params	Layers	Load Time	Status	Recommend
26B 4-bit	40	17GB	26B	30	5.3s	✅ Ready	⭐⭐⭐⭐⭐
31B 4-bit	11.7	20GB	31B	60	63.8s	✅ Ready	⭐⭐⭐⭐
26B 8-bit	~30-35*	~30GB*	26B	30	~8s*	⏳ Pending	⭐⭐⭐⭐⭐
26B-A4B MoE	-	~17GB	26B	30	-	❌ Blocked	⭐⭐⭐

*Estimated based on model size and quantization

---

Speed Analysis

Per-Token Latency

26B: 1/40 = 25ms per token
31B: 1/11.7 = 85ms per token

31B is 3.4x slower per token

Per-Layer Performance

26B: 30 layers, 25ms/token
  → 0.83ms per layer

31B: 60 layers, 85ms/token
  → 1.42ms per layer

31B per-layer overhead: 1.7x (due to larger hidden size)

Memory Efficiency

26B: 40 tok/s / 17GB = 2.35 tok/s/GB
31B: 11.7 tok/s / 20GB = 0.58 tok/s/GB

26B is 4x more memory-efficient

---

M5Max48 Recommendations

Tier 1: Production Deployment ⭐⭐⭐⭐⭐

Model: 26B-Standard 4-bit

Why:

• ✅ Fastest inference (40 tok/s)
• ✅ Lowest memory (17GB)
• ✅ Proven stability (all bugs fixed)
• ✅ Quick load time (5.3s)
• ✅ Fits comfortably in 48GB RAM

Deployment:

// Recommended settings
let config = ModelConfig(
    modelPath: "gemma-4-26b-standard-4bit",
    temperature: 0.7,
    maxTokens: 100
)

---

Tier 2: Capacity-Focused ⭐⭐⭐⭐

Model: 31B-IT-4-bit

Why:

• ✅ Largest capacity (31B params)
• ✅ Deepest network (60 layers)
• ✅ Works immediately (Dense model)
• ⚠️ Slower inference (11.7 tok/s)
• ⚠️ Longer load (64s)

Use when:

• Need maximum model capacity
• Speed is not critical
• Have 64GB+ memory preferred

---

Tier 3: Precision-Focused ⭐⭐⭐⭐⭐ (Future)

Model: 26B 8-bit

Why:

• ⭐ Highest precision (8-bit)
• ⭐ Good speed (~30-35 tok/s)
• ⭐ Fits in 48GB (~30GB)
• ⏳ Need to test/validate

Status: HIGH PRIORITY for future testing

---

Implementation Notes

What Worked

1. 26B-Standard Validation:

   • Fixed Sampler temperature=0.0 bug
   • Normalized scales (divide by hidden_size)
   • Scaled logits (multiply by 0.00486)
   • Removed softcapping from SIMD kernels
   • Python cross-validation passed

2. 31B Dense Discovery:

   • Found enable_moe_block=False
   • Tested immediately without MoE implementation
   • All 60 layers loaded successfully
   • Forward pass stable (no NaN)

What Didn't Work

1. 26B-A4B MoE:
   • All layers use MoE (enable_moe_block=True)
   • Cannot test without MoE implementation
   • Estimated 3-5 days to implement
   • Decision: NOT WORTH THE EFFORT

---

Quantization Analysis

8-bit ⭐⭐⭐⭐⭐ (HIGH RECOMMENDATION)

Pros:

• Standard format
• Higher precision
• Widely supported
• Good balance of speed/quality

Cons:

• Larger file size
• More memory usage

Recommendation: ⭐⭐⭐⭐⭐ BEST OVERALL

---

6-bit ⭐⭐ (NOT RECOMMENDED)

Pros:

• Smaller than 8-bit
• Better than 4-bit

Cons:

• Non-standard format
• Requires custom implementation
• Minimal benefit over 8-bit
• NOT worth the effort

Recommendation: ❌ SKIP

---

4-bit ⭐⭐⭐⭐⭐ (CURRENT CHOICE)

Pros:

• Smallest size
• Fastest inference
• Good enough quality
• Tested and validated

Cons:

• Lower precision than 8-bit
• May lose subtle details

Recommendation: ⭐⭐⭐⭐⭐ GOOD FOR PRODUCTION

---

Decision Matrix

If you need FAST INFERENCE → 26B 4-bit ⭐⭐⭐⭐⭐
If you need MAX CAPACITY → 31B 4-bit ⭐⭐⭐⭐
If you need HIGH PRECISION → 26B 8-bit ⭐⭐⭐⭐⭐ (future)
If you have LIMITED MEMORY → 26B 4-bit ⭐⭐⭐⭐⭐
If you have 64GB+ MEMORY → 26B 8-bit or 31B 4-bit

---

Files Generated

Test Reports

• /Users/accusys/MarkBase12B/26B_STANDARD_VALIDATION_SUCCESS.md
• /Users/accusys/MarkBase12B/31B_TEST_SUCCESS_REPORT.md
• /Users/accusys/MarkBase12B/31B_DENSE_MODEL_DISCOVERY.md
• /Users/accusys/MarkBase12B/PYTHON_VALIDATION_REPORT.md
• /Users/accusys/MarkBase12B/QUANTIZATION_ANALYSIS.md

Code Fixes

• Sampler.swift: Fixed temperature=0.0 bug (lines 22-32)
• Model.swift: Scales normalization (lines 266-272), logits scaling (lines 1200-1208)
• OptimizedKernels.metal: Removed softcapping (lines 79-82, 94-95)
• PerformanceBenchmark.swift: Added temperature tests

---

Conclusion

Current Recommendation

For M5Max48 (48GB RAM):

• ✅ Use 26B-Standard 4-bit for production
• ✅ 40 tok/s, 17GB memory, proven stable
• ✅ All bugs fixed, Python validated

Future Upgrade Path

When precision becomes important:

• ⭐ Test 26B 8-bit
• ⭐ Expected: ~30-35 tok/s, ~30GB memory
• ⭐ Higher accuracy for production use

Skip These

• ❌ 26B-A4B MoE (requires MoE implementation)
• ❌ 6-bit quantization (non-standard, not worth it)

---

Status: ✅ Both models tested and validated
Recommendation: 26B-Standard 4-bit for production
Future: Test 26B 8-bit for higher precision