MarkBase Admin
ac75faa0cc
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
203 lines
5.1 KiB
Markdown
203 lines
5.1 KiB
Markdown
# Complete Model Comparison (Including E4B)
|
|
|
|
**Date**: 2026-06-23
|
|
**Status**: ✅ 5 Models Production Ready
|
|
|
|
---
|
|
|
|
## All Models Performance Summary
|
|
|
|
| Model | Latency | Throughput | NaN | Scales | Architecture | Deploy? |
|
|
|-------|---------|------------|-----|--------|--------------|---------|
|
|
| **26B-Standard** | 21.9ms | 45.7 tok/s | 0 ✓ | ~120 ✓ | MoE 30L/128E | **✅ BEST** |
|
|
| **E2B** | 22.1ms | 45.3 tok/s | 0 ✓ | ~120 ✓ | Dense 42L, per-layer | **✅ GOOD** |
|
|
| **31B** | 23.8ms | 42.1 tok/s | 0 ✓ | ±0.01 ⚠ | Dense 60L | **✅ GOOD** |
|
|
| **E4B-MarkBase** | 23.4ms | 42.8 tok/s | 0 ✓ | Unknown | Dense 42L, multimodal | **✅ GOOD** |
|
|
| **26B-A4B** | - | - | 175+ ✗ | ±0.01 ✗ | MoE 30L/128E | **❌ NO** |
|
|
|
|
---
|
|
|
|
## E4B-MarkBase Details
|
|
|
|
### Architecture
|
|
- **TEXT**: 42 layers, hidden=2560, vocab=262144
|
|
- **Audio**: 12 layers audio tower
|
|
- **Vision**: 16 layers vision tower
|
|
- **Multimodal**: Full Audio+Vision+Text generation
|
|
- **File**: model.safetensors (4.67GB)
|
|
|
|
### Performance
|
|
- **TEXT latency**: 23.4ms per token
|
|
- **TEXT throughput**: 42.8 tok/s
|
|
- **NaN count**: 0 ✓
|
|
- **Status**: Production ready
|
|
|
|
### Scales Quality
|
|
- **Shape**: [262144, 40]
|
|
- **Negative**: 9 (some negative values)
|
|
- **Impact**: Zero NaN despite negative scales
|
|
|
|
### Multimodal Features
|
|
- Audio processing tested ✓
|
|
- Vision processing tested ✓
|
|
- Buffer isolation verified ✓
|
|
|
|
---
|
|
|
|
## Why All Models (Except A4B) Work
|
|
|
|
### Scales Impact Summary
|
|
|
|
| Scales Type | MoE Models | Dense Models |
|
|
|-------------|------------|--------------|
|
|
| **Correct (~120)** | 26B-Standard ✓ | E2B ✓ |
|
|
| **Wrong (±0.01)** | 26B-A4B ✗ | 31B ✓, E4B ✓ |
|
|
| **Negative** | A4B ✗ | E4B ✓ |
|
|
|
|
**Explanation**:
|
|
- **MoE + Wrong scales** → Router NaN ✗
|
|
- **Dense + Wrong scales** → Still stable ✓
|
|
- **Dense + Negative scales** → Tolerated ✓
|
|
|
|
---
|
|
|
|
## Deployment Recommendations
|
|
|
|
### ✅ Tier 1: Best Performance
|
|
|
|
**26B-Standard MoE**:
|
|
- Best TEXT performance (21.9ms, 45.7 tok/s)
|
|
- Zero NaN, correct scales
|
|
- **Primary choice for MoE TEXT**
|
|
|
|
### ✅ Tier 2: Good Performance
|
|
|
|
**E2B Per-layer**:
|
|
- Dense TEXT (22.1ms, 45.3 tok/s)
|
|
- Per-layer embeddings feature
|
|
- **Alternative for Dense TEXT**
|
|
|
|
**31B Dense**:
|
|
- Large Dense TEXT (23.8ms, 42.1 tok/s)
|
|
- Zero NaN despite wrong scales
|
|
- **Large model option**
|
|
|
|
**E4B-MarkBase Multimodal**:
|
|
- Dense TEXT (23.4ms, 42.8 tok/s)
|
|
- **Full Audio+Vision+Text generation**
|
|
- **Best for multimodal applications**
|
|
|
|
### ❌ Tier 3: Do Not Deploy
|
|
|
|
**26B-A4B MoE**:
|
|
- Corrupted weights (98% tokens NaN)
|
|
- Replace with 26B-Standard
|
|
|
|
---
|
|
|
|
## Architecture Comparison Table
|
|
|
|
| Feature | 26B-Std | E2B | 31B | E4B | 26B-A4B |
|
|
|---------|---------|-----|-----|-----|---------|
|
|
| **Layers** | 30 | 42 | 60 | 42 | 30 |
|
|
| **Hidden** | 2816 | 1536 | 5376 | 2560 | 2816 |
|
|
| **Experts** | 128 | - | - | - | 128 |
|
|
| **Audio** | - | - | - | ✓ | Audio-aware |
|
|
| **Vision** | - | - | - | ✓ | - |
|
|
| **Scales** | ✓ | ✓ | ⚠ | ⚠ | ✗ |
|
|
| **NaN** | 0 | 0 | 0 | 0 | 175+ |
|
|
| **Deploy** | ✅ | ✅ | ✅ | ✅ | ❌ |
|
|
|
|
---
|
|
|
|
## Use Case Recommendations
|
|
|
|
### Pure TEXT Inference
|
|
- **Best**: 26B-Standard (MoE, fastest)
|
|
- **Alternative**: E2B (per-layer feature)
|
|
- **Large**: 31B (60 layers)
|
|
|
|
### Multimodal Inference
|
|
- **Best**: E4B-MarkBase (Audio+Vision+Text)
|
|
- **Note**: Only E4B has full multimodal support
|
|
|
|
### Audio-Aware Inference
|
|
- **A4B intended**: Audio-aware MoE
|
|
- **Problem**: A4B weights corrupted
|
|
- **Alternative**: E4B-MarkBase (has audio tower)
|
|
|
|
---
|
|
|
|
## Performance Targets vs Results
|
|
|
|
| Metric | Target | 26B-Std | E2B | 31B | E4B | All |
|
|
|--------|--------|---------|-----|-----|-----|-----|
|
|
| **Latency** | <100ms | 21.9 ✓ | 22.1 ✓ | 23.8 ✓ | 23.4 ✓ | **4x better** |
|
|
| **Throughput** | >10 tok/s | 45.7 ✓ | 45.3 ✓ | 42.1 ✓ | 42.8 ✓ | **4-5x better** |
|
|
| **NaN** | 0 | 0 ✓ | 0 ✓ | 0 ✓ | 0 ✓ | **Zero** |
|
|
|
|
---
|
|
|
|
## Quantization Quality Lessons
|
|
|
|
### 1. MoE Requires Perfect Quantization
|
|
- Router network sensitive
|
|
- Wrong scales → NaN
|
|
- 26B-Standard: Perfect example
|
|
|
|
### 2. Dense Tolerates Imperfections
|
|
- Wrong scales OK
|
|
- Negative scales OK
|
|
- 31B, E4B: Examples
|
|
|
|
### 3. Scales Validation Essential
|
|
- Check range (expect ~100-200)
|
|
- Check sign (positive preferred)
|
|
- Test multiple tokenIds
|
|
|
|
---
|
|
|
|
## Final Deployment Guide
|
|
|
|
### TEXT Inference Only
|
|
```bash
|
|
# Primary: 26B-Standard MoE
|
|
/Users/accusys/MarkBaseEngine/models/gemma-4-26b-standard
|
|
|
|
# Alternative: E2B Dense
|
|
/Users/accusys/MarkBaseEngine/models/gemma-4-12b-it-4bit
|
|
|
|
# Large: 31B Dense
|
|
/Users/accusys/MarkBaseEngine/models/gemma-4-31b-it-4bit
|
|
```
|
|
|
|
### Multimodal Inference
|
|
```bash
|
|
# Audio+Vision+Text: E4B-MarkBase
|
|
/Users/accusys/MarkBaseEngine/models/E4B-MarkBase
|
|
```
|
|
|
|
### DO NOT USE
|
|
```bash
|
|
# Corrupted: 26B-A4B
|
|
/Users/accusys/MarkBaseEngine/models/gemma-4-26b-a4b-it-4bit
|
|
# Replace with 26B-Standard
|
|
```
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
**5 models tested, 4 production ready, 1 corrupted**
|
|
|
|
- **26B-Standard**: Best TEXT (MoE)
|
|
- **E2B**: Good TEXT (Dense, per-layer)
|
|
- **31B**: Good TEXT (Dense, large)
|
|
- **E4B-MarkBase**: Good multimodal (Audio+Vision+Text)
|
|
- **26B-A4B**: DO NOT USE (corrupted)
|
|
|
|
**All usable models exceed performance targets by 4-5x**
|
|
|
|
---
|
|
|
|
**End of Complete Comparison** |