markbaseengine/AVAILABLE_MODELS_SUMMARY.md

# Available Models Summary
## Tested and Ready for Use

**Date**: 2026-06-20  
**Device**: M5Max48 (48GB RAM)  

---

## ✅ Production Ready Models

### 1. Gemma-4-26B-Standard-4bit ✅ TESTED & RECOMMENDED

**Location**: `/Users/accusys/MarkBase12B/models/gemma-4-26b-standard/`

**Details**:
- Format: 4-bit quantized (bits=4, group_size=32, quant_method=custom)
- Size: 15GB (model.safetensors)
- Status: ✅ PRODUCTION READY

**Performance**:
- Speed: **40 tok/s** ⭐⭐⭐⭐⭐
- Memory: ~17GB
- Load time: 5.3s
- Hidden size: 2816
- Layers: 30

**Recommendation**: ⭐⭐⭐⭐⭐ BEST CHOICE for M5Max48

**Note**: Despite the name "standard", this is already 4-bit quantized (verified in config.json).

---

### 2. Gemma-4-26B-A4B-IT-4bit (MoE)

**Location**: `/Users/accusys/MarkBase12B/models/gemma-4-26b-a4b-it-4bit/`

**Details**:
- Format: 4-bit quantized
- Size: ~15.6GB (split into 3 parts)
- Structure: MoE on all 30 layers
- Status: ❌ BLOCKED (requires MoE implementation)

**Note**: All layers use Mixture of Experts (MoE). Cannot test without implementing MoE support.

---

### 3. Gemma-4-31B-IT-4bit ✅ TESTED

**Location**: `/Users/accusys/MarkBase12B/models/gemma-4-31b-it-4bit/`

**Details**:
- Format: 4-bit quantized
- Size: 18.4GB (split into 4 parts)
- Structure: Dense (no MoE!)
- Layers: 60
- Hidden size: 5376
- Status: ✅ WORKING

**Performance**:
- Speed: 11.7 tok/s
- Memory: ~20GB
- Load time: 63.8s

**Recommendation**: ⭐⭐⭐⭐ (Good for capacity, slower speed)

---

### 4. E4B-MarkBase (Reference)

**Location**: `/Users/accusys/MarkBase12B/models/E4B-MarkBase/`

**Details**:
- Format: Original
- Status: Reference model for comparison

---

## ❌ Missing Models

### Gemma-4-26B-8bit

**Status**: ❌ NOT AVAILABLE

**Expected**:
- Format: 8-bit quantized
- Size: ~15GB
- Speed: ~30-35 tok/s
- Memory: ~30GB

**Action Needed**: 
- Quantize from original 26B
- Or download from HuggingFace

---

### Gemma-4-26B-8bit

**Status**: ❌ NOT AVAILABLE

**Expected**:
- Format: 8-bit quantized
- Size: ~15GB
- Speed: ~30-35 tok/s
- Memory: ~30GB

**Action Needed**:
- Quantize from 26B-standard (15GB)
- Or download from HuggingFace

---

## Summary Table

| Model | Format | Size | Status | Speed | Recommend |
|-------|--------|------|--------|-------|-----------|
| **26B-Standard** | **4-bit** | **15GB** | **✅ Ready** | **40 tok/s** | **⭐⭐⭐⭐⭐** |
| 26B-A4B-IT | 4-bit MoE | 15.6GB | ❌ Blocked | - | ❌ |
| **31B-IT** | **4-bit** | **18.4GB** | **✅ Ready** | **11.7 tok/s** | **⭐⭐⭐⭐** |
| 26B-8bit | 8-bit | ~15GB | ❌ Missing | - | ⭐⭐⭐⭐⭐ (future) |
| E4B-MarkBase | Original | - | Reference | - | - |

---

## Current Best Options

### ✅ Available Now

**Gemma-4-26B-Standard-4bit** (RECOMMENDED):
- ✅ Works immediately
- ✅ Fastest speed (40 tok/s)
- ✅ Lowest memory (17GB)
- ✅ Quick load (5.3s)
- ✅ Production validated

**Gemma-4-31B-IT-4bit**:
- ✅ Works immediately
- ✅ Dense structure (no MoE)
- ✅ More capacity (31B params)
- ⚠️ Slower (11.7 tok/s)
- ⚠️ Longer load (64s)

---

### 🔧 Need to Obtain

**Gemma-4-26B-Standard-4bit** (RECOMMENDED):
- Expected speed: 40+ tok/s
- Expected memory: ~17GB
- Expected load: ~5s
- Status: Need to quantize or download

**Gemma-4-26B-8bit** (HIGH PRIORITY):
- Expected speed: ~30-35 tok/s
- Expected memory: ~30GB
- Expected precision: Better than 4-bit
- Status: Need to quantize or download

---

## Next Steps

### Option 1: Use 26B-Standard Now (RECOMMENDED)

**Action**: Use the available 26B-Standard-4bit model

**Pros**:
- ✅ Available immediately
- ✅ Fastest speed (40 tok/s)
- ✅ Lowest memory (17GB)
- ✅ Production validated

**Usage**:
```bash
cd /Users/accusys/MarkBase12B
swift run G12BServer --model 26b-standard
```

---

### Option 2: Use 31B-IT for Capacity

**Action**: Use 31B-IT-4bit when you need more capacity

**Pros**:
- ✅ Available immediately
- ✅ Larger capacity (31B)
- ✅ Deeper network (60 layers)

**Cons**:
- ⚠️ Slower (11.7 tok/s)
- ⚠️ Longer load (64s)

**Usage**:
```bash
cd /Users/accusys/MarkBase12B
swift run G12BServer --model 31b-it
```

---

### Option 3: Obtain 26B-8bit for Higher Precision (Future)

**Action**: Download or quantize 26B-8bit model

**Steps**:
1. Search HuggingFace for "gemma-4-26b-8bit"
2. Or quantize from original 26B
3. Test 26B-8bit (expected: 30-35 tok/s, better precision)

**Pros**:
- ✅ Higher precision (8-bit)
- ✅ Good speed (30-35 tok/s)
- ✅ Better quality outputs

**Cons**:
- ⏳ Need to obtain model
- ⏳ Need to test and validate

---

## Recommendation

**Immediate**: ✅ Use 26B-Standard-4bit (PRODUCTION READY)

**Why**:
- ✅ Fastest speed (40 tok/s)
- ✅ Lowest memory (17GB)
- ✅ Production validated
- ✅ All bugs fixed

**Alternative**: Use 31B-IT-4bit when you need more capacity (slower but larger)

**Future**: Obtain 26B-8bit for higher precision (better quality, still fast)

---

**Clarification**: The "26B-Standard" model is ALREADY 4-bit quantized (verified in config.json with "bits": 4). It's ready for production use with 40 tok/s speed.