markbaseengine/A4B_MODEL_SOURCE_ANALYSIS.md

# 26B-A4B Model Source Analysis

**Date**: 2026-06-23  
**Purpose**: Trace origin of problematic 26B-A4B model

---

## Model Sources Comparison

### 26B-A4B (Problematic)

**Origin**: HuggingFace MLX Community
- **Repository**: `mlx-community/gemma-4-26b-a4b-it-4bit`
- **Base Model**: `google/gemma-4-26b-a4b-it` (Google official)
- **Converter**: `mlx-vlm` version 0.4.3
- **Framework**: MLX (Apple's ML framework)
- **Library**: mlx
- **License**: Apache 2.0 (Gemma license)

**Quantization Config**:
```json
{
  "group_size": 64,
  "bits": 4,
  "mode": "affine",
  "mixed_precision": true  // Some layers use INT8
}
```

**File Format**:
- Sharded: model-00001-of-00003.safetensors (4.9GB)
- Sharded: model-00002-of-00003.safetensors (4.9GB)
- Sharded: model-00003-of-00003.safetensors (4.7GB)
- Total: 14.5GB

**Creation Date**: 19 Jun 10:20 (downloaded to local)

---

### 26B-Standard (Correct)

**Origin**: Unknown (possibly custom quantization)
- **No README.md** (no HuggingFace metadata)
- **Config**: Simple JSON (no mlx-vlm metadata)
- **Quant Method**: "custom"

**Quantization Config**:
```json
{
  "bits": 4,
  "group_size": 32,
  "quant_method": "custom"
}
```

**File Format**:
- Single file: model.safetensors (15.6GB)

**Creation Date**: 19 Jun 08:28 (downloaded/quantized locally)

---

## Key Differences

| Aspect | 26B-A4B | 26B-Standard |
|--------|---------|--------------|
| **Source** | HuggingFace MLX | Unknown/Custom |
| **Converter** | mlx-vlm 0.4.3 | Custom script? |
| **Group Size** | 64 | 32 |
| **Quant Mode** | affine | custom |
| **Scales Range** | ±0.01 ✗ | ~120 ✓ |
| **Scales Sign** | Negative ✗ | Positive ✓ |
| **File Size** | 14.5GB (sharded) | 15.6GB (single) |
| **Layers** | 30 | 30 |
| **Experts** | 128 | 128 |

---

## Problem Root Cause

### MLX Quantization Bug (mlx-vlm 0.4.3)

**Symptoms**:
1. Scales too small (±0.01 instead of ~120)
2. Negative scales (invalid for affine quantization)
3. Result: 98% tokens produce NaN

**Evidence**:
- 26B-Standard (custom quant): scales correct ~120 ✓
- 26B-A4B (mlx-vlm 0.4.3): scales wrong ±0.01 ✗

**Hypothesis**:
- mlx-vlm 0.4.3 has bug in affine quantization
- Generates wrong scales magnitude
- Missing normalization or wrong formula

---

## MLX Affine Quantization Theory

### Formula (Expected)
```
weight = (int4_value - zero_point) * scale + bias
```

**Correct Implementation**:
- scale = (weight_max - weight_min) / 15 (range for INT4)
- zero_point = intermediate value
- bias = weight_min

**Expected scales**:
- For typical weights: scale ≈ 50-200
- For group_size=64: similar range

**26B-A4B scales**:
- scale ≈ 0.01 (100x too small)
- Negative values (invalid)
- Bug in mlx-vlm quantization logic

---

## MLX-vlm Version Analysis

### mlx-vlm 0.4.3 (Used for 26B-A4B)
- Release date: Unknown (need check HuggingFace)
- Known issues: Quantization bugs?
- Affine mode: Problematic?

### Alternative Versions
- mlx-vlm latest: May have fixes
- Custom quantization: More control

---

## Recommended Actions

### 1. Check MLX-vlm Issues

**Search**:
- HuggingFace mlx-community repo issues
- GitHub mlx-vlm issues for "affine quantization"
- Look for scales bug reports

### 2. Re-quantize with Fixed Script

**If MLX-vlm fixed**:
- Download latest mlx-vlm
- Re-quantize from `google/gemma-4-26b-a4b-it`
- Verify scales range (~120)

**If custom script**:
- Use same method as 26B-Standard
- group_size=32, custom quant
- Manual scales verification

### 3. Report Issue

**To MLX Community**:
- HuggingFace: mlx-community/gemma-4-26b-a4b-it-4bit
- GitHub: mlx-vlm issue tracker
- Describe: scales too small + negative values
- Evidence: scales sample comparison

---

## Model Card Information

### Google Gemma-4-26B-A4B-IT

**Official Model** (pre-quantized):
- **Publisher**: Google
- **License**: Gemma license (Apache-style)
- **Architecture**: MoE (Mixture of Experts)
- **Layers**: 30
- **Experts**: 128 per layer
- **Parameters**: ~26B (active params)
- **Special**: A4B variant (Audio-Aware)

**HuggingFace**: `google/gemma-4-26b-a4b-it`
- BF16 weights (original)
- Used as base for MLX conversion

---

## Alternative: Google Gemma-4-27B-IT

**26B-Standard equivalent**:
- **Architecture**: MoE, 30 layers, 128 experts
- **Parameters**: ~27B (similar to 26B-A4B)
- **License**: Same Gemma license
- **Status**: Available in BF16

**If 26B-Standard is Gemma-4-27B-IT**:
- Same architecture family
- Custom quantization (group_size=32)
- Correct scales ✓

---

## Conclusion

**26B-A4B problem traced to MLX-vlm 0.4.3 quantization bug**

- **Source**: `mlx-community/gemma-4-26b-a4b-it-4bit`
- **Converter**: mlx-vlm 0.4.3 (buggy)
- **Result**: Wrong scales magnitude + negative values
- **Solution**: Use 26B-Standard (custom quant, correct scales)

---

## Next Steps

1. **Check HuggingFace**:
   - `mlx-community/gemma-4-26b-a4b-it-4bit` issues
   - Look for reports of quantization bugs

2. **Check GitHub**:
   - `mlx-vlm` repository issues
   - Search "affine quantization" problems

3. **Test MLX-vlm latest**:
   - Download newer version if available
   - Test quantization on small model

4. **Report Issue**:
   - Provide scales sample evidence
   - Compare with custom quant (26B-Standard)

---

## Files

### A4B Model Files
```
/Users/accusys/MarkBaseEngine/models/gemma-4-26b-a4b-it-4bit/
  README.md: MLX metadata
  config.json: quantization config (group_size=64, affine)
  model-00001-of-00003.safetensors (4.9GB)
  model-00002-of-00003.safetensors (4.9GB)
  model-00003-of-00003.safetensors (4.7GB)
```

### Standard Model Files
```
/Users/accusys/MarkBaseEngine/models/gemma-4-26b-standard/
  config.json: quantization config (group_size=32, custom)
  model.safetensors (15.6GB)
  No README (custom origin)
```

---

**End of Source Analysis**