markbaseengine/A4B_MODEL_SOURCE_ANALYSIS.md

26B-A4B Model Source Analysis

Date: 2026-06-23
Purpose: Trace origin of problematic 26B-A4B model

---

Model Sources Comparison

26B-A4B (Problematic)

Origin: HuggingFace MLX Community

• Repository: mlx-community/gemma-4-26b-a4b-it-4bit
• Base Model: google/gemma-4-26b-a4b-it (Google official)
• Converter: mlx-vlm version 0.4.3
• Framework: MLX (Apple's ML framework)
• Library: mlx
• License: Apache 2.0 (Gemma license)

Quantization Config:

{
  "group_size": 64,
  "bits": 4,
  "mode": "affine",
  "mixed_precision": true  // Some layers use INT8
}

File Format:

• Sharded: model-00001-of-00003.safetensors (4.9GB)
• Sharded: model-00002-of-00003.safetensors (4.9GB)
• Sharded: model-00003-of-00003.safetensors (4.7GB)
• Total: 14.5GB

Creation Date: 19 Jun 10:20 (downloaded to local)

---

26B-Standard (Correct)

Origin: Unknown (possibly custom quantization)

• No README.md (no HuggingFace metadata)
• Config: Simple JSON (no mlx-vlm metadata)
• Quant Method: "custom"

Quantization Config:

{
  "bits": 4,
  "group_size": 32,
  "quant_method": "custom"
}

File Format:

• Single file: model.safetensors (15.6GB)

Creation Date: 19 Jun 08:28 (downloaded/quantized locally)

---

Key Differences

Aspect	26B-A4B	26B-Standard
Source	HuggingFace MLX	Unknown/Custom
Converter	mlx-vlm 0.4.3	Custom script?
Group Size	64	32
Quant Mode	affine	custom
Scales Range	±0.01 ✗	~120 ✓
Scales Sign	Negative ✗	Positive ✓
File Size	14.5GB (sharded)	15.6GB (single)
Layers	30	30
Experts	128	128

---

Problem Root Cause

MLX Quantization Bug (mlx-vlm 0.4.3)

Symptoms:

1. Scales too small (±0.01 instead of ~120)
2. Negative scales (invalid for affine quantization)
3. Result: 98% tokens produce NaN

Evidence:

• 26B-Standard (custom quant): scales correct ~120 ✓
• 26B-A4B (mlx-vlm 0.4.3): scales wrong ±0.01 ✗

Hypothesis:

• mlx-vlm 0.4.3 has bug in affine quantization
• Generates wrong scales magnitude
• Missing normalization or wrong formula

---

MLX Affine Quantization Theory

Formula (Expected)

weight = (int4_value - zero_point) * scale + bias

Correct Implementation:

• scale = (weight_max - weight_min) / 15 (range for INT4)
• zero_point = intermediate value
• bias = weight_min

Expected scales:

• For typical weights: scale ≈ 50-200
• For group_size=64: similar range

26B-A4B scales:

• scale ≈ 0.01 (100x too small)
• Negative values (invalid)
• Bug in mlx-vlm quantization logic

---

MLX-vlm Version Analysis

mlx-vlm 0.4.3 (Used for 26B-A4B)

• Release date: Unknown (need check HuggingFace)
• Known issues: Quantization bugs?
• Affine mode: Problematic?

Alternative Versions

• mlx-vlm latest: May have fixes
• Custom quantization: More control

---

Recommended Actions

1. Check MLX-vlm Issues

Search:

• HuggingFace mlx-community repo issues
• GitHub mlx-vlm issues for "affine quantization"
• Look for scales bug reports

2. Re-quantize with Fixed Script

If MLX-vlm fixed:

• Download latest mlx-vlm
• Re-quantize from google/gemma-4-26b-a4b-it
• Verify scales range (~120)

If custom script:

• Use same method as 26B-Standard
• group_size=32, custom quant
• Manual scales verification

3. Report Issue

To MLX Community:

• HuggingFace: mlx-community/gemma-4-26b-a4b-it-4bit
• GitHub: mlx-vlm issue tracker
• Describe: scales too small + negative values
• Evidence: scales sample comparison

---

Model Card Information

Google Gemma-4-26B-A4B-IT

Official Model (pre-quantized):

• Publisher: Google
• License: Gemma license (Apache-style)
• Architecture: MoE (Mixture of Experts)
• Layers: 30
• Experts: 128 per layer
• Parameters: ~26B (active params)
• Special: A4B variant (Audio-Aware)

HuggingFace: google/gemma-4-26b-a4b-it

• BF16 weights (original)
• Used as base for MLX conversion

---

Alternative: Google Gemma-4-27B-IT

26B-Standard equivalent:

• Architecture: MoE, 30 layers, 128 experts
• Parameters: ~27B (similar to 26B-A4B)
• License: Same Gemma license
• Status: Available in BF16

If 26B-Standard is Gemma-4-27B-IT:

• Same architecture family
• Custom quantization (group_size=32)
• Correct scales ✓

---

Conclusion

26B-A4B problem traced to MLX-vlm 0.4.3 quantization bug

• Source: mlx-community/gemma-4-26b-a4b-it-4bit
• Converter: mlx-vlm 0.4.3 (buggy)
• Result: Wrong scales magnitude + negative values
• Solution: Use 26B-Standard (custom quant, correct scales)

---

Next Steps

1. Check HuggingFace:

   • mlx-community/gemma-4-26b-a4b-it-4bit issues
   • Look for reports of quantization bugs

2. Check GitHub:

   • mlx-vlm repository issues
   • Search "affine quantization" problems

3. Test MLX-vlm latest:

   • Download newer version if available
   • Test quantization on small model

4. Report Issue:

   • Provide scales sample evidence
   • Compare with custom quant (26B-Standard)

---

Files

A4B Model Files

/Users/accusys/MarkBaseEngine/models/gemma-4-26b-a4b-it-4bit/
  README.md: MLX metadata
  config.json: quantization config (group_size=64, affine)
  model-00001-of-00003.safetensors (4.9GB)
  model-00002-of-00003.safetensors (4.9GB)
  model-00003-of-00003.safetensors (4.7GB)

Standard Model Files

/Users/accusys/MarkBaseEngine/models/gemma-4-26b-standard/
  config.json: quantization config (group_size=32, custom)
  model.safetensors (15.6GB)
  No README (custom origin)

---

End of Source Analysis