warren/markbaseengine
1
0
Fork 0
Code Issues Pull Requests Actions 18 Packages Projects Releases Wiki Activity
Files
285dc4bce400b4406e673f3b0c16e88d54da8372
markbaseengine/M5MAX48_DEPLOYMENT_ASSESSMENT.md
T
MarkBase Admin ac75faa0cc
CI / build-and-test (push) Has been cancelled
Details
Initial commit: E4B-MarkBase model integration with passing tests 
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully
- All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance)
- All stress tests passed (5/5 in 127.6s)
  - Concurrent inference
  - Memory stress (67.5 tok/s, 0 NaN)
  - Continuous generation
  - Batch processing
  - Long-running stability
- Swift Metal inference engine with multimodal support
2026-06-23 18:12:35 +08:00

7.2 KiB
Raw Blame History

M5Max48 LLM Deployment Assessment

Target: 192.168.110.201 (M5Max48)
Date: 2026-06-23
Status: Assessment Complete

System Specifications

Hardware

  • Hostname: M5Max48
  • Memory: 48GB unified (51539607552 bytes)
  • Disk: 1.8TB APFS, 12GB used, 47GB available
  • OS: macOS 26.5.1

Current Usage

Total disk: 1.8TB
Used: 12GB (thin provisioning)
Available: 47GB for deployment

Current Models Inventory

GGUF Models (llama.cpp format)

gemma-4-31B-it-Q5_K_M.gguf        20GB  ✓ (31B deployed)
google_gemma-4-26B-A4B-it-Q5_K_M   18GB  (A4B GGUF, not MLX)
gemma-4-E4B-it-Q4_K_M.gguf        5GB   ✓ (E4B GGUF)
mmproj-models                      1GB   (multimodal projections)

MLX Models

gemma-4-e4b-it-4bit                4.9GB  ✓ (MLX E4B)
mlx-gemma4-e4b-it-4bit             7.7GB  ✓ (Alternative E4B)
mlx-gemma4-e4b-it-8bit             8.4GB  ✓ (8-bit variant)

HuggingFace Cache

models--google--gemma-4-12B-it     31MB   (metadata only, not full model)
models--google--gemma-4-e2b-it     191MB  (metadata only)
mlx-community--gemma-4-e4b-it-4bit 4.9GB  ✓ (MLX cached)
mlx-community--gemma-4-e2b-it-8bit 3.1GB  ✓ (E2B 8-bit cached)
paligemma models                   27GB   (vision models)

Deployment Requirements

Models to Deploy (from MarkBaseEngine)

Model Size Source Status on M5Max48
E4B-MarkBase 4.67GB E4B-MarkBase dir Use existing mlx-gemma4-e4b (7.7GB) ✓
12B Standard ~4GB MLX 12B cache Need download (~4GB)
26B-Standard 15.6GB Local copy Need copy (15.6GB)
31B MLX ~20GB Optional Use existing GGUF (20GB) ✓

Total Deployment Space

Required:

  • 12B: ~4GB
  • 26B: ~15.6GB
  • MarkBaseEngine: ~200MB
  • Total new: ~20GB

Available: 47GB ✓ (sufficient)

Deployment Strategy

Option 1: Use Existing MLX Models

E4B: Use mlx-gemma4-e4b-it-4bit (7.7GB) ✓
31B: Use gemma-4-31B-it-Q5_K_M.gguf (20GB) ✓
Deploy: 12B + 26B-Standard (~20GB)

Option 2: Full MLX Deployment

Deploy all 4 models in MLX format:
- E4B-MarkBase: 4.67GB (copy)
- 12B Standard: 4GB (copy)
- 26B-Standard: 15.6GB (copy)
- 31B MLX: 20GB (optional, use GGUF)

Deployment Plan

Phase 1: MarkBaseEngine Setup (5 min)

ssh 192.168.110.201
cd ~
git clone [MarkBaseEngine repo]
swift build

Phase 2: Use Existing Models (immediate)

E4B: ~/models/mlx-gemma4-e4b-it-4bit (7.7GB)
31B: ~/models/gemma-4-31B-it-Q5_K_M.gguf (20GB, GGUF)

Phase 3: Deploy Missing Models (30-60 min)

# Copy from local MarkBaseEngine
scp -r models/gemma-4-26b-standard 192.168.110.201:~/models/

# Download 12B MLX (if needed)
ssh 192.168.110.201 "cd ~/models && huggingface-cli download mlx-community/gemma-4-12B-it-4bit"

Space Optimization

Clean Up Recommendations

# Remove duplicate E4B (keep largest)
rm ~/models/gemma-4-e4b-it-4bit  # 4.9GB duplicate
rm ~/models/gemma-4-E4B-it-Q4_K_M.gguf  # 5GB GGUF (use MLX)

# Remove unused vision models (if not needed)
rm ~/.cache/huggingface/hub/models--google--paligemma-*  # 27GB

# Keep essential:
- mlx-gemma4-e4b-it-4bit (7.7GB) - E4B MLX
- gemma-4-31B-it-Q5_K_M.gguf (20GB) - 31B GGUF

Space freed: ~32GB → 79GB available ✓

Model Paths on M5Max48

Existing (Verified)

E4B: /Users/accusys/models/mlx-gemma4-e4b-it-4bit/
31B: /Users/accusys/models/gemma-4-31B-it-Q5_K_M.gguf
E2B: ~/.cache/huggingface/hub/models--mlx-community--gemma-4-e2b-it-8bit/

To Deploy

26B: ~/models/gemma-4-26b-standard/  (copy from local)
12B: ~/models/gemma-4-12b-it-4bit/   (download)

Deployment Commands

Step 1: Clone MarkBaseEngine

ssh 192.168.110.201
cd ~
git clone https://github.com/[repo]/MarkBaseEngine.git
cd MarkBaseEngine
swift build -c release

Step 2: Copy 26B-Standard (from local)

# From local machine
scp -r /Users/accusys/coder/models/gemma-4-26b-standard \
      192.168.110.201:/Users/accusys/models/

# Or use rsync for large files
rsync -avh --progress \
      /Users/accusys/coder/models/gemma-4-26b-standard \
      192.168.110.201:/Users/accusys/models/

Step 3: Copy 12B Standard (from local)

# From local MarkBaseEngine
scp -r /Users/accusys/MarkBaseEngine/models/E4B-MarkBase \
      192.168.110.201:/Users/accusys/models/

# Or use HuggingFace cache
scp -r ~/.cache/huggingface/hub/models--mlx-community--gemma-4-12B-it-4bit \
      192.168.110.201:/Users/accusys/.cache/huggingface/hub/

Network Transfer Estimates

Bandwidth

  • Local network: ~100Mbps (WiFi) or ~1Gbps (Ethernet)
  • Transfer time estimates:
Model Size WiFi (100Mbps) Ethernet (1Gbps)
26B 15.6GB ~20 min ~2 min
12B 4GB ~5 min ~30 sec
E4B 4.67GB ~6 min ~40 sec

Total: ~30 min (WiFi) or ~3 min (Ethernet)

Testing Commands

Verify Models

ssh 192.168.110.201
cd ~/MarkBaseEngine
swift test --filter E4BMarkBaseTest
swift test --filter Model31BForwardTest
swift test --filter InferenceSpeedTest

Performance Check

# TEXT inference speed
swift run MarkBaseServer --model ~/models/mlx-gemma4-e4b-it-4bit

# Expected: <30ms/token, >30 tok/s (48GB memory)

Deployment Status

Model Local Status M5Max48 Status Action
E4B ✓ Ready (E4B-MarkBase) ✓ Existing (mlx-gemma4) Use existing
12B ✓ Ready (Standard) ⚠ Metadata only Deploy needed
26B-Standard ✓ Ready ✗ Missing Deploy needed
31B ✓ Ready ✓ GGUF existing Use GGUF

Recommendations

Immediate Actions

  1. Clone MarkBaseEngine to M5Max48 (~5 min)
  2. Use existing E4B (mlx-gemma4-e4b-it-4bit)
  3. Copy 26B-Standard (15.6GB, ~20 min WiFi)
  4. Copy 12B Standard (4GB, ~5 min WiFi)
  5. Use existing 31B GGUF (no copy needed)

Space Optimization

  • Clean up duplicate E4B models (free ~5GB)
  • Clean up unused paligemma (free ~27GB) if not needed
  • Total freed: ~32GB → 79GB available

Testing

  • Run speed tests on M5Max48 (verify <30ms/token)
  • Compare performance with local (M5 128GB)
  • Validate zero NaN on all models

Deployment Timeline

Phase Task Duration
1 Clone MarkBaseEngine 5 min
2 Build Swift project 3 min
3 Copy 26B-Standard 20 min (WiFi)
4 Copy 12B Standard 5 min (WiFi)
5 Test models 5 min
Total Full deployment ~40 min

Final Checklist

  • ✅ System specs verified (48GB memory, 47GB space)
  • ✅ Existing models inventoried (E4B, 31B GGUF)
  • ⚠️ MarkBaseEngine not installed (need clone)
  • ⚠️ 12B Standard missing (need copy)
  • ⚠️ 26B-Standard missing (need copy)
  • ✅ Deployment plan ready (~40 min)

Next Steps

  1. Clone MarkBaseEngine → ssh 192.168.110.201 && git clone [repo]
  2. Copy models → scp -r models/* 192.168.110.201:~/models/
  3. Build and test → swift build && swift test

End of Deployment Assessment

Reference in New Issue View Git Blame Copy Permalink