MarkBase Admin
ac75faa0cc
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
298 lines
7.2 KiB
Markdown
298 lines
7.2 KiB
Markdown
# M5Max48 LLM Deployment Assessment
|
|
|
|
**Target**: 192.168.110.201 (M5Max48)
|
|
**Date**: 2026-06-23
|
|
**Status**: Assessment Complete
|
|
|
|
---
|
|
|
|
## System Specifications
|
|
|
|
### Hardware
|
|
- **Hostname**: M5Max48
|
|
- **Memory**: 48GB unified (51539607552 bytes)
|
|
- **Disk**: 1.8TB APFS, 12GB used, **47GB available**
|
|
- **OS**: macOS 26.5.1
|
|
|
|
### Current Usage
|
|
```
|
|
Total disk: 1.8TB
|
|
Used: 12GB (thin provisioning)
|
|
Available: 47GB for deployment
|
|
```
|
|
|
|
---
|
|
|
|
## Current Models Inventory
|
|
|
|
### GGUF Models (llama.cpp format)
|
|
```
|
|
gemma-4-31B-it-Q5_K_M.gguf 20GB ✓ (31B deployed)
|
|
google_gemma-4-26B-A4B-it-Q5_K_M 18GB (A4B GGUF, not MLX)
|
|
gemma-4-E4B-it-Q4_K_M.gguf 5GB ✓ (E4B GGUF)
|
|
mmproj-models 1GB (multimodal projections)
|
|
```
|
|
|
|
### MLX Models
|
|
```
|
|
gemma-4-e4b-it-4bit 4.9GB ✓ (MLX E4B)
|
|
mlx-gemma4-e4b-it-4bit 7.7GB ✓ (Alternative E4B)
|
|
mlx-gemma4-e4b-it-8bit 8.4GB ✓ (8-bit variant)
|
|
```
|
|
|
|
### HuggingFace Cache
|
|
```
|
|
models--google--gemma-4-12B-it 31MB (metadata only, not full model)
|
|
models--google--gemma-4-e2b-it 191MB (metadata only)
|
|
mlx-community--gemma-4-e4b-it-4bit 4.9GB ✓ (MLX cached)
|
|
mlx-community--gemma-4-e2b-it-8bit 3.1GB ✓ (E2B 8-bit cached)
|
|
paligemma models 27GB (vision models)
|
|
```
|
|
|
|
---
|
|
|
|
## Deployment Requirements
|
|
|
|
### Models to Deploy (from MarkBaseEngine)
|
|
|
|
| Model | Size | Source | Status on M5Max48 |
|
|
|-------|------|--------|-------------------|
|
|
| **E4B-MarkBase** | 4.67GB | E4B-MarkBase dir | Use existing mlx-gemma4-e4b (7.7GB) ✓ |
|
|
| **12B Standard** | ~4GB | MLX 12B cache | Need download (~4GB) |
|
|
| **26B-Standard** | 15.6GB | Local copy | Need copy (15.6GB) |
|
|
| **31B MLX** | ~20GB | Optional | Use existing GGUF (20GB) ✓ |
|
|
|
|
### Total Deployment Space
|
|
|
|
**Required**:
|
|
- 12B: ~4GB
|
|
- 26B: ~15.6GB
|
|
- MarkBaseEngine: ~200MB
|
|
- **Total new**: ~20GB
|
|
|
|
**Available**: 47GB ✓ (sufficient)
|
|
|
|
---
|
|
|
|
## Deployment Strategy
|
|
|
|
### Option 1: Use Existing MLX Models
|
|
```
|
|
E4B: Use mlx-gemma4-e4b-it-4bit (7.7GB) ✓
|
|
31B: Use gemma-4-31B-it-Q5_K_M.gguf (20GB) ✓
|
|
Deploy: 12B + 26B-Standard (~20GB)
|
|
```
|
|
|
|
### Option 2: Full MLX Deployment
|
|
```
|
|
Deploy all 4 models in MLX format:
|
|
- E4B-MarkBase: 4.67GB (copy)
|
|
- 12B Standard: 4GB (copy)
|
|
- 26B-Standard: 15.6GB (copy)
|
|
- 31B MLX: 20GB (optional, use GGUF)
|
|
```
|
|
|
|
---
|
|
|
|
## Deployment Plan
|
|
|
|
### Phase 1: MarkBaseEngine Setup (5 min)
|
|
```bash
|
|
ssh 192.168.110.201
|
|
cd ~
|
|
git clone [MarkBaseEngine repo]
|
|
swift build
|
|
```
|
|
|
|
### Phase 2: Use Existing Models (immediate)
|
|
```
|
|
E4B: ~/models/mlx-gemma4-e4b-it-4bit (7.7GB)
|
|
31B: ~/models/gemma-4-31B-it-Q5_K_M.gguf (20GB, GGUF)
|
|
```
|
|
|
|
### Phase 3: Deploy Missing Models (30-60 min)
|
|
```bash
|
|
# Copy from local MarkBaseEngine
|
|
scp -r models/gemma-4-26b-standard 192.168.110.201:~/models/
|
|
|
|
# Download 12B MLX (if needed)
|
|
ssh 192.168.110.201 "cd ~/models && huggingface-cli download mlx-community/gemma-4-12B-it-4bit"
|
|
```
|
|
|
|
---
|
|
|
|
## Space Optimization
|
|
|
|
### Clean Up Recommendations
|
|
```bash
|
|
# Remove duplicate E4B (keep largest)
|
|
rm ~/models/gemma-4-e4b-it-4bit # 4.9GB duplicate
|
|
rm ~/models/gemma-4-E4B-it-Q4_K_M.gguf # 5GB GGUF (use MLX)
|
|
|
|
# Remove unused vision models (if not needed)
|
|
rm ~/.cache/huggingface/hub/models--google--paligemma-* # 27GB
|
|
|
|
# Keep essential:
|
|
- mlx-gemma4-e4b-it-4bit (7.7GB) - E4B MLX
|
|
- gemma-4-31B-it-Q5_K_M.gguf (20GB) - 31B GGUF
|
|
```
|
|
|
|
**Space freed**: ~32GB → **79GB available** ✓
|
|
|
|
---
|
|
|
|
## Model Paths on M5Max48
|
|
|
|
### Existing (Verified)
|
|
```
|
|
E4B: /Users/accusys/models/mlx-gemma4-e4b-it-4bit/
|
|
31B: /Users/accusys/models/gemma-4-31B-it-Q5_K_M.gguf
|
|
E2B: ~/.cache/huggingface/hub/models--mlx-community--gemma-4-e2b-it-8bit/
|
|
```
|
|
|
|
### To Deploy
|
|
```
|
|
26B: ~/models/gemma-4-26b-standard/ (copy from local)
|
|
12B: ~/models/gemma-4-12b-it-4bit/ (download)
|
|
```
|
|
|
|
---
|
|
|
|
## Deployment Commands
|
|
|
|
### Step 1: Clone MarkBaseEngine
|
|
```bash
|
|
ssh 192.168.110.201
|
|
cd ~
|
|
git clone https://github.com/[repo]/MarkBaseEngine.git
|
|
cd MarkBaseEngine
|
|
swift build -c release
|
|
```
|
|
|
|
### Step 2: Copy 26B-Standard (from local)
|
|
```bash
|
|
# From local machine
|
|
scp -r /Users/accusys/coder/models/gemma-4-26b-standard \
|
|
192.168.110.201:/Users/accusys/models/
|
|
|
|
# Or use rsync for large files
|
|
rsync -avh --progress \
|
|
/Users/accusys/coder/models/gemma-4-26b-standard \
|
|
192.168.110.201:/Users/accusys/models/
|
|
```
|
|
|
|
### Step 3: Copy 12B Standard (from local)
|
|
```bash
|
|
# From local MarkBaseEngine
|
|
scp -r /Users/accusys/MarkBaseEngine/models/E4B-MarkBase \
|
|
192.168.110.201:/Users/accusys/models/
|
|
|
|
# Or use HuggingFace cache
|
|
scp -r ~/.cache/huggingface/hub/models--mlx-community--gemma-4-12B-it-4bit \
|
|
192.168.110.201:/Users/accusys/.cache/huggingface/hub/
|
|
```
|
|
|
|
---
|
|
|
|
## Network Transfer Estimates
|
|
|
|
### Bandwidth
|
|
- Local network: ~100Mbps (WiFi) or ~1Gbps (Ethernet)
|
|
- Transfer time estimates:
|
|
|
|
| Model | Size | WiFi (100Mbps) | Ethernet (1Gbps) |
|
|
|-------|------|----------------|-------------------|
|
|
| 26B | 15.6GB | ~20 min | ~2 min |
|
|
| 12B | 4GB | ~5 min | ~30 sec |
|
|
| E4B | 4.67GB | ~6 min | ~40 sec |
|
|
|
|
**Total**: ~30 min (WiFi) or ~3 min (Ethernet)
|
|
|
|
---
|
|
|
|
## Testing Commands
|
|
|
|
### Verify Models
|
|
```bash
|
|
ssh 192.168.110.201
|
|
cd ~/MarkBaseEngine
|
|
swift test --filter E4BMarkBaseTest
|
|
swift test --filter Model31BForwardTest
|
|
swift test --filter InferenceSpeedTest
|
|
```
|
|
|
|
### Performance Check
|
|
```bash
|
|
# TEXT inference speed
|
|
swift run MarkBaseServer --model ~/models/mlx-gemma4-e4b-it-4bit
|
|
|
|
# Expected: <30ms/token, >30 tok/s (48GB memory)
|
|
```
|
|
|
|
---
|
|
|
|
## Deployment Status
|
|
|
|
| Model | Local Status | M5Max48 Status | Action |
|
|
|-------|--------------|----------------|--------|
|
|
| **E4B** | ✓ Ready (E4B-MarkBase) | ✓ Existing (mlx-gemma4) | Use existing |
|
|
| **12B** | ✓ Ready (Standard) | ⚠ Metadata only | **Deploy needed** |
|
|
| **26B-Standard** | ✓ Ready | ✗ Missing | **Deploy needed** |
|
|
| **31B** | ✓ Ready | ✓ GGUF existing | Use GGUF |
|
|
|
|
---
|
|
|
|
## Recommendations
|
|
|
|
### Immediate Actions
|
|
1. **Clone MarkBaseEngine** to M5Max48 (~5 min)
|
|
2. **Use existing E4B** (mlx-gemma4-e4b-it-4bit)
|
|
3. **Copy 26B-Standard** (15.6GB, ~20 min WiFi)
|
|
4. **Copy 12B Standard** (4GB, ~5 min WiFi)
|
|
5. **Use existing 31B GGUF** (no copy needed)
|
|
|
|
### Space Optimization
|
|
- Clean up duplicate E4B models (free ~5GB)
|
|
- Clean up unused paligemma (free ~27GB) if not needed
|
|
- **Total freed**: ~32GB → **79GB available**
|
|
|
|
### Testing
|
|
- Run speed tests on M5Max48 (verify <30ms/token)
|
|
- Compare performance with local (M5 128GB)
|
|
- Validate zero NaN on all models
|
|
|
|
---
|
|
|
|
## Deployment Timeline
|
|
|
|
| Phase | Task | Duration |
|
|
|-------|------|----------|
|
|
| **1** | Clone MarkBaseEngine | 5 min |
|
|
| **2** | Build Swift project | 3 min |
|
|
| **3** | Copy 26B-Standard | 20 min (WiFi) |
|
|
| **4** | Copy 12B Standard | 5 min (WiFi) |
|
|
| **5** | Test models | 5 min |
|
|
| **Total** | **Full deployment** | **~40 min** |
|
|
|
|
---
|
|
|
|
## Final Checklist
|
|
|
|
- ✅ System specs verified (48GB memory, 47GB space)
|
|
- ✅ Existing models inventoried (E4B, 31B GGUF)
|
|
- ⚠️ MarkBaseEngine not installed (need clone)
|
|
- ⚠️ 12B Standard missing (need copy)
|
|
- ⚠️ 26B-Standard missing (need copy)
|
|
- ✅ Deployment plan ready (~40 min)
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
1. **Clone MarkBaseEngine** → `ssh 192.168.110.201 && git clone [repo]`
|
|
2. **Copy models** → `scp -r models/* 192.168.110.201:~/models/`
|
|
3. **Build and test** → `swift build && swift test`
|
|
|
|
---
|
|
|
|
**End of Deployment Assessment** |