markbaseengine/INTEGRATION_GUIDE.md

# MarkBaseEngine Integration Guide
## For momentry_core (Rust Backend) & momentry_studio (Frontend)

---

## Overview

MarkBaseEngine provides a high-performance inference engine for multimodal AI models (Text, Vision, Audio) on Apple Silicon. This guide explains how to integrate MarkBaseServer with your Rust backend and frontend.

---

## Architecture

```
┌─────────────────────────────────────────────────────────────────┐
│                      momentry_studio (Frontend)                  │
│                   TypeScript/React/Svelte/etc.                  │
└────────────────────────┬────────────────────────────────────────┘
                         │ HTTP/WebSocket
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                     momentry_core (Rust Backend)                 │
│              API Gateway, Auth, Business Logic                   │
└────────────────────────┬────────────────────────────────────────┘
                         │ HTTP REST API
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                    MarkBaseServer (Swift)                        │
│         OpenAI-Compatible API: Text/Vision/Audio                │
│         Port: 8080 (or 8083-8097 for dev)                       │
└────────────────────────┬────────────────────────────────────────┘
                         │ Metal GPU
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│                    MarkBaseEngine (Core)                         │
│        Model Loading, Inference, KV Cache, Multimodal           │
│        Models: E4B-MarkBase, 12B, 26B-Standard, 31B             │
└─────────────────────────────────────────────────────────────────┘
```

---

## MarkBaseServer API Endpoints

### Base URL
- **Local**: `http://127.0.0.1:8080/v1`
- **Production**: `http://10.10.10.201:8080/v1`

### Endpoints

| Endpoint | Method | Description |
|----------|--------|-------------|
| `/health` | GET | Server health check |
| `/v1/models` | GET | List available models |
| `/v1/chat/completions` | POST | Text generation |
| `/v1/multimodal/chat/completions` | POST | Vision+Audio+Text generation |

---

## 1. Text Model Integration

### Rust Backend (momentry_core)

```rust
use reqwest::Client;
use serde::{Deserialize, Serialize};

#[derive(Serialize)]
struct ChatRequest {
    model: String,
    messages: Vec<Message>,
    max_tokens: Option<u32>,
    temperature: Option<f32>,
    stream: Option<bool>,
}

#[derive(Serialize, Deserialize)]
struct Message {
    role: String,
    content: String,
}

#[derive(Deserialize)]
struct ChatResponse {
    id: String,
    choices: Vec<Choice>,
    usage: Usage,
}

#[derive(Deserialize)]
struct Choice {
    message: Message,
    finish_reason: String,
}

#[derive(Deserialize)]
struct Usage {
    prompt_tokens: u32,
    completion_tokens: u32,
    total_tokens: u32,
}

// Call MarkBaseServer for text generation
async fn generate_text(prompt: &str, model: &str) -> Result<String, Box<dyn std::error::Error>> {
    let client = Client::new();
    
    let request = ChatRequest {
        model: model.to_string(),
        messages: vec![
            Message { role: "user".to_string(), content: prompt.to_string() }
        ],
        max_tokens: Some(100),
        temperature: Some(0.7),
        stream: Some(false),
    };
    
    let response = client
        .post("http://10.10.10.201:8080/v1/chat/completions")
        .json(&request)
        .send()
        .await?
        .json::<ChatResponse>()
        .await?;
    
    Ok(response.choices[0].message.content)
}

// Available models
const MODELS: &[&str] = &[
    "gemma-4-e4b-markbase",   // 4B, optimized for speed
    "gemma-4-12b-it-4bit",    // 12B, balanced
    "gemma-4-26b-standard",   // 26B, high quality
    "gemma-4-31b",            // 31B, highest quality
];
```

### Frontend (momentry_studio)

```typescript
interface ChatRequest {
  model: string;
  messages: Array<{role: string, content: string}>;
  max_tokens?: number;
  temperature?: number;
  stream?: boolean;
}

interface ChatResponse {
  id: string;
  choices: Array<{
    message: {role: string, content: string};
    finish_reason: string;
  }>;
  usage: {
    prompt_tokens: number;
    completion_tokens: number;
    total_tokens: number;
  };
}

// Call via momentry_core backend proxy
async function generateText(prompt: string, model: string = 'gemma-4-e4b-markbase'): Promise<string> {
  const response = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model,
      messages: [{ role: 'user', content: prompt }],
      max_tokens: 100,
      temperature: 0.7,
    }),
  });
  
  const data: ChatResponse = await response.json();
  return data.choices[0].message.content;
}
```

---

## 2. Vision Model Integration

### Input Format
Vision models accept images encoded as base64 or URLs.

### Rust Backend

```rust
#[derive(Serialize)]
struct MultimodalChatRequest {
    model: String,
    messages: Vec<MultimodalMessage>,
    max_tokens: Option<u32>,
}

#[derive(Serialize)]
struct MultimodalMessage {
    role: String,
    content: Vec<ContentPart>,
}

#[derive(Serialize)]
#[serde(tag = "type")]
enum ContentPart {
    #[serde(rename = "text")]
    Text { text: String },
    #[serde(rename = "image_url")]
    ImageUrl { image_url: ImageUrl },
}

#[derive(Serialize)]
struct ImageUrl {
    url: String,  // base64 data URI or HTTP URL
}

// Vision inference
async fn analyze_image(image_path: &str, prompt: &str) -> Result<String, Box<dyn std::error::Error>> {
    let client = Client::new();
    
    // Read and encode image as base64
    let image_data = std::fs::read(image_path)?;
    let base64 = base64::encode(&image_data);
    let data_uri = format!("data:image/jpeg;base64,{}", base64);
    
    let request = MultimodalChatRequest {
        model: "gemma-4-12b-it-4bit".to_string(),
        messages: vec![
            MultimodalMessage {
                role: "user".to_string(),
                content: vec![
                    ContentPart::ImageUrl { 
                        image_url: ImageUrl { url: data_uri } 
                    },
                    ContentPart::Text { text: prompt.to_string() },
                ],
            },
        ],
        max_tokens: Some(200),
    };
    
    let response = client
        .post("http://10.10.10.201:8080/v1/multimodal/chat/completions")
        .json(&request)
        .send()
        .await?
        .json::<ChatResponse>()
        .await?;
    
    Ok(response.choices[0].message.content)
}
```

### Frontend

```typescript
interface MultimodalMessage {
  role: string;
  content: Array<{type: 'text', text: string} | {type: 'image_url', image_url: {url: string}}>;
}

async function analyzeImage(imageFile: File, prompt: string): Promise<string> {
  // Convert image to base64
  const base64 = await new Promise<string>((resolve) => {
    const reader = new FileReader();
    reader.onload = () => resolve(reader.result as string);
    reader.readAsDataURL(imageFile);
  });
  
  const response = await fetch('/api/multimodal/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'gemma-4-12b-it-4bit',
      messages: [{
        role: 'user',
        content: [
          { type: 'image_url', image_url: { url: base64 } },
          { type: 'text', text: prompt },
        ],
      }],
      max_tokens: 200,
    }),
  });
  
  const data = await response.json();
  return data.choices[0].message.content;
}
```

---

## 3. Audio Model Integration

### Audio Input Format
Audio models accept audio files (WAV, MP3, AAC) encoded as base64.

### Rust Backend

```rust
#[derive(Serialize)]
struct AudioChatRequest {
    model: String,
    messages: Vec<AudioMessage>,
    max_tokens: Option<u32>,
}

#[derive(Serialize)]
struct AudioMessage {
    role: String,
    content: Vec<AudioContentPart>,
}

#[derive(Serialize)]
#[serde(tag = "type")]
enum AudioContentPart {
    #[serde(rename = "text")]
    Text { text: String },
    #[serde(rename = "audio_url")]
    AudioUrl { audio_url: AudioUrl },
}

#[derive(Serialize)]
struct AudioUrl {
    url: String,  // base64 data URI
}

// Audio transcription/analysis
async fn process_audio(audio_path: &str, prompt: &str) -> Result<String, Box<dyn std::error::Error>> {
    let client = Client::new();
    
    let audio_data = std::fs::read(audio_path)?;
    let base64 = base64::encode(&audio_data);
    let data_uri = format!("data:audio/wav;base64,{}", base64);
    
    let request = AudioChatRequest {
        model: "gemma-4-12b-it-4bit".to_string(),
        messages: vec![
            AudioMessage {
                role: "user".to_string(),
                content: vec![
                    AudioContentPart::AudioUrl { 
                        audio_url: AudioUrl { url: data_uri } 
                    },
                    AudioContentPart::Text { text: prompt.to_string() },
                ],
            },
        ],
        max_tokens: Some(100),
    };
    
    let response = client
        .post("http://10.10.10.201:8080/v1/multimodal/chat/completions")
        .json(&request)
        .send()
        .await?
        .json::<ChatResponse>()
        .await?;
    
    Ok(response.choices[0].message.content)
}
```

### Frontend

```typescript
async function processAudio(audioFile: File, prompt: string): Promise<string> {
  const base64 = await new Promise<string>((resolve) => {
    const reader = new FileReader();
    reader.onload = () => resolve(reader.result as string);
    reader.readAsDataURL(audioFile);
  });
  
  const response = await fetch('/api/multimodal/chat', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'gemma-4-12b-it-4bit',
      messages: [{
        role: 'user',
        content: [
          { type: 'audio_url', audio_url: { url: base64 } },
          { type: 'text', text: prompt },
        ],
      }],
      max_tokens: 100,
    }),
  });
  
  const data = await response.json();
  return data.choices[0].message.content;
}
```

---

## 4. Streaming Responses

### Server-Sent Events (SSE)

MarkBaseServer supports streaming via SSE when `stream: true` is set.

### Rust Backend

```rust
use futures::StreamExt;

async fn stream_text(prompt: &str, model: &str) -> Result<String, Box<dyn std::error::Error>> {
    let client = Client::new();
    
    let request = ChatRequest {
        model: model.to_string(),
        messages: vec![Message { role: "user".to_string(), content: prompt.to_string() }],
        max_tokens: Some(100),
        stream: Some(true),
    };
    
    let mut stream = client
        .post("http://10.10.10.201:8080/v1/chat/completions")
        .json(&request)
        .send()
        .await?
        .bytes_stream();
    
    let mut full_text = String::new();
    
    while let Some(chunk) = stream.next().await {
        let chunk = chunk?;
        let text = String::from_utf8_lossy(&chunk);
        
        // Parse SSE format: "data: {...}\n\n"
        for line in text.lines() {
            if line.starts_with("data: ") {
                let json_str = &line[6..];
                if json_str == "[DONE]" { break; }
                
                let chunk_data: serde_json::Value = serde_json::from_str(json_str)?;
                if let Some(content) = chunk_data["choices"][0]["delta"]["content"].as_str() {
                    full_text.push_str(content);
                    // Send to frontend via WebSocket
                }
            }
        }
    }
    
    Ok(full_text)
}
```

### Frontend

```typescript
async function streamText(prompt: string, onChunk: (text: string) => void): Promise<void> {
  const response = await fetch('/api/chat/stream', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      model: 'gemma-4-e4b-markbase',
      messages: [{ role: 'user', content: prompt }],
      stream: true,
    }),
  });
  
  const reader = response.body?.getReader();
  const decoder = new TextDecoder();
  
  while (reader) {
    const { done, value } = await reader.read();
    if (done) break;
    
    const text = decoder.decode(value);
    for (const line of text.split('\n')) {
      if (line.startsWith('data: ')) {
        const json = line.slice(6);
        if (json === '[DONE]') break;
        
        const data = JSON.parse(json);
        const content = data.choices[0]?.delta?.content || '';
        onChunk(content);
      }
    }
  }
}
```

---

## 5. Model Selection Guide

| Model | Size | Speed | Quality | Use Case |
|-------|------|-------|---------|----------|
| E4B-MarkBase | 4.4GB | 49ms/token | Good | Real-time chat, quick responses |
| 12B | 6.3GB | 6ms/token (158 tok/s) | Better | Balanced speed/quality |
| 26B-Standard | 15GB | 30ms/token | High | Complex reasoning, code generation |
| 31B | 17GB | 38ms/token | Highest | Deep analysis, expert tasks |

### Recommendation Matrix

| Scenario | Recommended Model |
|----------|-------------------|
| Chat UI autocomplete | E4B-MarkBase |
| Document summarization | 12B or 26B-Standard |
| Code generation | 26B-Standard |
| Vision analysis | 12B (has VisionTower12B) |
| Audio transcription | 12B (has AudioTower12B) |
| Expert reasoning | 31B |

---

## 6. Performance Optimization

### KV Cache Management
MarkBaseServer automatically manages KV cache. For long conversations:

```rust
// Clear context for new conversation
async fn reset_context(session_id: &str) {
    // MarkBaseServer handles this internally
    // Just start a new messages array
}
```

### Concurrent Requests
MarkBaseServer handles concurrent requests efficiently:

- **Text**: Up to 10 concurrent streams
- **Vision**: 2-3 concurrent (GPU intensive)
- **Audio**: 2-3 concurrent (GPU intensive)

### Memory Limits
- **M5Max48 (48GB)**: Max 3 models loaded concurrently
- **M5 (128GB)**: All 4 models can be loaded

---

## 7. Deployment Configuration

### MarkBaseServer Startup

```bash
# Local development (M5 128GB)
cd ~/MarkBaseEngine
./start_server.sh

# Production (M5Max48 via TBT5)
# Deploy models first:
rsync -avP ~/MarkBaseEngine/models/ 10.10.10.201:/Volumes/TBT5/models/

# Start server on M5Max48:
ssh 10.10.10.201
cd /Volumes/TBT5/MarkBaseEngine
./build/release/MarkBaseServer ./models/E4B-MarkBase 8080 gemma-4-e4b-markbase
```

### Rust Backend Configuration

```rust
// config.rs
pub struct MarkBaseConfig {
    pub base_url: String,
    pub default_model: String,
    pub timeout_ms: u64,
}

impl Default for MarkBaseConfig {
    fn default() -> Self {
        Self {
            base_url: "http://10.10.10.201:8080/v1".to_string(),
            default_model: "gemma-4-e4b-markbase".to_string(),
            timeout_ms: 30000,
        }
    }
}
```

---

## 8. Error Handling

### Common Errors

| Error | Cause | Solution |
|-------|-------|----------|
| Connection refused | Server not running | Check `./start_server.sh` |
| Model not found | Wrong model name | Check `/v1/models` endpoint |
| Timeout | Large input/slow model | Increase timeout, use faster model |
| GPU memory limit | Too many concurrent | Reduce concurrent requests |
| NaN output | Forward pass bug | Report to MarkBaseEngine team |

### Rust Error Handling

```rust
use thiserror::Error;

#[derive(Error, Debug)]
pub enum MarkBaseError {
    #[error("Connection failed: {0}")]
    ConnectionFailed(String),
    #[error("Model not found: {0}")]
    ModelNotFound(String),
    #[error("Timeout after {0}ms")]
    Timeout(u64),
    #[error("Invalid response: {0}")]
    InvalidResponse(String),
}

impl From<reqwest::Error> for MarkBaseError {
    fn from(e: reqwest::Error) -> Self {
        if e.is_timeout() {
            MarkBaseError::Timeout(30000)
        } else if e.is_connect() {
            MarkBaseError::ConnectionFailed(e.to_string())
        } else {
            MarkBaseError::InvalidResponse(e.to_string())
        }
    }
}
```

---

## 9. Testing & Validation

### Health Check

```rust
async fn check_health() -> bool {
    let client = Client::new();
    let response = client
        .get("http://10.10.10.201:8080/health")
        .send()
        .await;
    
    response.is_ok()
}
```

### Model List

```rust
async fn list_models() -> Result<Vec<String>, Box<dyn std::error::Error>> {
    let client = Client::new();
    let response = client
        .get("http://10.10.10.201:8080/v1/models")
        .send()
        .await?
        .json::<serde_json::Value>()
        .await?;
    
    let models = response["data"]
        .as_array()
        .unwrap_or(&vec![])
        .iter()
        .filter_map(|m| m["id"].as_str().map(|s| s.to_string()))
        .collect();
    
    Ok(models)
}
```

---

## 10. Security Considerations

### API Gateway (momentry_core)

```rust
// Add authentication layer
use actix_web::{web, HttpRequest, HttpResponse};

async fn chat_proxy(
    req: HttpRequest,
    body: web::Json<ChatRequest>,
) -> HttpResponse {
    // Validate auth token
    let auth = req.headers().get("Authorization");
    if !validate_auth(auth) {
        return HttpResponse::Unauthorized().finish();
    }
    
    // Rate limiting
    if !check_rate_limit(&req) {
        return HttpResponse::TooManyRequests().finish();
    }
    
    // Forward to MarkBaseServer
    let response = forward_to_markbase(body.into_inner());
    
    HttpResponse::Ok().json(response)
}
```

### Input Validation

```rust
fn validate_chat_request(req: &ChatRequest) -> Result<(), String> {
    if req.messages.is_empty() {
        return Err("Messages array cannot be empty".to_string());
    }
    
    if req.max_tokens.unwrap_or(100) > 2048 {
        return Err("max_tokens cannot exceed 2048".to_string());
    }
    
    Ok(())
}
```

---

## 11. Complete Example: momentry_core Integration

```rust
// src/markbase_client.rs
use reqwest::Client;
use serde::{Deserialize, Serialize};
use std::time::Duration;

pub struct MarkBaseClient {
    client: Client,
    base_url: String,
    default_model: String,
}

impl MarkBaseClient {
    pub fn new(base_url: &str, default_model: &str) -> Self {
        let client = Client::builder()
            .timeout(Duration::from_secs(30))
            .build()
            .unwrap();
        
        Self {
            client,
            base_url: base_url.to_string(),
            default_model: default_model.to_string(),
        }
    }
    
    pub async fn chat(&self, prompt: &str) -> Result<String, MarkBaseError> {
        self.chat_with_model(prompt, &self.default_model).await
    }
    
    pub async fn chat_with_model(&self, prompt: &str, model: &str) -> Result<String, MarkBaseError> {
        let request = ChatRequest {
            model: model.to_string(),
            messages: vec![Message { role: "user".to_string(), content: prompt.to_string() }],
            max_tokens: Some(100),
            temperature: Some(0.7),
            stream: Some(false),
        };
        
        let url = format!("{}{}", self.base_url, "/chat/completions");
        let response = self.client
            .post(&url)
            .json(&request)
            .send()
            .await?
            .json::<ChatResponse>()
            .await?;
        
        Ok(response.choices[0].message.content)
    }
    
    pub async fn vision(&self, image_base64: &str, prompt: &str) -> Result<String, MarkBaseError> {
        let request = MultimodalChatRequest {
            model: self.default_model.clone(),
            messages: vec![
                MultimodalMessage {
                    role: "user".to_string(),
                    content: vec![
                        ContentPart::ImageUrl { 
                            image_url: ImageUrl { url: format!("data:image/jpeg;base64,{}", image_base64) } 
                        },
                        ContentPart::Text { text: prompt.to_string() },
                    ],
                },
            ],
            max_tokens: Some(200),
        };
        
        let url = format!("{}{}", self.base_url, "/multimodal/chat/completions");
        let response = self.client
            .post(&url)
            .json(&request)
            .send()
            .await?
            .json::<ChatResponse>()
            .await?;
        
        Ok(response.choices[0].message.content)
    }
    
    pub async fn audio(&self, audio_base64: &str, prompt: &str) -> Result<String, MarkBaseError> {
        let request = AudioChatRequest {
            model: self.default_model.clone(),
            messages: vec![
                AudioMessage {
                    role: "user".to_string(),
                    content: vec![
                        AudioContentPart::AudioUrl { 
                            audio_url: AudioUrl { url: format!("data:audio/wav;base64,{}", audio_base64) } 
                        },
                        AudioContentPart::Text { text: prompt.to_string() },
                    ],
                },
            ],
            max_tokens: Some(100),
        };
        
        let url = format!("{}{}", self.base_url, "/multimodal/chat/completions");
        let response = self.client
            .post(&url)
            .json(&request)
            .send()
            .await?
            .json::<ChatResponse>()
            .await?;
        
        Ok(response.choices[0].message.content)
    }
    
    pub async fn health_check(&self) -> bool {
        let url = format!("{}{}", self.base_url.replace("/v1", ""), "/health");
        self.client.get(&url).send().await.is_ok()
    }
}

// Usage in main.rs
#[actix_web::main]
async fn main() -> std::io::Result<()> {
    let markbase = MarkBaseClient::new(
        "http://10.10.10.201:8080/v1",
        "gemma-4-e4b-markbase",
    );
    
    // Test connection
    if !markbase.health_check().await {
        eprintln!("MarkBaseServer not responding!");
    }
    
    // Use in routes
    HttpServer::new(|| {
        App::new()
            .app_data(web::Data::new(markbase.clone()))
            .route("/api/chat", web::post().to(chat_handler))
            .route("/api/vision", web::post().to(vision_handler))
            .route("/api/audio", web::post().to(audio_handler))
    })
    .bind("127.0.0.1:3000")?
    .run()
    .await
}
```

---

## 12. Monitoring & Logging

### Performance Metrics

```rust
use std::time::Instant;

async fn monitored_chat(client: &MarkBaseClient, prompt: &str) -> Result<(String, u64), MarkBaseError> {
    let start = Instant::now();
    let response = client.chat(prompt).await?;
    let latency_ms = start.elapsed().as_millis() as u64;
    
    // Log to monitoring system
    log::info!("Chat latency: {}ms, tokens: {}", latency_ms, response.len());
    
    Ok((response, latency_ms))
}
```

### Structured Logging

```rust
use serde_json::json;

fn log_request(model: &str, prompt_len: usize, latency_ms: u64) {
    let log_entry = json!({
        "timestamp": chrono::Utc::now().to_rfc3339(),
        "model": model,
        "prompt_length": prompt_len,
        "latency_ms": latency_ms,
        "server": "MarkBaseServer",
    });
    
    println!("{}", log_entry);
}
```

---

## Summary

This guide provides complete integration patterns for:

1. **Text Models**: Simple chat completion via `/v1/chat/completions`
2. **Vision Models**: Image analysis via `/v1/multimodal/chat/completions` with base64 images
3. **Audio Models**: Audio processing via `/v1/multimodal/chat/completions` with base64 audio
4. **Streaming**: SSE support for real-time UI updates
5. **Model Selection**: Choose based on speed/quality tradeoff
6. **Performance**: Optimized for Apple Silicon Metal GPU

### Next Steps

1. Set up MarkBaseServer on production server (M5Max48)
2. Integrate Rust client into momentry_core
3. Build frontend UI with streaming support
4. Add authentication and rate limiting
5. Deploy and monitor performance

---

**Document Version**: 1.0
**Last Updated**: 2026-06-23
**Author**: MarkBaseEngine Team