Initial commit: E4B-MarkBase model integration with passing tests
CI / build-and-test (push) Has been cancelled
CI / build-and-test (push) Has been cancelled
- E4B-MarkBase model (42 layers, 4.4GB) loaded successfully - All Phase 1-6 tests passed (model loading, forward pass, vision/audio towers, token generation, performance) - All stress tests passed (5/5 in 127.6s) - Concurrent inference - Memory stress (67.5 tok/s, 0 NaN) - Continuous generation - Batch processing - Long-running stability - Swift Metal inference engine with multimodal support
This commit is contained in:
MarkBase Admin
@@ -0,0 +1,918 @@
|
||||
# MarkBaseEngine Integration Guide
|
||||
## For momentry_core (Rust Backend) & momentry_studio (Frontend)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
MarkBaseEngine provides a high-performance inference engine for multimodal AI models (Text, Vision, Audio) on Apple Silicon. This guide explains how to integrate MarkBaseServer with your Rust backend and frontend.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ momentry_studio (Frontend) │
|
||||
│ TypeScript/React/Svelte/etc. │
|
||||
└────────────────────────┬────────────────────────────────────────┘
|
||||
│ HTTP/WebSocket
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ momentry_core (Rust Backend) │
|
||||
│ API Gateway, Auth, Business Logic │
|
||||
└────────────────────────┬────────────────────────────────────────┘
|
||||
│ HTTP REST API
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ MarkBaseServer (Swift) │
|
||||
│ OpenAI-Compatible API: Text/Vision/Audio │
|
||||
│ Port: 8080 (or 8083-8097 for dev) │
|
||||
└────────────────────────┬────────────────────────────────────────┘
|
||||
│ Metal GPU
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────────┐
|
||||
│ MarkBaseEngine (Core) │
|
||||
│ Model Loading, Inference, KV Cache, Multimodal │
|
||||
│ Models: E4B-MarkBase, 12B, 26B-Standard, 31B │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MarkBaseServer API Endpoints
|
||||
|
||||
### Base URL
|
||||
- **Local**: `http://127.0.0.1:8080/v1`
|
||||
- **Production**: `http://10.10.10.201:8080/v1`
|
||||
|
||||
### Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| `/health` | GET | Server health check |
|
||||
| `/v1/models` | GET | List available models |
|
||||
| `/v1/chat/completions` | POST | Text generation |
|
||||
| `/v1/multimodal/chat/completions` | POST | Vision+Audio+Text generation |
|
||||
|
||||
---
|
||||
|
||||
## 1. Text Model Integration
|
||||
|
||||
### Rust Backend (momentry_core)
|
||||
|
||||
```rust
|
||||
use reqwest::Client;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct ChatRequest {
|
||||
model: String,
|
||||
messages: Vec<Message>,
|
||||
max_tokens: Option<u32>,
|
||||
temperature: Option<f32>,
|
||||
stream: Option<bool>,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
struct Message {
|
||||
role: String,
|
||||
content: String,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct ChatResponse {
|
||||
id: String,
|
||||
choices: Vec<Choice>,
|
||||
usage: Usage,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct Choice {
|
||||
message: Message,
|
||||
finish_reason: String,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct Usage {
|
||||
prompt_tokens: u32,
|
||||
completion_tokens: u32,
|
||||
total_tokens: u32,
|
||||
}
|
||||
|
||||
// Call MarkBaseServer for text generation
|
||||
async fn generate_text(prompt: &str, model: &str) -> Result<String, Box<dyn std::error::Error>> {
|
||||
let client = Client::new();
|
||||
|
||||
let request = ChatRequest {
|
||||
model: model.to_string(),
|
||||
messages: vec![
|
||||
Message { role: "user".to_string(), content: prompt.to_string() }
|
||||
],
|
||||
max_tokens: Some(100),
|
||||
temperature: Some(0.7),
|
||||
stream: Some(false),
|
||||
};
|
||||
|
||||
let response = client
|
||||
.post("http://10.10.10.201:8080/v1/chat/completions")
|
||||
.json(&request)
|
||||
.send()
|
||||
.await?
|
||||
.json::<ChatResponse>()
|
||||
.await?;
|
||||
|
||||
Ok(response.choices[0].message.content)
|
||||
}
|
||||
|
||||
// Available models
|
||||
const MODELS: &[&str] = &[
|
||||
"gemma-4-e4b-markbase", // 4B, optimized for speed
|
||||
"gemma-4-12b-it-4bit", // 12B, balanced
|
||||
"gemma-4-26b-standard", // 26B, high quality
|
||||
"gemma-4-31b", // 31B, highest quality
|
||||
];
|
||||
```
|
||||
|
||||
### Frontend (momentry_studio)
|
||||
|
||||
```typescript
|
||||
interface ChatRequest {
|
||||
model: string;
|
||||
messages: Array<{role: string, content: string}>;
|
||||
max_tokens?: number;
|
||||
temperature?: number;
|
||||
stream?: boolean;
|
||||
}
|
||||
|
||||
interface ChatResponse {
|
||||
id: string;
|
||||
choices: Array<{
|
||||
message: {role: string, content: string};
|
||||
finish_reason: string;
|
||||
}>;
|
||||
usage: {
|
||||
prompt_tokens: number;
|
||||
completion_tokens: number;
|
||||
total_tokens: number;
|
||||
};
|
||||
}
|
||||
|
||||
// Call via momentry_core backend proxy
|
||||
async function generateText(prompt: string, model: string = 'gemma-4-e4b-markbase'): Promise<string> {
|
||||
const response = await fetch('/api/chat', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model,
|
||||
messages: [{ role: 'user', content: prompt }],
|
||||
max_tokens: 100,
|
||||
temperature: 0.7,
|
||||
}),
|
||||
});
|
||||
|
||||
const data: ChatResponse = await response.json();
|
||||
return data.choices[0].message.content;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Vision Model Integration
|
||||
|
||||
### Input Format
|
||||
Vision models accept images encoded as base64 or URLs.
|
||||
|
||||
### Rust Backend
|
||||
|
||||
```rust
|
||||
#[derive(Serialize)]
|
||||
struct MultimodalChatRequest {
|
||||
model: String,
|
||||
messages: Vec<MultimodalMessage>,
|
||||
max_tokens: Option<u32>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct MultimodalMessage {
|
||||
role: String,
|
||||
content: Vec<ContentPart>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
#[serde(tag = "type")]
|
||||
enum ContentPart {
|
||||
#[serde(rename = "text")]
|
||||
Text { text: String },
|
||||
#[serde(rename = "image_url")]
|
||||
ImageUrl { image_url: ImageUrl },
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct ImageUrl {
|
||||
url: String, // base64 data URI or HTTP URL
|
||||
}
|
||||
|
||||
// Vision inference
|
||||
async fn analyze_image(image_path: &str, prompt: &str) -> Result<String, Box<dyn std::error::Error>> {
|
||||
let client = Client::new();
|
||||
|
||||
// Read and encode image as base64
|
||||
let image_data = std::fs::read(image_path)?;
|
||||
let base64 = base64::encode(&image_data);
|
||||
let data_uri = format!("data:image/jpeg;base64,{}", base64);
|
||||
|
||||
let request = MultimodalChatRequest {
|
||||
model: "gemma-4-12b-it-4bit".to_string(),
|
||||
messages: vec![
|
||||
MultimodalMessage {
|
||||
role: "user".to_string(),
|
||||
content: vec![
|
||||
ContentPart::ImageUrl {
|
||||
image_url: ImageUrl { url: data_uri }
|
||||
},
|
||||
ContentPart::Text { text: prompt.to_string() },
|
||||
],
|
||||
},
|
||||
],
|
||||
max_tokens: Some(200),
|
||||
};
|
||||
|
||||
let response = client
|
||||
.post("http://10.10.10.201:8080/v1/multimodal/chat/completions")
|
||||
.json(&request)
|
||||
.send()
|
||||
.await?
|
||||
.json::<ChatResponse>()
|
||||
.await?;
|
||||
|
||||
Ok(response.choices[0].message.content)
|
||||
}
|
||||
```
|
||||
|
||||
### Frontend
|
||||
|
||||
```typescript
|
||||
interface MultimodalMessage {
|
||||
role: string;
|
||||
content: Array<{type: 'text', text: string} | {type: 'image_url', image_url: {url: string}}>;
|
||||
}
|
||||
|
||||
async function analyzeImage(imageFile: File, prompt: string): Promise<string> {
|
||||
// Convert image to base64
|
||||
const base64 = await new Promise<string>((resolve) => {
|
||||
const reader = new FileReader();
|
||||
reader.onload = () => resolve(reader.result as string);
|
||||
reader.readAsDataURL(imageFile);
|
||||
});
|
||||
|
||||
const response = await fetch('/api/multimodal/chat', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: 'gemma-4-12b-it-4bit',
|
||||
messages: [{
|
||||
role: 'user',
|
||||
content: [
|
||||
{ type: 'image_url', image_url: { url: base64 } },
|
||||
{ type: 'text', text: prompt },
|
||||
],
|
||||
}],
|
||||
max_tokens: 200,
|
||||
}),
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
return data.choices[0].message.content;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Audio Model Integration
|
||||
|
||||
### Audio Input Format
|
||||
Audio models accept audio files (WAV, MP3, AAC) encoded as base64.
|
||||
|
||||
### Rust Backend
|
||||
|
||||
```rust
|
||||
#[derive(Serialize)]
|
||||
struct AudioChatRequest {
|
||||
model: String,
|
||||
messages: Vec<AudioMessage>,
|
||||
max_tokens: Option<u32>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct AudioMessage {
|
||||
role: String,
|
||||
content: Vec<AudioContentPart>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
#[serde(tag = "type")]
|
||||
enum AudioContentPart {
|
||||
#[serde(rename = "text")]
|
||||
Text { text: String },
|
||||
#[serde(rename = "audio_url")]
|
||||
AudioUrl { audio_url: AudioUrl },
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
struct AudioUrl {
|
||||
url: String, // base64 data URI
|
||||
}
|
||||
|
||||
// Audio transcription/analysis
|
||||
async fn process_audio(audio_path: &str, prompt: &str) -> Result<String, Box<dyn std::error::Error>> {
|
||||
let client = Client::new();
|
||||
|
||||
let audio_data = std::fs::read(audio_path)?;
|
||||
let base64 = base64::encode(&audio_data);
|
||||
let data_uri = format!("data:audio/wav;base64,{}", base64);
|
||||
|
||||
let request = AudioChatRequest {
|
||||
model: "gemma-4-12b-it-4bit".to_string(),
|
||||
messages: vec![
|
||||
AudioMessage {
|
||||
role: "user".to_string(),
|
||||
content: vec![
|
||||
AudioContentPart::AudioUrl {
|
||||
audio_url: AudioUrl { url: data_uri }
|
||||
},
|
||||
AudioContentPart::Text { text: prompt.to_string() },
|
||||
],
|
||||
},
|
||||
],
|
||||
max_tokens: Some(100),
|
||||
};
|
||||
|
||||
let response = client
|
||||
.post("http://10.10.10.201:8080/v1/multimodal/chat/completions")
|
||||
.json(&request)
|
||||
.send()
|
||||
.await?
|
||||
.json::<ChatResponse>()
|
||||
.await?;
|
||||
|
||||
Ok(response.choices[0].message.content)
|
||||
}
|
||||
```
|
||||
|
||||
### Frontend
|
||||
|
||||
```typescript
|
||||
async function processAudio(audioFile: File, prompt: string): Promise<string> {
|
||||
const base64 = await new Promise<string>((resolve) => {
|
||||
const reader = new FileReader();
|
||||
reader.onload = () => resolve(reader.result as string);
|
||||
reader.readAsDataURL(audioFile);
|
||||
});
|
||||
|
||||
const response = await fetch('/api/multimodal/chat', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: 'gemma-4-12b-it-4bit',
|
||||
messages: [{
|
||||
role: 'user',
|
||||
content: [
|
||||
{ type: 'audio_url', audio_url: { url: base64 } },
|
||||
{ type: 'text', text: prompt },
|
||||
],
|
||||
}],
|
||||
max_tokens: 100,
|
||||
}),
|
||||
});
|
||||
|
||||
const data = await response.json();
|
||||
return data.choices[0].message.content;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Streaming Responses
|
||||
|
||||
### Server-Sent Events (SSE)
|
||||
|
||||
MarkBaseServer supports streaming via SSE when `stream: true` is set.
|
||||
|
||||
### Rust Backend
|
||||
|
||||
```rust
|
||||
use futures::StreamExt;
|
||||
|
||||
async fn stream_text(prompt: &str, model: &str) -> Result<String, Box<dyn std::error::Error>> {
|
||||
let client = Client::new();
|
||||
|
||||
let request = ChatRequest {
|
||||
model: model.to_string(),
|
||||
messages: vec![Message { role: "user".to_string(), content: prompt.to_string() }],
|
||||
max_tokens: Some(100),
|
||||
stream: Some(true),
|
||||
};
|
||||
|
||||
let mut stream = client
|
||||
.post("http://10.10.10.201:8080/v1/chat/completions")
|
||||
.json(&request)
|
||||
.send()
|
||||
.await?
|
||||
.bytes_stream();
|
||||
|
||||
let mut full_text = String::new();
|
||||
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let chunk = chunk?;
|
||||
let text = String::from_utf8_lossy(&chunk);
|
||||
|
||||
// Parse SSE format: "data: {...}\n\n"
|
||||
for line in text.lines() {
|
||||
if line.starts_with("data: ") {
|
||||
let json_str = &line[6..];
|
||||
if json_str == "[DONE]" { break; }
|
||||
|
||||
let chunk_data: serde_json::Value = serde_json::from_str(json_str)?;
|
||||
if let Some(content) = chunk_data["choices"][0]["delta"]["content"].as_str() {
|
||||
full_text.push_str(content);
|
||||
// Send to frontend via WebSocket
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(full_text)
|
||||
}
|
||||
```
|
||||
|
||||
### Frontend
|
||||
|
||||
```typescript
|
||||
async function streamText(prompt: string, onChunk: (text: string) => void): Promise<void> {
|
||||
const response = await fetch('/api/chat/stream', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({
|
||||
model: 'gemma-4-e4b-markbase',
|
||||
messages: [{ role: 'user', content: prompt }],
|
||||
stream: true,
|
||||
}),
|
||||
});
|
||||
|
||||
const reader = response.body?.getReader();
|
||||
const decoder = new TextDecoder();
|
||||
|
||||
while (reader) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
|
||||
const text = decoder.decode(value);
|
||||
for (const line of text.split('\n')) {
|
||||
if (line.startsWith('data: ')) {
|
||||
const json = line.slice(6);
|
||||
if (json === '[DONE]') break;
|
||||
|
||||
const data = JSON.parse(json);
|
||||
const content = data.choices[0]?.delta?.content || '';
|
||||
onChunk(content);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Model Selection Guide
|
||||
|
||||
| Model | Size | Speed | Quality | Use Case |
|
||||
|-------|------|-------|---------|----------|
|
||||
| E4B-MarkBase | 4.4GB | 49ms/token | Good | Real-time chat, quick responses |
|
||||
| 12B | 6.3GB | 6ms/token (158 tok/s) | Better | Balanced speed/quality |
|
||||
| 26B-Standard | 15GB | 30ms/token | High | Complex reasoning, code generation |
|
||||
| 31B | 17GB | 38ms/token | Highest | Deep analysis, expert tasks |
|
||||
|
||||
### Recommendation Matrix
|
||||
|
||||
| Scenario | Recommended Model |
|
||||
|----------|-------------------|
|
||||
| Chat UI autocomplete | E4B-MarkBase |
|
||||
| Document summarization | 12B or 26B-Standard |
|
||||
| Code generation | 26B-Standard |
|
||||
| Vision analysis | 12B (has VisionTower12B) |
|
||||
| Audio transcription | 12B (has AudioTower12B) |
|
||||
| Expert reasoning | 31B |
|
||||
|
||||
---
|
||||
|
||||
## 6. Performance Optimization
|
||||
|
||||
### KV Cache Management
|
||||
MarkBaseServer automatically manages KV cache. For long conversations:
|
||||
|
||||
```rust
|
||||
// Clear context for new conversation
|
||||
async fn reset_context(session_id: &str) {
|
||||
// MarkBaseServer handles this internally
|
||||
// Just start a new messages array
|
||||
}
|
||||
```
|
||||
|
||||
### Concurrent Requests
|
||||
MarkBaseServer handles concurrent requests efficiently:
|
||||
|
||||
- **Text**: Up to 10 concurrent streams
|
||||
- **Vision**: 2-3 concurrent (GPU intensive)
|
||||
- **Audio**: 2-3 concurrent (GPU intensive)
|
||||
|
||||
### Memory Limits
|
||||
- **M5Max48 (48GB)**: Max 3 models loaded concurrently
|
||||
- **M5 (128GB)**: All 4 models can be loaded
|
||||
|
||||
---
|
||||
|
||||
## 7. Deployment Configuration
|
||||
|
||||
### MarkBaseServer Startup
|
||||
|
||||
```bash
|
||||
# Local development (M5 128GB)
|
||||
cd ~/MarkBaseEngine
|
||||
./start_server.sh
|
||||
|
||||
# Production (M5Max48 via TBT5)
|
||||
# Deploy models first:
|
||||
rsync -avP ~/MarkBaseEngine/models/ 10.10.10.201:/Volumes/TBT5/models/
|
||||
|
||||
# Start server on M5Max48:
|
||||
ssh 10.10.10.201
|
||||
cd /Volumes/TBT5/MarkBaseEngine
|
||||
./build/release/MarkBaseServer ./models/E4B-MarkBase 8080 gemma-4-e4b-markbase
|
||||
```
|
||||
|
||||
### Rust Backend Configuration
|
||||
|
||||
```rust
|
||||
// config.rs
|
||||
pub struct MarkBaseConfig {
|
||||
pub base_url: String,
|
||||
pub default_model: String,
|
||||
pub timeout_ms: u64,
|
||||
}
|
||||
|
||||
impl Default for MarkBaseConfig {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
base_url: "http://10.10.10.201:8080/v1".to_string(),
|
||||
default_model: "gemma-4-e4b-markbase".to_string(),
|
||||
timeout_ms: 30000,
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. Error Handling
|
||||
|
||||
### Common Errors
|
||||
|
||||
| Error | Cause | Solution |
|
||||
|-------|-------|----------|
|
||||
| Connection refused | Server not running | Check `./start_server.sh` |
|
||||
| Model not found | Wrong model name | Check `/v1/models` endpoint |
|
||||
| Timeout | Large input/slow model | Increase timeout, use faster model |
|
||||
| GPU memory limit | Too many concurrent | Reduce concurrent requests |
|
||||
| NaN output | Forward pass bug | Report to MarkBaseEngine team |
|
||||
|
||||
### Rust Error Handling
|
||||
|
||||
```rust
|
||||
use thiserror::Error;
|
||||
|
||||
#[derive(Error, Debug)]
|
||||
pub enum MarkBaseError {
|
||||
#[error("Connection failed: {0}")]
|
||||
ConnectionFailed(String),
|
||||
#[error("Model not found: {0}")]
|
||||
ModelNotFound(String),
|
||||
#[error("Timeout after {0}ms")]
|
||||
Timeout(u64),
|
||||
#[error("Invalid response: {0}")]
|
||||
InvalidResponse(String),
|
||||
}
|
||||
|
||||
impl From<reqwest::Error> for MarkBaseError {
|
||||
fn from(e: reqwest::Error) -> Self {
|
||||
if e.is_timeout() {
|
||||
MarkBaseError::Timeout(30000)
|
||||
} else if e.is_connect() {
|
||||
MarkBaseError::ConnectionFailed(e.to_string())
|
||||
} else {
|
||||
MarkBaseError::InvalidResponse(e.to_string())
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 9. Testing & Validation
|
||||
|
||||
### Health Check
|
||||
|
||||
```rust
|
||||
async fn check_health() -> bool {
|
||||
let client = Client::new();
|
||||
let response = client
|
||||
.get("http://10.10.10.201:8080/health")
|
||||
.send()
|
||||
.await;
|
||||
|
||||
response.is_ok()
|
||||
}
|
||||
```
|
||||
|
||||
### Model List
|
||||
|
||||
```rust
|
||||
async fn list_models() -> Result<Vec<String>, Box<dyn std::error::Error>> {
|
||||
let client = Client::new();
|
||||
let response = client
|
||||
.get("http://10.10.10.201:8080/v1/models")
|
||||
.send()
|
||||
.await?
|
||||
.json::<serde_json::Value>()
|
||||
.await?;
|
||||
|
||||
let models = response["data"]
|
||||
.as_array()
|
||||
.unwrap_or(&vec![])
|
||||
.iter()
|
||||
.filter_map(|m| m["id"].as_str().map(|s| s.to_string()))
|
||||
.collect();
|
||||
|
||||
Ok(models)
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 10. Security Considerations
|
||||
|
||||
### API Gateway (momentry_core)
|
||||
|
||||
```rust
|
||||
// Add authentication layer
|
||||
use actix_web::{web, HttpRequest, HttpResponse};
|
||||
|
||||
async fn chat_proxy(
|
||||
req: HttpRequest,
|
||||
body: web::Json<ChatRequest>,
|
||||
) -> HttpResponse {
|
||||
// Validate auth token
|
||||
let auth = req.headers().get("Authorization");
|
||||
if !validate_auth(auth) {
|
||||
return HttpResponse::Unauthorized().finish();
|
||||
}
|
||||
|
||||
// Rate limiting
|
||||
if !check_rate_limit(&req) {
|
||||
return HttpResponse::TooManyRequests().finish();
|
||||
}
|
||||
|
||||
// Forward to MarkBaseServer
|
||||
let response = forward_to_markbase(body.into_inner());
|
||||
|
||||
HttpResponse::Ok().json(response)
|
||||
}
|
||||
```
|
||||
|
||||
### Input Validation
|
||||
|
||||
```rust
|
||||
fn validate_chat_request(req: &ChatRequest) -> Result<(), String> {
|
||||
if req.messages.is_empty() {
|
||||
return Err("Messages array cannot be empty".to_string());
|
||||
}
|
||||
|
||||
if req.max_tokens.unwrap_or(100) > 2048 {
|
||||
return Err("max_tokens cannot exceed 2048".to_string());
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 11. Complete Example: momentry_core Integration
|
||||
|
||||
```rust
|
||||
// src/markbase_client.rs
|
||||
use reqwest::Client;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::time::Duration;
|
||||
|
||||
pub struct MarkBaseClient {
|
||||
client: Client,
|
||||
base_url: String,
|
||||
default_model: String,
|
||||
}
|
||||
|
||||
impl MarkBaseClient {
|
||||
pub fn new(base_url: &str, default_model: &str) -> Self {
|
||||
let client = Client::builder()
|
||||
.timeout(Duration::from_secs(30))
|
||||
.build()
|
||||
.unwrap();
|
||||
|
||||
Self {
|
||||
client,
|
||||
base_url: base_url.to_string(),
|
||||
default_model: default_model.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn chat(&self, prompt: &str) -> Result<String, MarkBaseError> {
|
||||
self.chat_with_model(prompt, &self.default_model).await
|
||||
}
|
||||
|
||||
pub async fn chat_with_model(&self, prompt: &str, model: &str) -> Result<String, MarkBaseError> {
|
||||
let request = ChatRequest {
|
||||
model: model.to_string(),
|
||||
messages: vec![Message { role: "user".to_string(), content: prompt.to_string() }],
|
||||
max_tokens: Some(100),
|
||||
temperature: Some(0.7),
|
||||
stream: Some(false),
|
||||
};
|
||||
|
||||
let url = format!("{}{}", self.base_url, "/chat/completions");
|
||||
let response = self.client
|
||||
.post(&url)
|
||||
.json(&request)
|
||||
.send()
|
||||
.await?
|
||||
.json::<ChatResponse>()
|
||||
.await?;
|
||||
|
||||
Ok(response.choices[0].message.content)
|
||||
}
|
||||
|
||||
pub async fn vision(&self, image_base64: &str, prompt: &str) -> Result<String, MarkBaseError> {
|
||||
let request = MultimodalChatRequest {
|
||||
model: self.default_model.clone(),
|
||||
messages: vec![
|
||||
MultimodalMessage {
|
||||
role: "user".to_string(),
|
||||
content: vec![
|
||||
ContentPart::ImageUrl {
|
||||
image_url: ImageUrl { url: format!("data:image/jpeg;base64,{}", image_base64) }
|
||||
},
|
||||
ContentPart::Text { text: prompt.to_string() },
|
||||
],
|
||||
},
|
||||
],
|
||||
max_tokens: Some(200),
|
||||
};
|
||||
|
||||
let url = format!("{}{}", self.base_url, "/multimodal/chat/completions");
|
||||
let response = self.client
|
||||
.post(&url)
|
||||
.json(&request)
|
||||
.send()
|
||||
.await?
|
||||
.json::<ChatResponse>()
|
||||
.await?;
|
||||
|
||||
Ok(response.choices[0].message.content)
|
||||
}
|
||||
|
||||
pub async fn audio(&self, audio_base64: &str, prompt: &str) -> Result<String, MarkBaseError> {
|
||||
let request = AudioChatRequest {
|
||||
model: self.default_model.clone(),
|
||||
messages: vec![
|
||||
AudioMessage {
|
||||
role: "user".to_string(),
|
||||
content: vec![
|
||||
AudioContentPart::AudioUrl {
|
||||
audio_url: AudioUrl { url: format!("data:audio/wav;base64,{}", audio_base64) }
|
||||
},
|
||||
AudioContentPart::Text { text: prompt.to_string() },
|
||||
],
|
||||
},
|
||||
],
|
||||
max_tokens: Some(100),
|
||||
};
|
||||
|
||||
let url = format!("{}{}", self.base_url, "/multimodal/chat/completions");
|
||||
let response = self.client
|
||||
.post(&url)
|
||||
.json(&request)
|
||||
.send()
|
||||
.await?
|
||||
.json::<ChatResponse>()
|
||||
.await?;
|
||||
|
||||
Ok(response.choices[0].message.content)
|
||||
}
|
||||
|
||||
pub async fn health_check(&self) -> bool {
|
||||
let url = format!("{}{}", self.base_url.replace("/v1", ""), "/health");
|
||||
self.client.get(&url).send().await.is_ok()
|
||||
}
|
||||
}
|
||||
|
||||
// Usage in main.rs
|
||||
#[actix_web::main]
|
||||
async fn main() -> std::io::Result<()> {
|
||||
let markbase = MarkBaseClient::new(
|
||||
"http://10.10.10.201:8080/v1",
|
||||
"gemma-4-e4b-markbase",
|
||||
);
|
||||
|
||||
// Test connection
|
||||
if !markbase.health_check().await {
|
||||
eprintln!("MarkBaseServer not responding!");
|
||||
}
|
||||
|
||||
// Use in routes
|
||||
HttpServer::new(|| {
|
||||
App::new()
|
||||
.app_data(web::Data::new(markbase.clone()))
|
||||
.route("/api/chat", web::post().to(chat_handler))
|
||||
.route("/api/vision", web::post().to(vision_handler))
|
||||
.route("/api/audio", web::post().to(audio_handler))
|
||||
})
|
||||
.bind("127.0.0.1:3000")?
|
||||
.run()
|
||||
.await
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 12. Monitoring & Logging
|
||||
|
||||
### Performance Metrics
|
||||
|
||||
```rust
|
||||
use std::time::Instant;
|
||||
|
||||
async fn monitored_chat(client: &MarkBaseClient, prompt: &str) -> Result<(String, u64), MarkBaseError> {
|
||||
let start = Instant::now();
|
||||
let response = client.chat(prompt).await?;
|
||||
let latency_ms = start.elapsed().as_millis() as u64;
|
||||
|
||||
// Log to monitoring system
|
||||
log::info!("Chat latency: {}ms, tokens: {}", latency_ms, response.len());
|
||||
|
||||
Ok((response, latency_ms))
|
||||
}
|
||||
```
|
||||
|
||||
### Structured Logging
|
||||
|
||||
```rust
|
||||
use serde_json::json;
|
||||
|
||||
fn log_request(model: &str, prompt_len: usize, latency_ms: u64) {
|
||||
let log_entry = json!({
|
||||
"timestamp": chrono::Utc::now().to_rfc3339(),
|
||||
"model": model,
|
||||
"prompt_length": prompt_len,
|
||||
"latency_ms": latency_ms,
|
||||
"server": "MarkBaseServer",
|
||||
});
|
||||
|
||||
println!("{}", log_entry);
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
This guide provides complete integration patterns for:
|
||||
|
||||
1. **Text Models**: Simple chat completion via `/v1/chat/completions`
|
||||
2. **Vision Models**: Image analysis via `/v1/multimodal/chat/completions` with base64 images
|
||||
3. **Audio Models**: Audio processing via `/v1/multimodal/chat/completions` with base64 audio
|
||||
4. **Streaming**: SSE support for real-time UI updates
|
||||
5. **Model Selection**: Choose based on speed/quality tradeoff
|
||||
6. **Performance**: Optimized for Apple Silicon Metal GPU
|
||||
|
||||
### Next Steps
|
||||
|
||||
1. Set up MarkBaseServer on production server (M5Max48)
|
||||
2. Integrate Rust client into momentry_core
|
||||
3. Build frontend UI with streaming support
|
||||
4. Add authentication and rate limiting
|
||||
5. Deploy and monitor performance
|
||||
|
||||
---
|
||||
|
||||
**Document Version**: 1.0
|
||||
**Last Updated**: 2026-06-23
|
||||
**Author**: MarkBaseEngine Team
|
||||
Reference in New Issue
Block a user