---
document_type: "architecture_design"
service: "MOMENTRY_CORE"
title: "Identity Agent Design Specification"
date: "2026-04-28"
version: "V2.0"
status: "active"
owner: "Warren"
created_by: "OpenCode"
tags:
  - "identity-agent"
  - "agent"
  - "face-clustering"
  - "embedding-matching"
  - "multi-file-aggregation"
ai_query_hints:
  - "Identity Agent design specification"
  - "Face to Identity inference flow"
  - "Multi-file identity aggregation"
  - "Embedding matching with pose adaptation"
related_documents:
  - "AI_AGENTS/CORE/AGENT_SPEC.md"
  - "AI_AGENTS/IDENTITY/IDENTITY_MANAGEMENT_API.md"
  - "FILE_IDENTITIES_TABLE_SPEC.md"
---

# Identity Agent Design Specification

| Item | Content |
|------|---------|
| Creator | OpenCode |
| Date | 2026-04-28 |
| Version | V2.0 (Two-layer Architecture) |

---

## Version History

| Version | Date | Changes | Author |
|---------|------|---------|--------|
| V2.0 | 2026-04-28 | Two-layer architecture (Face → Identity) | OpenCode |
| V1.0 | 2026-04-27 | Initial design (three-layer) | OpenCode |

---

## Overview

Identity Agent is an L3 Agent in Momentry Core, responsible for inferring "Who is Who" from Face Processor outputs and aggregating identities across multiple files.

---

## Architecture Change (V1.0 → V2.0)

| Aspect | V1.0 (Deprecated) | V2.0 (Current) |
|--------|-------------------|----------------|
| **Layers** | Face → Person → Identity | Face → Identity (2 layers) |
| **person_identities** | Required table | Removed (deprecated) |
| **Binding** | Person → Identity | Face → Identity (direct) |
| **Chunks** | Person → Chunk | Face → Chunk (auto-bind by time) |

---

## Current Status

| Component | Status |
|-----------|--------|
| Face Processor | ✅ Implemented (InsightFace) |
| Face Tracker | ✅ Implemented (trace_id) |
| ASRX Processor | ✅ Implemented (WhisperX) |
| Identity Agent | 🔧 Pending implementation |

---

## 1. Agent Goals

### 1.1 Core Problem

**Question**: How to infer global Identity from Face embeddings across multiple files?

**Challenges**:
1. **Same person in different files**: Need cross-file matching
2. **Different poses**: frontal vs profile have different thresholds
3. **Temporal alignment**: Chunks need time-based binding
4. **Quality variance**: Low-quality faces need filtering

---

### 1.2 Agent Goals

Aggregate evidence across files to create/maintain global Identities:

| Evidence Source | Input | Output |
|-----------------|-------|--------|
| **Face Processor** | Face embedding + pose_angle | Face → identity_id |
| **Face Tracker** | trace_id (face tracking) | Trace statistics |
| **ASRX Processor** | Speaker segments | Chunk → identity_id (auto-bind) |
| **Identity Agent** | Face + trace + time | **Identity** (global) |

---

## 2. Data Flow (Two-layer)

```
File → InsightFace → face_full_traced.json
            ↓
        face_id + embedding + pose_angle + trace_id
            ↓
        Identity Agent
            ↓
        ┌─────────────────────────────────────┐
        │  Step 1: Select unregistered face   │
        │  Step 2: Register identity          │
        │  Step 3: Embedding matching         │
        │  Step 4: Bind faces → identity_id   │
        │  Step 5: Auto-bind chunks           │
        └─────────────────────────────────────┘
            ↓
        identities + file_identities tables
```

---

## 3. Input Data

### 3.1 Face Data Structure

```json
{
  "file_uuid": "384b0ff44aaaa1f14cb2cd63b3fea966",
  "fps": 59.94,
  "metadata": {
    "trace_stats": {
      "total_traces": 4,
      "long_traces": 3
    }
  },
  "frames": {
    "100": {
      "faces": [
        {
          "face_id": "face_100",
          "confidence": 0.92,
          "embedding": [512-dim vector],
          "pose_angle": {
            "angle": "frontal",
            "yaw": -5.2,
            "pitch": 2.1,
            "confidence": 0.95
          },
          "trace_id": 2,
          "identity_id": null
        }
      ]
    }
  },
  "traces": {
    "2": {
      "trace_id": 2,
      "total_appearances": 143,
      "avg_confidence": 0.86,
      "pose_distribution": {
        "frontal": 20,
        "profile_right": 125
      }
    }
  }
}
```

---

### 3.2 Data Sources

| Data | Source File | Description |
|------|--------------|-------------|
| **Face frames** | `{uuid}.face_full_traced_v2.json` | Face detection + embedding + trace |
| **Speaker segments** | `{uuid}.asrx.json` | Speaker time segments |
| **Chunks** | `chunks` table | Sentence chunks (from pre_chunks) |

---

## 4. Core Logic

### 4.1 Inference Flow

```
┌─────────────────────────────────────────────────────────────────┐
│                     Identity Agent Workflow                     │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Step 1: Candidates Query                                       │
│  ─────────────────────────────                                  │
│  Query: GET /api/v1/faces/candidates                            │
│  Filter: identity_id = NULL, confidence >= 0.8                  │
│  Result: Unregistered faces list                                │
│                                                                 │
│  Step 2: AI Suggestion                                          │
│  ─────────────────                                              │
│  Query: POST /api/v1/agents/suggest/clustering                  │
│  Input: Unregistered faces                                      │
│  Output: Cluster suggestions + recommended primary face         │
│                                                                 │
│  Step 3: Identity Registration                                  │
│  ─────────────────────────────                                  │
│  Query: POST /api/v1/identities/register                        │
│  Input: face_ids + name                                         │
│  Output: identity_uuid                                          │
│                                                                 │
│  Step 4: Face Binding                                           │
│  ─────────────────                                              │
│  For each face in same trace:                                   │
│    Calculate: embedding_similarity(face, identity.embedding)    │
│    Apply: adaptive_threshold(pose_angle)                        │
│    If similarity > threshold:                                    │
│      UPDATE face_detections SET identity_id = identity.id       │
│                                                                 │
│  Step 5: Chunk Auto-Binding                                     │
│  ─────────────────────────────                                  │
│  For each face with identity_id:                                │
│    Query: chunks WHERE time overlaps face timestamp             │
│    Update: chunk.metadata.identity_id = identity.uuid           │
│    Update: chunk.metadata.chunk_identity.faces.push(face_id)    │
│                                                                 │
│  Step 6: Statistics Aggregation                                 │
│  ───────────────────────────────                                │
│  Update: file_identities (face_count, speaker_count)            │
│  Update: identities.metadata (global stats)                     │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘
```

---

### 4.2 Adaptive Threshold

**Pose-based threshold strategy**:

```python
def get_adaptive_threshold(pose_angle: str) -> float:
    """Get matching threshold based on pose angle"""
    thresholds = {
        "frontal": 0.90,        # Strict for frontal
        "three_quarter": 0.85,  # Moderate
        "profile_left": 0.80,   # Relaxed for profile
        "profile_right": 0.80,
    }
    return thresholds.get(pose_angle, 0.75)
```

**Reasoning**:
- Frontal faces have best embedding quality → strict threshold
- Profile faces have distorted embedding → relaxed threshold
- Three_quarter is intermediate

---

### 4.3 Embedding Matching

```python
def match_face_to_identity(
    face_embedding: List[float],
    identity_embedding: List[float],
    pose_angle: str
) -> Tuple[bool, float]:
    """Match face to identity with pose-adaptive threshold"""
    
    similarity = cosine_similarity(face_embedding, identity_embedding)
    threshold = get_adaptive_threshold(pose_angle)
    
    is_match = similarity > threshold
    return is_match, similarity
```

---

### 4.4 Chunk Auto-Binding

```python
def bind_chunks_to_identity(
    identity_id: int,
    file_uuid: str,
    pool: PgPool
) -> int:
    """Auto-bind chunks by time alignment"""
    
    # Get face time ranges
    faces = sqlx::query(
        "SELECT timestamp, pose_angle 
         FROM face_detections 
         WHERE identity_id = $1 AND file_uuid = $2"
    ).bind(identity_id).bind(file_uuid).fetch_all(pool)
    
    # Find overlapping chunks
    chunks_updated = 0
    for face in faces:
        chunks = sqlx::query(
            "UPDATE chunks 
             SET metadata = jsonb_set(
               metadata, '{chunk_identity}',
               jsonb_build_object(
                 'identity_id', $1::text,
                 'binding_source', 'auto'
               )
             )
             WHERE file_uuid = $2 
             AND ABS(start_time - $3) < 2.0"
        ).bind(identity_id).bind(file_uuid).bind(face.timestamp)
        .execute(pool)
        
        chunks_updated += chunks.rowcount()
    
    return chunks_updated
```

---

## 5. Database Schema

### 5.1 identities Table

| Field | Type | Description |
|-------|------|-------------|
| `uuid` | UUID | identity_uuid (global) |
| `name` | VARCHAR | Identity name |
| `face_embedding` | VECTOR(512) | Reference embedding |
| `reference_data` | JSONB | Multi-angle reference vectors |
| `metadata` | JSONB | Global statistics |

---

### 5.2 file_identities Table (N:N)

| Field | Type | Description |
|-------|------|-------------|
| `file_uuid` | UUID | File UUID |
| `identity_id` | BIGINT | Identity ID |
| `face_count` | INT | Faces in this file |
| `speaker_count` | INT | Speaker segments |
| `first_appearance` | FLOAT | First appearance time |
| `last_appearance` | FLOAT | Last appearance time |
| `confidence` | FLOAT | Avg confidence |

---

### 5.3 face_detections Table

| Field | Type | Description |
|-------|------|-------------|
| `identity_id` | BIGINT | Bound identity (direct) |
| `file_uuid` | UUID | File UUID |
| `pose_angle` | VARCHAR | Pose angle |
| `embedding` | VECTOR(512) | Face embedding |
| `trace_id` | INT | Trace ID (from Face Tracker) |

---

### 5.4 chunks.metadata Structure

```json
{
  "chunk_identity": {
    "faces": [100, 150],
    "speakers": ["SPEAKER_0"],
    "identity_id": "a9a90105-...",
    "confidence": 0.88,
    "binding_source": "auto"
  }
}
```

---

## 6. API Design

### 6.1 Candidates API

```http
GET /api/v1/faces/candidates
  ?min_confidence=0.8
  &pose_angle=frontal
  &page=1
  &page_size=15
  &limit=100
```

**Response**:
```json
{
  "candidates": [
    {
      "face_id": "face_100",
      "pose_angle": "frontal",
      "confidence": 0.92,
      "trace_id": 2
    }
  ]
}
```

---

### 6.2 Suggest API

```http
POST /api/v1/agents/suggest/clustering
{
  "min_confidence": 0.8,
  "max_suggestions": 5
}
```

**Response**:
```json
{
  "suggestions": [
    {
      "cluster_type": "high_confidence",
      "recommended_faces": ["face_100"],
      "action": "register"
    }
  ]
}
```

---

### 6.3 Register API

```http
POST /api/v1/identities/register
{
  "face_ids": ["face_100"],
  "name": "Person A",
  "auto_bind_chunks": true
}
```

---

## 7. Multi-File Aggregation

### 7.1 Cross-File Matching

When a new file is processed:

1. **Query existing identities**: `SELECT * FROM identities`
2. **For each unregistered face**:
   - Calculate similarity with all identity.face_embedding
   - Apply adaptive threshold
   - If match: bind to existing identity
3. **If no match**: create new identity

---

### 7.2 Statistics Update

```sql
-- Update file_identities after binding
INSERT INTO file_identities (
  file_uuid, identity_id, face_count, confidence
)
SELECT 
  file_uuid,
  identity_id,
  COUNT(*),
  AVG(confidence)
FROM face_detections
WHERE identity_id IS NOT NULL
GROUP BY file_uuid, identity_id
ON CONFLICT (file_uuid, identity_id) 
DO UPDATE SET 
  face_count = EXCLUDED.face_count,
  confidence = EXCLUDED.confidence;
```

---

## 8. Implementation Plan

### 8.1 Phase 1: Core Matching

| Task | Status |
|------|--------|
| Adaptive threshold function | Pending |
| Embedding matching logic | Pending |
| Face → Identity binding | Pending |
| Chunk auto-binding | Pending |

---

### 8.2 Phase 2: Candidates API

| Task | Status |
|------|--------|
| Candidates query endpoint | Pending |
| Pose distribution statistics | Pending |
| Trace-based filtering | Pending |

---

### 8.3 Phase 3: Suggest API

| Task | Status |
|------|--------|
| Clustering suggestion logic | Pending |
| Primary face recommendation | Pending |
| Merge suggestion | Pending |

---

### 8.4 Phase 4: Statistics

| Task | Status |
|------|--------|
| file_identities aggregation | Pending |
| identities.metadata update | Pending |
| Cross-file identity stats | Pending |

---

## 9. Key Decisions

| Decision | Reason |
|----------|--------|
| **Remove person_identities** | Middle layer adds complexity, unused (303 records, 0 registered) |
| **Face → Identity direct** | Simpler, embedding comparison is sufficient |
| **Adaptive threshold** | Pose affects embedding quality |
| **Chunk auto-bind** | Chunks follow faces by time alignment |
| **file_identities table** | Needed for N:N relationship tracking |

---

## 10. Metrics

| Metric | Target |
|--------|--------|
| **Matching accuracy** | > 90% for frontal |
| **False positive rate** | < 5% |
| **Processing speed** | 1000 faces/second |
| **Cross-file recall** | > 85% |

---

## Version Information

- Version: V2.0
- Architecture: Two-layer (Face → Identity)
- Date: 2026-04-28
- Status: Specification complete, implementation pending
