# Unified Brain — Integration Requirements
## Xilos `[brain]` routing + MPR agents (consumers) ↔ Brain API service (producer)

**Author:** Milly (consumer side — Xilos routing layer + agent direct access)
**Owner of target system:** Eve (Brain API service build, per BRAIN-ARCHITECTURE-PLAN.md)
**Date:** 2026-08-18
**Status:** DRAFT — awaiting Eve's sign-off / counter-spec
**Produced via:** `/cross-system-integration` skill (spec-first workflow)

---

## 1. System Map

| | |
|---|---|
| **This system** | (a) Xilos `[brain]` routing rule — intercepts `model="[brain]"` chat completions, queries Brain API, injects context, routes to fast model. (b) MPR agents (Milly, Eve) calling Brain API directly for structured CRUD. |
| **That system** | Brain API service — FastAPI + SQLite + sqlite-vec, indexing the Obsidian vault (per `/root/grammerly-extension/BRAIN-ARCHITECTURE-PLAN.md`) |
| **Auth boundary** | Two paths: user/org JWT from Xilos `/api/v1/auth/login` (for Xilos server-side calls), static per-agent API keys (for agent scripts/cron) |
| **Data flow** | Vault → git push → webhook → Brain API index. Xilos/agents → REST → Brain API. Xilos `[brain]` → enriched prompt → upstream model → answer. |

---

## 2. Auth Flow

| Path | Mechanism | Used by |
|------|-----------|---------|
| Org JWT | `POST /api/v1/auth/login` (Xilos) → `Authorization: Bearer <jwt>` | Xilos server when executing `[brain]` rule; org resolution from token |
| Agent service keys | `X-Brain-Key: *** static key, one per agent (milly, eve), org-scoped to MPR | Agent scripts, cron jobs, migration tooling — no interactive login |

**Requirements on Brain API:**
- Accept BOTH auth forms on every endpoint.
- Service keys are provisioned by Eve at deploy; keys delivered via secure channel (NOT in this doc, NOT in git).
- All data is org-scoped; MPR is the only org for MVP.

---

## 3. API Contract (MVP subset the consumers need)

Base URL: `https://brain.millpondresearch.com/api/v1/brain` (proposed — see §8)

### 3.1 Query (the critical path — Xilos `[brain]` calls this on EVERY routed request)

```
GET /query?q=<urlencoded question>&limit=5&type=person,tags=leadership
```

Response (contract — Xilos parses this exact shape):
```json
{
  "results": [
    {
      "slug": "pete-shimshock",
      "title": "Pete Shimshock",
      "type": "person",
      "score": 1.0,
      "snippet": "...Co-founder and CAIO of Mill Pond Research...",
      "highlights": ["Co-founder", "CAIO", "Mill Pond Research"]
    }
  ],
  "took_ms": 87
}
```

Contract details that MUST hold:
- `score` normalized to **[0.0, 1.0]**, top result in each response = 1.0 (raw RRF scores are tiny fractions — normalize before returning; plan example shows 0.89, so this matches intent).
- `results` sorted by score desc. Empty result set → `{"results": [], "took_ms": N}` with **HTTP 200**, never an error.
- `snippet` ≤ 300 chars, centered on the best-matching passage.
- `limit` default 5, max 20.
- Optional filters: `type` (comma-separated), `tags` (comma-separated). Unknown filter values are ignored, not errors.

### 3.2 Pages

| Call | Contract |
|------|----------|
| `GET /pages/{slug}` | Full page: `{slug, type, title, created, updated, tags[], body_markdown, links_out[]}`. 404 if absent. |
| `PUT /pages/{slug}` | Upsert. JSON body: `{type, title, tags[], body_markdown, effective_date?, sources?[]}`. Returns 200 + stored page. Slug in path wins over anything else. Re-embeds on write (see §5). |
| `DELETE /pages/{slug}` | 204 on success, 404 if absent. Cleans up links both directions. |
| `POST /pages/{slug}/import` | Raw markdown with YAML frontmatter (`Content-Type: text/markdown` or multipart). Server parses frontmatter; slug from frontmatter wins, path slug is fallback. 422 with field errors if frontmatter invalid. |

Validation errors shape (all 422s):
```json
{"error": "validation_failed", "fields": [{"field": "type", "message": "must be one of: company, person, concept, meeting, deal, source, note"}]}
```

### 3.3 Search (keyword only, no LLM path)

`GET /search?q=keyword&limit=10` → same result shape as `/query`. Used by agent skills for exact-term lookups.

### 3.4 Sync

| Call | Contract |
|------|----------|
| `POST /sync` | Trigger re-index now. 202 `{"status": "queued"}`. Reads stay live during sync. |
| `GET /sync/status` | `{"last_sync": "ISO8601", "last_result": "ok|partial|failed", "pages_changed": N, "errors": []}` |
| `POST /sync/webhook` | GitHub push payload. Verified via `X-Hub-Signature-256` (HMAC-SHA256, shared secret provisioned out-of-band). Bad signature → 403. Valid → 202. Idempotent on repeat deliveries. |

### 3.5 Health & stats (Milly wires these into uptime monitoring)

| Call | Contract |
|------|----------|
| `GET /health` | `{"status": "ok|degraded", "pages": N, "links": N, "embedded_pct": 0-100, "last_sync": "ISO8601", "stale_chunks": N}` — 200 when serving, 503 when not. No auth required. |
| `GET /stats` | Same fields + tag/type breakdowns. Auth required. |

### 3.6 Graph & timeline (Phase 2 — NOT in MVP, contract reserved)

As defined in the architecture plan (§Graph, §Timeline). Milly's agent-integration phase (plan Phase 8) will build against those shapes; no changes expected.

---

## 4. Latency & Capacity Contract

This is the part that breaks silently if we don't pin it down now:

| Metric | Requirement | Why |
|--------|-------------|-----|
| `GET /query` p95 | **< 400ms** (local, same droplet) | `[brain]` adds a hop before the LLM call; users feel >1s of pre-LLM delay |
| `GET /pages/{slug}` p95 | < 100ms | Agent lookups happen mid-task |
| Write → queryable | **< 5s** (embed-on-write) | Agents write meeting notes then immediately query them |
| Concurrent load | 10 req/s sustained is plenty | 2 agents + Xilos routing; no public exposure |
| Index size | 10K pages without degradation | MVP seeds ~7; room to grow |

---

## 5. Expected Behavior

- **Embed-on-write:** every `PUT`/`import` re-embeds that page's chunks synchronously before returning 200. Nightly backfill sweep catches orphans (reported as `stale_chunks` in `/health`).
- **Sync semantics:** new → index; modified → re-parse + re-embed; deleted → remove + link cleanup; rename = delete + create (slug from frontmatter is identity).
- **Sync failures are per-page, never global.** Partial failures listed in `/sync/status.errors`; index keeps serving last-good state.
- **Xilos `[brain]` behavior (Milly's side, for Eve's awareness):** on `model="[brain]"` → call `/query` with the user's latest message, 2s timeout → inject top-5 results with score ≥ 0.4 into system prompt (format per plan §Phase 4) → route to fast model → if Brain API is down/times out/returns empty, **proceed WITHOUT context and never fail the user's chat request** (log it, expose via usage logs).

---

## 6. Error Handling (consumer behavior per code)

| Code | Meaning | Xilos `[brain]` behavior | Agent behavior |
|------|---------|--------------------------|----------------|
| 401 | Bad/expired token | Proceed without context; alert admin | Re-auth / flag key rotation |
| 403 | Out of org scope / bad webhook signature | Proceed without context | Bug — investigate |
| 404 | Page/rule not found | N/A for /query | Report "not in brain yet", offer to create |
| 422 | Validation failure | N/A | Surface field errors verbatim to operator |
| 429 | Rate limited | Proceed without context | Respect `Retry-After`, back off |
| 5xx / timeout | Brain down | **Proceed without context** — graceful degradation, never block chat | Retry once, then queue + alert |

**Required:** 429 responses include `Retry-After` header.

---

## 7. Configuration Required

| Item | Who | Notes |
|------|-----|-------|
| Xilos routing rule `[brain]` | Milly (Xilos dashboard) | Model-field trigger `[brain]`, target = fast model, context template per plan §Phase 4 |
| Model-field matching in routing rules | Xilos engineering (Milly) | Same capability `[brand-voice]` needs — **dependency: confirm whether it shipped with the Grammerly spec work; if not, it's on my build list before `[brain]` can exist** |
| Brain API service keys | Eve | One per agent, delivered out-of-band |
| Webhook HMAC secret | Eve provisions, Milly configures GitHub | Out-of-band exchange |
| DNS + nginx + systemd + monitoring | Milly | brain.millpondresearch.com → localhost port (proposed below) |

---

## 8. Milly's Answers to the Plan's Open Questions

1. **Sync mechanism:** Webhook-first, with a 15-minute polling fallback cron as safety net. Webhook needs a public endpoint — the droplet already serves several behind nginx; not a blocker.
2. **Embedding refresh:** Embed-on-write (local 384-dim model is cheap; staleness actively hurts agents who write-then-query). Nightly batch only as backfill/consistency sweep.
3. **Storage location:** Same droplet. systemd unit + nginx reverse proxy, proposed subdomain `brain.millpondresearch.com` (keeps it off the agent.* path namespace, clean TLS, easy monitoring target). I own DNS/nginx wiring; Eve ships the service.
4. **Obsidian vault location:** Canonical git clone on the droplet. Eve edits locally, commits + pushes; webhook triggers re-index. Matches plan Phase 3 and keeps the sync service simple.
5. **Rate limits:** Same org-based RPM as Xilos for read endpoints; `/sync` and `/import` exempt or capped much higher (bulk migration will hammer them).

---

## 9. What Eve's System Must Build

**Must-have (MVP, plan Phases 1–4):**
1. Everything in §3.1–3.5 exactly as contracted (shapes, normalization, status codes)
2. Dual auth (JWT + service keys)
3. Embed-on-write with <5s write→queryable
4. Score normalization to [0,1]
5. GitHub webhook receiver with HMAC verification
6. `/health` unauthenticated (monitoring requirement)

**Nice-to-have (post-MVP):**
7. Graph + timeline endpoints (plan Phase 7 — shapes already agreed)
8. Write-back to vault git (plan Phase 3 bidirectional)
9. Per-agent usage stats in `/stats`

---

## 10. What Milly Builds in Return

1. Xilos `[brain]` routing rule + context injection (§5 behavior, plan Phase 4)
2. Verify/build model-field matching in Xilos routing rules (dependency check)
3. `brain.millpondresearch.com` DNS, nginx proxy, systemd hardening (OOMScoreAdjust per droplet conventions), uptime monitoring wired to my alerting
4. Agent integration: replace `gbrain-query` wrapper with Brain API client skill; update handoffs (plan Phase 8)
5. Migration verification harness (§11) run from my side

---

## 11. Acceptance Tests (both sides run before gbrain decommission)

1. `POST /sync` → `GET /stats` shows pages ≥ 7, links ≥ 6, `embedded_pct` = 100
2. `GET /query?q=who founded Mill Pond Research` → top-2 includes founder pages
3. Round-trip: `PUT /pages/test-roundtrip` → `GET` returns it → `/query?q=<unique term from body>` finds it within 5s → `DELETE` → 404 after
4. Error paths: no auth → 401; `GET /pages/nope` → 404; `PUT` with `type=bogus` → 422 with field errors; webhook with bad signature → 403
5. Latency: 50 consecutive `/query` calls, p95 < 400ms
6. Degradation: stop Brain API → Xilos `[brain]` chat still answers (without context) and logs the miss

---

## Handoff Question for Eve

Does anything in §3 (shapes, auth, normalization) or §4 (latency budget) conflict with your current build direction? Specifically:
1. Can FastAPI + sqlite-vec hit <400ms p95 on hybrid `/query` at 10K pages on this droplet, or do we need an index strategy change now?
2. Embed-on-write synchronous (block the PUT response until embedded) — acceptable, or do you need async with the 5s SLA measured differently?
3. Any objection to dual auth (JWT + static service keys), or does that complicate your design?

Spec is the shared source of truth from here: if either side changes shape, update this doc first, then code.