DATA_LAYER_API

MONITOR Data Layer API

Complete API contract for interacting with MONITOR's multi-database data layer.

Overview

The Data Layer is a service interface between agents and the five storage systems. Agents interact with data exclusively through these APIs, never directly with databases.

Key principle: Data layer is stateless and agent-agnostic. It validates, enforces authority, and ensures consistency.

API Architecture

┌─────────────────────────────────────────────┐
│           AGENT LAYER                       │
│  (Orchestrator, Narrator, CanonKeeper...)  │
└────────────────┬────────────────────────────┘
                 │
                 ▼ (MCP or gRPC)
┌─────────────────────────────────────────────┐
│        DATA LAYER API                       │
│  - Validation                               │
│  - Authority enforcement                    │
│  - Cross-DB coordination                    │
└─┬───────┬────────┬────────┬────────┬────────┘
  │       │        │        │        │
  ▼       ▼        ▼        ▼        ▼
┌────┐ ┌────┐ ┌────┐ ┌────┐ ┌────┐
│Neo4│ │Mongo│ │Qdrant│ │OpenS│ │MinIO│
└────┘ └────┘ └────┘ └────┘ └────┘

1. Neo4j Canonical Operations

1.1 Universe & World Structure

CreateUniverse

interface CreateUniverseRequest {
  multiverse_id: UUID;
  name: string;
  description: string;
  genre?: string;
  tone?: string;
  tech_level?: string;
  authority: "source" | "gm" | "system";
}

interface CreateUniverseResponse {
  universe_id: UUID;
  created_at: timestamp;
}

Authority: CanonKeeper only Validation: multiverse_id must exist, name required

GetUniverse

interface GetUniverseRequest {
  universe_id: UUID;
}

interface GetUniverseResponse {
  universe_id: UUID;
  name: string;
  description: string;
  genre: string;
  tone: string;
  tech_level: string;
  canon_level: "proposed" | "canon" | "retconned";
  created_at: timestamp;
}

Authority: Any agent (read-only)

ListUniverses

interface ListUniversesRequest {
  multiverse_id?: UUID;  // filter by multiverse
  canon_level?: "proposed" | "canon" | "retconned";
  limit?: number;
  offset?: number;
}

interface ListUniversesResponse {
  universes: Universe[];
  total: number;
}

Authority: Any agent (read-only)

1.2 Entity Operations

CreateEntity

interface CreateEntityRequest {
  entity_class: "EntityArchetype" | "EntityInstance";
  universe_id: UUID;
  name: string;
  entity_type: "character" | "faction" | "location" | "object" | "concept" | "organization";
  description: string;
  properties: Record<string, any>;
  state_tags?: string[];  // EntityInstance only
  derives_from?: UUID;  // EntityInstance only, optional EntityArchetype reference
  confidence: number;  // 0.0-1.0
  authority: "source" | "gm" | "player" | "system";
  evidence_refs: string[];  // ["source:uuid", "turn:uuid", ...]
}

interface CreateEntityResponse {
  entity_id: UUID;
  canon_level: "proposed" | "canon";
  created_at: timestamp;
}

Authority: CanonKeeper only Validation:

universe_id must exist
confidence ∈ [0.0, 1.0]
state_tags only for EntityInstance
derives_from must reference EntityArchetype of same type

GetEntity

interface GetEntityRequest {
  entity_id: UUID;
  include_relationships?: boolean;
  include_state_history?: boolean;
}

interface GetEntityResponse {
  entity_id: UUID;
  entity_class: "EntityArchetype" | "EntityInstance";
  universe_id: UUID;
  name: string;
  entity_type: string;
  description: string;
  properties: Record<string, any>;
  state_tags?: string[];
  canon_level: "proposed" | "canon" | "retconned";
  confidence: number;
  created_at: timestamp;
  updated_at?: timestamp;
  relationships?: Relationship[];  // if requested
}

Authority: Any agent (read-only)

UpdateEntityState

interface UpdateEntityStateRequest {
  entity_id: UUID;
  state_tag_changes: {
    add?: string[];
    remove?: string[];
  };
  authority: "gm" | "player" | "system";
  evidence_refs: string[];
}

interface UpdateEntityStateResponse {
  entity_id: UUID;
  new_state_tags: string[];
  fact_ids: UUID[];  // created Fact nodes documenting changes
}

Authority: CanonKeeper only Validation:

entity must be EntityInstance
Creates Fact nodes for each state change

QueryEntities

interface QueryEntitiesRequest {
  universe_id?: UUID;
  entity_type?: string;
  entity_class?: "EntityArchetype" | "EntityInstance";
  canon_level?: "proposed" | "canon" | "retconned";
  state_tags?: {
    all_of?: string[];  // has ALL these tags
    any_of?: string[];  // has ANY of these tags
    none_of?: string[];  // has NONE of these tags
  };
  name_pattern?: string;  // regex or LIKE
  limit?: number;
  offset?: number;
}

interface QueryEntitiesResponse {
  entities: Entity[];
  total: number;
}

Authority: Any agent (read-only)

1.3 Fact & Event Operations

CreateFact

interface CreateFactRequest {
  universe_id: UUID;
  statement: string;
  time_ref?: timestamp;
  duration?: number;
  involved_entity_ids: UUID[];
  confidence: number;
  authority: "source" | "gm" | "player" | "system";
  evidence_refs: string[];  // ["source:uuid", "scene:uuid", "turn:uuid"]
}

interface CreateFactResponse {
  fact_id: UUID;
  canon_level: "proposed" | "canon";
  created_at: timestamp;
}

Authority: CanonKeeper only Validation:

Creates INVOLVES edges to entities
Creates SUPPORTED_BY edges to evidence

CreateEvent

interface CreateEventRequest {
  scene_id?: UUID;
  universe_id: UUID;
  title: string;
  description: string;
  time_ref?: timestamp;
  severity: number;  // 0-10
  involved_entity_ids: UUID[];
  causes_event_ids?: UUID[];  // causal edges
  confidence: number;
  authority: "source" | "gm" | "player" | "system";
  evidence_refs: string[];
}

interface CreateEventResponse {
  event_id: UUID;
  canon_level: "proposed" | "canon";
  created_at: timestamp;
}

Authority: CanonKeeper only Validation:

Creates CAUSES edges (must be acyclic)
Creates INVOLVES edges to entities

QueryFacts

interface QueryFactsRequest {
  universe_id?: UUID;
  entity_id?: UUID;  // facts involving this entity
  time_range?: { start: timestamp; end: timestamp };
  canon_level?: "proposed" | "canon" | "retconned";
  authority?: "source" | "gm" | "player" | "system";
  limit?: number;
  offset?: number;
}

interface QueryFactsResponse {
  facts: Fact[];
  total: number;
}

Authority: Any agent (read-only)

1.4 Story & Scene Operations

CreateStory

interface CreateStoryRequest {
  universe_id: UUID;
  title: string;
  story_type: "campaign" | "arc" | "episode" | "one_shot";
  theme?: string;
  premise?: string;
  parent_story_id?: UUID;  // for arcs within campaigns
  start_time_ref?: timestamp;
}

interface CreateStoryResponse {
  story_id: UUID;
  created_at: timestamp;
}

Authority: CanonKeeper (or Orchestrator for planning)

CreateCanonicalScene

interface CreateCanonicalSceneRequest {
  story_id: UUID;
  title: string;
  purpose?: string;
  order: number;
  time_ref?: timestamp;
  participating_entity_ids: UUID[];
}

interface CreateCanonicalSceneResponse {
  scene_id: UUID;
  created_at: timestamp;
}

Authority: CanonKeeper only Note: Most scenes stay MongoDB-only. Only create in Neo4j if needed for timeline/continuity.

1.5 Provenance Operations

CreateSource

interface CreateSourceRequest {
  universe_id: UUID;
  doc_id: string;  // MongoDB reference
  title: string;
  edition?: string;
  provenance?: string;  // ISBN, URL, etc.
  source_type: "manual" | "rulebook" | "lore" | "session";
  canon_level: "proposed" | "canon" | "authoritative";
}

interface CreateSourceResponse {
  source_id: UUID;
  created_at: timestamp;
}

Authority: CanonKeeper only

LinkEvidence

interface LinkEvidenceRequest {
  canonical_id: UUID;  // Fact/Event/Entity/Axiom
  canonical_type: "Fact" | "Event" | "Entity" | "Axiom";
  evidence_id: UUID;
  evidence_type: "Source" | "Scene" | "Turn";
}

interface LinkEvidenceResponse {
  edge_id: string;
}

Authority: CanonKeeper only Validation: Creates SUPPORTED_BY edge

2. MongoDB Narrative Operations

2.1 Scene Operations

CreateScene

interface CreateSceneRequest {
  story_id: UUID;  // Neo4j reference
  universe_id: UUID;  // Neo4j reference
  title: string;
  purpose?: string;
  order?: number;  // optional ordering within the Story
  location_ref?: UUID;  // EntityInstance ID
  participating_entities: UUID[];  // EntityInstance IDs
}

interface CreateSceneResponse {
  scene_id: UUID;
  status: "active";
  created_at: Date;
}

Authority: Orchestrator only Storage: MongoDB scenes collection

AppendTurn

interface AppendTurnRequest {
  scene_id: UUID;
  speaker: "user" | "gm" | "entity";
  entity_id?: UUID;  // if speaker is entity
  text: string;
  resolution_ref?: UUID;
}

interface AppendTurnResponse {
  turn_id: UUID;
  timestamp: Date;
}

Authority: Narrator, Orchestrator Storage: Appends to scenes.turns array or separate turns collection

GetScene

interface GetSceneRequest {
  scene_id: UUID;
  include_turns?: boolean;
  include_proposals?: boolean;
  turn_limit?: number;  // last N turns
}

interface GetSceneResponse {
  scene_id: UUID;
  story_id: UUID;
  universe_id: UUID;
  title: string;
  status: "active" | "finalizing" | "completed";
  order?: number;
  location_ref?: UUID;
  participating_entities: UUID[];
  turns?: Turn[];
  proposed_changes?: UUID[];
  canonical_outcomes?: UUID[];
  summary?: string;
  created_at: Date;
  updated_at: Date;
  completed_at?: Date;
}

Authority: Any agent (read-only)

FinalizeScene

interface FinalizeSceneRequest {
  scene_id: UUID;
  canonical_outcome_ids: UUID[];  // Neo4j Fact/Event IDs
  summary: string;
}

interface FinalizeSceneResponse {
  scene_id: UUID;
  status: "completed";
  completed_at: Date;
}

Authority: CanonKeeper (after canonization) Side effects:

Updates scene.status = "completed"
Sets canonical_outcomes
Triggers Indexer to embed summary

2.2 ProposedChange Operations

CreateProposedChange

interface CreateProposedChangeRequest {
  scene_id: UUID;
  turn_id?: UUID;  // optional for ingest/system proposals
  type: "fact" | "entity" | "relationship" | "state_change" | "event";
  content: Record<string, any>;  // structure depends on type
  evidence: Array<{
    type: "turn" | "snippet" | "source" | "rule";
    ref_id: UUID;
  }>;
  confidence: number;
  authority: "source" | "gm" | "player" | "system";
}

interface CreateProposedChangeResponse {
  proposal_id: UUID;
  status: "pending";
  created_at: Date;
}

Authority: Resolver, Narrator, any agent proposing changes Storage: MongoDB proposed_changes collection

EvaluateProposal

interface EvaluateProposalRequest {
  proposal_id: UUID;
  decision: "accepted" | "rejected";
  rationale?: string;
  canonical_id?: UUID;  // if accepted, the Neo4j node/edge ID
}

interface EvaluateProposalResponse {
  proposal_id: UUID;
  status: "accepted" | "rejected";
  evaluated_at: Date;
}

Authority: CanonKeeper only Side effects:

Updates proposal status
If accepted, links to canonical_id

GetPendingProposals

interface GetPendingProposalsRequest {
  scene_id?: UUID;
  type?: "fact" | "entity" | "relationship" | "state_change" | "event";
  limit?: number;
}

interface GetPendingProposalsResponse {
  proposals: ProposedChange[];
  total: number;
}

Authority: CanonKeeper (for evaluation)

2.3 Memory Operations

CreateCharacterMemory

interface CreateCharacterMemoryRequest {
  entity_id: UUID;  // Neo4j EntityInstance
  text: string;
  linked_fact_id?: UUID;  // optional Neo4j Fact anchor
  scene_id?: UUID;
  emotional_valence: number;  // -1.0 to 1.0
  importance: number;  // 0.0-1.0
  certainty: number;  // 0.0-1.0
}

interface CreateCharacterMemoryResponse {
  memory_id: UUID;
  created_at: Date;
}

Authority: MemoryManager only Side effects: Triggers Indexer to embed memory

RetrieveCharacterMemories

interface RetrieveCharacterMemoriesRequest {
  entity_id: UUID;
  limit?: number;
  min_importance?: number;
  semantic_query?: string;  // if provided, uses Qdrant
}

interface RetrieveCharacterMemoriesResponse {
  memories: Memory[];
  total: number;
}

Authority: ContextAssembly, MemoryManager

2.4 Document & Snippet Operations

CreateDocument

interface CreateDocumentRequest {
  source_id: UUID;  // Neo4j Source
  universe_id: UUID;
  minio_ref: string;
  title: string;
  filename: string;
  file_type: string;
}

interface CreateDocumentResponse {
  doc_id: UUID;
  extraction_status: "pending";
  created_at: Date;
}

Authority: Ingest pipeline

CreateSnippet

interface CreateSnippetRequest {
  doc_id: UUID;
  source_id: UUID;
  text: string;
  page?: number;
  section?: string;
  chunk_index: number;
}

interface CreateSnippetResponse {
  snippet_id: UUID;
  created_at: Date;
}

Authority: Ingest pipeline Side effects: Triggers Indexer to embed snippet

3. Qdrant Semantic Operations

3.1 Embedding Operations

EmbedSceneSummary

interface EmbedSceneSummaryRequest {
  scene_id: UUID;
  story_id: UUID;
  universe_id: UUID;
  text: string;
  timestamp: Date;
}

interface EmbedSceneSummaryResponse {
  vector_id: UUID;
  collection: "scene_chunks";
}

Authority: Indexer only

EmbedMemory

interface EmbedMemoryRequest {
  memory_id: UUID;
  entity_id: UUID;
  text: string;
  importance: number;
  timestamp: Date;
}

interface EmbedMemoryResponse {
  vector_id: UUID;
  collection: "memory_chunks";
}

Authority: Indexer only

3.2 Retrieval Operations

SemanticSearch

interface SemanticSearchRequest {
  query_text: string;
  collection: "scene_chunks" | "memory_chunks" | "snippet_chunks";
  filters?: {
    universe_id?: UUID;
    entity_id?: UUID;  // for memories
    source_id?: UUID;  // for snippets
  };
  limit?: number;
  min_score?: number;
}

interface SemanticSearchResponse {
  results: Array<{
    id: UUID;
    score: number;
    payload: Record<string, any>;
    text: string;
  }>;
}

Authority: ContextAssembly, any retrieval agent

4. Composite Operations (Cross-DB)

4.1 Context Assembly

AssembleSceneContext

interface AssembleSceneContextRequest {
  scene_id: UUID;
  include_canonical?: boolean;
  include_narrative?: boolean;
  include_semantic?: boolean;
  semantic_query?: string;
}

interface AssembleSceneContextResponse {
  canonical: {
    entities: Entity[];
    facts: Fact[];
    relations: Relationship[];
  };
  narrative: {
    prior_turns: Turn[];
    scene_summary?: string;
    gm_notes?: string;
  };
  recalled: {
    similar_scenes?: Scene[];
    character_memories?: Memory[];
    rule_excerpts?: Snippet[];
  };
  metadata: {
    universe_id: UUID;
    story_id: UUID;
    scene_id: UUID;
    timestamp: Date;
  };
}

Authority: ContextAssembly agent Data sources:

Neo4j: canonical state
MongoDB: narrative logs
Qdrant: semantic recall

4.2 Canonization

CanonizeScene

interface CanonizeSceneRequest {
  scene_id: UUID;
  evaluate_proposals?: boolean;  // default true
}

interface CanonizeSceneResponse {
  scene_id: UUID;
  accepted_proposals: UUID[];
  rejected_proposals: UUID[];
  canonical_fact_ids: UUID[];
  canonical_event_ids: UUID[];
  canonical_entity_ids: UUID[];
}

Authority: CanonKeeper only Operations:

Fetch pending proposals from MongoDB
Evaluate each (authority + confidence checks)
Write accepted to Neo4j (Facts/Events/Entities)
Create SUPPORTED_BY edges
Update MongoDB proposals status
Finalize scene in MongoDB
Trigger Indexer

5. Authority Enforcement

Authority Matrix

Operation	Allowed Agents	Validation
CreateEntity	CanonKeeper	Requires evidence_refs
CreateFact	CanonKeeper	Requires evidence_refs, involved entities
CreateProposedChange	Resolver, Narrator, any	None (staging)
EvaluateProposal	CanonKeeper	Authority + confidence checks
CreateScene	Orchestrator	Requires valid story_id
CreateStory	CanonKeeper, Orchestrator	Planning writes; canonical container
AppendTurn	Narrator, Orchestrator	Scene must be active
UpdateEntityState	CanonKeeper	Creates Fact nodes
EmbedMemory	Indexer	Requires valid memory_id
SemanticSearch	Any	Read-only

Enforcement Mechanism

interface APIRequest {
  agent_id: string;
  agent_type: "Orchestrator" | "CanonKeeper" | "Narrator" | ...;
  operation: string;
  params: Record<string, any>;
}

function enforceAuthority(request: APIRequest): boolean {
  const allowed = AUTHORITY_MATRIX[request.operation];
  return allowed.includes(request.agent_type);
}

6. Transaction Semantics

6.1 Scene Canonization Transaction

Scope: End of scene batch commit

Atomicity:

All proposals evaluated atomically (all-or-nothing per proposal)
If Neo4j write fails, proposal stays "pending"
MongoDB scene state reflects last successful canonization

Isolation:

Concurrent scenes can canonize independently
Same scene cannot canonize concurrently (lock scene_id)

Durability:

Neo4j writes are durable once committed
MongoDB proposals track status
Qdrant updates are eventual (can retry)

6.2 Entity State Update Transaction

Scope: Updating entity state tags

Operations:

Update EntityInstance.state_tags (Neo4j)
Create Fact documenting change (Neo4j)
Link INVOLVES edge (Neo4j)
Link SUPPORTED_BY evidence (Neo4j)

Rollback: If any step fails, rollback all (Neo4j transaction)

7. Use Case Examples

P-1: Start New Story

Data flow:

1. CreateStory (Neo4j)
   → story_id

2. CreateScene (MongoDB)
   → scene_id, status=active

3. Optional: CreateCanonicalScene (Neo4j)
   → canonical scene_id for timeline

P-3: User Turn in Active Scene

Data flow:

1. AppendTurn (MongoDB)
   → turn_id

2. CreateProposedChange (MongoDB) - if action implies changes
   → proposal_id, status=pending

3. No Neo4j writes (deferred)

P-8: End Scene (Canonization)

Data flow:

1. GetPendingProposals (MongoDB)
   → proposals[]

2. For each proposal:
   a. EvaluateProposal (CanonKeeper logic)
   b. If accepted:
      - CreateFact/CreateEvent (Neo4j)
      - LinkEvidence (Neo4j)
      - EvaluateProposal status=accepted (MongoDB)
   c. If rejected:
      - EvaluateProposal status=rejected (MongoDB)

3. FinalizeScene (MongoDB)
   → status=completed, canonical_outcomes=fact_ids

4. EmbedSceneSummary (Qdrant)
   → indexed for recall

I-1: Upload Document

Data flow:

1. CreateSource (Neo4j)
   → source_id

2. CreateDocument (MongoDB)
   → doc_id, minio_ref

3. CreateSnippet × N (MongoDB)
   → snippet_ids[]

4. EmbedSnippet × N (Qdrant)
   → indexed

5. CreateProposedChange × M (MongoDB)
   → proposals for axioms/entities

6. User review → EvaluateProposal × M
   → accepted proposals

7. CreateEntity/CreateAxiom (Neo4j) for accepted
   → canonical_ids

Q-1: Semantic Search

Data flow:

1. SemanticSearch (Qdrant)
   → candidate IDs

2. GetEntity / QueryFacts (Neo4j)
   → canonical data

3. Optional: GetScene (MongoDB) for narrative details
   → narrative context

4. Return composed result

8. Error Handling

Error Codes

Code	Meaning	Recovery
`UNAUTHORIZED`	Agent lacks authority for operation	Reject request
`NOT_FOUND`	Referenced ID doesn't exist	Check references
`VALIDATION_ERROR`	Invalid parameters	Fix parameters
`CONSTRAINT_VIOLATION`	DB constraint failed	Check invariants
`TRANSACTION_FAILED`	DB write failed	Retry or rollback
`ALREADY_CANONIZED`	Scene already finalized	Cannot modify

Retry Policy

Idempotent operations (reads): Safe to retry
Non-idempotent writes (creates): Use unique IDs to detect duplicates
Transactions: Rollback on failure, retry entire transaction

9. Performance Considerations

Caching Strategy

What to cache:

Frequently accessed entities (PCs, active NPCs)
Current scene canonical state
Universe/Story metadata

Cache invalidation:

On entity state update
On scene canonization
TTL: 5 minutes for canonical data

Batch Operations

CreateProposedChange bulk:

interface CreateProposedChangesBulkRequest {
  proposals: CreateProposedChangeRequest[];
}

Reduces round-trips for multi-change turns.

10. API Versioning

Current version: v1

Breaking changes require v2:

Changing request/response schemas
Removing operations
Changing authority requirements

Non-breaking changes (v1.x):

Adding optional parameters
Adding new operations
Extending response data

11. Implementation Checklist

To implement this API:

References

DATABASE_INTEGRATION.md - Data layer architecture
AGENT_ORCHESTRATION.md - Agent roles and authority
ONTOLOGY.md - Data model specification
CONVERSATIONAL_LOOPS.md - Loop workflows

DATA_LAYER_API

MONITOR Data Layer API

Overview

API Architecture

1. Neo4j Canonical Operations

1.1 Universe & World Structure

CreateUniverse

GetUniverse

ListUniverses

1.2 Entity Operations

CreateEntity

GetEntity

UpdateEntityState

QueryEntities

1.3 Fact & Event Operations

CreateFact

CreateEvent

QueryFacts

1.4 Story & Scene Operations

CreateStory

CreateCanonicalScene

1.5 Provenance Operations

CreateSource

LinkEvidence

2. MongoDB Narrative Operations

2.1 Scene Operations

CreateScene

AppendTurn

GetScene

FinalizeScene

2.2 ProposedChange Operations

CreateProposedChange

EvaluateProposal

GetPendingProposals

2.3 Memory Operations

CreateCharacterMemory

RetrieveCharacterMemories

2.4 Document & Snippet Operations

CreateDocument

CreateSnippet

3. Qdrant Semantic Operations

3.1 Embedding Operations

EmbedSceneSummary

EmbedMemory

3.2 Retrieval Operations

SemanticSearch

4. Composite Operations (Cross-DB)

4.1 Context Assembly

AssembleSceneContext

4.2 Canonization

CanonizeScene

5. Authority Enforcement

Authority Matrix

Enforcement Mechanism

6. Transaction Semantics

6.1 Scene Canonization Transaction

6.2 Entity State Update Transaction

7. Use Case Examples

P-1: Start New Story

P-3: User Turn in Active Scene

P-8: End Scene (Canonization)

I-1: Upload Document

Q-1: Semantic Search

8. Error Handling

Error Codes

Retry Policy

9. Performance Considerations

Caching Strategy

Batch Operations

10. API Versioning

11. Implementation Checklist

References

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!