Skip to content

data-durability.md

Latest commit

 

History

History
65 lines (41 loc) · 3.61 KB

data-durability.md

File metadata and controls

65 lines (41 loc) · 3.61 KB
Raw

Data Durability

Sokratos is designed to avoid silent data loss across its memory pipelines. Every path from ingestion to storage has retry mechanisms backed by PostgreSQL, so in-flight work survives process crashes and transient backend failures.

Conversation Triage

When a user sends a message, the exchange is not triaged immediately. Instead:

  1. The message loop writes the raw exchange to the pending_triage table via EnqueueConversationTriage().
  2. During the next heartbeat tick, DrainPendingTriage() reads pending items and runs the full triage pipeline (truncation, adaptive threshold lookup, salience scoring, contradiction check, memory save).
  3. On success, the row is deleted.
  4. On failure, the retry counter is incremented. After 3 failures, the item is logged to failed_operations and deleted.

This ensures conversation data is never lost due to slot contention, LLM timeouts, or process crashes. The pending_triage table acts as a durable work queue.

Email Triage

Emails follow a similar pattern with an additional safety guarantee:

  1. New emails are fetched and filtered against processed_emails.
  2. Each email is triaged asynchronously via TriageAndSaveEmailAsync().
  3. The email is only marked as processed (via an onProcessed callback) after triage succeeds or is enqueued for retry.
  4. If triage fails and cannot be retried (no DB), the email is not marked as processed — it will be picked up on the next email check.

This prevents the scenario where an email is marked as "seen" but its content was never saved to memory.

Context Sliding & Distillation

When the conversation context exceeds the token budget, old messages are archived:

  1. SlideAndArchiveContext() formats the trimmed messages and sends them for distillation (LLM extraction of lasting facts).
  2. Distillation tries multiple backends in order: DTC queue → subagent queue → direct grammar call → direct subagent call.
  3. If all backends fail, the raw archive text is written to the pending_distillation table.
  4. During heartbeat maintenance, DrainPendingDistillation() retries pending items with available backends.
  5. After 3 failed retries, the raw text is saved directly to memory as a conversation_archive — less granular than distilled facts, but the data is preserved.

Failure Observability

All final-attempt failures are logged to the failed_operations table with operation type, label, error message, and context data. The heartbeat includes recent failures in its context XML (<recent_failures>), so the supervisor is aware of persistent issues.

Database Tables

Table Purpose Drained by
pending_triage Deferred and failed triage items Heartbeat tick (DrainPendingTriage)
pending_distillation Failed archive distillation Heartbeat maintenance (DrainPendingDistillation)
failed_operations Terminal failures (observability) Manual inspection / heartbeat context

What Can Still Be Lost

  • Process crash during heartbeat drain: Items currently being processed (mid-triage, mid-distillation) may fail without being re-queued. The next drain will skip them since they're deleted only on success.
  • Database unavailable: If PostgreSQL is down, enqueue calls log a warning but the item is not persisted. This is an infrastructure failure outside the application's control.
  • Tool results in conversation: Tool call parameters and raw results are not archived during context sliding — only the assistant's natural language synthesis. If the assistant's response is vague, the raw tool data is lost.