From 966b70daa1da39ebdf17a87c890b03fae9fefc0e Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:27 +0100 Subject: [PATCH 01/63] docs: archive Utopia Architecture.md --- docs/history/original-architecture.md | 1818 +++++++++++++++++++++++++ 1 file changed, 1818 insertions(+) create mode 100644 docs/history/original-architecture.md diff --git a/docs/history/original-architecture.md b/docs/history/original-architecture.md new file mode 100644 index 0000000..626418f --- /dev/null +++ b/docs/history/original-architecture.md @@ -0,0 +1,1818 @@ +Yes. Here is the version that treats **Utopia as a serious software architecture**, not a vague product idea. + +--- + +# **Utopia** + +**Utopia is a private cognitive operating system and judgment refinery for a non-linear operator.** + +Its purpose is to do four things at once: + +1. preserve continuity across time +2. protect direction against drift +3. convert knowledge into usable judgment +4. collapse the present into the smallest correct move for the operator’s real condition + +That move is not always “do the task.” + +Sometimes the correct move is: + +* recover +* preserve continuity +* re-enter a thread +* reduce ambiguity +* ask one decisive question +* close a loop +* avoid a false interpretation +* act + +Utopia exists because the real failure is not forgetfulness. It is **continuity failure under state variability**, plus **knowledge-to-action failure under complexity**. + +The system is built for an operator whose cognition is high, but whose access to that cognition is unstable across time, fatigue, interruption, ambiguity, and physiology. + +--- + +# **1\. The problem Utopia is actually solving** + +Most software in this category assumes a stable operator. + +It assumes that if the right task is visible, the person can do it. +It assumes that motivation is roughly consistent. +It assumes that context can be reconstructed cheaply. +It assumes that state is noise. + +Those assumptions are wrong for the kind of mind Utopia is designed for. + +The actual problem looks more like this: + +* the past self and present self do not hand off context cleanly +* attention is discontinuous +* re-entry cost is high +* the next step is often too abstract to enter +* physiological depletion gets misread as character failure +* ambiguity feels like incapacity +* reflection can become self-referential sludge +* books and ideas are consumed, but do not become live decision advantage +* complex problems are often attacked with the wrong frame because the right questions never get surfaced + +So Utopia is solving **two coupled failures**: + +## **Continuity failure** + +The present self cannot inherit the past self’s context, edge, momentum, and truth without paying a high reconstruction cost. + +## **Judgment failure** + +Valuable knowledge and experience do not automatically become the right questions, the right distinctions, or the right action policy at the moment of need. + +That is why Utopia has to be more than productivity software. + +It is not just trying to help the operator remember. + +It is trying to help the operator **remain aligned, interpret correctly, and move cleanly under variable internal conditions**. + +--- + +# **2\. The product in one sentence** + +**Utopia is a private, state-aware, physiology-informed, model-agnostic cognitive control architecture that preserves direction, reconstructs continuity, refines knowledge into reusable judgment, and selects the smallest correct move for the operator’s actual present state.** + +That is the cleanest accurate definition. + +--- + +# **3\. What Utopia is not** + +This matters because the architecture collapses if the product drifts into the wrong category. + +Utopia is not: + +* a generic note-taking tool +* a second-brain scrapbook +* a diary +* a mood tracker +* a searchable highlight archive +* a generic AI assistant over documents +* a motivational habit app +* a task manager with better copy +* an agent swarm pretending to be intelligence + +It is a **private operating system for continuity, judgment, and action**. + +Its retention standard is severe: + +**Only preserve what improves future orientation, diagnosis, judgment, execution, calibration, or correction.** + +Everything else is noise tax. + +--- + +# **4\. The governing design philosophy** + +Utopia only works if it obeys hard architectural laws. + +## **Reality over narrative** + +The system must not trust momentary self-story as ground truth. + +## **Direction over activity** + +Motion without vector is drift. + +## **One correct move over option abundance** + +Too many choices create internal negotiation and energy leakage. + +## **Continuity over productivity theater** + +The true bottleneck is often re-entry cost, not raw willingness. + +## **Compression over accumulation** + +The system should get sharper over time, not larger. + +## **State-aware, not moralistic** + +Low usable capacity is not proof of weakness. + +## **Questions before false certainty** + +When the problem is structurally underdefined, the correct move is often a better question, not more action. + +## **Privacy or failure** + +If the system is not trusted, the data becomes edited. +If the data becomes edited, the system becomes fake. +If the system becomes fake, the whole architecture collapses. + +--- + +# **5\. The top-level system model** + +At the highest level, Utopia has four jobs: + +1. **sense reality** +2. **interpret reality** +3. **select policy** +4. **learn from outcome** + +The full system looks like this: + +CONSTITUTION / BOUNDARY + ↓ +VECTOR + ↓ +EVIDENCE PLANE + ├─ subjective signals + ├─ behavioral signals + ├─ contextual signals + ├─ physiological signals (WHOOP \+ optional biomarker layer) + └─ external knowledge sources + ↓ +NORMALIZATION / FEATURE EXTRACTION / COMPRESSION + ↓ +AETHER + ├─ truth memory + ├─ conceptual memory + ├─ procedural memory + ├─ episodic traces + ├─ calibration memory + └─ knowledge graph / lens packs + ↓ +AI FABRIC + ├─ Router + ├─ State Estimator + ├─ Blocker Classifier + ├─ Problem Structurer + ├─ Context Retriever + ├─ Physiology Interpreter + ├─ Interrogator + ├─ Contradiction Checker + ├─ Vector Arbiter + ├─ Council + └─ Policy Selector + ↓ +SCHRÖDINGER + ↓ +ONE CORRECT MOVE + ├─ recover + ├─ preserve + ├─ orient + ├─ re-enter + ├─ clarify + ├─ ask + ├─ execute + ├─ close loop + └─ review + ↓ +TRACE + ↓ +REVIEW / COMPRESSION / MODEL UPDATE + ↓ +PERSONAL EXECUTION MODEL + +That is the organism. + +--- + +# **6\. The deepest structural idea** + +Utopia is not one loop. It is **two loops sharing one memory and one control plane**. + +## **Loop A: continuity and execution** + +This loop answers: + +* what state am I in? +* what is blocking motion? +* what is the smallest correct move? +* how do I re-enter without paying a massive context tax? + +## **Loop B: judgment and strategic reasoning** + +This loop answers: + +* what kind of problem is this really? +* what matters? +* what is uncertain? +* what is hidden? +* which extracted models apply? +* what questions should be asked before committing to action? + +These loops share: + +* Vector +* Aether +* the Evidence Plane +* the AI Fabric +* Trace +* Review +* the Personal Execution Model + +This is why Utopia is stronger than a task system on one side and stronger than a passive research archive on the other. + +It is a **control system plus reasoning system**. + +--- + +# **7\. Constitution and Boundary Layer** + +This is the invisible legal framework of the system. + +It decides: + +* what enters +* what gets discarded +* what the system may infer +* when the system must expose uncertainty +* when preservation outranks execution +* when reflection has become waste +* which integrations are allowed +* how much authority the system has relative to user override + +## **What Utopia should store** + +It should preserve things like: + +* missions +* threads +* blockers +* re-entry artifacts +* action-outcome traces +* evidence-backed rules +* state transitions +* useful physiological priors +* extracted mechanisms from books +* heuristics +* question trees +* decision protocols +* contradictions that improve future judgment + +## **What Utopia should reject** + +It should reject: + +* every passing feeling +* decorative introspection +* generic summaries +* vanity reading logs +* quote hoarding +* untyped thought piles +* bloated reflections with no future use + +The boundary layer is what stops Utopia from degenerating into sludge. + +--- + +# **8\. Vector: the directional control plane** + +**Vector** is the layer that preserves direction. + +It answers: + +**What matters, at what horizon, relative to what direction?** + +Without Vector, the rest of the system can become highly competent at moving fast in the wrong direction. + +Vector preserves: + +* life arc +* seasonal focus +* weekly priority logic +* mission hierarchy +* anti-goals +* definitions of drift +* acceptable sacrifice boundaries +* active strategic threads + +## **Why Vector matters** + +Most productivity systems optimize local activity. +Vector exists to block: + +* busy drift +* local optimization +* false urgency +* avoidance disguised as work +* repeated abandonment of the deep thread + +## **Time horizons inside Vector** + +A proper Vector model should operate at multiple levels: + +* life arc +* season +* month or sprint +* week +* mission +* thread + +This gives the system the ability to distinguish between: + +* what is important in principle +* what is important now +* what is important relative to the current operating state + +Vector is not task storage. +It is **directional governance**. + +--- + +# **9\. Ontology: the semantic spine** + +A serious system needs strict internal language. + +If the terms are vague, the memory becomes fuzzy and the policies become incoherent. + +Utopia therefore needs a precise ontology. + +## **Core execution objects** + +### **Mission** + +A strategically meaningful objective. + +### **Thread** + +A live line of work within a mission. + +### **State** + +The current operating condition of the operator. + +### **Blocker** + +The dominant reason motion is failing. + +### **Mode** + +The type of intervention currently required. + +### **Action Primitive** + +The smallest executable intervention unit. + +### **Re-entry Artifact** + +A bridge object for future reactivation of a thread. + +### **Trace** + +A compressed post-action record. + +### **Pattern** + +A recurring structure across traces. + +### **Rule** + +A compressed truth suitable for reuse. + +### **Drift** + +Resource expenditure without vector alignment. + +### **Distortion** + +Mismatch between narrative and evidence. + +### **Closure** + +Deliberate finishing and packaging of a loop. + +## **Knowledge and judgment objects** + +### **Source** + +A book, paper, essay, transcript, note bundle, or case file. + +### **Extraction** + +A structured distillation from a source. + +### **Concept** + +A reusable idea or term. + +### **Mechanism** + +A causal process. + +### **Tradeoff** + +A meaningful tension between competing goods or constraints. + +### **Failure Mode** + +A repeatable way systems break. + +### **Heuristic** + +A compressed decision aid. + +### **Diagnostic Question** + +A question that materially reduces uncertainty. + +### **Decision Protocol** + +A reusable reasoning flow for a class of problems. + +### **Lens** + +A domain-specific way of seeing. + +### **Case** + +An internal or external example instance. + +### **Assumption** + +A relied-upon proposition that remains unverified. + +### **Constraint** + +A real limit on action. + +### **Decision Brief** + +The structured output Utopia produces for a live problem. + +The ontology is not decoration. +It is the interoperability contract for the whole system. + +--- + +# **10\. The Evidence Plane** + +The Evidence Plane answers: + +**What evidence exists about this moment, this operator, and this problem?** + +It has five main classes. + +## **1\. Subjective signals** + +Fast self-reports such as: + +* energy +* clarity +* resistance +* overwhelm +* emotional load +* perceived urgency +* subjective confidence + +## **2\. Behavioral signals** + +Observed behavior such as: + +* repeated failed starts +* switching frequency +* completion rate +* time since last meaningful action +* open-without-action patterns +* unfinished action count +* avoidance near completion +* thread neglect + +## **3\. Contextual signals** + +Structural facts about the environment: + +* time of day +* environment +* current obligations +* interruption history +* current mission +* current thread +* time since last thread touch + +## **4\. Physiological signals** + +This is where **WHOOP becomes a first-class subsystem**. + +WHOOP’s developer API currently exposes OAuth scopes for recovery, sleep, cycles, workouts, profile, and body measurements; recovery includes daily readiness-style metrics such as score, HRV, and resting heart rate, and WHOOP supports webhooks for event-driven updates. WHOOP’s own docs also define Recovery as a daily percentage calculated when the user wakes up, not as a continuously changing intraday mood meter. ([WHOOP for Developers](https://developer.whoop.com/api/?utm_source=chatgpt.com)) + +The architectural implication is simple: + +WHOOP should not sit off to the side as a nice-to-have wellness plugin. + +It belongs inside the core evidence plane as the **physiological prior subsystem**. + +## **5\. External knowledge** + +This includes: + +* books +* technical material +* essays +* transcripts +* notes +* postmortems +* prior cases +* extracted frameworks + +External knowledge is not a library here. +It is raw material for judgment refinement. + +--- + +# **11\. WHOOP’s exact role inside Utopia** + +WHOOP should be treated as: + +**a physiological prior engine for capacity, fragility, and depth selection** + +Not the truth. +Not the brain. +Not the policy authority. +A prior. + +Its job is to help Utopia distinguish: + +* depletion from ambiguity +* fragility from laziness +* low depth from zero capacity +* a bad day for open-ended work from a perfectly viable day for deterministic work + +WHOOP’s official material states that Recovery is based on measurements such as resting heart rate, HRV, respiratory rate, sleep-related measures, and other physiological indicators, and WHOOP measures HRV during the deepest period of sleep rather than continuously across the day. ([WHOOP for Developers](https://developer.whoop.com/docs/whoop-101/?utm_source=chatgpt.com)) + +That matters because Utopia is trying to answer a very specific control question: + +**Is the operator blocked because the work is wrong, because the state is wrong, or because the self-story is wrong?** + +WHOOP improves that inference. + +## **The WHOOP subsystem should contain:** + +### **WHOOP Connector** + +Handles OAuth, token refresh, scope negotiation, webhook registration, and ingestion. + +### **WHOOP Signal Store** + +Stores raw recovery, sleep, cycle, and workout events in time-series form. + +### **Physiology Interpreter** + +Converts raw metrics into operational inferences such as: + +* capacity envelope +* fragility risk +* action-depth ceiling +* recovery mismatch +* depletion likelihood +* recommended caution level + +### **Physiological Feature Deriver** + +Builds rolling features such as: + +* 3-day recovery trend +* recovery volatility +* sleep debt slope +* HRV deviation from personal baseline +* strain-to-recovery mismatch +* late exertion carryover +* sleep regularity + +### **Policy Constraint Layer** + +Feeds those features into state estimation, blocker classification, policy selection, and trigger generation. + +That is the right technical role. + +## **WHOOP should influence these specific parts of Utopia** + +### **State estimation** + +WHOOP helps decide whether “Deep Work” is realistic, whether “Preserve” should dominate, and whether “Clarify” is smarter than “Execute.” WHOOP’s docs explicitly position Recovery as a daily measure of how prepared the body is to perform. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/user-data/recovery/?utm_source=chatgpt.com)) + +### **Blocker classification** + +Low readiness should raise the prior probability of physiological depletion without forcing that conclusion. + +### **Policy selection** + +Schrödinger should scale action depth and work type using physiological constraints. + +### **Trigger engine** + +WHOOP webhooks allow the system to react when new recovery, sleep, or activity data arrives, rather than relying on the operator to remember to self-assess. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/webhooks/?utm_source=chatgpt.com)) + +### **Contradiction detection** + +WHOOP gives the system a way to challenge narratives like: +“I’m lazy,” +“I’m fine,” +“I can push,” +when the physiological evidence points elsewhere. + +### **Personal execution model** + +Over time, Utopia should learn how WHOOP signals correlate with this specific operator’s true performance envelope, rather than using generic assumptions. + +## **What WHOOP must not be allowed to do** + +WHOOP should not: + +* override Vector +* become an excuse engine +* become the sole explanation of low output +* dictate identity-level conclusions +* automatically cancel action on low-recovery days + +WHOOP is a prior, not a judge. + +## **Cortisol and biomarker layer** + +If you later add WHOOP Advanced Labs, that should sit in a separate **biomarker calibration layer**, not be conflated with wearable data. WHOOP’s Advanced Labs materials describe clinical lab-result integration with WHOOP data, and WHOOP support materials list cortisol among Advanced Labs biomarkers. That means wearable WHOOP data and lab biomarker data should be architecturally separate. ([WHOOP](https://www.whoop.com/us/en/advanced-labs/?srsltid=AfmBOoqKF3AaZUk2QVpqqJwlC0hiPizWN5iBizkJtUyw6lucZL9uGj-9&utm_source=chatgpt.com)) + +So the clean split is: + +* **WHOOP wearable data** \= operational daily physiology +* **Advanced Labs biomarkers** \= occasional calibration and deeper health context + +That distinction is important. + +--- + +# **12\. Normalization and Feature Extraction** + +Raw inputs are too noisy to be useful. + +This layer standardizes, denoises, and compresses evidence into usable features. + +Responsibilities include: + +* normalize subjective scales +* separate fresh from stale inputs +* compute re-entry risk +* estimate thread decay +* derive drift probability +* quantify ambiguity pressure +* summarize physiology into policy-relevant signals +* attach confidence and freshness metadata + +Examples of derived features: + +* repeated app opens without action → drift or ambiguity candidate +* long time since last thread touch → high re-entry cost +* repeated late-stage abandonment → completion aversion candidate +* high subjective exhaustion \+ decent recovery \+ repeated unclear tasks → ambiguity over depletion candidate +* low recovery \+ sleep disruption \+ high task openness → reduced action-depth recommendation + +This is a crucial layer. +Without it, everything above is reasoning over noise. + +--- + +# **13\. Aether: the selective memory organ** + +**Aether** is the epistemic core of Utopia. + +It is not a notebook. +It is not a diary. +It is not a quote archive. + +It is a **selective memory organ for truth, judgment, and continuity**. + +Its job is to preserve what improves future policy. + +## **Aether’s memory classes** + +### **Constitutional memory** + +Boundary rules, anti-goals, permissions, preservation rules. + +### **Directional memory** + +Vector, mission hierarchy, active season, drift definitions. + +### **Conceptual memory** + +Extracted intelligence from books, papers, notes, and source material. + +### **Procedural memory** + +Protocols and policy flows the system can execute. + +### **Episodic memory** + +Compressed traces of what actually happened. + +### **Calibration memory** + +What was believed, what happened, what corrected the model. + +### **Physiological calibration memory** + +How body-state signals correlate with actual usable capacity for this operator. + +## **What Aether stores** + +Aether should store: + +* observations +* rules +* patterns +* contradictions +* mechanisms +* decision protocols +* extracted lenses +* re-entry artifacts +* prior cases +* effective interventions +* false interpretations and later corrections + +The standard is brutal: +if it does not improve future diagnosis, judgment, or action, it should not survive. + +--- + +# **14\. Knowledge Refinery** + +This is the subsystem that turns reading into leverage. + +The Knowledge Refinery takes sources such as: + +* DDIA +* systems architecture books +* strategy books +* psychology books +* negotiation books +* your own case notes +* postmortems +* essays and papers + +and converts them into **reusable reasoning primitives**. + +It should not primarily emit summaries. + +It should emit: + +* concepts +* mechanisms +* causal logic +* tradeoffs +* invariants +* failure modes +* anti-patterns +* diagnostic questions +* decision rules +* heuristics +* boundary conditions +* cases +* contradictions +* applicability scope + +This is the difference between “AI that has read your books” and “a private council of extracted thinking systems.” + +For example, a technical book like DDIA should not live inside Utopia as text chunks first. +It should live as things like: + +* source-of-truth questions +* consistency tradeoffs +* idempotency checks +* recovery semantics +* durability concerns +* bottleneck patterns +* schema evolution constraints +* observability heuristics + +That is the right storage unit. + +--- + +# **15\. Lens Packs** + +A clean way to package extracted knowledge inside Utopia is as **lens packs**. + +A lens pack is a domain-specific reasoning bundle. + +Each lens pack should contain: + +* canonical concepts +* mechanisms +* tradeoffs +* failure modes +* heuristics +* diagnostic questions +* decision protocols +* boundary conditions +* examples + +Examples: + +## **DDIA lens** + +Teaches Utopia to ask: + +* what is the source of truth? +* what failure boundary exists? +* what needs durability? +* where can duplication happen? +* what recovery semantics exist? +* what consistency actually matters? + +## **Strategy lens** + +Teaches Utopia to ask: + +* where is leverage? +* what is being mistaken for motion? +* whose incentives shape the surface behavior? +* what is actually scarce? + +## **Psychology lens** + +Teaches Utopia to ask: + +* is state being misread as identity? +* what emotion is being turned into a belief? +* what is avoidance protecting? + +## **Negotiation lens** + +Teaches Utopia to ask: + +* what is each actor optimizing for? +* what is the BATNA? +* what is said versus unsayable? +* what would a hostile counterpart exploit? + +This makes books usable at runtime. + +--- + +# **16\. AI Fabric** + +The AI Fabric is Utopia’s orchestration layer. + +It is not “chat in the app.” +It is not a monolithic assistant. + +It is a **model-agnostic reasoning fabric** that decides what kind of cognition is needed, loads the right context, invokes the right modules, and returns structured outputs. + +## **Its jobs** + +* classify the moment +* choose the required reasoning path +* retrieve only the needed context +* call the right specialist modules +* constrain output shape +* escalate to deeper reasoning only when needed +* preserve deterministic rules wherever possible + +## **Why model-agnostic matters** + +Provider capabilities differ. + +OpenAI’s current Responses API is positioned as a primary interface for model responses and supports tool use and stateful interactions; OpenAI’s docs also expose streaming and schema-constrained structured outputs. Anthropic’s docs support tool use and prompt caching, including caching tools, system messages, and tool results. Gemini’s official docs support function calling and structured outputs, and Gemini 3 uses thought signatures that must be preserved in certain tool-calling flows. Bedrock’s Converse API provides a more consistent interface across models while still allowing provider-specific parameters when needed. ([OpenAI Platform](https://platform.openai.com/docs/api-reference/responses/list?ref=test-ippon.ghost.io&utm_source=chatgpt.com)) + +So the correct design is: + +**Utopia’s internal abstraction is the source of truth.** +**Provider SDKs are transport layers.** + +--- + +# **17\. Provider abstraction layer** + +A proper implementation should define capability-based internal contracts. + +Not “send this to OpenAI.” +Not “run this on Gemini.” + +Instead, define contracts such as: + +* `ReasoningTask` +* `StructuredDecisionTask` +* `ExtractionTask` +* `CompressionTask` +* `ToolCallTask` +* `RetrievalSelectionTask` +* `LowLatencyClassifierTask` + +And define capability profiles such as: + +* supports JSON schema output +* supports tool use +* supports streaming +* supports caching +* supports long context +* supports special reasoning-state passthrough +* latency tier +* cost tier +* reliability tier +* privacy/compliance tier + +Then create thin adapters for each provider. + +That architecture is durable. +It also lets Utopia swap backends without rewriting the cognition layer. + +--- + +# **18\. AI execution tiers** + +Utopia should not burn the biggest model on everything. + +It should have execution tiers. + +## **Tier 1: classifier / router** + +Cheap, fast, structured. + +Used for: + +* request typing +* blocker hypothesis +* simple confidence estimation +* trigger triage + +## **Tier 2: reasoning / policy** + +More capable model. + +Used for: + +* contradiction analysis +* complex re-entry synthesis +* policy selection +* problem structuring +* strategic ambiguity resolution + +## **Tier 3: extraction / compression** + +Cheap but structured. + +Used for: + +* source distillation +* trace compaction +* pattern candidates +* archive cleanup + +## **Tier 4: deterministic validators** + +No model where possible. + +Used for: + +* schema validation +* rule precedence +* confidence thresholds +* permission checks +* conflict resolution +* irreversible-action gating + +This is the correct cost-performance architecture. + +--- + +# **19\. The Router** + +The Router is the metacognitive dispatcher. + +It answers: + +**What kind of cognition is required right now?** + +Examples: + +* state problem +* re-entry problem +* blocker problem +* contradiction problem +* vector conflict +* reasoning problem +* review problem +* source extraction problem + +The Router should not freestyle. +It should classify and dispatch. + +A good router payload includes: + +* request type +* urgency level +* evidence sufficiency +* required depth +* safe output shape +* required modules +* context budget +* deterministic guardrails + +This keeps the rest of the system legible. + +--- + +# **20\. State Estimator** + +The State Estimator answers: + +**What operating condition is the operator most likely in right now?** + +A useful state set includes: + +* Recover +* Preserve +* Orient +* Clarify +* Re-enter +* Execute +* Deep Work +* Close Loop +* Review +* Drift + +The State Estimator should combine: + +* subjective evidence +* behavioral evidence +* contextual evidence +* WHOOP-derived priors +* recent traces +* thread characteristics + +Its output should include: + +* primary state +* confidence +* dominant blocker hypothesis +* contributing factors +* action-depth recommendation +* caution flags + +That output then constrains downstream policy. + +--- + +# **21\. Blocker Classifier** + +The Blocker Classifier answers: + +**Why is motion failing?** + +Core blocker classes should include: + +* ambiguity +* scope overload +* physiological depletion +* emotional threat +* context fracture +* vector conflict +* environmental friction +* stimulation hijack +* narrative distortion +* completion aversion +* problem misclassification +* decision fog + +This matters because every blocker implies a different intervention. + +Examples: + +* ambiguity → shrink problem / define next edge +* depletion → reduce depth / preserve continuity +* context fracture → generate re-entry artifact +* vector conflict → escalate to Vector Arbiter +* emotional threat → lower exposure / reduce symbolic load +* completion aversion → close-loop protocol + +Without blocker typing, the system will recommend the wrong thing while sounding intelligent. + +--- + +# **22\. Problem Structurer** + +This is one of the most important modules in the entire architecture. + +The Problem Structurer takes a messy real-world situation and turns it into a usable internal representation. + +It should extract: + +* objective +* stakes +* actors +* incentives +* constraints +* resources +* assumptions +* unknowns +* time horizon +* irreversibilities +* bottlenecks +* observable facts +* narrative layer +* possible distortions + +That lets Utopia distinguish between: + +* a technical problem +* a coordination problem +* an incentive problem +* a narrative problem +* a timing problem +* a capability problem +* a scope problem + +This is how the system avoids attacking the wrong battlefield. + +--- + +# **23\. Context Retriever** + +The Context Retriever exists because most advice engines over-retrieve. + +Utopia should retrieve only what is relevant. + +That means: + +* current thread summary +* last meaningful truth +* relevant Vector constraints +* matching rules from Aether +* similar prior cases +* lens packs relevant to the problem class +* recent physiological context if relevant + +The retriever should optimize for: + +* relevance +* recency +* confidence +* diversity of useful lenses +* low token waste + +This is where the memory system becomes operational instead of archival. + +--- + +# **24\. Interrogator** + +The Interrogator is one of the highest-value modules in Utopia. + +Its purpose is not to ask many questions. +Its purpose is to ask the **minimum set of high-leverage questions** that most reduce uncertainty. + +Question classes include: + +## **Problem typing** + +What kind of problem is this really? + +## **Objective clarity** + +What counts as success? + +## **Constraint discovery** + +What is truly fixed? + +## **Hidden variable detection** + +What unknown would most change the decision? + +## **Assumption exposure** + +What is being relied on without proof? + +## **Failure analysis** + +How does this most likely fail? + +## **Time and reversibility** + +What becomes harder if delayed? +What is irreversible? + +## **Distortion detection** + +Is state being mistaken for truth? +Is the symbolic meaning of the problem distorting the diagnosis? + +The Interrogator is where extracted books become live cognition. + +--- + +# **25\. Contradiction Checker** + +The Contradiction Checker compares: + +* self-story +* behavior +* physiology +* prior traces +* Vector +* actual outcomes + +Its function is to detect mismatches such as: + +* “I’m lazy” vs low recovery and broken sleep +* “I can push” vs fragile body-state and recent collapses +* “This is impossible” vs repeated ambiguity patterns +* “This matters” vs Vector showing it is drift + +This is one of the modules that makes Utopia dangerous in a good way. +It helps prevent false self-interpretation. + +--- + +# **26\. Vector Arbiter** + +The Vector Arbiter exists because urgency and importance are not the same. + +It checks whether the proposed move aligns with direction. + +Its output may be: + +* aligned +* misaligned +* conditionally aligned +* locally useful but globally drifting +* requires explicit override + +This prevents Utopia from becoming an elegant executor of accidental drift. + +--- + +# **27\. The Council** + +The Council is the multi-perspective reasoning layer. + +It should be narrow and explicit, not vague. + +A good configuration is: + +## **Cartographer** + +Maps the structure of the problem. + +## **Historian** + +Finds similar cases, prior patterns, and relevant extracted models. + +## **Skeptic** + +Attacks assumptions and overconfidence. + +## **Adversary** + +Models hostile incentives and exploitation paths. + +## **Strategist** + +Generates leverage points, sequencing, and option paths. + +## **Operator** + +Converts reasoning into executable action shape. + +## **Calibrator** + +Separates fact, inference, assumption, and speculation. + +The Council does not replace Schrödinger. +It prepares better policy for Schrödinger to collapse. + +--- + +# **28\. Schrödinger: the action-policy engine** + +**Schrödinger** is the subsystem that turns all of the above into one correct move. + +Its governing question is: + +**Given the current state, blocker, direction, uncertainty level, remembered truth, and physiological condition, what is the smallest correct move now?** + +The move may be: + +* recover +* preserve +* orient +* re-enter +* clarify +* ask one question +* execute +* close loop +* review + +This is important: + +Schrödinger should not assume that “action” is always the right output. + +Sometimes the right move is a question. +Sometimes it is continuity preservation. +Sometimes it is deliberate non-expansion. + +## **Inputs** + +* state estimate +* blocker classification +* Vector constraints +* Aether retrieval +* problem structure +* WHOOP-based capacity prior +* environment context +* recent action history + +## **Outputs** + +* mode +* action type +* action depth +* one next move +* rationale +* confidence +* caution flags + +Schrödinger is not a task manager. +It is a **policy engine**. + +--- + +# **29\. Re-entry Artifacts** + +A re-entry artifact is one of the most valuable objects in the whole system. + +Its purpose is to reduce the tax of interruption. + +A good re-entry artifact contains: + +* thread identity +* last completed step +* unresolved edge +* smallest next move +* relevant constraints +* trap to avoid on return + +Example shape: + +* Thread: parser normalization +* Last done: enum mismatch identified +* Next edge: inspect five failing rows +* Trap: do not reopen the whole architecture +* Context: mapping issue tied to downstream schema validator + +This object is how Utopia preserves continuity across state changes. + +--- + +# **30\. Trace Layer** + +The Trace Layer records what happened after the move. + +Each trace should be compressed and typed. + +A trace may include: + +* timestamp +* state +* blocker +* move chosen +* outcome +* revealed truth +* contradiction observed +* next edge +* confidence delta + +The trace is not a diary. +It is training data for Aether and the Personal Execution Model. + +--- + +# **31\. Review and Compression** + +This is Utopia’s immune system. + +Without compression, the system decays into information obesity. + +Responsibilities include: + +* promote repeated truths into rules +* merge duplicate patterns +* archive dead threads +* weaken rules that stop holding +* upgrade reliable question trees +* recalibrate intervention priors +* refine physiological interpretations +* remove low-value noise + +## **Review horizons** + +* micro-review after action +* daily review +* weekly compression +* periodic deep maintenance + +Compression is what turns memory into intelligence. + +--- + +# **32\. Personal Execution Model** + +Over time, Utopia should build a model of how this operator actually functions. + +Not generic humans. +This operator. + +It should learn: + +* which blockers recur most +* which interventions work best under which states +* when low recovery actually matters +* when subjective exhaustion is false +* which environments support depth +* where continuity decays fastest +* when completion aversion appears +* which problem classes are often misclassified +* how action-depth should be scaled by state and physiology + +This is where Utopia becomes asymmetrical. + +At first it is a framework. +Later it becomes a **personal operating model of cognition and execution**. + +--- + +# **33\. Storage architecture** + +A serious implementation should separate storage by semantic role. + +## **1\. Relational core** + +Use a relational database for canonical objects: + +* missions +* threads +* states +* traces +* rules +* blockers +* re-entry artifacts +* provider configs +* permissions +* schedules +* review logs + +This is the operational source of truth. + +## **2\. Vector index** + +Used for semantic retrieval across: + +* extracted concepts +* cases +* prior traces +* rules +* problem briefs +* source fragments when needed + +This supports relevance-based recall. + +## **3\. Graph layer** + +Optional early, important later. + +Used for explicit relations like: + +* concept → mechanism +* mechanism → failure mode +* lens → question +* case → pattern +* rule → applies\_to +* rule → contradicted\_by + +This is where cross-domain reasoning gets stronger. + +## **4\. Time-series / event store** + +Useful for physiological signals, action timing, behavior patterns, and triggers. + +WHOOP and action telemetry fit naturally here. + +## **5\. Object store** + +For source documents, raw imports, snapshots, and large artifacts. + +That is the right decomposition. + +--- + +# **34\. Runtime architecture** + +A clean runtime design would split into the following services. + +## **API Gateway** + +Unified entry point for clients. + +## **Identity / Permissions Service** + +Controls local auth, provider tokens, access classes, revocation. + +## **Vector Service** + +Owns missions, direction, anti-goals, and drift logic. + +## **Evidence Service** + +Ingests subjective, behavioral, contextual, and physiological data. + +## **WHOOP Integration Service** + +Handles OAuth, webhooks, sync, normalization, and physiological feature derivation. WHOOP’s API surface and webhook model support this event-driven design. ([WHOOP for Developers](https://developer.whoop.com/api/?utm_source=chatgpt.com)) + +## **Knowledge Refinery Service** + +Runs ingestion, extraction, schema validation, and lens-pack generation. + +## **Aether Service** + +Owns memory objects, rules, patterns, traces, and retrieval surfaces. + +## **Orchestrator / AI Fabric Service** + +Runs routing, state estimation, problem structuring, question generation, and policy synthesis. + +## **Schrödinger Policy Service** + +Produces one move and action constraints. + +## **Review / Compression Service** + +Runs maintenance jobs, rule promotion, archive cleanup, and calibration. + +## **Trigger Engine** + +Listens to events and decides whether intervention is warranted. + +This service split makes the system implementable. + +--- + +# **35\. Event-driven design** + +Utopia should be heavily event-driven. + +Important event types include: + +* thread touched +* action started +* action completed +* repeated failed start +* re-entry gap exceeded +* new WHOOP recovery available +* sleep event ingested +* source extracted +* rule upgraded +* blocker distribution changed +* mission neglected +* contradiction detected + +This allows the system to react to real changes instead of relying only on polling and manual check-ins. + +WHOOP’s webhook support makes it especially suitable for this model. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/webhooks/?utm_source=chatgpt.com)) + +--- + +# **36\. Conflict resolution and precedence** + +The system will routinely receive conflicting signals. + +A clean internal precedence order is: + +1. external hard constraints +2. safety and preservation needs +3. Vector priorities +4. present-state feasibility +5. evidence-backed rules from Aether +6. observed behavior +7. physiological prior +8. subjective narrative +9. momentary impulse + +You can change the exact ranking, but you need one. + +Otherwise the system becomes whichever signal is loudest. + +WHOOP should be strong evidence, but not sovereign authority. + +--- + +# **37\. Security and privacy architecture** + +Privacy is not optional here. +It is functional infrastructure. + +Utopia should be designed as: + +* local-first where possible +* encrypted at rest +* explicit about every integration +* tokenized and revocable for external providers +* transparent about data classes +* configurable about retention +* conservative about third-party exposure + +The architecture should treat data domains separately: + +* strategic data +* physiological data +* source documents +* action traces +* provider metadata + +This lets the user selectively revoke or isolate domains without destroying the whole system. + +For model providers, the fabric should support per-provider privacy policies and transport settings rather than assuming one global trust model. Bedrock’s documentation, for example, describes Converse as a unified interface and notes content is used to generate responses rather than stored by Bedrock, while other providers have their own operational assumptions that should be isolated behind adapters. ([AWS Documentation](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html?utm_source=chatgpt.com)) + +--- + +# **38\. The four main software surfaces** + +As a product, Utopia should likely present itself through four main surfaces. + +## **1\. Control Surface** + +The live cockpit. + +It shows: + +* current state +* blocker hypothesis +* capacity envelope +* one mode +* one next move +* re-entry doorway +* caution flags + +WHOOP belongs here as context, not decoration. + +## **2\. Vector Surface** + +The directional layer. + +It shows: + +* life arc +* seasonal focus +* mission stack +* thread hierarchy +* drift risk +* neglected high-value lines + +## **3\. Problem Room** + +The reasoning layer. + +You bring in a live problem. +Utopia returns: + +* problem classification +* relevant lens packs +* key unknowns +* high-leverage questions +* blind spots +* risks +* option paths +* recommended action policy + +## **4\. Refinery / Review Surface** + +The compounding layer. + +It handles: + +* source ingestion +* extraction review +* pattern merges +* rule upgrades +* archive cleanup +* calibration review +* physiological interpretation adjustments + +That gives the product a clear shape. + +--- + +# **39\. Evaluation and metrics** + +A serious system needs evals. + +Not vanity metrics. +Real ones. + +## **Continuity metrics** + +* median re-entry time +* thread decay rate +* % of returns with valid next edge +* abandonment rate of active threads + +## **Policy metrics** + +* % of selected moves executed +* % of moves later judged correct +* action-depth mismatch rate +* false-push rate +* false-preserve rate + +## **Judgment metrics** + +* problem classification accuracy +* question usefulness score +* reduction in uncertainty after interrogation +* option quality score +* contradiction detection precision + +## **Knowledge metrics** + +* extraction reuse rate +* rule promotion rate +* stale knowledge ratio +* lens-pack retrieval usefulness + +## **Physiology metrics** + +* correlation between WHOOP-informed capacity estimate and realized usable depth +* low-recovery false-negative rate +* low-recovery false-excuse rate +* recovery trend usefulness for policy selection + +Without evals, Utopia becomes elegant theater. + +--- + +# **40\. The technical essence of WHOOP inside Utopia** + +To make sure WHOOP is fully embedded and not merely mentioned, here is the concise engineering truth: + +WHOOP supplies: + +* daily readiness-style recovery signal +* sleep structure and duration signals +* cycle / strain context +* workout / exertion events +* webhook-triggerable updates +* stable personal longitudinal physiology data + +WHOOP does not directly tell Utopia what to do. + +It tells Utopia how much confidence it should have in different kinds of action, how fragile the operator may be, and whether “I can’t do this” is more likely body-state, ambiguity, or self-story. The developer docs expose the relevant data scopes and webhook mechanisms, and WHOOP’s concept docs clarify that Recovery is a daily preparedness measure and that HRV is measured during deepest sleep, which makes it appropriate as a stable policy prior rather than a continuously fluctuating command signal. ([WHOOP for Developers](https://developer.whoop.com/api/?utm_source=chatgpt.com)) + +That is exactly where WHOOP belongs. + +--- + +# **41\. The final architecture in one line** + +**Utopia is a private, model-agnostic, physiology-aware cognitive control system and judgment refinery that preserves direction through Vector, senses reality through an evidence plane that includes WHOOP, refines truth and knowledge through Aether, routes cognition through a modular AI Fabric, and uses Schrödinger to select the smallest correct move for the operator’s real present condition.** + +And if compressed even further: + +**Utopia is software for preserving truth, continuity, and judgment under variable internal conditions** + From fe72c3abd44cb72ad50703b6a19b8e1d12ce8174 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:31 +0100 Subject: [PATCH 02/63] docs: move Utopia Architecture.md into history --- Utopia Architecture.md | 1818 ---------------------------------------- 1 file changed, 1818 deletions(-) delete mode 100644 Utopia Architecture.md diff --git a/Utopia Architecture.md b/Utopia Architecture.md deleted file mode 100644 index 626418f..0000000 --- a/Utopia Architecture.md +++ /dev/null @@ -1,1818 +0,0 @@ -Yes. Here is the version that treats **Utopia as a serious software architecture**, not a vague product idea. - ---- - -# **Utopia** - -**Utopia is a private cognitive operating system and judgment refinery for a non-linear operator.** - -Its purpose is to do four things at once: - -1. preserve continuity across time -2. protect direction against drift -3. convert knowledge into usable judgment -4. collapse the present into the smallest correct move for the operator’s real condition - -That move is not always “do the task.” - -Sometimes the correct move is: - -* recover -* preserve continuity -* re-enter a thread -* reduce ambiguity -* ask one decisive question -* close a loop -* avoid a false interpretation -* act - -Utopia exists because the real failure is not forgetfulness. It is **continuity failure under state variability**, plus **knowledge-to-action failure under complexity**. - -The system is built for an operator whose cognition is high, but whose access to that cognition is unstable across time, fatigue, interruption, ambiguity, and physiology. - ---- - -# **1\. The problem Utopia is actually solving** - -Most software in this category assumes a stable operator. - -It assumes that if the right task is visible, the person can do it. -It assumes that motivation is roughly consistent. -It assumes that context can be reconstructed cheaply. -It assumes that state is noise. - -Those assumptions are wrong for the kind of mind Utopia is designed for. - -The actual problem looks more like this: - -* the past self and present self do not hand off context cleanly -* attention is discontinuous -* re-entry cost is high -* the next step is often too abstract to enter -* physiological depletion gets misread as character failure -* ambiguity feels like incapacity -* reflection can become self-referential sludge -* books and ideas are consumed, but do not become live decision advantage -* complex problems are often attacked with the wrong frame because the right questions never get surfaced - -So Utopia is solving **two coupled failures**: - -## **Continuity failure** - -The present self cannot inherit the past self’s context, edge, momentum, and truth without paying a high reconstruction cost. - -## **Judgment failure** - -Valuable knowledge and experience do not automatically become the right questions, the right distinctions, or the right action policy at the moment of need. - -That is why Utopia has to be more than productivity software. - -It is not just trying to help the operator remember. - -It is trying to help the operator **remain aligned, interpret correctly, and move cleanly under variable internal conditions**. - ---- - -# **2\. The product in one sentence** - -**Utopia is a private, state-aware, physiology-informed, model-agnostic cognitive control architecture that preserves direction, reconstructs continuity, refines knowledge into reusable judgment, and selects the smallest correct move for the operator’s actual present state.** - -That is the cleanest accurate definition. - ---- - -# **3\. What Utopia is not** - -This matters because the architecture collapses if the product drifts into the wrong category. - -Utopia is not: - -* a generic note-taking tool -* a second-brain scrapbook -* a diary -* a mood tracker -* a searchable highlight archive -* a generic AI assistant over documents -* a motivational habit app -* a task manager with better copy -* an agent swarm pretending to be intelligence - -It is a **private operating system for continuity, judgment, and action**. - -Its retention standard is severe: - -**Only preserve what improves future orientation, diagnosis, judgment, execution, calibration, or correction.** - -Everything else is noise tax. - ---- - -# **4\. The governing design philosophy** - -Utopia only works if it obeys hard architectural laws. - -## **Reality over narrative** - -The system must not trust momentary self-story as ground truth. - -## **Direction over activity** - -Motion without vector is drift. - -## **One correct move over option abundance** - -Too many choices create internal negotiation and energy leakage. - -## **Continuity over productivity theater** - -The true bottleneck is often re-entry cost, not raw willingness. - -## **Compression over accumulation** - -The system should get sharper over time, not larger. - -## **State-aware, not moralistic** - -Low usable capacity is not proof of weakness. - -## **Questions before false certainty** - -When the problem is structurally underdefined, the correct move is often a better question, not more action. - -## **Privacy or failure** - -If the system is not trusted, the data becomes edited. -If the data becomes edited, the system becomes fake. -If the system becomes fake, the whole architecture collapses. - ---- - -# **5\. The top-level system model** - -At the highest level, Utopia has four jobs: - -1. **sense reality** -2. **interpret reality** -3. **select policy** -4. **learn from outcome** - -The full system looks like this: - -CONSTITUTION / BOUNDARY - ↓ -VECTOR - ↓ -EVIDENCE PLANE - ├─ subjective signals - ├─ behavioral signals - ├─ contextual signals - ├─ physiological signals (WHOOP \+ optional biomarker layer) - └─ external knowledge sources - ↓ -NORMALIZATION / FEATURE EXTRACTION / COMPRESSION - ↓ -AETHER - ├─ truth memory - ├─ conceptual memory - ├─ procedural memory - ├─ episodic traces - ├─ calibration memory - └─ knowledge graph / lens packs - ↓ -AI FABRIC - ├─ Router - ├─ State Estimator - ├─ Blocker Classifier - ├─ Problem Structurer - ├─ Context Retriever - ├─ Physiology Interpreter - ├─ Interrogator - ├─ Contradiction Checker - ├─ Vector Arbiter - ├─ Council - └─ Policy Selector - ↓ -SCHRÖDINGER - ↓ -ONE CORRECT MOVE - ├─ recover - ├─ preserve - ├─ orient - ├─ re-enter - ├─ clarify - ├─ ask - ├─ execute - ├─ close loop - └─ review - ↓ -TRACE - ↓ -REVIEW / COMPRESSION / MODEL UPDATE - ↓ -PERSONAL EXECUTION MODEL - -That is the organism. - ---- - -# **6\. The deepest structural idea** - -Utopia is not one loop. It is **two loops sharing one memory and one control plane**. - -## **Loop A: continuity and execution** - -This loop answers: - -* what state am I in? -* what is blocking motion? -* what is the smallest correct move? -* how do I re-enter without paying a massive context tax? - -## **Loop B: judgment and strategic reasoning** - -This loop answers: - -* what kind of problem is this really? -* what matters? -* what is uncertain? -* what is hidden? -* which extracted models apply? -* what questions should be asked before committing to action? - -These loops share: - -* Vector -* Aether -* the Evidence Plane -* the AI Fabric -* Trace -* Review -* the Personal Execution Model - -This is why Utopia is stronger than a task system on one side and stronger than a passive research archive on the other. - -It is a **control system plus reasoning system**. - ---- - -# **7\. Constitution and Boundary Layer** - -This is the invisible legal framework of the system. - -It decides: - -* what enters -* what gets discarded -* what the system may infer -* when the system must expose uncertainty -* when preservation outranks execution -* when reflection has become waste -* which integrations are allowed -* how much authority the system has relative to user override - -## **What Utopia should store** - -It should preserve things like: - -* missions -* threads -* blockers -* re-entry artifacts -* action-outcome traces -* evidence-backed rules -* state transitions -* useful physiological priors -* extracted mechanisms from books -* heuristics -* question trees -* decision protocols -* contradictions that improve future judgment - -## **What Utopia should reject** - -It should reject: - -* every passing feeling -* decorative introspection -* generic summaries -* vanity reading logs -* quote hoarding -* untyped thought piles -* bloated reflections with no future use - -The boundary layer is what stops Utopia from degenerating into sludge. - ---- - -# **8\. Vector: the directional control plane** - -**Vector** is the layer that preserves direction. - -It answers: - -**What matters, at what horizon, relative to what direction?** - -Without Vector, the rest of the system can become highly competent at moving fast in the wrong direction. - -Vector preserves: - -* life arc -* seasonal focus -* weekly priority logic -* mission hierarchy -* anti-goals -* definitions of drift -* acceptable sacrifice boundaries -* active strategic threads - -## **Why Vector matters** - -Most productivity systems optimize local activity. -Vector exists to block: - -* busy drift -* local optimization -* false urgency -* avoidance disguised as work -* repeated abandonment of the deep thread - -## **Time horizons inside Vector** - -A proper Vector model should operate at multiple levels: - -* life arc -* season -* month or sprint -* week -* mission -* thread - -This gives the system the ability to distinguish between: - -* what is important in principle -* what is important now -* what is important relative to the current operating state - -Vector is not task storage. -It is **directional governance**. - ---- - -# **9\. Ontology: the semantic spine** - -A serious system needs strict internal language. - -If the terms are vague, the memory becomes fuzzy and the policies become incoherent. - -Utopia therefore needs a precise ontology. - -## **Core execution objects** - -### **Mission** - -A strategically meaningful objective. - -### **Thread** - -A live line of work within a mission. - -### **State** - -The current operating condition of the operator. - -### **Blocker** - -The dominant reason motion is failing. - -### **Mode** - -The type of intervention currently required. - -### **Action Primitive** - -The smallest executable intervention unit. - -### **Re-entry Artifact** - -A bridge object for future reactivation of a thread. - -### **Trace** - -A compressed post-action record. - -### **Pattern** - -A recurring structure across traces. - -### **Rule** - -A compressed truth suitable for reuse. - -### **Drift** - -Resource expenditure without vector alignment. - -### **Distortion** - -Mismatch between narrative and evidence. - -### **Closure** - -Deliberate finishing and packaging of a loop. - -## **Knowledge and judgment objects** - -### **Source** - -A book, paper, essay, transcript, note bundle, or case file. - -### **Extraction** - -A structured distillation from a source. - -### **Concept** - -A reusable idea or term. - -### **Mechanism** - -A causal process. - -### **Tradeoff** - -A meaningful tension between competing goods or constraints. - -### **Failure Mode** - -A repeatable way systems break. - -### **Heuristic** - -A compressed decision aid. - -### **Diagnostic Question** - -A question that materially reduces uncertainty. - -### **Decision Protocol** - -A reusable reasoning flow for a class of problems. - -### **Lens** - -A domain-specific way of seeing. - -### **Case** - -An internal or external example instance. - -### **Assumption** - -A relied-upon proposition that remains unverified. - -### **Constraint** - -A real limit on action. - -### **Decision Brief** - -The structured output Utopia produces for a live problem. - -The ontology is not decoration. -It is the interoperability contract for the whole system. - ---- - -# **10\. The Evidence Plane** - -The Evidence Plane answers: - -**What evidence exists about this moment, this operator, and this problem?** - -It has five main classes. - -## **1\. Subjective signals** - -Fast self-reports such as: - -* energy -* clarity -* resistance -* overwhelm -* emotional load -* perceived urgency -* subjective confidence - -## **2\. Behavioral signals** - -Observed behavior such as: - -* repeated failed starts -* switching frequency -* completion rate -* time since last meaningful action -* open-without-action patterns -* unfinished action count -* avoidance near completion -* thread neglect - -## **3\. Contextual signals** - -Structural facts about the environment: - -* time of day -* environment -* current obligations -* interruption history -* current mission -* current thread -* time since last thread touch - -## **4\. Physiological signals** - -This is where **WHOOP becomes a first-class subsystem**. - -WHOOP’s developer API currently exposes OAuth scopes for recovery, sleep, cycles, workouts, profile, and body measurements; recovery includes daily readiness-style metrics such as score, HRV, and resting heart rate, and WHOOP supports webhooks for event-driven updates. WHOOP’s own docs also define Recovery as a daily percentage calculated when the user wakes up, not as a continuously changing intraday mood meter. ([WHOOP for Developers](https://developer.whoop.com/api/?utm_source=chatgpt.com)) - -The architectural implication is simple: - -WHOOP should not sit off to the side as a nice-to-have wellness plugin. - -It belongs inside the core evidence plane as the **physiological prior subsystem**. - -## **5\. External knowledge** - -This includes: - -* books -* technical material -* essays -* transcripts -* notes -* postmortems -* prior cases -* extracted frameworks - -External knowledge is not a library here. -It is raw material for judgment refinement. - ---- - -# **11\. WHOOP’s exact role inside Utopia** - -WHOOP should be treated as: - -**a physiological prior engine for capacity, fragility, and depth selection** - -Not the truth. -Not the brain. -Not the policy authority. -A prior. - -Its job is to help Utopia distinguish: - -* depletion from ambiguity -* fragility from laziness -* low depth from zero capacity -* a bad day for open-ended work from a perfectly viable day for deterministic work - -WHOOP’s official material states that Recovery is based on measurements such as resting heart rate, HRV, respiratory rate, sleep-related measures, and other physiological indicators, and WHOOP measures HRV during the deepest period of sleep rather than continuously across the day. ([WHOOP for Developers](https://developer.whoop.com/docs/whoop-101/?utm_source=chatgpt.com)) - -That matters because Utopia is trying to answer a very specific control question: - -**Is the operator blocked because the work is wrong, because the state is wrong, or because the self-story is wrong?** - -WHOOP improves that inference. - -## **The WHOOP subsystem should contain:** - -### **WHOOP Connector** - -Handles OAuth, token refresh, scope negotiation, webhook registration, and ingestion. - -### **WHOOP Signal Store** - -Stores raw recovery, sleep, cycle, and workout events in time-series form. - -### **Physiology Interpreter** - -Converts raw metrics into operational inferences such as: - -* capacity envelope -* fragility risk -* action-depth ceiling -* recovery mismatch -* depletion likelihood -* recommended caution level - -### **Physiological Feature Deriver** - -Builds rolling features such as: - -* 3-day recovery trend -* recovery volatility -* sleep debt slope -* HRV deviation from personal baseline -* strain-to-recovery mismatch -* late exertion carryover -* sleep regularity - -### **Policy Constraint Layer** - -Feeds those features into state estimation, blocker classification, policy selection, and trigger generation. - -That is the right technical role. - -## **WHOOP should influence these specific parts of Utopia** - -### **State estimation** - -WHOOP helps decide whether “Deep Work” is realistic, whether “Preserve” should dominate, and whether “Clarify” is smarter than “Execute.” WHOOP’s docs explicitly position Recovery as a daily measure of how prepared the body is to perform. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/user-data/recovery/?utm_source=chatgpt.com)) - -### **Blocker classification** - -Low readiness should raise the prior probability of physiological depletion without forcing that conclusion. - -### **Policy selection** - -Schrödinger should scale action depth and work type using physiological constraints. - -### **Trigger engine** - -WHOOP webhooks allow the system to react when new recovery, sleep, or activity data arrives, rather than relying on the operator to remember to self-assess. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/webhooks/?utm_source=chatgpt.com)) - -### **Contradiction detection** - -WHOOP gives the system a way to challenge narratives like: -“I’m lazy,” -“I’m fine,” -“I can push,” -when the physiological evidence points elsewhere. - -### **Personal execution model** - -Over time, Utopia should learn how WHOOP signals correlate with this specific operator’s true performance envelope, rather than using generic assumptions. - -## **What WHOOP must not be allowed to do** - -WHOOP should not: - -* override Vector -* become an excuse engine -* become the sole explanation of low output -* dictate identity-level conclusions -* automatically cancel action on low-recovery days - -WHOOP is a prior, not a judge. - -## **Cortisol and biomarker layer** - -If you later add WHOOP Advanced Labs, that should sit in a separate **biomarker calibration layer**, not be conflated with wearable data. WHOOP’s Advanced Labs materials describe clinical lab-result integration with WHOOP data, and WHOOP support materials list cortisol among Advanced Labs biomarkers. That means wearable WHOOP data and lab biomarker data should be architecturally separate. ([WHOOP](https://www.whoop.com/us/en/advanced-labs/?srsltid=AfmBOoqKF3AaZUk2QVpqqJwlC0hiPizWN5iBizkJtUyw6lucZL9uGj-9&utm_source=chatgpt.com)) - -So the clean split is: - -* **WHOOP wearable data** \= operational daily physiology -* **Advanced Labs biomarkers** \= occasional calibration and deeper health context - -That distinction is important. - ---- - -# **12\. Normalization and Feature Extraction** - -Raw inputs are too noisy to be useful. - -This layer standardizes, denoises, and compresses evidence into usable features. - -Responsibilities include: - -* normalize subjective scales -* separate fresh from stale inputs -* compute re-entry risk -* estimate thread decay -* derive drift probability -* quantify ambiguity pressure -* summarize physiology into policy-relevant signals -* attach confidence and freshness metadata - -Examples of derived features: - -* repeated app opens without action → drift or ambiguity candidate -* long time since last thread touch → high re-entry cost -* repeated late-stage abandonment → completion aversion candidate -* high subjective exhaustion \+ decent recovery \+ repeated unclear tasks → ambiguity over depletion candidate -* low recovery \+ sleep disruption \+ high task openness → reduced action-depth recommendation - -This is a crucial layer. -Without it, everything above is reasoning over noise. - ---- - -# **13\. Aether: the selective memory organ** - -**Aether** is the epistemic core of Utopia. - -It is not a notebook. -It is not a diary. -It is not a quote archive. - -It is a **selective memory organ for truth, judgment, and continuity**. - -Its job is to preserve what improves future policy. - -## **Aether’s memory classes** - -### **Constitutional memory** - -Boundary rules, anti-goals, permissions, preservation rules. - -### **Directional memory** - -Vector, mission hierarchy, active season, drift definitions. - -### **Conceptual memory** - -Extracted intelligence from books, papers, notes, and source material. - -### **Procedural memory** - -Protocols and policy flows the system can execute. - -### **Episodic memory** - -Compressed traces of what actually happened. - -### **Calibration memory** - -What was believed, what happened, what corrected the model. - -### **Physiological calibration memory** - -How body-state signals correlate with actual usable capacity for this operator. - -## **What Aether stores** - -Aether should store: - -* observations -* rules -* patterns -* contradictions -* mechanisms -* decision protocols -* extracted lenses -* re-entry artifacts -* prior cases -* effective interventions -* false interpretations and later corrections - -The standard is brutal: -if it does not improve future diagnosis, judgment, or action, it should not survive. - ---- - -# **14\. Knowledge Refinery** - -This is the subsystem that turns reading into leverage. - -The Knowledge Refinery takes sources such as: - -* DDIA -* systems architecture books -* strategy books -* psychology books -* negotiation books -* your own case notes -* postmortems -* essays and papers - -and converts them into **reusable reasoning primitives**. - -It should not primarily emit summaries. - -It should emit: - -* concepts -* mechanisms -* causal logic -* tradeoffs -* invariants -* failure modes -* anti-patterns -* diagnostic questions -* decision rules -* heuristics -* boundary conditions -* cases -* contradictions -* applicability scope - -This is the difference between “AI that has read your books” and “a private council of extracted thinking systems.” - -For example, a technical book like DDIA should not live inside Utopia as text chunks first. -It should live as things like: - -* source-of-truth questions -* consistency tradeoffs -* idempotency checks -* recovery semantics -* durability concerns -* bottleneck patterns -* schema evolution constraints -* observability heuristics - -That is the right storage unit. - ---- - -# **15\. Lens Packs** - -A clean way to package extracted knowledge inside Utopia is as **lens packs**. - -A lens pack is a domain-specific reasoning bundle. - -Each lens pack should contain: - -* canonical concepts -* mechanisms -* tradeoffs -* failure modes -* heuristics -* diagnostic questions -* decision protocols -* boundary conditions -* examples - -Examples: - -## **DDIA lens** - -Teaches Utopia to ask: - -* what is the source of truth? -* what failure boundary exists? -* what needs durability? -* where can duplication happen? -* what recovery semantics exist? -* what consistency actually matters? - -## **Strategy lens** - -Teaches Utopia to ask: - -* where is leverage? -* what is being mistaken for motion? -* whose incentives shape the surface behavior? -* what is actually scarce? - -## **Psychology lens** - -Teaches Utopia to ask: - -* is state being misread as identity? -* what emotion is being turned into a belief? -* what is avoidance protecting? - -## **Negotiation lens** - -Teaches Utopia to ask: - -* what is each actor optimizing for? -* what is the BATNA? -* what is said versus unsayable? -* what would a hostile counterpart exploit? - -This makes books usable at runtime. - ---- - -# **16\. AI Fabric** - -The AI Fabric is Utopia’s orchestration layer. - -It is not “chat in the app.” -It is not a monolithic assistant. - -It is a **model-agnostic reasoning fabric** that decides what kind of cognition is needed, loads the right context, invokes the right modules, and returns structured outputs. - -## **Its jobs** - -* classify the moment -* choose the required reasoning path -* retrieve only the needed context -* call the right specialist modules -* constrain output shape -* escalate to deeper reasoning only when needed -* preserve deterministic rules wherever possible - -## **Why model-agnostic matters** - -Provider capabilities differ. - -OpenAI’s current Responses API is positioned as a primary interface for model responses and supports tool use and stateful interactions; OpenAI’s docs also expose streaming and schema-constrained structured outputs. Anthropic’s docs support tool use and prompt caching, including caching tools, system messages, and tool results. Gemini’s official docs support function calling and structured outputs, and Gemini 3 uses thought signatures that must be preserved in certain tool-calling flows. Bedrock’s Converse API provides a more consistent interface across models while still allowing provider-specific parameters when needed. ([OpenAI Platform](https://platform.openai.com/docs/api-reference/responses/list?ref=test-ippon.ghost.io&utm_source=chatgpt.com)) - -So the correct design is: - -**Utopia’s internal abstraction is the source of truth.** -**Provider SDKs are transport layers.** - ---- - -# **17\. Provider abstraction layer** - -A proper implementation should define capability-based internal contracts. - -Not “send this to OpenAI.” -Not “run this on Gemini.” - -Instead, define contracts such as: - -* `ReasoningTask` -* `StructuredDecisionTask` -* `ExtractionTask` -* `CompressionTask` -* `ToolCallTask` -* `RetrievalSelectionTask` -* `LowLatencyClassifierTask` - -And define capability profiles such as: - -* supports JSON schema output -* supports tool use -* supports streaming -* supports caching -* supports long context -* supports special reasoning-state passthrough -* latency tier -* cost tier -* reliability tier -* privacy/compliance tier - -Then create thin adapters for each provider. - -That architecture is durable. -It also lets Utopia swap backends without rewriting the cognition layer. - ---- - -# **18\. AI execution tiers** - -Utopia should not burn the biggest model on everything. - -It should have execution tiers. - -## **Tier 1: classifier / router** - -Cheap, fast, structured. - -Used for: - -* request typing -* blocker hypothesis -* simple confidence estimation -* trigger triage - -## **Tier 2: reasoning / policy** - -More capable model. - -Used for: - -* contradiction analysis -* complex re-entry synthesis -* policy selection -* problem structuring -* strategic ambiguity resolution - -## **Tier 3: extraction / compression** - -Cheap but structured. - -Used for: - -* source distillation -* trace compaction -* pattern candidates -* archive cleanup - -## **Tier 4: deterministic validators** - -No model where possible. - -Used for: - -* schema validation -* rule precedence -* confidence thresholds -* permission checks -* conflict resolution -* irreversible-action gating - -This is the correct cost-performance architecture. - ---- - -# **19\. The Router** - -The Router is the metacognitive dispatcher. - -It answers: - -**What kind of cognition is required right now?** - -Examples: - -* state problem -* re-entry problem -* blocker problem -* contradiction problem -* vector conflict -* reasoning problem -* review problem -* source extraction problem - -The Router should not freestyle. -It should classify and dispatch. - -A good router payload includes: - -* request type -* urgency level -* evidence sufficiency -* required depth -* safe output shape -* required modules -* context budget -* deterministic guardrails - -This keeps the rest of the system legible. - ---- - -# **20\. State Estimator** - -The State Estimator answers: - -**What operating condition is the operator most likely in right now?** - -A useful state set includes: - -* Recover -* Preserve -* Orient -* Clarify -* Re-enter -* Execute -* Deep Work -* Close Loop -* Review -* Drift - -The State Estimator should combine: - -* subjective evidence -* behavioral evidence -* contextual evidence -* WHOOP-derived priors -* recent traces -* thread characteristics - -Its output should include: - -* primary state -* confidence -* dominant blocker hypothesis -* contributing factors -* action-depth recommendation -* caution flags - -That output then constrains downstream policy. - ---- - -# **21\. Blocker Classifier** - -The Blocker Classifier answers: - -**Why is motion failing?** - -Core blocker classes should include: - -* ambiguity -* scope overload -* physiological depletion -* emotional threat -* context fracture -* vector conflict -* environmental friction -* stimulation hijack -* narrative distortion -* completion aversion -* problem misclassification -* decision fog - -This matters because every blocker implies a different intervention. - -Examples: - -* ambiguity → shrink problem / define next edge -* depletion → reduce depth / preserve continuity -* context fracture → generate re-entry artifact -* vector conflict → escalate to Vector Arbiter -* emotional threat → lower exposure / reduce symbolic load -* completion aversion → close-loop protocol - -Without blocker typing, the system will recommend the wrong thing while sounding intelligent. - ---- - -# **22\. Problem Structurer** - -This is one of the most important modules in the entire architecture. - -The Problem Structurer takes a messy real-world situation and turns it into a usable internal representation. - -It should extract: - -* objective -* stakes -* actors -* incentives -* constraints -* resources -* assumptions -* unknowns -* time horizon -* irreversibilities -* bottlenecks -* observable facts -* narrative layer -* possible distortions - -That lets Utopia distinguish between: - -* a technical problem -* a coordination problem -* an incentive problem -* a narrative problem -* a timing problem -* a capability problem -* a scope problem - -This is how the system avoids attacking the wrong battlefield. - ---- - -# **23\. Context Retriever** - -The Context Retriever exists because most advice engines over-retrieve. - -Utopia should retrieve only what is relevant. - -That means: - -* current thread summary -* last meaningful truth -* relevant Vector constraints -* matching rules from Aether -* similar prior cases -* lens packs relevant to the problem class -* recent physiological context if relevant - -The retriever should optimize for: - -* relevance -* recency -* confidence -* diversity of useful lenses -* low token waste - -This is where the memory system becomes operational instead of archival. - ---- - -# **24\. Interrogator** - -The Interrogator is one of the highest-value modules in Utopia. - -Its purpose is not to ask many questions. -Its purpose is to ask the **minimum set of high-leverage questions** that most reduce uncertainty. - -Question classes include: - -## **Problem typing** - -What kind of problem is this really? - -## **Objective clarity** - -What counts as success? - -## **Constraint discovery** - -What is truly fixed? - -## **Hidden variable detection** - -What unknown would most change the decision? - -## **Assumption exposure** - -What is being relied on without proof? - -## **Failure analysis** - -How does this most likely fail? - -## **Time and reversibility** - -What becomes harder if delayed? -What is irreversible? - -## **Distortion detection** - -Is state being mistaken for truth? -Is the symbolic meaning of the problem distorting the diagnosis? - -The Interrogator is where extracted books become live cognition. - ---- - -# **25\. Contradiction Checker** - -The Contradiction Checker compares: - -* self-story -* behavior -* physiology -* prior traces -* Vector -* actual outcomes - -Its function is to detect mismatches such as: - -* “I’m lazy” vs low recovery and broken sleep -* “I can push” vs fragile body-state and recent collapses -* “This is impossible” vs repeated ambiguity patterns -* “This matters” vs Vector showing it is drift - -This is one of the modules that makes Utopia dangerous in a good way. -It helps prevent false self-interpretation. - ---- - -# **26\. Vector Arbiter** - -The Vector Arbiter exists because urgency and importance are not the same. - -It checks whether the proposed move aligns with direction. - -Its output may be: - -* aligned -* misaligned -* conditionally aligned -* locally useful but globally drifting -* requires explicit override - -This prevents Utopia from becoming an elegant executor of accidental drift. - ---- - -# **27\. The Council** - -The Council is the multi-perspective reasoning layer. - -It should be narrow and explicit, not vague. - -A good configuration is: - -## **Cartographer** - -Maps the structure of the problem. - -## **Historian** - -Finds similar cases, prior patterns, and relevant extracted models. - -## **Skeptic** - -Attacks assumptions and overconfidence. - -## **Adversary** - -Models hostile incentives and exploitation paths. - -## **Strategist** - -Generates leverage points, sequencing, and option paths. - -## **Operator** - -Converts reasoning into executable action shape. - -## **Calibrator** - -Separates fact, inference, assumption, and speculation. - -The Council does not replace Schrödinger. -It prepares better policy for Schrödinger to collapse. - ---- - -# **28\. Schrödinger: the action-policy engine** - -**Schrödinger** is the subsystem that turns all of the above into one correct move. - -Its governing question is: - -**Given the current state, blocker, direction, uncertainty level, remembered truth, and physiological condition, what is the smallest correct move now?** - -The move may be: - -* recover -* preserve -* orient -* re-enter -* clarify -* ask one question -* execute -* close loop -* review - -This is important: - -Schrödinger should not assume that “action” is always the right output. - -Sometimes the right move is a question. -Sometimes it is continuity preservation. -Sometimes it is deliberate non-expansion. - -## **Inputs** - -* state estimate -* blocker classification -* Vector constraints -* Aether retrieval -* problem structure -* WHOOP-based capacity prior -* environment context -* recent action history - -## **Outputs** - -* mode -* action type -* action depth -* one next move -* rationale -* confidence -* caution flags - -Schrödinger is not a task manager. -It is a **policy engine**. - ---- - -# **29\. Re-entry Artifacts** - -A re-entry artifact is one of the most valuable objects in the whole system. - -Its purpose is to reduce the tax of interruption. - -A good re-entry artifact contains: - -* thread identity -* last completed step -* unresolved edge -* smallest next move -* relevant constraints -* trap to avoid on return - -Example shape: - -* Thread: parser normalization -* Last done: enum mismatch identified -* Next edge: inspect five failing rows -* Trap: do not reopen the whole architecture -* Context: mapping issue tied to downstream schema validator - -This object is how Utopia preserves continuity across state changes. - ---- - -# **30\. Trace Layer** - -The Trace Layer records what happened after the move. - -Each trace should be compressed and typed. - -A trace may include: - -* timestamp -* state -* blocker -* move chosen -* outcome -* revealed truth -* contradiction observed -* next edge -* confidence delta - -The trace is not a diary. -It is training data for Aether and the Personal Execution Model. - ---- - -# **31\. Review and Compression** - -This is Utopia’s immune system. - -Without compression, the system decays into information obesity. - -Responsibilities include: - -* promote repeated truths into rules -* merge duplicate patterns -* archive dead threads -* weaken rules that stop holding -* upgrade reliable question trees -* recalibrate intervention priors -* refine physiological interpretations -* remove low-value noise - -## **Review horizons** - -* micro-review after action -* daily review -* weekly compression -* periodic deep maintenance - -Compression is what turns memory into intelligence. - ---- - -# **32\. Personal Execution Model** - -Over time, Utopia should build a model of how this operator actually functions. - -Not generic humans. -This operator. - -It should learn: - -* which blockers recur most -* which interventions work best under which states -* when low recovery actually matters -* when subjective exhaustion is false -* which environments support depth -* where continuity decays fastest -* when completion aversion appears -* which problem classes are often misclassified -* how action-depth should be scaled by state and physiology - -This is where Utopia becomes asymmetrical. - -At first it is a framework. -Later it becomes a **personal operating model of cognition and execution**. - ---- - -# **33\. Storage architecture** - -A serious implementation should separate storage by semantic role. - -## **1\. Relational core** - -Use a relational database for canonical objects: - -* missions -* threads -* states -* traces -* rules -* blockers -* re-entry artifacts -* provider configs -* permissions -* schedules -* review logs - -This is the operational source of truth. - -## **2\. Vector index** - -Used for semantic retrieval across: - -* extracted concepts -* cases -* prior traces -* rules -* problem briefs -* source fragments when needed - -This supports relevance-based recall. - -## **3\. Graph layer** - -Optional early, important later. - -Used for explicit relations like: - -* concept → mechanism -* mechanism → failure mode -* lens → question -* case → pattern -* rule → applies\_to -* rule → contradicted\_by - -This is where cross-domain reasoning gets stronger. - -## **4\. Time-series / event store** - -Useful for physiological signals, action timing, behavior patterns, and triggers. - -WHOOP and action telemetry fit naturally here. - -## **5\. Object store** - -For source documents, raw imports, snapshots, and large artifacts. - -That is the right decomposition. - ---- - -# **34\. Runtime architecture** - -A clean runtime design would split into the following services. - -## **API Gateway** - -Unified entry point for clients. - -## **Identity / Permissions Service** - -Controls local auth, provider tokens, access classes, revocation. - -## **Vector Service** - -Owns missions, direction, anti-goals, and drift logic. - -## **Evidence Service** - -Ingests subjective, behavioral, contextual, and physiological data. - -## **WHOOP Integration Service** - -Handles OAuth, webhooks, sync, normalization, and physiological feature derivation. WHOOP’s API surface and webhook model support this event-driven design. ([WHOOP for Developers](https://developer.whoop.com/api/?utm_source=chatgpt.com)) - -## **Knowledge Refinery Service** - -Runs ingestion, extraction, schema validation, and lens-pack generation. - -## **Aether Service** - -Owns memory objects, rules, patterns, traces, and retrieval surfaces. - -## **Orchestrator / AI Fabric Service** - -Runs routing, state estimation, problem structuring, question generation, and policy synthesis. - -## **Schrödinger Policy Service** - -Produces one move and action constraints. - -## **Review / Compression Service** - -Runs maintenance jobs, rule promotion, archive cleanup, and calibration. - -## **Trigger Engine** - -Listens to events and decides whether intervention is warranted. - -This service split makes the system implementable. - ---- - -# **35\. Event-driven design** - -Utopia should be heavily event-driven. - -Important event types include: - -* thread touched -* action started -* action completed -* repeated failed start -* re-entry gap exceeded -* new WHOOP recovery available -* sleep event ingested -* source extracted -* rule upgraded -* blocker distribution changed -* mission neglected -* contradiction detected - -This allows the system to react to real changes instead of relying only on polling and manual check-ins. - -WHOOP’s webhook support makes it especially suitable for this model. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/webhooks/?utm_source=chatgpt.com)) - ---- - -# **36\. Conflict resolution and precedence** - -The system will routinely receive conflicting signals. - -A clean internal precedence order is: - -1. external hard constraints -2. safety and preservation needs -3. Vector priorities -4. present-state feasibility -5. evidence-backed rules from Aether -6. observed behavior -7. physiological prior -8. subjective narrative -9. momentary impulse - -You can change the exact ranking, but you need one. - -Otherwise the system becomes whichever signal is loudest. - -WHOOP should be strong evidence, but not sovereign authority. - ---- - -# **37\. Security and privacy architecture** - -Privacy is not optional here. -It is functional infrastructure. - -Utopia should be designed as: - -* local-first where possible -* encrypted at rest -* explicit about every integration -* tokenized and revocable for external providers -* transparent about data classes -* configurable about retention -* conservative about third-party exposure - -The architecture should treat data domains separately: - -* strategic data -* physiological data -* source documents -* action traces -* provider metadata - -This lets the user selectively revoke or isolate domains without destroying the whole system. - -For model providers, the fabric should support per-provider privacy policies and transport settings rather than assuming one global trust model. Bedrock’s documentation, for example, describes Converse as a unified interface and notes content is used to generate responses rather than stored by Bedrock, while other providers have their own operational assumptions that should be isolated behind adapters. ([AWS Documentation](https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Converse.html?utm_source=chatgpt.com)) - ---- - -# **38\. The four main software surfaces** - -As a product, Utopia should likely present itself through four main surfaces. - -## **1\. Control Surface** - -The live cockpit. - -It shows: - -* current state -* blocker hypothesis -* capacity envelope -* one mode -* one next move -* re-entry doorway -* caution flags - -WHOOP belongs here as context, not decoration. - -## **2\. Vector Surface** - -The directional layer. - -It shows: - -* life arc -* seasonal focus -* mission stack -* thread hierarchy -* drift risk -* neglected high-value lines - -## **3\. Problem Room** - -The reasoning layer. - -You bring in a live problem. -Utopia returns: - -* problem classification -* relevant lens packs -* key unknowns -* high-leverage questions -* blind spots -* risks -* option paths -* recommended action policy - -## **4\. Refinery / Review Surface** - -The compounding layer. - -It handles: - -* source ingestion -* extraction review -* pattern merges -* rule upgrades -* archive cleanup -* calibration review -* physiological interpretation adjustments - -That gives the product a clear shape. - ---- - -# **39\. Evaluation and metrics** - -A serious system needs evals. - -Not vanity metrics. -Real ones. - -## **Continuity metrics** - -* median re-entry time -* thread decay rate -* % of returns with valid next edge -* abandonment rate of active threads - -## **Policy metrics** - -* % of selected moves executed -* % of moves later judged correct -* action-depth mismatch rate -* false-push rate -* false-preserve rate - -## **Judgment metrics** - -* problem classification accuracy -* question usefulness score -* reduction in uncertainty after interrogation -* option quality score -* contradiction detection precision - -## **Knowledge metrics** - -* extraction reuse rate -* rule promotion rate -* stale knowledge ratio -* lens-pack retrieval usefulness - -## **Physiology metrics** - -* correlation between WHOOP-informed capacity estimate and realized usable depth -* low-recovery false-negative rate -* low-recovery false-excuse rate -* recovery trend usefulness for policy selection - -Without evals, Utopia becomes elegant theater. - ---- - -# **40\. The technical essence of WHOOP inside Utopia** - -To make sure WHOOP is fully embedded and not merely mentioned, here is the concise engineering truth: - -WHOOP supplies: - -* daily readiness-style recovery signal -* sleep structure and duration signals -* cycle / strain context -* workout / exertion events -* webhook-triggerable updates -* stable personal longitudinal physiology data - -WHOOP does not directly tell Utopia what to do. - -It tells Utopia how much confidence it should have in different kinds of action, how fragile the operator may be, and whether “I can’t do this” is more likely body-state, ambiguity, or self-story. The developer docs expose the relevant data scopes and webhook mechanisms, and WHOOP’s concept docs clarify that Recovery is a daily preparedness measure and that HRV is measured during deepest sleep, which makes it appropriate as a stable policy prior rather than a continuously fluctuating command signal. ([WHOOP for Developers](https://developer.whoop.com/api/?utm_source=chatgpt.com)) - -That is exactly where WHOOP belongs. - ---- - -# **41\. The final architecture in one line** - -**Utopia is a private, model-agnostic, physiology-aware cognitive control system and judgment refinery that preserves direction through Vector, senses reality through an evidence plane that includes WHOOP, refines truth and knowledge through Aether, routes cognition through a modular AI Fabric, and uses Schrödinger to select the smallest correct move for the operator’s real present condition.** - -And if compressed even further: - -**Utopia is software for preserving truth, continuity, and judgment under variable internal conditions** - From 0c4853c7ef3017513a022cf121f9907f7ba1f952 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:34 +0100 Subject: [PATCH 03/63] docs: archive Utopia Formal Architecture DB etc.md --- docs/history/original-database-design.md | 1531 ++++++++++++++++++++++ 1 file changed, 1531 insertions(+) create mode 100644 docs/history/original-database-design.md diff --git a/docs/history/original-database-design.md b/docs/history/original-database-design.md new file mode 100644 index 0000000..575155e --- /dev/null +++ b/docs/history/original-database-design.md @@ -0,0 +1,1531 @@ +For a **serious private Utopia system**, I would not design the database as “one big notes table plus embeddings.” +That would kill the product. + +Utopia needs a **typed memory architecture** with clear separation between: + +* directional truth +* live execution state +* evidence about the operator +* physiological priors +* extracted knowledge +* reasoning artifacts +* action traces +* calibration and policy learning +* integration and audit infrastructure + +The right mental model is: + +**Postgres as canonical truth** +**vector index for semantic recall** +**graph edges for explicit conceptual relationships** +**event log for replay/audit/triggering** +**object store for raw artifacts** +**time-series for dense physiological / behavioral telemetry** + +That is the serious shape. + +--- + +# **1\. Core storage strategy** + +I would split the system into five storage roles. + +## **A. Relational core** + +This is the source of truth for all canonical entities. + +Use it for: + +* missions +* threads +* states +* blockers +* re-entry artifacts +* rules +* traces +* decision briefs +* sources +* extractions +* concepts +* mechanisms +* lens packs +* permissions +* OAuth connections +* review records + +This should be PostgreSQL. + +## **B. Vector retrieval layer** + +This exists for semantic retrieval across: + +* extracted concepts +* cases +* prior traces +* source chunks +* decision briefs +* rules +* diagnostic questions + +Keep this close to the relational core unless scale forces separation. +For a private serious system, I would start with **pgvector inside Postgres**, not an external vector database. + +## **C. Graph layer** + +Use this for explicit relationships, not as the primary store. + +Graph is useful for: + +* concept → mechanism +* mechanism → failure mode +* lens → question +* rule → applies\_to +* case → pattern +* pattern → intervention +* contradiction → corrected\_by + +You can model this in relational first with an `edges` table. +Only move to a dedicated graph database if traversal actually becomes central. + +## **D. Event / telemetry layer** + +This captures: + +* user actions +* thread touches +* failed starts +* trigger firings +* AI orchestration events +* webhook receipts +* provider calls +* policy decisions + +This can still live in Postgres initially as append-only event tables. + +## **E. Object storage** + +Use for: + +* raw source files +* PDFs +* original imported documents +* extraction snapshots +* large JSON payload archives +* encrypted export bundles + +That should not be the operational DB. + +--- + +# **2\. Database design principles** + +The entire schema should obey a few principles. + +## **Typed over generic** + +Do not store everything in `notes`. +Use separate typed tables for distinct cognitive objects. + +## **Append where truth unfolds** + +Traces, estimates, webhook receipts, and orchestration runs should often be append-only. + +## **Current-state views over mutable sludge** + +Keep immutable history, then materialize current state via views or snapshot tables. + +## **JSONB for edges, not for the core** + +Use JSONB for flexible payloads, model outputs, or provider-specific metadata. +Do not hide core product semantics inside JSON blobs. + +## **Single-user first, multi-tenant safe** + +Even if Utopia is private and single-user, design tables with `operator_id` or `workspace_id` so the system is structurally clean. + +## **Explicit confidence everywhere** + +Any inferred object should store: + +* confidence +* evidence\_count +* provenance +* last\_validated\_at + +Because Utopia is a reasoning system, not just a CRUD app. + +--- + +# **3\. Logical schema namespaces** + +I would separate the database into logical schemas like this: + +* `core` +* `vector` +* `evidence` +* `physiology` +* `vector_ctrl` +* `aether` +* `reasoning` +* `execution` +* `integration` +* `system` + +That gives you clean domain boundaries. + +--- + +# **4\. Global conventions** + +Every major object should have: + +* `id` as UUIDv7 +* `operator_id` +* `created_at` +* `updated_at` +* `archived_at` nullable +* `version` +* `source_kind` +* `provenance` +* `confidence` where applicable + +Suggested enums: + +* `status` +* `state_kind` +* `blocker_kind` +* `source_kind` +* `memory_class` +* `evidence_kind` +* `intervention_kind` +* `decision_kind` + +Use UUIDv7 or time-ordered UUIDs because Utopia is event-heavy and benefits from sortable IDs. + +--- + +# **5\. Core identity and system tables** + +Even for a private system, you want these. + +## **`core.operators`** + +Represents the human operator the system is modeling. + +create table core.operators ( + id uuid primary key, + display\_name text not null, + timezone text not null, + locale text, + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`core.devices`** + +Tracks trusted devices for local-first syncing. + +create table core.devices ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + device\_name text not null, + device\_type text not null, + public\_key\_fingerprint text, + last\_seen\_at timestamptz, + created\_at timestamptz not null default now() +); + +## **`integration.oauth_connections`** + +Stores external integration connections. + +create table integration.oauth\_connections ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + provider text not null, \-- whoop, openai, anthropic, etc. + provider\_user\_id text, + scopes text\[\] not null default '{}', + status text not null, \-- active, revoked, expired + encrypted\_access\_token bytea, + encrypted\_refresh\_token bytea, + token\_expires\_at timestamptz, + metadata jsonb not null default '{}', + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`integration.permissions`** + +Fine-grained permissions ledger. + +create table integration.permissions ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + provider text not null, + permission\_key text not null, + granted boolean not null, + granted\_at timestamptz, + revoked\_at timestamptz, + details jsonb not null default '{}' +); + +--- + +# **6\. Vector schema: direction and control plane** + +This is where Utopia stores what matters. + +## **`vector_ctrl.life_arcs`** + +Top-level directional framing. + +create table vector\_ctrl.life\_arcs ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + title text not null, + description text, + status text not null, \-- active, dormant, archived + horizon\_start date, + horizon\_end date, + success\_definition text, + anti\_goals text\[\] not null default '{}', + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`vector_ctrl.seasons`** + +A season is a bounded phase of focus. + +create table vector\_ctrl.seasons ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + life\_arc\_id uuid references vector\_ctrl.life\_arcs(id), + title text not null, + thesis text, + start\_date date not null, + end\_date date, + priority\_stack jsonb not null default '\[\]', + status text not null, \-- planned, active, closed + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`vector_ctrl.missions`** + +Strategically meaningful objectives. + +create table vector\_ctrl.missions ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + season\_id uuid references vector\_ctrl.seasons(id), + title text not null, + description text, + mission\_kind text not null, \-- strategic, technical, personal, recovery, etc. + priority\_score numeric(6,3), + status text not null, \-- active, paused, completed, abandoned + success\_definition text, + failure\_definition text, + drift\_definition text, + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`vector_ctrl.threads`** + +Live lines of work within missions. + +create table vector\_ctrl.threads ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + mission\_id uuid not null references vector\_ctrl.missions(id), + parent\_thread\_id uuid references vector\_ctrl.threads(id), + title text not null, + description text, + thread\_kind text not null, \-- build, research, decision, admin, etc. + status text not null, \-- active, blocked, paused, closed + complexity\_score numeric(6,3), + ambiguity\_score numeric(6,3), + reentry\_risk\_score numeric(6,3), + last\_meaningful\_touch\_at timestamptz, + next\_edge\_summary text, + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`vector_ctrl.thread_constraints`** + +Explicit constraints on a thread. + +create table vector\_ctrl.thread\_constraints ( + id uuid primary key, + thread\_id uuid not null references vector\_ctrl.threads(id), + constraint\_type text not null, \-- time, dependency, budget, skill, legal + description text not null, + hardness text not null, \-- hard, soft, assumed + created\_at timestamptz not null default now() +); + +## **`vector_ctrl.anti_goals`** + +What must not happen. + +create table vector\_ctrl.anti\_goals ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + scope\_type text not null, \-- life\_arc, season, mission, thread + scope\_id uuid not null, + description text not null, + created\_at timestamptz not null default now() +); + +--- + +# **7\. Evidence schema: subjective, behavioral, contextual** + +This is the live sensing layer. + +## **`evidence.subjective_checkins`** + +create table evidence.subjective\_checkins ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid references vector\_ctrl.threads(id), + energy smallint check (energy between 0 and 100), + clarity smallint check (clarity between 0 and 100), + resistance smallint check (resistance between 0 and 100), + overwhelm smallint check (overwhelm between 0 and 100), + emotional\_load smallint check (emotional\_load between 0 and 100), + perceived\_urgency smallint check (perceived\_urgency between 0 and 100), + free\_text text, + recorded\_at timestamptz not null, + created\_at timestamptz not null default now() +); + +## **`evidence.behavior_events`** + +create table evidence.behavior\_events ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid references vector\_ctrl.threads(id), + event\_type text not null, \-- thread\_opened, action\_started, action\_abandoned, etc. + event\_at timestamptz not null, + duration\_ms bigint, + metadata jsonb not null default '{}' +); + +## **`evidence.context_snapshots`** + +create table evidence.context\_snapshots ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid references vector\_ctrl.threads(id), + local\_time timestamptz not null, + environment\_label text, + interruption\_count int, + obligation\_load smallint, + available\_minutes int, + active\_window text, \-- morning, afternoon, evening + metadata jsonb not null default '{}' +); + +## **`evidence.derived_features`** + +Computed features from raw evidence. + +create table evidence.derived\_features ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid references vector\_ctrl.threads(id), + feature\_name text not null, + feature\_value numeric, + feature\_json jsonb, + feature\_window text, \-- last\_2h, last\_24h, last\_7d + confidence numeric(5,4), + observed\_at timestamptz not null, + created\_at timestamptz not null default now() +); + +This is where you store things like: + +* failed\_start\_rate\_24h +* thread\_decay\_hours +* drift\_probability +* completion\_aversion\_score +* ambiguity\_candidate\_score + +--- + +# **8\. Physiology schema: WHOOP as a first-class subsystem** + +This domain should be explicit because WHOOP is not just metadata. +WHOOP’s API provides OAuth-scoped data for recovery, cycles, sleep, workouts, profile, and body measurements, and its webhook model sends events such as `recovery.updated`, `sleep.updated`, and `workout.updated` with HMAC-verifiable signatures. Recovery is described by WHOOP as a daily 0–100 score derived from signals such as HRV and resting heart rate. ([WHOOP for Developers](https://developer.whoop.com/api/)) + +## **`physiology.whoop_connections`** + +create table physiology.whoop\_connections ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + oauth\_connection\_id uuid not null references integration.oauth\_connections(id), + whoop\_user\_id bigint, + webhook\_secret\_fingerprint text, + status text not null, \-- active, revoked, errored + last\_sync\_at timestamptz, + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`physiology.whoop_body_measurements`** + +create table physiology.whoop\_body\_measurements ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + measured\_at timestamptz not null, + height\_meter numeric(6,4), + weight\_kg numeric(6,3), + max\_heart\_rate int, + source\_payload jsonb not null default '{}', + created\_at timestamptz not null default now() +); + +## **`physiology.whoop_cycles`** + +create table physiology.whoop\_cycles ( + id uuid primary key, \-- internal UUID + operator\_id uuid not null references core.operators(id), + provider\_cycle\_id bigint not null unique, + whoop\_user\_id bigint, + cycle\_start timestamptz not null, + cycle\_end timestamptz, + timezone\_offset text, + score\_state text, \-- SCORED, PENDING\_SCORE, UNSCORABLE + strain numeric(8,4), + kilojoule numeric(12,3), + average\_heart\_rate int, + max\_heart\_rate int, + source\_payload jsonb not null default '{}', + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`physiology.whoop_sleeps`** + +create table physiology.whoop\_sleeps ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + provider\_sleep\_id uuid not null unique, + provider\_cycle\_id bigint, + sleep\_start timestamptz, + sleep\_end timestamptz, + sleep\_performance\_pct numeric(5,2), + total\_in\_bed\_ms bigint, + total\_asleep\_ms bigint, + slow\_wave\_ms bigint, + rem\_ms bigint, + light\_ms bigint, + awake\_ms bigint, + source\_payload jsonb not null default '{}', + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`physiology.whoop_recoveries`** + +Recovery objects should mirror WHOOP’s actual structure closely enough to avoid lossy mapping. WHOOP’s recovery docs describe fields including `cycle_id`, `sleep_id`, `user_id`, `created_at`, `updated_at`, `score_state`, and score values like `recovery_score`, `resting_heart_rate`, and `hrv_rmssd_milli`. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/user-data/recovery/)) + +create table physiology.whoop\_recoveries ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + provider\_cycle\_id bigint not null, + provider\_sleep\_id uuid not null, + whoop\_user\_id bigint, + recorded\_at timestamptz not null, + updated\_at\_source timestamptz, + score\_state text not null, + recovery\_score smallint, + resting\_heart\_rate int, + hrv\_rmssd\_milli numeric(12,6), + spo2\_percentage numeric(6,3), + skin\_temp\_celsius numeric(6,3), + user\_calibrating boolean, + source\_payload jsonb not null default '{}', + created\_at timestamptz not null default now(), + unique (operator\_id, provider\_cycle\_id, provider\_sleep\_id) +); + +## **`physiology.whoop_workouts`** + +create table physiology.whoop\_workouts ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + provider\_workout\_id uuid not null unique, + workout\_type text, + workout\_start timestamptz, + workout\_end timestamptz, + strain numeric(8,4), + average\_heart\_rate int, + max\_heart\_rate int, + source\_payload jsonb not null default '{}', + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`physiology.physiology_features`** + +This is where raw WHOOP data becomes operational priors. + +create table physiology.physiology\_features ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + feature\_date date not null, + feature\_name text not null, \-- capacity\_envelope, fragility\_risk, sleep\_debt\_slope, etc. + feature\_value numeric, + feature\_json jsonb, + confidence numeric(5,4), + computed\_at timestamptz not null default now(), + unique (operator\_id, feature\_date, feature\_name) +); + +Useful features: + +* recovery\_trend\_3d +* recovery\_volatility\_7d +* strain\_carryover +* sleep\_regularity\_score +* fragility\_risk +* depth\_ceiling\_score +* depletion\_prior +* recovery\_mismatch\_score + +## **`physiology.biomarker_panels`** + +For Advanced Labs or any later biomarker layer, keep separate from wearable data. + +create table physiology.biomarker\_panels ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + panel\_date date not null, + provider text not null, + panel\_type text not null, \-- advanced\_labs, manual\_lab\_import + summary jsonb not null, + source\_payload jsonb not null default '{}', + created\_at timestamptz not null default now() +); + +## **`integration.webhook_receipts`** + +WHOOP webhook bodies include fields like `user_id`, `id`, `type`, and `trace_id`, and WHOOP documents signature validation using `X-WHOOP-Signature` and `X-WHOOP-Signature-Timestamp`. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/webhooks/)) + +create table integration.webhook\_receipts ( + id uuid primary key, + provider text not null, + event\_type text not null, + provider\_user\_id text, + provider\_object\_id text, + trace\_id text, + received\_at timestamptz not null default now(), + signature\_valid boolean, + headers jsonb not null, + raw\_body jsonb not null, + processed\_at timestamptz, + processing\_status text not null default 'pending' +); + +--- + +# **9\. Aether schema: memory and extracted intelligence** + +This is the most important part of the system. + +Do not make Aether one table. +It should be a family of typed memory objects. + +## **`aether.sources`** + +create table aether.sources ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + source\_kind text not null, \-- book, paper, essay, transcript, note\_bundle, case\_file + title text not null, + author text, + published\_at date, + ingest\_status text not null, + canonical\_uri text, + storage\_uri text, + checksum text, + metadata jsonb not null default '{}', + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now() +); + +## **`aether.source_chunks`** + +Only if you need chunk-level retrieval. + +create table aether.source\_chunks ( + id uuid primary key, + source\_id uuid not null references aether.sources(id), + chunk\_index int not null, + raw\_text text not null, + token\_count int, + semantic\_label text, + metadata jsonb not null default '{}' +); + +## **`aether.extractions`** + +A structured distillation pass for a source. + +create table aether.extractions ( + id uuid primary key, + source\_id uuid not null references aether.sources(id), + extraction\_version text not null, + extraction\_status text not null, + thesis text, + summary text, + confidence numeric(5,4), + extracted\_at timestamptz not null default now(), + model\_run\_id uuid, + unique (source\_id, extraction\_version) +); + +## **`aether.concepts`** + +create table aether.concepts ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + canonical\_name text not null, + definition text, + domain text, + source\_count int not null default 0, + confidence numeric(5,4), + created\_at timestamptz not null default now(), + updated\_at timestamptz not null default now(), + unique (operator\_id, canonical\_name) +); + +## **`aether.mechanisms`** + +create table aether.mechanisms ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + name text not null, + description text not null, + causal\_logic text, + domain text, + confidence numeric(5,4), + created\_at timestamptz not null default now() +); + +## **`aether.tradeoffs`** + +create table aether.tradeoffs ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + name text not null, + pole\_a text not null, + pole\_b text not null, + description text, + domain text, + created\_at timestamptz not null default now() +); + +## **`aether.failure_modes`** + +create table aether.failure\_modes ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + name text not null, + description text not null, + early\_signals text\[\], + domain text, + severity text, + created\_at timestamptz not null default now() +); + +## **`aether.heuristics`** + +create table aether.heuristics ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + statement text not null, + domain text, + applicability text, + failure\_conditions text, + confidence numeric(5,4), + created\_at timestamptz not null default now() +); + +## **`aether.diagnostic_questions`** + +create table aether.diagnostic\_questions ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + question\_text text not null, + question\_class text not null, \-- objective, constraint, risk, hidden\_variable, etc. + domain text, + usefulness\_score numeric(6,3), + created\_at timestamptz not null default now() +); + +## **`aether.protocols`** + +Reusable reasoning flows. + +create table aether.protocols ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + protocol\_name text not null, + domain text not null, + purpose text, + steps jsonb not null, \-- ordered protocol structure + applicability text, + created\_at timestamptz not null default now() +); + +## **`aether.lens_packs`** + +create table aether.lens\_packs ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + name text not null, + domain text not null, + description text, + version text not null, + source\_basis jsonb not null default '\[\]', + created\_at timestamptz not null default now() +); + +## **`aether.lens_pack_items`** + +create table aether.lens\_pack\_items ( + id uuid primary key, + lens\_pack\_id uuid not null references aether.lens\_packs(id), + item\_kind text not null, \-- concept, mechanism, question, heuristic, protocol, failure\_mode + item\_id uuid not null, + weight numeric(6,3), + metadata jsonb not null default '{}' +); + +## **`aether.cases`** + +Internal or external cases. + +create table aether.cases ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + title text not null, + case\_kind text not null, \-- internal, external + summary text, + outcome text, + lessons jsonb not null default '\[\]', + created\_at timestamptz not null default now() +); + +## **`aether.rules`** + +Durable truths that survived compression. + +create table aether.rules ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + rule\_text text not null, + rule\_kind text not null, \-- execution, reasoning, physiology, strategic, personal + evidence\_count int not null default 0, + confidence numeric(5,4), + first\_observed\_at timestamptz, + last\_validated\_at timestamptz, + active boolean not null default true, + created\_at timestamptz not null default now() +); + +## **`aether.patterns`** + +create table aether.patterns ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + pattern\_name text not null, + description text, + pattern\_kind text not null, \-- blocker, timing, physiology, narrative, execution + recurrence\_count int not null default 0, + confidence numeric(5,4), + created\_at timestamptz not null default now() +); + +## **`aether.edges`** + +Relational graph layer. + +create table aether.edges ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + src\_kind text not null, + src\_id uuid not null, + edge\_type text not null, \-- supports, contradicts, applies\_to, predicts, resembles, etc. + dst\_kind text not null, + dst\_id uuid not null, + weight numeric(6,3), + provenance jsonb not null default '{}', + created\_at timestamptz not null default now() +); + +This table becomes surprisingly powerful if kept clean. + +--- + +# **10\. Reasoning schema: problem room and decision artifacts** + +This is where Utopia turns live problems into structured thinking. + +## **`reasoning.problems`** + +create table reasoning.problems ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + title text not null, + raw\_prompt text not null, + problem\_kind text, \-- strategic, technical, social, legal, execution, etc. + urgency\_score numeric(6,3), + stakes\_score numeric(6,3), + uncertainty\_score numeric(6,3), + state\_at\_creation text, + thread\_id uuid references vector\_ctrl.threads(id), + created\_at timestamptz not null default now() +); + +## **`reasoning.problem_structures`** + +create table reasoning.problem\_structures ( + id uuid primary key, + problem\_id uuid not null references reasoning.problems(id), + objective text, + stakes text, + actors jsonb not null default '\[\]', + incentives jsonb not null default '\[\]', + constraints jsonb not null default '\[\]', + assumptions jsonb not null default '\[\]', + unknowns jsonb not null default '\[\]', + irreversibilities jsonb not null default '\[\]', + bottlenecks jsonb not null default '\[\]', + observable\_facts jsonb not null default '\[\]', + narrative\_layer jsonb not null default '\[\]', + distortion\_candidates jsonb not null default '\[\]', + confidence numeric(5,4), + generated\_at timestamptz not null default now() +); + +## **`reasoning.interrogations`** + +create table reasoning.interrogations ( + id uuid primary key, + problem\_id uuid not null references reasoning.problems(id), + interrogation\_kind text not null, \-- initial, followup, contradiction, pressure\_test + questions jsonb not null, \-- ordered high-leverage questions + rationale jsonb not null default '{}', + generated\_at timestamptz not null default now() +); + +## **`reasoning.decision_briefs`** + +create table reasoning.decision\_briefs ( + id uuid primary key, + problem\_id uuid not null references reasoning.problems(id), + classification text, + summary text, + key\_unknowns jsonb not null default '\[\]', + blind\_spots jsonb not null default '\[\]', + risks jsonb not null default '\[\]', + relevant\_lens\_pack\_ids uuid\[\] not null default '{}', + recommendation\_summary text, + confidence numeric(5,4), + generated\_at timestamptz not null default now() +); + +## **`reasoning.option_paths`** + +create table reasoning.option\_paths ( + id uuid primary key, + decision\_brief\_id uuid not null references reasoning.decision\_briefs(id), + option\_label text not null, \-- conservative, aggressive, asymmetric + description text not null, + expected\_upside text, + expected\_downside text, + reversibility text, + risk\_score numeric(6,3), + recommendation\_rank int, + created\_at timestamptz not null default now() +); + +## **`reasoning.contradiction_reports`** + +create table reasoning.contradiction\_reports ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + problem\_id uuid references reasoning.problems(id), + contradiction\_kind text not null, \-- narrative\_vs\_behavior, physiology\_vs\_claim, vector\_vs\_action + description text not null, + evidence jsonb not null default '\[\]', + severity numeric(6,3), + created\_at timestamptz not null default now() +); + +--- + +# **11\. Execution schema: Schrödinger, re-entry, traces** + +This is the action layer. + +## **`execution.state_estimates`** + +create table execution.state\_estimates ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid references vector\_ctrl.threads(id), + state\_kind text not null, \-- recover, preserve, orient, clarify, execute, etc. + confidence numeric(5,4) not null, + contributing\_factors jsonb not null default '\[\]', + generated\_at timestamptz not null default now() +); + +## **`execution.blocker_estimates`** + +create table execution.blocker\_estimates ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid references vector\_ctrl.threads(id), + blocker\_kind text not null, + confidence numeric(5,4) not null, + supporting\_evidence jsonb not null default '\[\]', + generated\_at timestamptz not null default now() +); + +## **`execution.reentry_artifacts`** + +create table execution.reentry\_artifacts ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid not null references vector\_ctrl.threads(id), + last\_completed\_step text, + unresolved\_edge text, + next\_smallest\_move text, + trap\_to\_avoid text, + relevant\_context jsonb not null default '{}', + freshness\_score numeric(6,3), + created\_at timestamptz not null default now(), + superseded\_by uuid references execution.reentry\_artifacts(id) +); + +## **`execution.policy_decisions`** + +This is the actual Schrödinger output record. + +create table execution.policy\_decisions ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid references vector\_ctrl.threads(id), + problem\_id uuid references reasoning.problems(id), + state\_estimate\_id uuid references execution.state\_estimates(id), + blocker\_estimate\_id uuid references execution.blocker\_estimates(id), + mode text not null, \-- recover, preserve, re-enter, clarify, ask, execute, close\_loop, review + intervention\_kind text not null, + action\_depth text not null, \-- tiny, narrow, moderate, deep + next\_move text not null, + rationale text, + confidence numeric(5,4), + caution\_flags jsonb not null default '\[\]', + created\_at timestamptz not null default now() +); + +## **`execution.traces`** + +create table execution.traces ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid references vector\_ctrl.threads(id), + policy\_decision\_id uuid references execution.policy\_decisions(id), + trace\_kind text not null, \-- action, question, preserve, recovery, closure + action\_taken text, + outcome text, + truth\_revealed text, + next\_edge text, + completion\_score numeric(6,3), + subjective\_after jsonb not null default '{}', + created\_at timestamptz not null default now() +); + +## **`execution.closures`** + +create table execution.closures ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + thread\_id uuid not null references vector\_ctrl.threads(id), + closure\_type text not null, \-- complete, archive, pause, merge + summary text, + lessons jsonb not null default '\[\]', + created\_at timestamptz not null default now() +); + +--- + +# **12\. Review and calibration schema** + +This is the immune system. + +## **`execution.review_sessions`** + +create table execution.review\_sessions ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + review\_scope text not null, \-- micro, daily, weekly, monthly + started\_at timestamptz not null, + completed\_at timestamptz, + summary text, + metadata jsonb not null default '{}' +); + +## **`execution.rule_promotions`** + +create table execution.rule\_promotions ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + rule\_id uuid references aether.rules(id), + source\_trace\_ids uuid\[\] not null default '{}', + promotion\_reason text, + created\_at timestamptz not null default now() +); + +## **`execution.pattern_updates`** + +create table execution.pattern\_updates ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + pattern\_id uuid references aether.patterns(id), + supporting\_trace\_ids uuid\[\] not null default '{}', + recurrence\_delta int not null default 1, + created\_at timestamptz not null default now() +); + +## **`execution.calibration_records`** + +create table execution.calibration\_records ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + policy\_decision\_id uuid references execution.policy\_decisions(id), + predicted\_outcome text, + actual\_outcome text, + calibration\_error numeric(6,3), + lesson text, + created\_at timestamptz not null default now() +); + +## **`execution.physiology_calibrations`** + +Critical for personal WHOOP learning. + +create table execution.physiology\_calibrations ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + feature\_date date not null, + physiology\_summary jsonb not null, + realized\_capacity\_score numeric(6,3), + work\_type text, \-- deep, narrow, admin, recovery, exploratory + mismatch\_score numeric(6,3), + lesson text, + created\_at timestamptz not null default now() +); + +This is how Utopia learns that low recovery may still permit narrow deterministic work, while open-ended work collapses. + +--- + +# **13\. AI orchestration schema** + +If this is a serious private system, you need auditability around model behavior. + +## **`system.model_providers`** + +create table system.model\_providers ( + id uuid primary key, + provider text not null unique, + capability\_profile jsonb not null, + active boolean not null default true, + created\_at timestamptz not null default now() +); + +## **`system.model_runs`** + +create table system.model\_runs ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + provider text not null, + model\_name text not null, + task\_kind text not null, \-- extraction, routing, reasoning, compression + input\_object\_kind text, + input\_object\_id uuid, + output\_object\_kind text, + output\_object\_id uuid, + latency\_ms int, + prompt\_tokens int, + completion\_tokens int, + cache\_hit boolean, + success boolean not null, + error\_text text, + created\_at timestamptz not null default now() +); + +## **`system.retrieval_runs`** + +create table system.retrieval\_runs ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + query\_text text not null, + query\_kind text not null, \-- problem, reentry, lens, contradiction + retrieved\_objects jsonb not null, + created\_at timestamptz not null default now() +); + +## **`system.event_log`** + +create table system.event\_log ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + event\_type text not null, + aggregate\_kind text, + aggregate\_id uuid, + payload jsonb not null, + occurred\_at timestamptz not null default now() +); + +## **`system.outbox_events`** + +For reliable async processing. + +create table system.outbox\_events ( + id uuid primary key, + event\_type text not null, + aggregate\_kind text, + aggregate\_id uuid, + payload jsonb not null, + created\_at timestamptz not null default now(), + processed\_at timestamptz +); + +--- + +# **14\. Vector / embedding schema** + +Use this only for semantic recall, not as truth storage. + +## **`vector.embeddings`** + +create table vector.embeddings ( + id uuid primary key, + operator\_id uuid not null references core.operators(id), + object\_kind text not null, \-- source\_chunk, concept, rule, case, brief, trace + object\_id uuid not null, + embedding vector(1536), \-- or chosen dimension + model\_name text not null, + created\_at timestamptz not null default now() +); + +Index with HNSW or IVF depending on scale. +For a private system, HNSW is usually fine. + +--- + +# **15\. Materialized views / derived read models** + +You should not compute everything on the fly. + +I would create these derived views: + +## **`execution.current_thread_state_mv`** + +Current effective state per active thread. + +Contains: + +* last state estimate +* last blocker +* last touch +* re-entry risk +* current next edge +* last policy decision + +## **`physiology.daily_capacity_mv`** + +Daily physiology summary. + +Contains: + +* latest recovery +* latest sleep summary +* derived fragility risk +* depth ceiling +* recent strain carryover + +## **`vector.active_focus_mv`** + +What matters now. + +Contains: + +* active season +* active missions +* neglected high-value threads +* drift candidates + +## **`reasoning.latest_briefs_mv`** + +Latest problem-room outputs. + +Contains: + +* open problem +* classification +* key unknowns +* current recommendation +* unresolved questions + +These views make the product fast and legible. + +--- + +# **16\. Recommended indexes** + +At minimum: + +* `(operator_id, created_at desc)` on almost every append-heavy table +* `unique` on provider object IDs +* `btree` on active status columns +* `gin` on JSONB fields that need querying +* vector index on embeddings +* `(thread_id, created_at desc)` on traces, policy decisions, checkins +* `(feature_date, feature_name)` on physiology features +* `(problem_id)` on all reasoning children +* `(source_id)` on all extraction children + +For event tables: + +* `(processing_status, received_at)` +* `(event_type, occurred_at desc)` + +--- + +# **17\. Data lifecycle and retention** + +A serious private system needs explicit retention policy. + +## **Hot operational data** + +Keep immediately accessible: + +* active missions +* active threads +* latest states +* latest recoveries +* recent traces +* current re-entry artifacts +* active rules + +## **Warm memory** + +Frequently retrievable but not in the cockpit: + +* past decision briefs +* old traces +* historical patterns +* old problems +* lens-pack history + +## **Cold archive** + +Store encrypted and compressed: + +* raw source payloads +* old webhook bodies +* raw import snapshots +* superseded extraction runs + +Do not delete what might matter for calibration, but do aggressively archive what should not clutter runtime. + +--- + +# **18\. What the actual core aggregates are** + +If I compress the entire schema into true product aggregates, they are: + +## **Directional aggregates** + +* LifeArc +* Season +* Mission +* Thread + +## **Evidence aggregates** + +* SubjectiveCheckin +* BehaviorEvent +* ContextSnapshot +* PhysiologyDay + +## **Knowledge aggregates** + +* Source +* Extraction +* Concept +* Mechanism +* Rule +* LensPack +* Case +* Pattern + +## **Reasoning aggregates** + +* Problem +* ProblemStructure +* Interrogation +* DecisionBrief +* OptionPath + +## **Execution aggregates** + +* StateEstimate +* BlockerEstimate +* ReentryArtifact +* PolicyDecision +* Trace +* Closure + +## **Learning aggregates** + +* ReviewSession +* CalibrationRecord +* PhysiologyCalibration +* RulePromotion +* PatternUpdate + +## **System aggregates** + +* OAuthConnection +* WebhookReceipt +* ModelRun +* RetrievalRun +* EventLog + +That is the serious internal shape. + +--- + +# **19\. The most important non-obvious design choice** + +The most important design choice is this: + +**Do not store only conclusions. Store the chain between evidence, inference, policy, and outcome.** + +That means for every serious recommendation, you want to be able to recover: + +* what evidence existed +* what state was inferred +* what blocker was inferred +* which rules/lenses were retrieved +* what policy was selected +* what action was actually taken +* what happened afterward +* what got updated + +That chain is the real moat. + +Because that is how Utopia compounds judgment instead of becoming a smart-looking black box. + +--- + +# **20\. If I had to reduce this to the minimum serious v1 schema** + +If you want the smallest schema that is still architecturally correct, build these first: + +**core** + +* operators +* oauth\_connections + +**vector\_ctrl** + +* seasons +* missions +* threads + +**evidence** + +* subjective\_checkins +* behavior\_events +* context\_snapshots + +**physiology** + +* whoop\_recoveries +* whoop\_sleeps +* physiology\_features + +**aether** + +* sources +* extractions +* concepts +* mechanisms +* rules +* lens\_packs +* edges + +**reasoning** + +* problems +* problem\_structures +* interrogations +* decision\_briefs +* option\_paths + +**execution** + +* state\_estimates +* blocker\_estimates +* reentry\_artifacts +* policy\_decisions +* traces +* calibration\_records + +**system** + +* model\_runs +* webhook\_receipts +* event\_log + +That is enough to build a real Utopia, not a toy. + +--- + +# **21\. Final compression** + +The database for Utopia should not look like “notes \+ tasks \+ embeddings.” + +It should look like a **typed cognitive architecture**: + +* Vector stores what matters +* Evidence stores what is true about the moment +* Physiology stores body-state priors, including WHOOP recovery/sleep/cycle/workout-derived signals and webhook events. ([WHOOP for Developers](https://developer.whoop.com/api/)) +* Aether stores durable truth and extracted intelligence +* Reasoning stores problem structures and decision artifacts +* Execution stores state, blockers, policy, re-entry, and traces +* Review stores calibration and learning +* System tables store orchestration, integrations, and audit + +That is the serious private system as a whole. + +The next correct step is to turn this into: + +1. exact enum definitions, +2. Pydantic or TypeScript domain models, +3. SQL migration files, +4. event contracts, +5. and the service boundaries for the API. + From 5b37a5352d90c4431f13601c9ef12b46aee913f4 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:37 +0100 Subject: [PATCH 04/63] docs: move Utopia Formal Architecture DB etc.md into history --- Utopia Formal Architecture DB etc.md | 1531 -------------------------- 1 file changed, 1531 deletions(-) delete mode 100644 Utopia Formal Architecture DB etc.md diff --git a/Utopia Formal Architecture DB etc.md b/Utopia Formal Architecture DB etc.md deleted file mode 100644 index 575155e..0000000 --- a/Utopia Formal Architecture DB etc.md +++ /dev/null @@ -1,1531 +0,0 @@ -For a **serious private Utopia system**, I would not design the database as “one big notes table plus embeddings.” -That would kill the product. - -Utopia needs a **typed memory architecture** with clear separation between: - -* directional truth -* live execution state -* evidence about the operator -* physiological priors -* extracted knowledge -* reasoning artifacts -* action traces -* calibration and policy learning -* integration and audit infrastructure - -The right mental model is: - -**Postgres as canonical truth** -**vector index for semantic recall** -**graph edges for explicit conceptual relationships** -**event log for replay/audit/triggering** -**object store for raw artifacts** -**time-series for dense physiological / behavioral telemetry** - -That is the serious shape. - ---- - -# **1\. Core storage strategy** - -I would split the system into five storage roles. - -## **A. Relational core** - -This is the source of truth for all canonical entities. - -Use it for: - -* missions -* threads -* states -* blockers -* re-entry artifacts -* rules -* traces -* decision briefs -* sources -* extractions -* concepts -* mechanisms -* lens packs -* permissions -* OAuth connections -* review records - -This should be PostgreSQL. - -## **B. Vector retrieval layer** - -This exists for semantic retrieval across: - -* extracted concepts -* cases -* prior traces -* source chunks -* decision briefs -* rules -* diagnostic questions - -Keep this close to the relational core unless scale forces separation. -For a private serious system, I would start with **pgvector inside Postgres**, not an external vector database. - -## **C. Graph layer** - -Use this for explicit relationships, not as the primary store. - -Graph is useful for: - -* concept → mechanism -* mechanism → failure mode -* lens → question -* rule → applies\_to -* case → pattern -* pattern → intervention -* contradiction → corrected\_by - -You can model this in relational first with an `edges` table. -Only move to a dedicated graph database if traversal actually becomes central. - -## **D. Event / telemetry layer** - -This captures: - -* user actions -* thread touches -* failed starts -* trigger firings -* AI orchestration events -* webhook receipts -* provider calls -* policy decisions - -This can still live in Postgres initially as append-only event tables. - -## **E. Object storage** - -Use for: - -* raw source files -* PDFs -* original imported documents -* extraction snapshots -* large JSON payload archives -* encrypted export bundles - -That should not be the operational DB. - ---- - -# **2\. Database design principles** - -The entire schema should obey a few principles. - -## **Typed over generic** - -Do not store everything in `notes`. -Use separate typed tables for distinct cognitive objects. - -## **Append where truth unfolds** - -Traces, estimates, webhook receipts, and orchestration runs should often be append-only. - -## **Current-state views over mutable sludge** - -Keep immutable history, then materialize current state via views or snapshot tables. - -## **JSONB for edges, not for the core** - -Use JSONB for flexible payloads, model outputs, or provider-specific metadata. -Do not hide core product semantics inside JSON blobs. - -## **Single-user first, multi-tenant safe** - -Even if Utopia is private and single-user, design tables with `operator_id` or `workspace_id` so the system is structurally clean. - -## **Explicit confidence everywhere** - -Any inferred object should store: - -* confidence -* evidence\_count -* provenance -* last\_validated\_at - -Because Utopia is a reasoning system, not just a CRUD app. - ---- - -# **3\. Logical schema namespaces** - -I would separate the database into logical schemas like this: - -* `core` -* `vector` -* `evidence` -* `physiology` -* `vector_ctrl` -* `aether` -* `reasoning` -* `execution` -* `integration` -* `system` - -That gives you clean domain boundaries. - ---- - -# **4\. Global conventions** - -Every major object should have: - -* `id` as UUIDv7 -* `operator_id` -* `created_at` -* `updated_at` -* `archived_at` nullable -* `version` -* `source_kind` -* `provenance` -* `confidence` where applicable - -Suggested enums: - -* `status` -* `state_kind` -* `blocker_kind` -* `source_kind` -* `memory_class` -* `evidence_kind` -* `intervention_kind` -* `decision_kind` - -Use UUIDv7 or time-ordered UUIDs because Utopia is event-heavy and benefits from sortable IDs. - ---- - -# **5\. Core identity and system tables** - -Even for a private system, you want these. - -## **`core.operators`** - -Represents the human operator the system is modeling. - -create table core.operators ( - id uuid primary key, - display\_name text not null, - timezone text not null, - locale text, - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`core.devices`** - -Tracks trusted devices for local-first syncing. - -create table core.devices ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - device\_name text not null, - device\_type text not null, - public\_key\_fingerprint text, - last\_seen\_at timestamptz, - created\_at timestamptz not null default now() -); - -## **`integration.oauth_connections`** - -Stores external integration connections. - -create table integration.oauth\_connections ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - provider text not null, \-- whoop, openai, anthropic, etc. - provider\_user\_id text, - scopes text\[\] not null default '{}', - status text not null, \-- active, revoked, expired - encrypted\_access\_token bytea, - encrypted\_refresh\_token bytea, - token\_expires\_at timestamptz, - metadata jsonb not null default '{}', - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`integration.permissions`** - -Fine-grained permissions ledger. - -create table integration.permissions ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - provider text not null, - permission\_key text not null, - granted boolean not null, - granted\_at timestamptz, - revoked\_at timestamptz, - details jsonb not null default '{}' -); - ---- - -# **6\. Vector schema: direction and control plane** - -This is where Utopia stores what matters. - -## **`vector_ctrl.life_arcs`** - -Top-level directional framing. - -create table vector\_ctrl.life\_arcs ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - title text not null, - description text, - status text not null, \-- active, dormant, archived - horizon\_start date, - horizon\_end date, - success\_definition text, - anti\_goals text\[\] not null default '{}', - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`vector_ctrl.seasons`** - -A season is a bounded phase of focus. - -create table vector\_ctrl.seasons ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - life\_arc\_id uuid references vector\_ctrl.life\_arcs(id), - title text not null, - thesis text, - start\_date date not null, - end\_date date, - priority\_stack jsonb not null default '\[\]', - status text not null, \-- planned, active, closed - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`vector_ctrl.missions`** - -Strategically meaningful objectives. - -create table vector\_ctrl.missions ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - season\_id uuid references vector\_ctrl.seasons(id), - title text not null, - description text, - mission\_kind text not null, \-- strategic, technical, personal, recovery, etc. - priority\_score numeric(6,3), - status text not null, \-- active, paused, completed, abandoned - success\_definition text, - failure\_definition text, - drift\_definition text, - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`vector_ctrl.threads`** - -Live lines of work within missions. - -create table vector\_ctrl.threads ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - mission\_id uuid not null references vector\_ctrl.missions(id), - parent\_thread\_id uuid references vector\_ctrl.threads(id), - title text not null, - description text, - thread\_kind text not null, \-- build, research, decision, admin, etc. - status text not null, \-- active, blocked, paused, closed - complexity\_score numeric(6,3), - ambiguity\_score numeric(6,3), - reentry\_risk\_score numeric(6,3), - last\_meaningful\_touch\_at timestamptz, - next\_edge\_summary text, - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`vector_ctrl.thread_constraints`** - -Explicit constraints on a thread. - -create table vector\_ctrl.thread\_constraints ( - id uuid primary key, - thread\_id uuid not null references vector\_ctrl.threads(id), - constraint\_type text not null, \-- time, dependency, budget, skill, legal - description text not null, - hardness text not null, \-- hard, soft, assumed - created\_at timestamptz not null default now() -); - -## **`vector_ctrl.anti_goals`** - -What must not happen. - -create table vector\_ctrl.anti\_goals ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - scope\_type text not null, \-- life\_arc, season, mission, thread - scope\_id uuid not null, - description text not null, - created\_at timestamptz not null default now() -); - ---- - -# **7\. Evidence schema: subjective, behavioral, contextual** - -This is the live sensing layer. - -## **`evidence.subjective_checkins`** - -create table evidence.subjective\_checkins ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid references vector\_ctrl.threads(id), - energy smallint check (energy between 0 and 100), - clarity smallint check (clarity between 0 and 100), - resistance smallint check (resistance between 0 and 100), - overwhelm smallint check (overwhelm between 0 and 100), - emotional\_load smallint check (emotional\_load between 0 and 100), - perceived\_urgency smallint check (perceived\_urgency between 0 and 100), - free\_text text, - recorded\_at timestamptz not null, - created\_at timestamptz not null default now() -); - -## **`evidence.behavior_events`** - -create table evidence.behavior\_events ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid references vector\_ctrl.threads(id), - event\_type text not null, \-- thread\_opened, action\_started, action\_abandoned, etc. - event\_at timestamptz not null, - duration\_ms bigint, - metadata jsonb not null default '{}' -); - -## **`evidence.context_snapshots`** - -create table evidence.context\_snapshots ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid references vector\_ctrl.threads(id), - local\_time timestamptz not null, - environment\_label text, - interruption\_count int, - obligation\_load smallint, - available\_minutes int, - active\_window text, \-- morning, afternoon, evening - metadata jsonb not null default '{}' -); - -## **`evidence.derived_features`** - -Computed features from raw evidence. - -create table evidence.derived\_features ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid references vector\_ctrl.threads(id), - feature\_name text not null, - feature\_value numeric, - feature\_json jsonb, - feature\_window text, \-- last\_2h, last\_24h, last\_7d - confidence numeric(5,4), - observed\_at timestamptz not null, - created\_at timestamptz not null default now() -); - -This is where you store things like: - -* failed\_start\_rate\_24h -* thread\_decay\_hours -* drift\_probability -* completion\_aversion\_score -* ambiguity\_candidate\_score - ---- - -# **8\. Physiology schema: WHOOP as a first-class subsystem** - -This domain should be explicit because WHOOP is not just metadata. -WHOOP’s API provides OAuth-scoped data for recovery, cycles, sleep, workouts, profile, and body measurements, and its webhook model sends events such as `recovery.updated`, `sleep.updated`, and `workout.updated` with HMAC-verifiable signatures. Recovery is described by WHOOP as a daily 0–100 score derived from signals such as HRV and resting heart rate. ([WHOOP for Developers](https://developer.whoop.com/api/)) - -## **`physiology.whoop_connections`** - -create table physiology.whoop\_connections ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - oauth\_connection\_id uuid not null references integration.oauth\_connections(id), - whoop\_user\_id bigint, - webhook\_secret\_fingerprint text, - status text not null, \-- active, revoked, errored - last\_sync\_at timestamptz, - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`physiology.whoop_body_measurements`** - -create table physiology.whoop\_body\_measurements ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - measured\_at timestamptz not null, - height\_meter numeric(6,4), - weight\_kg numeric(6,3), - max\_heart\_rate int, - source\_payload jsonb not null default '{}', - created\_at timestamptz not null default now() -); - -## **`physiology.whoop_cycles`** - -create table physiology.whoop\_cycles ( - id uuid primary key, \-- internal UUID - operator\_id uuid not null references core.operators(id), - provider\_cycle\_id bigint not null unique, - whoop\_user\_id bigint, - cycle\_start timestamptz not null, - cycle\_end timestamptz, - timezone\_offset text, - score\_state text, \-- SCORED, PENDING\_SCORE, UNSCORABLE - strain numeric(8,4), - kilojoule numeric(12,3), - average\_heart\_rate int, - max\_heart\_rate int, - source\_payload jsonb not null default '{}', - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`physiology.whoop_sleeps`** - -create table physiology.whoop\_sleeps ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - provider\_sleep\_id uuid not null unique, - provider\_cycle\_id bigint, - sleep\_start timestamptz, - sleep\_end timestamptz, - sleep\_performance\_pct numeric(5,2), - total\_in\_bed\_ms bigint, - total\_asleep\_ms bigint, - slow\_wave\_ms bigint, - rem\_ms bigint, - light\_ms bigint, - awake\_ms bigint, - source\_payload jsonb not null default '{}', - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`physiology.whoop_recoveries`** - -Recovery objects should mirror WHOOP’s actual structure closely enough to avoid lossy mapping. WHOOP’s recovery docs describe fields including `cycle_id`, `sleep_id`, `user_id`, `created_at`, `updated_at`, `score_state`, and score values like `recovery_score`, `resting_heart_rate`, and `hrv_rmssd_milli`. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/user-data/recovery/)) - -create table physiology.whoop\_recoveries ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - provider\_cycle\_id bigint not null, - provider\_sleep\_id uuid not null, - whoop\_user\_id bigint, - recorded\_at timestamptz not null, - updated\_at\_source timestamptz, - score\_state text not null, - recovery\_score smallint, - resting\_heart\_rate int, - hrv\_rmssd\_milli numeric(12,6), - spo2\_percentage numeric(6,3), - skin\_temp\_celsius numeric(6,3), - user\_calibrating boolean, - source\_payload jsonb not null default '{}', - created\_at timestamptz not null default now(), - unique (operator\_id, provider\_cycle\_id, provider\_sleep\_id) -); - -## **`physiology.whoop_workouts`** - -create table physiology.whoop\_workouts ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - provider\_workout\_id uuid not null unique, - workout\_type text, - workout\_start timestamptz, - workout\_end timestamptz, - strain numeric(8,4), - average\_heart\_rate int, - max\_heart\_rate int, - source\_payload jsonb not null default '{}', - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`physiology.physiology_features`** - -This is where raw WHOOP data becomes operational priors. - -create table physiology.physiology\_features ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - feature\_date date not null, - feature\_name text not null, \-- capacity\_envelope, fragility\_risk, sleep\_debt\_slope, etc. - feature\_value numeric, - feature\_json jsonb, - confidence numeric(5,4), - computed\_at timestamptz not null default now(), - unique (operator\_id, feature\_date, feature\_name) -); - -Useful features: - -* recovery\_trend\_3d -* recovery\_volatility\_7d -* strain\_carryover -* sleep\_regularity\_score -* fragility\_risk -* depth\_ceiling\_score -* depletion\_prior -* recovery\_mismatch\_score - -## **`physiology.biomarker_panels`** - -For Advanced Labs or any later biomarker layer, keep separate from wearable data. - -create table physiology.biomarker\_panels ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - panel\_date date not null, - provider text not null, - panel\_type text not null, \-- advanced\_labs, manual\_lab\_import - summary jsonb not null, - source\_payload jsonb not null default '{}', - created\_at timestamptz not null default now() -); - -## **`integration.webhook_receipts`** - -WHOOP webhook bodies include fields like `user_id`, `id`, `type`, and `trace_id`, and WHOOP documents signature validation using `X-WHOOP-Signature` and `X-WHOOP-Signature-Timestamp`. ([WHOOP for Developers](https://developer.whoop.com/docs/developing/webhooks/)) - -create table integration.webhook\_receipts ( - id uuid primary key, - provider text not null, - event\_type text not null, - provider\_user\_id text, - provider\_object\_id text, - trace\_id text, - received\_at timestamptz not null default now(), - signature\_valid boolean, - headers jsonb not null, - raw\_body jsonb not null, - processed\_at timestamptz, - processing\_status text not null default 'pending' -); - ---- - -# **9\. Aether schema: memory and extracted intelligence** - -This is the most important part of the system. - -Do not make Aether one table. -It should be a family of typed memory objects. - -## **`aether.sources`** - -create table aether.sources ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - source\_kind text not null, \-- book, paper, essay, transcript, note\_bundle, case\_file - title text not null, - author text, - published\_at date, - ingest\_status text not null, - canonical\_uri text, - storage\_uri text, - checksum text, - metadata jsonb not null default '{}', - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now() -); - -## **`aether.source_chunks`** - -Only if you need chunk-level retrieval. - -create table aether.source\_chunks ( - id uuid primary key, - source\_id uuid not null references aether.sources(id), - chunk\_index int not null, - raw\_text text not null, - token\_count int, - semantic\_label text, - metadata jsonb not null default '{}' -); - -## **`aether.extractions`** - -A structured distillation pass for a source. - -create table aether.extractions ( - id uuid primary key, - source\_id uuid not null references aether.sources(id), - extraction\_version text not null, - extraction\_status text not null, - thesis text, - summary text, - confidence numeric(5,4), - extracted\_at timestamptz not null default now(), - model\_run\_id uuid, - unique (source\_id, extraction\_version) -); - -## **`aether.concepts`** - -create table aether.concepts ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - canonical\_name text not null, - definition text, - domain text, - source\_count int not null default 0, - confidence numeric(5,4), - created\_at timestamptz not null default now(), - updated\_at timestamptz not null default now(), - unique (operator\_id, canonical\_name) -); - -## **`aether.mechanisms`** - -create table aether.mechanisms ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - name text not null, - description text not null, - causal\_logic text, - domain text, - confidence numeric(5,4), - created\_at timestamptz not null default now() -); - -## **`aether.tradeoffs`** - -create table aether.tradeoffs ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - name text not null, - pole\_a text not null, - pole\_b text not null, - description text, - domain text, - created\_at timestamptz not null default now() -); - -## **`aether.failure_modes`** - -create table aether.failure\_modes ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - name text not null, - description text not null, - early\_signals text\[\], - domain text, - severity text, - created\_at timestamptz not null default now() -); - -## **`aether.heuristics`** - -create table aether.heuristics ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - statement text not null, - domain text, - applicability text, - failure\_conditions text, - confidence numeric(5,4), - created\_at timestamptz not null default now() -); - -## **`aether.diagnostic_questions`** - -create table aether.diagnostic\_questions ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - question\_text text not null, - question\_class text not null, \-- objective, constraint, risk, hidden\_variable, etc. - domain text, - usefulness\_score numeric(6,3), - created\_at timestamptz not null default now() -); - -## **`aether.protocols`** - -Reusable reasoning flows. - -create table aether.protocols ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - protocol\_name text not null, - domain text not null, - purpose text, - steps jsonb not null, \-- ordered protocol structure - applicability text, - created\_at timestamptz not null default now() -); - -## **`aether.lens_packs`** - -create table aether.lens\_packs ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - name text not null, - domain text not null, - description text, - version text not null, - source\_basis jsonb not null default '\[\]', - created\_at timestamptz not null default now() -); - -## **`aether.lens_pack_items`** - -create table aether.lens\_pack\_items ( - id uuid primary key, - lens\_pack\_id uuid not null references aether.lens\_packs(id), - item\_kind text not null, \-- concept, mechanism, question, heuristic, protocol, failure\_mode - item\_id uuid not null, - weight numeric(6,3), - metadata jsonb not null default '{}' -); - -## **`aether.cases`** - -Internal or external cases. - -create table aether.cases ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - title text not null, - case\_kind text not null, \-- internal, external - summary text, - outcome text, - lessons jsonb not null default '\[\]', - created\_at timestamptz not null default now() -); - -## **`aether.rules`** - -Durable truths that survived compression. - -create table aether.rules ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - rule\_text text not null, - rule\_kind text not null, \-- execution, reasoning, physiology, strategic, personal - evidence\_count int not null default 0, - confidence numeric(5,4), - first\_observed\_at timestamptz, - last\_validated\_at timestamptz, - active boolean not null default true, - created\_at timestamptz not null default now() -); - -## **`aether.patterns`** - -create table aether.patterns ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - pattern\_name text not null, - description text, - pattern\_kind text not null, \-- blocker, timing, physiology, narrative, execution - recurrence\_count int not null default 0, - confidence numeric(5,4), - created\_at timestamptz not null default now() -); - -## **`aether.edges`** - -Relational graph layer. - -create table aether.edges ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - src\_kind text not null, - src\_id uuid not null, - edge\_type text not null, \-- supports, contradicts, applies\_to, predicts, resembles, etc. - dst\_kind text not null, - dst\_id uuid not null, - weight numeric(6,3), - provenance jsonb not null default '{}', - created\_at timestamptz not null default now() -); - -This table becomes surprisingly powerful if kept clean. - ---- - -# **10\. Reasoning schema: problem room and decision artifacts** - -This is where Utopia turns live problems into structured thinking. - -## **`reasoning.problems`** - -create table reasoning.problems ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - title text not null, - raw\_prompt text not null, - problem\_kind text, \-- strategic, technical, social, legal, execution, etc. - urgency\_score numeric(6,3), - stakes\_score numeric(6,3), - uncertainty\_score numeric(6,3), - state\_at\_creation text, - thread\_id uuid references vector\_ctrl.threads(id), - created\_at timestamptz not null default now() -); - -## **`reasoning.problem_structures`** - -create table reasoning.problem\_structures ( - id uuid primary key, - problem\_id uuid not null references reasoning.problems(id), - objective text, - stakes text, - actors jsonb not null default '\[\]', - incentives jsonb not null default '\[\]', - constraints jsonb not null default '\[\]', - assumptions jsonb not null default '\[\]', - unknowns jsonb not null default '\[\]', - irreversibilities jsonb not null default '\[\]', - bottlenecks jsonb not null default '\[\]', - observable\_facts jsonb not null default '\[\]', - narrative\_layer jsonb not null default '\[\]', - distortion\_candidates jsonb not null default '\[\]', - confidence numeric(5,4), - generated\_at timestamptz not null default now() -); - -## **`reasoning.interrogations`** - -create table reasoning.interrogations ( - id uuid primary key, - problem\_id uuid not null references reasoning.problems(id), - interrogation\_kind text not null, \-- initial, followup, contradiction, pressure\_test - questions jsonb not null, \-- ordered high-leverage questions - rationale jsonb not null default '{}', - generated\_at timestamptz not null default now() -); - -## **`reasoning.decision_briefs`** - -create table reasoning.decision\_briefs ( - id uuid primary key, - problem\_id uuid not null references reasoning.problems(id), - classification text, - summary text, - key\_unknowns jsonb not null default '\[\]', - blind\_spots jsonb not null default '\[\]', - risks jsonb not null default '\[\]', - relevant\_lens\_pack\_ids uuid\[\] not null default '{}', - recommendation\_summary text, - confidence numeric(5,4), - generated\_at timestamptz not null default now() -); - -## **`reasoning.option_paths`** - -create table reasoning.option\_paths ( - id uuid primary key, - decision\_brief\_id uuid not null references reasoning.decision\_briefs(id), - option\_label text not null, \-- conservative, aggressive, asymmetric - description text not null, - expected\_upside text, - expected\_downside text, - reversibility text, - risk\_score numeric(6,3), - recommendation\_rank int, - created\_at timestamptz not null default now() -); - -## **`reasoning.contradiction_reports`** - -create table reasoning.contradiction\_reports ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - problem\_id uuid references reasoning.problems(id), - contradiction\_kind text not null, \-- narrative\_vs\_behavior, physiology\_vs\_claim, vector\_vs\_action - description text not null, - evidence jsonb not null default '\[\]', - severity numeric(6,3), - created\_at timestamptz not null default now() -); - ---- - -# **11\. Execution schema: Schrödinger, re-entry, traces** - -This is the action layer. - -## **`execution.state_estimates`** - -create table execution.state\_estimates ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid references vector\_ctrl.threads(id), - state\_kind text not null, \-- recover, preserve, orient, clarify, execute, etc. - confidence numeric(5,4) not null, - contributing\_factors jsonb not null default '\[\]', - generated\_at timestamptz not null default now() -); - -## **`execution.blocker_estimates`** - -create table execution.blocker\_estimates ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid references vector\_ctrl.threads(id), - blocker\_kind text not null, - confidence numeric(5,4) not null, - supporting\_evidence jsonb not null default '\[\]', - generated\_at timestamptz not null default now() -); - -## **`execution.reentry_artifacts`** - -create table execution.reentry\_artifacts ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid not null references vector\_ctrl.threads(id), - last\_completed\_step text, - unresolved\_edge text, - next\_smallest\_move text, - trap\_to\_avoid text, - relevant\_context jsonb not null default '{}', - freshness\_score numeric(6,3), - created\_at timestamptz not null default now(), - superseded\_by uuid references execution.reentry\_artifacts(id) -); - -## **`execution.policy_decisions`** - -This is the actual Schrödinger output record. - -create table execution.policy\_decisions ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid references vector\_ctrl.threads(id), - problem\_id uuid references reasoning.problems(id), - state\_estimate\_id uuid references execution.state\_estimates(id), - blocker\_estimate\_id uuid references execution.blocker\_estimates(id), - mode text not null, \-- recover, preserve, re-enter, clarify, ask, execute, close\_loop, review - intervention\_kind text not null, - action\_depth text not null, \-- tiny, narrow, moderate, deep - next\_move text not null, - rationale text, - confidence numeric(5,4), - caution\_flags jsonb not null default '\[\]', - created\_at timestamptz not null default now() -); - -## **`execution.traces`** - -create table execution.traces ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid references vector\_ctrl.threads(id), - policy\_decision\_id uuid references execution.policy\_decisions(id), - trace\_kind text not null, \-- action, question, preserve, recovery, closure - action\_taken text, - outcome text, - truth\_revealed text, - next\_edge text, - completion\_score numeric(6,3), - subjective\_after jsonb not null default '{}', - created\_at timestamptz not null default now() -); - -## **`execution.closures`** - -create table execution.closures ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - thread\_id uuid not null references vector\_ctrl.threads(id), - closure\_type text not null, \-- complete, archive, pause, merge - summary text, - lessons jsonb not null default '\[\]', - created\_at timestamptz not null default now() -); - ---- - -# **12\. Review and calibration schema** - -This is the immune system. - -## **`execution.review_sessions`** - -create table execution.review\_sessions ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - review\_scope text not null, \-- micro, daily, weekly, monthly - started\_at timestamptz not null, - completed\_at timestamptz, - summary text, - metadata jsonb not null default '{}' -); - -## **`execution.rule_promotions`** - -create table execution.rule\_promotions ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - rule\_id uuid references aether.rules(id), - source\_trace\_ids uuid\[\] not null default '{}', - promotion\_reason text, - created\_at timestamptz not null default now() -); - -## **`execution.pattern_updates`** - -create table execution.pattern\_updates ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - pattern\_id uuid references aether.patterns(id), - supporting\_trace\_ids uuid\[\] not null default '{}', - recurrence\_delta int not null default 1, - created\_at timestamptz not null default now() -); - -## **`execution.calibration_records`** - -create table execution.calibration\_records ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - policy\_decision\_id uuid references execution.policy\_decisions(id), - predicted\_outcome text, - actual\_outcome text, - calibration\_error numeric(6,3), - lesson text, - created\_at timestamptz not null default now() -); - -## **`execution.physiology_calibrations`** - -Critical for personal WHOOP learning. - -create table execution.physiology\_calibrations ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - feature\_date date not null, - physiology\_summary jsonb not null, - realized\_capacity\_score numeric(6,3), - work\_type text, \-- deep, narrow, admin, recovery, exploratory - mismatch\_score numeric(6,3), - lesson text, - created\_at timestamptz not null default now() -); - -This is how Utopia learns that low recovery may still permit narrow deterministic work, while open-ended work collapses. - ---- - -# **13\. AI orchestration schema** - -If this is a serious private system, you need auditability around model behavior. - -## **`system.model_providers`** - -create table system.model\_providers ( - id uuid primary key, - provider text not null unique, - capability\_profile jsonb not null, - active boolean not null default true, - created\_at timestamptz not null default now() -); - -## **`system.model_runs`** - -create table system.model\_runs ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - provider text not null, - model\_name text not null, - task\_kind text not null, \-- extraction, routing, reasoning, compression - input\_object\_kind text, - input\_object\_id uuid, - output\_object\_kind text, - output\_object\_id uuid, - latency\_ms int, - prompt\_tokens int, - completion\_tokens int, - cache\_hit boolean, - success boolean not null, - error\_text text, - created\_at timestamptz not null default now() -); - -## **`system.retrieval_runs`** - -create table system.retrieval\_runs ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - query\_text text not null, - query\_kind text not null, \-- problem, reentry, lens, contradiction - retrieved\_objects jsonb not null, - created\_at timestamptz not null default now() -); - -## **`system.event_log`** - -create table system.event\_log ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - event\_type text not null, - aggregate\_kind text, - aggregate\_id uuid, - payload jsonb not null, - occurred\_at timestamptz not null default now() -); - -## **`system.outbox_events`** - -For reliable async processing. - -create table system.outbox\_events ( - id uuid primary key, - event\_type text not null, - aggregate\_kind text, - aggregate\_id uuid, - payload jsonb not null, - created\_at timestamptz not null default now(), - processed\_at timestamptz -); - ---- - -# **14\. Vector / embedding schema** - -Use this only for semantic recall, not as truth storage. - -## **`vector.embeddings`** - -create table vector.embeddings ( - id uuid primary key, - operator\_id uuid not null references core.operators(id), - object\_kind text not null, \-- source\_chunk, concept, rule, case, brief, trace - object\_id uuid not null, - embedding vector(1536), \-- or chosen dimension - model\_name text not null, - created\_at timestamptz not null default now() -); - -Index with HNSW or IVF depending on scale. -For a private system, HNSW is usually fine. - ---- - -# **15\. Materialized views / derived read models** - -You should not compute everything on the fly. - -I would create these derived views: - -## **`execution.current_thread_state_mv`** - -Current effective state per active thread. - -Contains: - -* last state estimate -* last blocker -* last touch -* re-entry risk -* current next edge -* last policy decision - -## **`physiology.daily_capacity_mv`** - -Daily physiology summary. - -Contains: - -* latest recovery -* latest sleep summary -* derived fragility risk -* depth ceiling -* recent strain carryover - -## **`vector.active_focus_mv`** - -What matters now. - -Contains: - -* active season -* active missions -* neglected high-value threads -* drift candidates - -## **`reasoning.latest_briefs_mv`** - -Latest problem-room outputs. - -Contains: - -* open problem -* classification -* key unknowns -* current recommendation -* unresolved questions - -These views make the product fast and legible. - ---- - -# **16\. Recommended indexes** - -At minimum: - -* `(operator_id, created_at desc)` on almost every append-heavy table -* `unique` on provider object IDs -* `btree` on active status columns -* `gin` on JSONB fields that need querying -* vector index on embeddings -* `(thread_id, created_at desc)` on traces, policy decisions, checkins -* `(feature_date, feature_name)` on physiology features -* `(problem_id)` on all reasoning children -* `(source_id)` on all extraction children - -For event tables: - -* `(processing_status, received_at)` -* `(event_type, occurred_at desc)` - ---- - -# **17\. Data lifecycle and retention** - -A serious private system needs explicit retention policy. - -## **Hot operational data** - -Keep immediately accessible: - -* active missions -* active threads -* latest states -* latest recoveries -* recent traces -* current re-entry artifacts -* active rules - -## **Warm memory** - -Frequently retrievable but not in the cockpit: - -* past decision briefs -* old traces -* historical patterns -* old problems -* lens-pack history - -## **Cold archive** - -Store encrypted and compressed: - -* raw source payloads -* old webhook bodies -* raw import snapshots -* superseded extraction runs - -Do not delete what might matter for calibration, but do aggressively archive what should not clutter runtime. - ---- - -# **18\. What the actual core aggregates are** - -If I compress the entire schema into true product aggregates, they are: - -## **Directional aggregates** - -* LifeArc -* Season -* Mission -* Thread - -## **Evidence aggregates** - -* SubjectiveCheckin -* BehaviorEvent -* ContextSnapshot -* PhysiologyDay - -## **Knowledge aggregates** - -* Source -* Extraction -* Concept -* Mechanism -* Rule -* LensPack -* Case -* Pattern - -## **Reasoning aggregates** - -* Problem -* ProblemStructure -* Interrogation -* DecisionBrief -* OptionPath - -## **Execution aggregates** - -* StateEstimate -* BlockerEstimate -* ReentryArtifact -* PolicyDecision -* Trace -* Closure - -## **Learning aggregates** - -* ReviewSession -* CalibrationRecord -* PhysiologyCalibration -* RulePromotion -* PatternUpdate - -## **System aggregates** - -* OAuthConnection -* WebhookReceipt -* ModelRun -* RetrievalRun -* EventLog - -That is the serious internal shape. - ---- - -# **19\. The most important non-obvious design choice** - -The most important design choice is this: - -**Do not store only conclusions. Store the chain between evidence, inference, policy, and outcome.** - -That means for every serious recommendation, you want to be able to recover: - -* what evidence existed -* what state was inferred -* what blocker was inferred -* which rules/lenses were retrieved -* what policy was selected -* what action was actually taken -* what happened afterward -* what got updated - -That chain is the real moat. - -Because that is how Utopia compounds judgment instead of becoming a smart-looking black box. - ---- - -# **20\. If I had to reduce this to the minimum serious v1 schema** - -If you want the smallest schema that is still architecturally correct, build these first: - -**core** - -* operators -* oauth\_connections - -**vector\_ctrl** - -* seasons -* missions -* threads - -**evidence** - -* subjective\_checkins -* behavior\_events -* context\_snapshots - -**physiology** - -* whoop\_recoveries -* whoop\_sleeps -* physiology\_features - -**aether** - -* sources -* extractions -* concepts -* mechanisms -* rules -* lens\_packs -* edges - -**reasoning** - -* problems -* problem\_structures -* interrogations -* decision\_briefs -* option\_paths - -**execution** - -* state\_estimates -* blocker\_estimates -* reentry\_artifacts -* policy\_decisions -* traces -* calibration\_records - -**system** - -* model\_runs -* webhook\_receipts -* event\_log - -That is enough to build a real Utopia, not a toy. - ---- - -# **21\. Final compression** - -The database for Utopia should not look like “notes \+ tasks \+ embeddings.” - -It should look like a **typed cognitive architecture**: - -* Vector stores what matters -* Evidence stores what is true about the moment -* Physiology stores body-state priors, including WHOOP recovery/sleep/cycle/workout-derived signals and webhook events. ([WHOOP for Developers](https://developer.whoop.com/api/)) -* Aether stores durable truth and extracted intelligence -* Reasoning stores problem structures and decision artifacts -* Execution stores state, blockers, policy, re-entry, and traces -* Review stores calibration and learning -* System tables store orchestration, integrations, and audit - -That is the serious private system as a whole. - -The next correct step is to turn this into: - -1. exact enum definitions, -2. Pydantic or TypeScript domain models, -3. SQL migration files, -4. event contracts, -5. and the service boundaries for the API. - From f1f146c602e790449ad555dfbb9f8efabeee042d Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:40 +0100 Subject: [PATCH 05/63] docs: define cleaned neurocognitive architecture --- docs/architecture/overview.md | 206 ++++++++++++++++++++++++++++++++++ 1 file changed, 206 insertions(+) create mode 100644 docs/architecture/overview.md diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md new file mode 100644 index 0000000..c69f31c --- /dev/null +++ b/docs/architecture/overview.md @@ -0,0 +1,206 @@ +# Neurocognitive System Architecture + +Neurocognitive System is the cleaned architectural interpretation of the historical Utopia prototype. + +The system is organized around one control loop: + +```text +Direction + ↓ +Sensing + ↓ +Observations + ↓ +Features + ↓ +Evidence Bundle + ↓ +Inference + ↓ +Policy + ↓ +Action / Re-entry + ↓ +Outcome Trace + ↓ +Review + ↓ +Calibration + ↺ +``` + +The key design rule is that these stages are not interchangeable. + +A sensor reading is not an inference. +An inference is not a policy. +A policy is not truth. +An outcome is evidence about whether the prior policy and inference were useful. + +## 1. Direction + +Direction defines what the system is trying to preserve or advance. + +Historical Utopia terminology called this layer **Vector**. + +The current implementation models: + +```text +life arc → season → mission → thread +``` + +Direction exists to distinguish useful motion from drift. + +## 2. Sensing + +Sensing is intentionally modality-agnostic. + +Historical implementation includes: + +- subjective check-ins +- behavioral events +- contextual snapshots +- WHOOP physiology + +The intended architecture was broader and was designed to support future sensor adapters, including neurophysiological sources such as EEG devices. + +A sensor adapter should emit normalized observations. It should never directly emit a domain conclusion such as "the operator is depleted." + +## 3. Observations + +An observation is a recorded measurement, report, or event. + +Examples: + +- subjective energy = 35 +- failed-start event +- WHOOP recovery measurement +- environment interruption count +- future EEG-derived feature window + +Observations preserve source, time, provenance, and freshness. + +## 4. Features + +Features are transformations over observations. + +Examples: + +- failed-start rate over 24 hours +- recovery trend +- re-entry risk +- context-switch density +- EEG-derived signal features + +Features remain measurements. They are not state labels. + +## 5. Evidence Bundles + +An evidence bundle is the exact set of observations and features used to evaluate a specific claim. + +This creates a stable boundary between sensing and reasoning and makes the system auditable. + +## 6. Inference + +Inference turns evidence into explicit hypotheses. + +Historical examples include: + +- state estimate +- blocker estimate +- contradiction report + +Every inference should eventually carry: + +- status +- confidence +- provenance +- evidence references +- model/rule version +- timestamp + +Provider failure or parse failure must not silently become a domain belief. + +## 7. Policy + +Policy proposes the next intervention given: + +- direction +- current evidence +- inferred state +- blocker hypothesis +- available capacity +- re-entry context +- constraints + +Historical Utopia called the policy selector **Schrödinger**. + +A policy decision is a testable proposal, not an oracle. + +## 8. Re-entry + +Re-entry remains a first-class concept. + +A re-entry artifact should preserve the minimum state necessary to resume an interrupted thread: + +- last completed step +- unresolved edge +- smallest next move +- relevant context +- trap to avoid +- freshness + +## 9. Outcome and Trace + +A trace records what actually happened after a policy was acted upon. + +The system should preserve the causal chain: + +```text +evidence → inference → policy → action → outcome +``` + +## 10. Review and Calibration + +Calibration compares predictions with outcomes. + +The historical implementation contains review sessions, rule promotions, pattern updates, and calibration records. The cleaned architecture treats those records as the beginning of a real feedback loop rather than the completion of one. + +The intended loop is: + +```text +prediction + ↓ +action + ↓ +outcome + ↓ +error / usefulness measurement + ↓ +reliability update + ↓ +future evidence weighting +``` + +## 11. Memory and Reasoning + +Historical **Aether** is the typed memory layer. It stores structured cognitive objects rather than an undifferentiated note archive. + +The reasoning layer stores explicit problem structures, unknowns, assumptions, decision briefs, options, and contradiction reports. + +Retrieved knowledge is evidence. It must never be treated as trusted instruction merely because it was retrieved. + +## 12. Trust boundary + +The historical codebase is a private prototype, not production multi-tenant software. + +The current API does not implement a complete authentication or authorization boundary. Do not expose it to untrusted networks. + +See [SECURITY.md](../../SECURITY.md). + +## 13. Medical boundary + +Neurocognitive System is designed around operational state estimation and continuity. + +It is not a medical diagnostic system. + +Physiological or future neurophysiological inputs should be treated as noisy evidence about operational capacity, not as a basis for diagnosing medical or psychiatric conditions. From 1a7bd5b517fa7a1e22810a91c470076937628e09 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:43 +0100 Subject: [PATCH 06/63] docs: add Neurocognitive System RFC --- docs/rfcs/RFC-0001-neurocognitive-system.md | 49 +++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 docs/rfcs/RFC-0001-neurocognitive-system.md diff --git a/docs/rfcs/RFC-0001-neurocognitive-system.md b/docs/rfcs/RFC-0001-neurocognitive-system.md new file mode 100644 index 0000000..50a5a7b --- /dev/null +++ b/docs/rfcs/RFC-0001-neurocognitive-system.md @@ -0,0 +1,49 @@ +# RFC-0001: Neurocognitive System + +**Status:** Draft +**Historical predecessor:** Utopia v0.1 + +## Summary + +This RFC defines the cleaned architecture extracted from Utopia. + +The historical implementation explored continuity, state-aware action, typed memory, physiology, reasoning, traces, and calibration. The redesign keeps the durable primitives while separating sensing, evidence, inference, policy, and feedback more rigorously. + +## Core claim + +The operator is not stable. + +Attention, context, energy, environment, and usable cognitive capacity change over time. A control system should therefore estimate state from imperfect evidence rather than assume a permanently stable executor. + +## System loop + +```text +Direction → Sensing → Observations → Features → Evidence + → Inference → Policy → Action → Outcome → Calibration +``` + +## Non-goals + +This system does not: + +- diagnose medical or psychiatric conditions +- treat any wearable or EEG signal as ground truth +- make model output equivalent to truth +- replace operator authority +- optimize activity without regard to direction + +## Historical compatibility + +The existing Python package remains `utopia` during the cleanup phase so the historical implementation remains inspectable. + +Historical names such as Vector, Aether, and Schrödinger are documented but are not required vocabulary for future architecture. + +## Migration principle + +Do not rewrite history to make the prototype appear more complete than it was. + +The cleaned architecture must clearly distinguish: + +1. implemented historical behavior +2. intended historical architecture +3. proposed future abstractions From 4ebf9838a0bc0bf2f3774963df9471ff2b9f91f1 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:46 +0100 Subject: [PATCH 07/63] docs: separate observation evidence and inference --- ...RFC-0002-observation-evidence-inference.md | 82 +++++++++++++++++++ 1 file changed, 82 insertions(+) create mode 100644 docs/rfcs/RFC-0002-observation-evidence-inference.md diff --git a/docs/rfcs/RFC-0002-observation-evidence-inference.md b/docs/rfcs/RFC-0002-observation-evidence-inference.md new file mode 100644 index 0000000..0320e70 --- /dev/null +++ b/docs/rfcs/RFC-0002-observation-evidence-inference.md @@ -0,0 +1,82 @@ +# RFC-0002: Observation, Evidence, and Inference + +**Status:** Draft + +## Problem + +The historical architecture sometimes collapses measurement, evidence, and model judgment into one object graph. + +That makes provenance harder to inspect and allows model failure to masquerade as domain truth. + +## Definitions + +### Observation + +A measurement, report, or event produced by a known source. + +An observation answers: + +> What was recorded? + +### Feature + +A deterministic or model-derived transformation over one or more observations. + +A feature answers: + +> What signal was derived? + +### Evidence + +An observation or feature used in support of or against a specific claim. + +Evidence answers: + +> Why is this claim being considered? + +### Inference + +A hypothesis produced from an evidence bundle. + +An inference answers: + +> What does the system currently believe, with what confidence and provenance? + +### Policy + +A proposed intervention based on direction, evidence, inference, constraints, and capacity. + +A policy answers: + +> What should be tried next? + +### Outcome + +A trace of what happened after action. + +An outcome answers: + +> What did reality do? + +## Required invariant + +No provider error, parse error, missing sensor value, or retrieval failure may silently become a valid inference. + +Future inference records should distinguish statuses such as: + +- valid +- low_confidence +- insufficient_evidence +- provider_error +- parse_error +- rejected + +## Sensor rule + +Sensor adapters emit observations. + +They do not emit operator identity claims, diagnoses, or policy decisions. + +## Calibration rule + +Confidence should become increasingly grounded in observed predictive usefulness rather than model self-reported certainty alone. From 840c92aed1a7c9fedb7dbd756cb79c80816444b8 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:49 +0100 Subject: [PATCH 08/63] docs: map historical terminology --- docs/history/terminology.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 docs/history/terminology.md diff --git a/docs/history/terminology.md b/docs/history/terminology.md new file mode 100644 index 0000000..99dd32c --- /dev/null +++ b/docs/history/terminology.md @@ -0,0 +1,18 @@ +# Historical terminology + +The repository began under the codename **Utopia**. Several internal names are preserved in code and historical documents. + +| Historical term | Cleaned interpretation | +| --- | --- | +| Utopia | Neurocognitive System | +| Vector | Direction | +| Evidence Plane | Sensing + Evidence | +| Aether | Typed Memory | +| AI Fabric | Inference / Reasoning Runtime | +| Schrödinger | Policy Selector | +| Trace | Outcome / execution trace | +| Review | Feedback + Calibration | + +The historical terminology is not being erased. It is kept so the original implementation and documents remain understandable. + +The cleaned vocabulary exists to make architectural boundaries more explicit. From e245d3ca9285f622f7744212bf02df82811707d1 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:52 +0100 Subject: [PATCH 09/63] docs: add history index --- docs/history/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100644 docs/history/README.md diff --git a/docs/history/README.md b/docs/history/README.md new file mode 100644 index 0000000..377b757 --- /dev/null +++ b/docs/history/README.md @@ -0,0 +1,11 @@ +# Historical Utopia material + +This directory preserves the original architecture documents from the Utopia prototype. + +They are intentionally retained with their original language, assumptions, and terminology. + +- [Original architecture](original-architecture.md) +- [Original database design](original-database-design.md) +- [Terminology map](terminology.md) + +These files are historical evidence, not statements that every described subsystem was implemented. From bb4a038d28d1d0cc5b51c2f6404c1ac3e590e29c Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:55 +0100 Subject: [PATCH 10/63] docs: record static audit --- docs/audit/2026-09-23.md | 66 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) create mode 100644 docs/audit/2026-09-23.md diff --git a/docs/audit/2026-09-23.md b/docs/audit/2026-09-23.md new file mode 100644 index 0000000..1b7a390 --- /dev/null +++ b/docs/audit/2026-09-23.md @@ -0,0 +1,66 @@ +# Static audit — 2026-09-23 + +This is a static audit of the historical Utopia codebase prior to the Neurocognitive System cleanup. + +## Strong existing primitives + +- explicit directional hierarchy +- separate evidence, inference, policy, trace, and review concepts +- re-entry as a first-class object +- typed memory rather than a generic notes store +- PostgreSQL + pgvector architecture +- structured AI modules +- WHOOP ingestion +- model/retrieval audit data models +- calibration data model +- reproducible migration history + +## Important gaps + +### Trust boundary + +The FastAPI surface does not currently implement a complete authentication and authorization boundary. + +The code should be treated as local/private prototype software. + +### Operator ownership + +Many lookups use bare object IDs rather than enforcing ownership by operator. Nested routes also historically allowed path IDs and body IDs to disagree. + +### Vector isolation + +Semantic retrieval historically allowed operator scoping to be omitted in some paths. + +### Inference failure semantics + +Several AI modules historically convert parse failure into low-confidence fallback domain values. Future architecture should preserve failure as failure. + +### Incomplete control loop + +The main assessment loop currently uses evidence and execution services, but does not automatically integrate all of Direction, stored Physiology, Memory, Re-entry, and Calibration. + +### Calibration + +Calibration objects are persisted, but calibration does not yet automatically change later model/rule weighting. + +### Audit wiring + +Model-run and retrieval-run tables exist, but model invocations are not automatically written to them. + +### Materialized views + +Three materialized views exist, but the repository does not implement an automatic refresh mechanism. + +### OAuth + +OAuth/token storage models exist, but a complete encrypted OAuth lifecycle and refresh flow is not implemented. WHOOP sync currently accepts an access token directly. + +### Verification + +The test suite covers important core paths but does not yet cover the full bounded-context surface. + +## Cleanup rule + +The redesign should fix high-confidence correctness problems while preserving the repository as an honest historical artifact. + +It should not retroactively claim production readiness. From 09cb24549c657a59bf32e42a4f92b6da9e2d4869 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:37:58 +0100 Subject: [PATCH 11/63] docs: document historical trust boundary --- SECURITY.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 SECURITY.md diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..29c298b --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,45 @@ +# Security + +## Status + +This repository contains a **historical private-system prototype**. + +It is not production-hardened and should not be exposed directly to an untrusted network. + +## Current trust model + +The historical FastAPI implementation does not contain a complete authentication or authorization boundary. + +Until that is redesigned: + +- bind services to a trusted local/private interface only +- do not expose the API publicly +- do not use production credentials +- do not store irreplaceable private data in an unreviewed deployment +- keep provider credentials in environment variables, never in the repository + +## Sensitive data + +The architecture can contain highly sensitive information, including: + +- subjective check-ins +- behavior traces +- physiological data +- biomarker information +- reasoning artifacts +- imported personal knowledge +- future neurophysiological observations + +A future deployment should treat these data classes as private by default. + +## Future sensor integrations + +Planned sensor-agnostic architecture may support sources such as EEG devices. + +Raw neurophysiological data should not automatically be forwarded to external model providers. Sensor data should be normalized locally into explicit observations/features with provenance and retention controls. + +## Medical boundary + +This project is not a medical diagnostic system. + +Physiological and neurophysiological signals should be treated as imperfect evidence about operational state, not as diagnostic truth. From 0ff6d8db6dd6bf5fbbc39937c3ae19921ae88a24 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:05 +0100 Subject: [PATCH 12/63] docs: clean Neurocognitive System README --- README.md | 498 ++++++++++++++++++------------------------------------ 1 file changed, 166 insertions(+), 332 deletions(-) diff --git a/README.md b/README.md index ba32de3..e5c0bc2 100644 --- a/README.md +++ b/README.md @@ -3,184 +3,181 @@ > **Historical codename: Utopia** > **Status: historical / inactive / preserved** -Neurocognitive System was an experimental, backend-first architecture for state-aware cognitive control. +Neurocognitive System is a historical experiment in **multimodal cognitive-state estimation, continuity, reasoning, and adaptive control**. -It was built around one assumption: +It began from one assumption: **the operator is not stable.** -Attention changes. Energy changes. Context disappears. Re-entry has a cost. Present-state narratives can be wrong. A system that assumes a permanently consistent operator will eventually recommend the wrong thing at the wrong depth. +Attention changes. Energy changes. Context disappears. Re-entry has a cost. Self-report is useful but imperfect. Physiological and behavioral signals are useful but imperfect. The intended architecture was therefore not a task manager or an AI that simply "knows" the user. -The experiment asked whether software could preserve continuity by maintaining explicit models of direction, evidence, state, blockers, policy, action, outcome, and calibration. - -Its central loop was: +It was a system for maintaining explicit, testable hypotheses about the operator's current state and deciding what depth of action was appropriate. ```text Direction - ↓ + ↓ +Sensing + ↓ +Observations + ↓ +Features + ↓ Evidence - ↓ -State + Blocker Estimate - ↓ + ↓ +Inference + ↓ Policy - ↓ + ↓ Action / Re-entry - ↓ -Trace - ↓ -Review / Calibration - ↺ + ↓ +Outcome + ↓ +Calibration + ↺ ``` -The project is no longer actively developed as a product. - -It remains public because several architectural ideas explored here continued to recur in later systems. - **The implementation is historical. The questions were not.** --- -## The problem - -Most productivity software assumes a relatively stable operator. - -If the correct task is visible, the user can execute it. - -If context is lost, they can reload it. - -If they stop moving, the problem is assumed to be motivation, planning, or discipline. +## What this repository contains -This system explored a different model. - -The failure may be one of continuity. - -The operator may still know what matters in the abstract while temporarily losing access to the context, state, energy, or cognitive depth required to act on it. - -That produces a different set of questions: - -- What direction actually matters right now? -- What evidence describes the operator's current condition? -- What state is the operator in? -- What is actually blocking motion? -- What action depth is appropriate for that state? -- What is the smallest correct next move? -- How can interrupted work be re-entered without reconstructing everything? -- What did reality reveal after the action? -- Was the system's previous estimate actually correct? - -The goal was not to maximize visible activity. +The historical implementation includes: -The goal was to preserve direction and choose the smallest defensible move given the available evidence. +- Python 3.12+ +- FastAPI +- PostgreSQL 16 + pgvector +- SQLAlchemy + Alembic +- 13 database migrations +- typed ORM and Pydantic models +- directional hierarchy: life arcs → seasons → missions → threads +- subjective, behavioral, contextual, and derived evidence +- WHOOP ingestion for sleep, recovery, cycles, workouts, and body measurements +- state estimation, blocker classification, and policy selection +- typed memory and semantic retrieval +- structured problem reasoning +- re-entry artifacts and execution traces +- review and calibration records +- model/retrieval audit data models +- materialized read models +- service, AI-module, and API tests + +The Python package remains named `utopia` so the historical implementation stays intact. --- -## Not a task manager - -Neurocognitive System was not designed as a conventional productivity application. - -It was not primarily: +## Multimodal intent -- a to-do list -- a note-taking system -- a mood tracker -- a habit app -- a diary -- a generic second brain -- a chatbot over personal data +WHOOP was the first implemented external sensor, not the intended endpoint. -The architecture treated cognition and execution as a control problem. +The longer-term design was sensor-agnostic. It was meant to accept additional evidence sources such as: -A recommendation was not supposed to be the end of the loop. +- behavioral telemetry +- environment/context sensors +- additional wearables +- biomarkers +- neurophysiological interfaces such as EEG devices, including systems in the class of Neurosity -It was a **claim** about what the operator should do next. +The goal was **not** to make any sensor authoritative. -That claim could later be compared with the outcome. +The intended model was: ```text -observation → inference → intervention → outcome → calibration +imperfect sensor + ↓ +observation + ↓ +derived feature + ↓ +evidence + ↓ +state hypothesis + ↓ +policy + ↓ +outcome + ↓ +calibration ``` -That feedback loop is the important part. +A future EEG integration would therefore contribute observations or derived features. It would not directly declare a cognitive state or diagnosis. --- ## Architecture -The repository separates the system into bounded contexts rather than storing everything inside generic notes or chat history. +The cleaned architecture is documented in: -### Vector +- [Architecture overview](docs/architecture/overview.md) +- [RFC-0001: Neurocognitive System](docs/rfcs/RFC-0001-neurocognitive-system.md) +- [RFC-0002: Observation, Evidence, and Inference](docs/rfcs/RFC-0002-observation-evidence-inference.md) +- [Historical terminology](docs/history/terminology.md) +- [Static codebase audit](docs/audit/2026-09-23.md) -Directional control. +The original Utopia documents are preserved unchanged under [docs/history](docs/history/README.md). -Vector models what matters across different horizons: +--- -```text -life arc → season → mission → thread -``` +## Core primitives -It exists to prevent motion from being mistaken for progress. +### Direction -### Evidence +Historical name: **Vector**. -The sensing layer. +Direction represents what matters: -It captures subjective, behavioral, contextual, and derived evidence about the operator's present condition. +```text +life arc → season → mission → thread +``` -The governing principle was: +It exists to distinguish useful motion from drift. -**evidence before narrative.** +### Sensing and Evidence -The system should not automatically treat a present-state self-interpretation as ground truth. +The system records subjective, behavioral, contextual, and physiological signals. -### Physiology +The cleaned architecture separates: -A physiological evidence channel. +**observation ≠ evidence ≠ inference** -The implementation includes WHOOP ingestion for cycles, sleep, recovery, workouts, body measurements, and derived physiology features. +A measurement is not automatically a conclusion. -Physiological data was intended to constrain interpretation, not define identity. +### Inference -### Execution +The historical AI Fabric estimates things such as: -The action-control layer. +- current operating state +- dominant blocker +- contradictions between narrative and evidence -Execution stores: +Inference should remain explicit about confidence, provenance, and failure. -- state estimates -- blocker estimates -- policy decisions -- re-entry artifacts -- traces +### Policy -The important chain is: +Historical name: **Schrödinger**. -```text -evidence → inference → policy → outcome -``` +Policy selects an intervention and action depth based on available evidence and inferred state. -A policy decision can be traced back to the estimates that produced it, and a later trace can record what actually happened. +A policy decision is a hypothesis about what to try next, not ground truth. ### Re-entry -Interruptions were treated as normal rather than exceptional. +Interrupted work is treated as normal. -A re-entry artifact preserves enough state to restart a thread without reconstructing the entire problem: +A re-entry artifact preserves: - last completed step - unresolved edge - smallest next move - trap to avoid - relevant context +- freshness -Re-entry was modeled as a first-class object because continuity loss was considered part of the system, not an edge case. +### Memory -### Aether +Historical name: **Aether**. -Typed memory and knowledge. - -Aether was designed to preserve reusable cognitive objects rather than accumulate an undifferentiated archive. - -Its model includes concepts such as: +The system stores typed cognitive objects rather than treating all memory as notes: - concepts - mechanisms @@ -192,289 +189,126 @@ Its model includes concepts such as: - cases - rules - patterns -- explicit edges between them - -The intention was to make memory useful for future judgment rather than merely searchable. - -### Reasoning - -Structured problem representation. - -The reasoning layer stores artifacts such as: - -- problems -- problem structures -- interrogations -- decision briefs -- option paths -- contradiction reports - -The underlying idea was that the system should improve the shape of the problem before rushing to solve it. - -### Review and Calibration - -The system's correction layer. - -Review stores: - -- closures -- review sessions -- rule promotions -- pattern updates -- calibration records - -Calibration records were designed to compare earlier estimates with later outcomes. - -That makes it possible, at least architecturally, for the system to ask: - -**Was my previous interpretation actually accurate?** - -### System Audit - -The AI and retrieval audit layer records model runs, retrieval runs, events, and outbox state so that reasoning does not disappear into an opaque chat transcript. - ---- - -## AI Fabric - -The implementation contains nine reasoning modules. - -### Core execution pipeline - -- **State Estimator** — estimates the operator's current operating state. -- **Blocker Classifier** — identifies the dominant reason motion is failing. -- **Policy Selector** — chooses an intervention and action depth appropriate to the estimated state. - -### Extended reasoning - -- **Router** — classifies incoming intent and chooses the appropriate reasoning path. -- **Problem Structurer** — turns raw problems into explicit objectives, constraints, unknowns, assumptions, and bottlenecks. -- **Context Retriever** — retrieves relevant material from the typed knowledge graph. -- **Physiology Interpreter** — translates physiological signals into bounded capacity information. -- **Contradiction Checker** — compares narrative against behavior, physiology, direction, and temporal patterns. -- **Council** — runs multiple reasoning lenses and synthesizes agreement and tension. - -Claude is used as the reasoning backend for these modules. OpenAI embeddings are used for semantic retrieval. - -The model layer was intended to remain subordinate to the typed system around it. +- explicit graph edges ---- - -## What was actually built +### Feedback and Calibration -This repository is more than an architecture document. - -The historical implementation includes: +The historical implementation records traces, reviews, rule promotions, pattern updates, and calibration records. -- Python 3.12+ -- FastAPI -- SQLAlchemy 2.x async -- PostgreSQL 16 -- pgvector -- Alembic -- 13 database migrations -- typed ORM models and Pydantic schemas -- bounded service layers -- API routes across the major contexts -- WHOOP API client, mapping, and synchronization -- semantic vector search -- Claude-backed reasoning modules -- OpenAI embedding support -- system-level model/retrieval audit records -- three materialized views for current state, active focus, and thread priority -- service, reasoning-module, and API-route tests - -The major bounded contexts represented in code are: +The intended end-state was a real feedback loop: ```text -core -integration -vector_ctrl -evidence -execution -physiology -aether -reasoning -review -system -vector +prediction → action → outcome → error → reliability update ``` ---- - -## Implementation vs. architecture - -The architecture documents in this repository describe a larger intended system than the implementation reached. - -That distinction is deliberate. - -Some ideas were implemented deeply enough to have migrations, models, services, routes, and tests. - -Others remained design directions. - -The documents are preserved because they show the evolution of the system's reasoning, but they should not be read as claims that every proposed subsystem became a finished product. - -There is no attempt here to rewrite the historical record into a cleaner story than it was. +The historical code models this loop more completely than it executes it. --- -## Architectural principles - -Several ideas mattered more than any individual feature. - -### State before planning - -The correct next action depends on the state of the operator executing it. - -### Evidence before narrative - -Self-interpretation is evidence, not unquestionable truth. +## What is implemented vs. what was intended -### Direction before motion +This repository intentionally distinguishes implementation from architecture. -Activity is not inherently progress. +### Implemented -### Re-entry as a first-class object +Substantial backend/domain infrastructure exists for: -A system designed for interruption should preserve the minimum state required to resume. +- Direction / Vector +- Evidence +- Execution +- Physiology +- Aether +- Reasoning +- Review +- System Audit +- vector search +- WHOOP ingestion +- AI reasoning modules -### Typed memory over accumulation +### Incomplete or architectural -If everything is a note, the system loses semantic structure. +Examples include: -### Explicit confidence and provenance +- a complete authentication/authorization boundary +- automatic model/retrieval audit wiring +- a full encrypted OAuth/token lifecycle +- automatic materialized-view refresh +- calibration automatically changing future model/rule weighting +- the complete cross-domain control loop +- general sensor-adapter abstractions +- EEG / Neurosity integration -Inferred state should remain distinguishable from observed fact. - -### Outcomes over persuasive explanations - -A recommendation that sounds intelligent is still only a hypothesis until reality answers. - -### Calibration over permanent assumptions - -The system should be able to compare previous estimates with subsequent outcomes and update accordingly. - -### Separation of observation, judgment, and execution - -Sensing reality, interpreting reality, and acting on reality are different operations and should remain inspectable. +These are documented as unfinished rather than presented as completed features. --- -## Why preserve this repository? - -Neurocognitive System is not the architecture I would build unchanged today. +## Trust boundary -That is part of its value. +**Do not expose the historical API directly to an untrusted network.** -It preserves an earlier attempt to formalize several problems that continued to matter: +The current implementation was built as private prototype software and does not contain a complete authentication and authorization boundary. -- continuity across interrupted work -- explicit system state -- evidence-backed inference -- re-entry cost -- typed memory -- confidence and provenance -- contradiction detection -- action selection under variable capacity -- outcome traces -- calibration -- review as a correction mechanism - -Looking backward, the vocabulary changed more than the underlying questions. - -The repository is useful as architecture archaeology: a record of how those ideas were represented before later systems refined them. - -**The implementation is historical. The questions were not.** +See [SECURITY.md](SECURITY.md). --- -## Historical terminology +## Medical boundary -Some names in the code and architecture documents are preserved because they were part of the original system. +Neurocognitive System is **not a medical diagnostic system**. -- **Utopia** — original project codename -- **Vector** — directional control plane -- **Aether** — typed knowledge and memory -- **Schrödinger** — original name for the policy-selection / "one correct move" concept - -These names should be read as historical terminology, not as a recommendation for how the same primitives would necessarily be named today. +Physiological or future neurophysiological inputs are treated as imperfect evidence about operational state and usable capacity. They are not a basis for diagnosing medical or psychiatric conditions. --- -## Repository structure +## Repository layout ```text +src/utopia/ + api/ # historical FastAPI surface + models/ # SQLAlchemy domain model + schemas/ # Pydantic contracts + services/ # bounded-context services + integrations/whoop/ # implemented sensor integration + ai/ # historical reasoning runtime + migrations/ - versions/ # 13 historical Alembic migrations + versions/ # 13 historical migrations -src/utopia/ - api/ # FastAPI application and routes - models/ # SQLAlchemy domain models - schemas/ # Pydantic request/response contracts - services/ # bounded-context service layer - integrations/ - whoop/ # WHOOP client, mapping, sync - ai/ # reasoning modules and providers - config.py - db.py - enums.py +docs/ + architecture/ # cleaned architecture + rfcs/ # redesign contracts + audit/ # static audit record + history/ # original Utopia design material + adhd-visual-prosthetic-thesis.md tests/ - conftest.py test_services.py test_ai_modules.py test_routes.py - -docs/ - adhd-visual-prosthetic-thesis.md - -Utopia Architecture.md -Utopia Formal Architecture DB etc.md ``` --- -## Local setup +## Local inspection -This is a historical repository, so setup instructions are preserved primarily for inspection and experimentation. - -### Requirements - -- Python 3.12+ -- Docker / Docker Compose -- PostgreSQL 16 with pgvector - -### Environment - -Copy the example environment file and provide only the integrations you intend to use: +This repository is preserved primarily for architecture inspection and experimentation. ```bash cp .env.example .env -``` - -Available configuration includes PostgreSQL, WHOOP, Anthropic, and OpenAI credentials. - -### Database - -```bash docker compose up -d +pip install -e ".[dev]" alembic upgrade head +pytest ``` -### Development - -Install the project and run the test suite using the configuration in `pyproject.toml`. +Provider keys are optional unless exercising the corresponding external integrations. --- ## Status -**Historical / inactive / preserved.** - -There is no active product roadmap for this repository. - -Parts of the architecture may be extracted, rewritten, or reappear elsewhere, but this repository represents the system as it existed during this experiment. +There is no active product roadmap for the historical Utopia application. -It is preserved as evidence of the design process, including ideas that worked, ideas that did not, and primitives that survived into later thinking. +The cleanup exists to expose the durable architectural primitives without rewriting the prototype into something it never was. **The implementation is historical. The questions were not.** From 4b07fc02690127538bd28934c9c5eef244ad1ad5 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:08 +0100 Subject: [PATCH 13/63] build: align package metadata and migration driver --- pyproject.toml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 3b15ad3..aae63b7 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,7 +1,7 @@ [project] name = "utopia" version = "0.1.0" -description = "Private cognitive operating system and judgment refinery" +description = "Historical multimodal neurocognitive state, continuity, and adaptive-control architecture" requires-python = ">=3.12" dependencies = [ "fastapi>=0.115.0", @@ -9,6 +9,7 @@ dependencies = [ "sqlalchemy[asyncio]>=2.0.36", "alembic>=1.14.0", "asyncpg>=0.30.0", + "psycopg[binary]>=3.2.0", "pgvector>=0.3.6", "pydantic>=2.10.0", "pydantic-settings>=2.6.0", From 8292885e42525488ee172328f792bea760df0703 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:11 +0100 Subject: [PATCH 14/63] docs: expose historical Neurocognitive System identity --- src/utopia/api/app.py | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/src/utopia/api/app.py b/src/utopia/api/app.py index 8dc51ee..1a05316 100644 --- a/src/utopia/api/app.py +++ b/src/utopia/api/app.py @@ -1,7 +1,4 @@ -"""Utopia FastAPI application. - -Minimal bootstrap — routes added per bounded context as they are built. -""" +"""Historical Neurocognitive System (Utopia) FastAPI application.""" from fastapi import FastAPI @@ -17,8 +14,11 @@ from utopia.api.routes.vector_search import router as vector_search_router app = FastAPI( - title="Utopia", - description="Private cognitive operating system and judgment refinery", + title="Neurocognitive System", + description=( + "Historical Utopia prototype: multimodal cognitive-state, continuity, " + "reasoning, and adaptive-control architecture. Not production hardened." + ), version="0.1.0", ) @@ -36,4 +36,8 @@ @app.get("/health") async def health() -> dict: - return {"status": "ok"} + return { + "status": "ok", + "system": "neurocognitive-system", + "historical_codename": "utopia", + } From fdb9343bf4a656e1bec18be76d97d5260da0a4c1 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:14 +0100 Subject: [PATCH 15/63] fix: use declared sync database driver --- src/utopia/config.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/utopia/config.py b/src/utopia/config.py index 15c3c56..09bb70b 100644 --- a/src/utopia/config.py +++ b/src/utopia/config.py @@ -1,4 +1,4 @@ -"""Utopia configuration loaded from environment variables.""" +"""Historical Utopia runtime configuration.""" from pydantic_settings import BaseSettings @@ -6,9 +6,9 @@ class Settings(BaseSettings): model_config = {"env_file": ".env", "env_file_encoding": "utf-8"} - # PostgreSQL (async driver for app, sync driver for alembic) + # PostgreSQL: async application driver + synchronous Alembic driver. database_url: str = "postgresql+asyncpg://utopia:utopia@localhost:5432/utopia" - database_url_sync: str = "postgresql+psycopg2://utopia:utopia@localhost:5432/utopia" + database_url_sync: str = "postgresql+psycopg://utopia:utopia@localhost:5432/utopia" # WHOOP integration whoop_client_id: str = "" From 7402bb441d8a7445517e0911ef3c8dc757722ed2 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:17 +0100 Subject: [PATCH 16/63] fix: align sync database example --- .env.example | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.env.example b/.env.example index 2fb57f8..ca082d4 100644 --- a/.env.example +++ b/.env.example @@ -1,12 +1,12 @@ # PostgreSQL DATABASE_URL=postgresql+asyncpg://utopia:utopia@localhost:5432/utopia -DATABASE_URL_SYNC=postgresql+psycopg2://utopia:utopia@localhost:5432/utopia +DATABASE_URL_SYNC=postgresql+psycopg://utopia:utopia@localhost:5432/utopia # WHOOP (placeholders — requires developer account) WHOOP_CLIENT_ID= WHOOP_CLIENT_SECRET= WHOOP_REDIRECT_URI=http://localhost:8000/integrations/whoop/callback -# AI Providers (placeholders) +# AI providers (placeholders) ANTHROPIC_API_KEY= OPENAI_API_KEY= From 777213ea5bb8d5fd3e63a42497f423abc8c4e81f Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:22 +0100 Subject: [PATCH 17/63] fix: align Alembic driver --- alembic.ini | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/alembic.ini b/alembic.ini index d76a706..9efd1f7 100644 --- a/alembic.ini +++ b/alembic.ini @@ -1,6 +1,6 @@ [alembic] script_location = migrations -sqlalchemy.url = postgresql+psycopg2://utopia:utopia@localhost:5432/utopia +sqlalchemy.url = postgresql+psycopg://utopia:utopia@localhost:5432/utopia [loggers] keys = root,sqlalchemy,alembic From 5403d96a888b3a9b5d06b7949826e69b94dcff48 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:25 +0100 Subject: [PATCH 18/63] fix: source Alembic URL from runtime settings --- migrations/env.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/migrations/env.py b/migrations/env.py index f0492e0..54ae4d8 100644 --- a/migrations/env.py +++ b/migrations/env.py @@ -5,12 +5,18 @@ from alembic import context from sqlalchemy import engine_from_config, pool +from utopia.config import settings + config = context.config + +# Keep one source of truth for migration connectivity. The historical +# alembic.ini value remains a local fallback, while .env can override it. +config.set_main_option("sqlalchemy.url", settings.database_url_sync) + if config.config_file_name is not None: fileConfig(config.config_file_name) -# We don't use autogenerate in this project — migrations are hand-written -# to match the architecture docs precisely. +# Historical migrations are hand-written; autogenerate is intentionally disabled. target_metadata = None From 254257674b6bd5e89e88b47a2f83a9026b582553 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:27 +0100 Subject: [PATCH 19/63] fix: preserve zero-valued evidence --- src/utopia/ai/assess.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/utopia/ai/assess.py b/src/utopia/ai/assess.py index e4a8672..ada865e 100644 --- a/src/utopia/ai/assess.py +++ b/src/utopia/ai/assess.py @@ -98,9 +98,9 @@ async def gather_evidence( evidence["derived_features"] = [ { "feature_name": f.feature_name, - "feature_value": str(f.feature_value) if f.feature_value else None, + "feature_value": str(f.feature_value) if f.feature_value is not None else None, "feature_window": f.feature_window, - "confidence": str(f.confidence) if f.confidence else None, + "confidence": str(f.confidence) if f.confidence is not None else None, } for f in features ] From 4d9ffda061976750ea6689965110dec9121e1051 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:51 +0100 Subject: [PATCH 20/63] fix: enforce nested resource identifiers --- src/utopia/api/routes/aether.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/utopia/api/routes/aether.py b/src/utopia/api/routes/aether.py index bc19b1c..b1a1112 100644 --- a/src/utopia/api/routes/aether.py +++ b/src/utopia/api/routes/aether.py @@ -75,6 +75,8 @@ async def add_source_chunk( data: SourceChunkCreate, svc: AetherService = Depends(get_aether_service), ) -> SourceChunkRead: + if data.source_id != source_id: + raise HTTPException(status_code=422, detail="source_id must match the path") chunk = await svc.add_source_chunk(data) await svc.commit() return SourceChunkRead.model_validate(chunk) @@ -86,6 +88,8 @@ async def record_extraction( data: ExtractionCreate, svc: AetherService = Depends(get_aether_service), ) -> ExtractionRead: + if data.source_id != source_id: + raise HTTPException(status_code=422, detail="source_id must match the path") extraction = await svc.record_extraction(data) await svc.commit() return ExtractionRead.model_validate(extraction) @@ -229,6 +233,8 @@ async def add_lens_pack_item( data: LensPackItemCreate, svc: AetherService = Depends(get_aether_service), ) -> LensPackItemRead: + if data.lens_pack_id != lens_pack_id: + raise HTTPException(status_code=422, detail="lens_pack_id must match the path") item = await svc.add_lens_pack_item(data) await svc.commit() return LensPackItemRead.model_validate(item) From a3476fc7893fb8c07e4b8290a9e2f6706e3d23f4 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:39:55 +0100 Subject: [PATCH 21/63] fix: enforce nested resource identifiers --- src/utopia/api/routes/reasoning.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/utopia/api/routes/reasoning.py b/src/utopia/api/routes/reasoning.py index 19c3308..ea7b5ef 100644 --- a/src/utopia/api/routes/reasoning.py +++ b/src/utopia/api/routes/reasoning.py @@ -68,6 +68,8 @@ async def create_problem_structure( data: ProblemStructureCreate, svc: ReasoningService = Depends(get_reasoning_service), ) -> ProblemStructureRead: + if data.problem_id != problem_id: + raise HTTPException(status_code=422, detail="problem_id must match the path") structure = await svc.create_problem_structure(data) await svc.commit() return ProblemStructureRead.model_validate(structure) @@ -94,6 +96,8 @@ async def create_interrogation( data: InterrogationCreate, svc: ReasoningService = Depends(get_reasoning_service), ) -> InterrogationRead: + if data.problem_id != problem_id: + raise HTTPException(status_code=422, detail="problem_id must match the path") interrogation = await svc.create_interrogation(data) await svc.commit() return InterrogationRead.model_validate(interrogation) @@ -118,6 +122,8 @@ async def create_decision_brief( data: DecisionBriefCreate, svc: ReasoningService = Depends(get_reasoning_service), ) -> DecisionBriefRead: + if data.problem_id != problem_id: + raise HTTPException(status_code=422, detail="problem_id must match the path") brief = await svc.create_decision_brief(data) await svc.commit() return DecisionBriefRead.model_validate(brief) @@ -144,6 +150,8 @@ async def add_option_path( data: OptionPathCreate, svc: ReasoningService = Depends(get_reasoning_service), ) -> OptionPathRead: + if data.decision_brief_id != brief_id: + raise HTTPException(status_code=422, detail="decision_brief_id must match the path") option = await svc.add_option_path(data) await svc.commit() return OptionPathRead.model_validate(option) From 1f0b437e4dd4ee498ff5a15789f89c9bb082742a Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:40:00 +0100 Subject: [PATCH 22/63] fix: enforce nested resource identifiers --- src/utopia/api/routes/review.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/utopia/api/routes/review.py b/src/utopia/api/routes/review.py index fc7b3a0..8e99eb4 100644 --- a/src/utopia/api/routes/review.py +++ b/src/utopia/api/routes/review.py @@ -108,6 +108,8 @@ async def record_rule_promotion( data: RulePromotionCreate, svc: ReviewService = Depends(get_review_service), ) -> RulePromotionRead: + if data.review_session_id != session_id: + raise HTTPException(status_code=422, detail="review_session_id must match the path") promotion = await svc.record_rule_promotion(data) await svc.commit() return RulePromotionRead.model_validate(promotion) @@ -139,6 +141,8 @@ async def record_pattern_update( data: PatternUpdateCreate, svc: ReviewService = Depends(get_review_service), ) -> PatternUpdateRead: + if data.review_session_id != session_id: + raise HTTPException(status_code=422, detail="review_session_id must match the path") update = await svc.record_pattern_update(data) await svc.commit() return PatternUpdateRead.model_validate(update) From 31c2910f7ecfbba005f09ac301fd980012710449 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:40:03 +0100 Subject: [PATCH 23/63] test: cover nested resource identifier integrity --- tests/test_routes.py | 45 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/tests/test_routes.py b/tests/test_routes.py index 1a02830..6e7b500 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -327,3 +327,48 @@ async def _mock(*, system, user, max_tokens, **kwargs): data = resp.json() assert len(data["perspectives"]) == 4 assert data["decision_readiness"] == "defer" + + +# --------------------------------------------------------------------------- +# Nested resource integrity +# --------------------------------------------------------------------------- + +class TestNestedResourceIntegrity: + + @pytest.mark.asyncio + async def test_aether_source_path_must_match_body(self, client): + path_id = uuid.uuid4() + body_id = uuid.uuid4() + resp = await client.post( + f"/aether/sources/{path_id}/chunks", + json={ + "source_id": str(body_id), + "chunk_index": 0, + "raw_text": "test", + }, + ) + assert resp.status_code == 422 + + @pytest.mark.asyncio + async def test_reasoning_problem_path_must_match_body(self, client): + path_id = uuid.uuid4() + body_id = uuid.uuid4() + resp = await client.post( + f"/reasoning/problems/{path_id}/structure", + json={"problem_id": str(body_id)}, + ) + assert resp.status_code == 422 + + @pytest.mark.asyncio + async def test_review_session_path_must_match_body(self, client): + path_id = uuid.uuid4() + body_id = uuid.uuid4() + resp = await client.post( + f"/review/sessions/{path_id}/rule-promotions", + json={ + "operator_id": str(uuid.uuid4()), + "review_session_id": str(body_id), + "rule_id": str(uuid.uuid4()), + }, + ) + assert resp.status_code == 422 From 2b65fc5a6a8729c8a41e5d0b529ca37771351621 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:40:48 +0100 Subject: [PATCH 24/63] docs: define sensor adapter architecture --- docs/architecture/sensors.md | 120 +++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 docs/architecture/sensors.md diff --git a/docs/architecture/sensors.md b/docs/architecture/sensors.md new file mode 100644 index 0000000..ee81a42 --- /dev/null +++ b/docs/architecture/sensors.md @@ -0,0 +1,120 @@ +# Sensor architecture + +The cleaned Neurocognitive System treats sensing as a replaceable boundary. + +The historical implementation contains WHOOP-specific ingestion. Future sensing should be implemented through a common adapter contract. + +## Design rule + +A sensor adapter emits **observations**. + +It does not emit diagnoses, identity claims, or policy decisions. + +```text +device / source + ↓ +adapter + ↓ +observation + ↓ +feature derivation + ↓ +evidence bundle + ↓ +inference +``` + +## Proposed SensorAdapter contract + +A future adapter should expose capabilities such as: + +- source identity +- device/session identity +- observation types +- time range +- sampling metadata +- freshness +- provenance +- optional raw-artifact reference + +Conceptually: + +```python +class SensorAdapter(Protocol): + source_kind: str + + async def capabilities(self) -> SensorCapabilities: ... + async def collect(self, request: CollectionRequest) -> list[Observation]: ... +``` + +The exact implementation is intentionally deferred. + +## High-frequency sources + +High-frequency sources such as EEG should not be represented as one PostgreSQL row per raw sample. + +The intended split is: + +```text +SensorSession + ↓ +raw artifact / stream + ↓ +feature windows + ↓ +normalized observations + ↓ +evidence +``` + +Raw signal artifacts can live in private object/file storage. PostgreSQL should store identity, provenance, time ranges, checksums, and derived features required by the control loop. + +## WHOOP + +WHOOP is the historical first sensor implementation. + +In the cleaned architecture it should eventually become an adapter that produces observations such as: + +- recovery +- sleep +- strain/cycle +- workout +- body measurement + +WHOOP-derived values remain evidence inputs, not policy authority. + +## EEG / neurophysiology + +The historical direction included future neurophysiological sensing, including devices in the class of Neurosity EEG. + +A future EEG adapter should preserve: + +- device identity +- session identity +- sample-rate/channel metadata +- time range +- processing pipeline/version +- raw-artifact checksum/reference +- derived feature definitions +- feature-window timestamps +- quality/confidence indicators + +EEG-derived features must not be translated directly into medical or psychiatric conclusions. + +The useful question for this architecture is operational: + +> Does this signal improve prediction of usable cognitive state or action depth for this operator? + +If repeated outcomes show that a signal is not useful, the calibration layer should reduce its weight. + +## Privacy + +Raw neurophysiological data is a sensitive data class. + +Default policy should be: + +- private/local storage +- explicit retention +- no automatic external-model upload +- derived features preferred over raw streams for routine inference +- provenance retained for every transformation From 2d3370e23295cc8c64d1a1d40f97504a32ee3cd3 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:40:52 +0100 Subject: [PATCH 25/63] docs: add sensor adapter RFC --- docs/rfcs/RFC-0003-sensor-adapter-contract.md | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 docs/rfcs/RFC-0003-sensor-adapter-contract.md diff --git a/docs/rfcs/RFC-0003-sensor-adapter-contract.md b/docs/rfcs/RFC-0003-sensor-adapter-contract.md new file mode 100644 index 0000000..16dcb55 --- /dev/null +++ b/docs/rfcs/RFC-0003-sensor-adapter-contract.md @@ -0,0 +1,48 @@ +# RFC-0003: Sensor Adapter Contract + +**Status:** Draft + +## Motivation + +The historical Utopia implementation contains WHOOP-specific integration logic, while the intended system was multimodal. + +A general sensor contract prevents the core architecture from becoming coupled to one wearable or one modality. + +## Contract boundary + +Adapters convert external sources into normalized observations. + +They MUST NOT: + +- declare operator state +- choose policy +- infer diagnosis +- silently discard provenance + +They SHOULD expose: + +- source kind +- source/device identifier +- collection/session identifier +- observed-at timestamps or ranges +- raw value/payload or artifact reference +- units +- quality metadata +- adapter/version metadata + +## High-frequency data + +High-frequency streams such as EEG should use a session + artifact + feature-window model instead of row-per-sample storage in the relational core. + +## Historical implementation + +WHOOP remains the first concrete integration. + +Future examples may include: + +- EEG systems such as Neurosity-class devices +- additional wearables +- environment sensors +- software/behavior telemetry + +These are extension points, not claims of current implementation. From 612d079cf13780a2fef88c505574dcdab9568e4a Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:41:00 +0100 Subject: [PATCH 26/63] docs: link sensor architecture --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index e5c0bc2..4ba2dcb 100644 --- a/README.md +++ b/README.md @@ -111,6 +111,8 @@ The cleaned architecture is documented in: - [Architecture overview](docs/architecture/overview.md) - [RFC-0001: Neurocognitive System](docs/rfcs/RFC-0001-neurocognitive-system.md) - [RFC-0002: Observation, Evidence, and Inference](docs/rfcs/RFC-0002-observation-evidence-inference.md) +- [Sensor architecture](docs/architecture/sensors.md) +- [RFC-0003: Sensor Adapter Contract](docs/rfcs/RFC-0003-sensor-adapter-contract.md) - [Historical terminology](docs/history/terminology.md) - [Static codebase audit](docs/audit/2026-09-23.md) From ae0fff26b4be1c4b1adf76e0683f2120aebd8470 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:41:05 +0100 Subject: [PATCH 27/63] test: align health metadata --- tests/test_routes.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/test_routes.py b/tests/test_routes.py index 6e7b500..1a90b9b 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -37,7 +37,7 @@ class TestHealth: async def test_health(self, client): resp = await client.get("/health") assert resp.status_code == 200 - assert resp.json() == {"status": "ok"} + assert resp.json() == {\n "status": "ok",\n "system": "neurocognitive-system",\n "historical_codename": "utopia",\n } # --------------------------------------------------------------------------- From f941647f85a1872c2aae21c6dd58030f2fb8b2e3 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:41:36 +0100 Subject: [PATCH 28/63] fix: correct health test formatting --- tests/test_routes.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/test_routes.py b/tests/test_routes.py index 1a90b9b..f177fda 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -37,7 +37,11 @@ class TestHealth: async def test_health(self, client): resp = await client.get("/health") assert resp.status_code == 200 - assert resp.json() == {\n "status": "ok",\n "system": "neurocognitive-system",\n "historical_codename": "utopia",\n } + assert resp.json() == { + "status": "ok", + "system": "neurocognitive-system", + "historical_codename": "utopia", + } # --------------------------------------------------------------------------- From 74fe7fbe360828bb8dedb46b6b4541a7decdefdf Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:42:35 +0100 Subject: [PATCH 29/63] fix: scope vector retrieval by operator --- src/utopia/services/vector_search_service.py | 31 ++++++++++++-------- 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/src/utopia/services/vector_search_service.py b/src/utopia/services/vector_search_service.py index a4ff20f..e567fcf 100644 --- a/src/utopia/services/vector_search_service.py +++ b/src/utopia/services/vector_search_service.py @@ -79,6 +79,7 @@ async def embed_entity( # Check for existing embedding with same hash stmt = ( select(Embedding) + .where(Embedding.operator_id == operator_id) .where(Embedding.entity_kind == entity_kind) .where(Embedding.entity_id == entity_id) ) @@ -137,6 +138,7 @@ async def embed_entities_batch( content_hash = _hash_content(entity["content_text"]) stmt = ( select(Embedding) + .where(Embedding.operator_id == operator_id) .where(Embedding.entity_kind == entity["entity_kind"]) .where(Embedding.entity_id == entity["entity_id"]) ) @@ -187,11 +189,12 @@ async def embed_entities_batch( return results async def delete_embedding( - self, entity_kind: str, entity_id: _uuid.UUID + self, operator_id: _uuid.UUID, entity_kind: str, entity_id: _uuid.UUID ) -> None: - """Delete an embedding for an entity.""" + """Delete an embedding owned by an operator.""" stmt = ( delete(Embedding) + .where(Embedding.operator_id == operator_id) .where(Embedding.entity_kind == entity_kind) .where(Embedding.entity_id == entity_id) ) @@ -205,7 +208,7 @@ async def search( self, query: str, *, - operator_id: _uuid.UUID | None = None, + operator_id: _uuid.UUID, entity_kinds: list[str] | None = None, top_k: int = 10, ) -> list[SearchResult]: @@ -213,7 +216,7 @@ async def search( Args: query: The natural language search query. - operator_id: Optional filter by operator. + operator_id: Operator that owns the searchable memory. entity_kinds: Optional filter to specific entity types. top_k: Number of results to return. @@ -226,12 +229,8 @@ async def search( # 1 - cosine_distance = cosine_similarity vector_literal = f"[{','.join(str(v) for v in query_vector)}]" - filters = [] - params: dict = {"top_k": top_k} - - if operator_id is not None: - filters.append("operator_id = :operator_id") - params["operator_id"] = str(operator_id) + filters = ["operator_id = :operator_id"] + params: dict = {"top_k": top_k, "operator_id": str(operator_id)} if entity_kinds: placeholders = ", ".join(f":ek_{i}" for i in range(len(entity_kinds))) @@ -268,6 +267,7 @@ async def search( async def find_similar( self, + operator_id: _uuid.UUID, entity_kind: str, entity_id: _uuid.UUID, *, @@ -281,6 +281,7 @@ async def find_similar( # Get the entity's embedding stmt = ( select(Embedding) + .where(Embedding.operator_id == operator_id) .where(Embedding.entity_kind == entity_kind) .where(Embedding.entity_id == entity_id) ) @@ -299,14 +300,20 @@ async def find_similar( content_text, 1 - (embedding <=> '{vector_literal}'::vector) AS similarity FROM vector.embeddings - WHERE NOT (entity_kind = :src_kind AND entity_id = :src_id) + WHERE operator_id = :operator_id + AND NOT (entity_kind = :src_kind AND entity_id = :src_id) ORDER BY embedding <=> '{vector_literal}'::vector LIMIT :top_k """ result = await self._session.execute( text(sql), - {"src_kind": entity_kind, "src_id": str(entity_id), "top_k": top_k}, + { + "operator_id": str(operator_id), + "src_kind": entity_kind, + "src_id": str(entity_id), + "top_k": top_k, + }, ) rows = result.fetchall() From f2c509a1cf38b73a654c03f8817910be88cea868 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:42:39 +0100 Subject: [PATCH 30/63] fix: require operator scope for vector API --- src/utopia/api/routes/vector_search.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/utopia/api/routes/vector_search.py b/src/utopia/api/routes/vector_search.py index 942c07d..9ee122f 100644 --- a/src/utopia/api/routes/vector_search.py +++ b/src/utopia/api/routes/vector_search.py @@ -39,7 +39,7 @@ class EmbeddingRead(BaseModel): class SearchRequest(BaseModel): query: str - operator_id: uuid.UUID | None = None + operator_id: uuid.UUID entity_kinds: list[str] | None = None top_k: int = Field(default=10, ge=1, le=100) @@ -52,6 +52,7 @@ class SearchResultRead(BaseModel): class SimilarRequest(BaseModel): + operator_id: uuid.UUID entity_kind: str entity_id: uuid.UUID top_k: int = Field(default=5, ge=1, le=50) @@ -144,7 +145,7 @@ async def find_similar( ) -> list[SearchResultRead]: """Find entities similar to a given entity.""" results = await svc.find_similar( - data.entity_kind, data.entity_id, top_k=data.top_k + data.operator_id, data.entity_kind, data.entity_id, top_k=data.top_k ) return [ SearchResultRead( From 0a4f435ca422f5fde2cb46a0a0d6ba5109f4df67 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:42:43 +0100 Subject: [PATCH 31/63] fix: require operator scope for context retrieval --- src/utopia/ai/context_retriever.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/utopia/ai/context_retriever.py b/src/utopia/ai/context_retriever.py index 0324442..af9f5fc 100644 --- a/src/utopia/ai/context_retriever.py +++ b/src/utopia/ai/context_retriever.py @@ -65,7 +65,7 @@ async def retrieve_context( vector_search_svc: VectorSearchService, query: str, *, - operator_id: uuid.UUID | None = None, + operator_id: uuid.UUID, entity_kinds: list[str] | None = None, top_k: int = 10, synthesize: bool = True, @@ -75,7 +75,7 @@ async def retrieve_context( Args: vector_search_svc: The vector search service for semantic retrieval. query: Natural language query to search for. - operator_id: Optional filter by operator. + operator_id: Operator whose memory may be retrieved. entity_kinds: Optional filter to specific Aether entity types. top_k: Number of raw results to retrieve. synthesize: Whether to run Claude synthesis on results (set False for raw search). From 8fd55945d67005a90ab5dab7f7a124ed44d00e58 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:42:45 +0100 Subject: [PATCH 32/63] fix: require operator scope for AI retrieval --- src/utopia/api/routes/ai.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/utopia/api/routes/ai.py b/src/utopia/api/routes/ai.py index c407d28..c8a5c6e 100644 --- a/src/utopia/api/routes/ai.py +++ b/src/utopia/api/routes/ai.py @@ -173,7 +173,7 @@ async def structure_problem( class RetrieveRequest(BaseModel): query: str - operator_id: uuid.UUID | None = None + operator_id: uuid.UUID entity_kinds: list[str] | None = None top_k: int = 10 synthesize: bool = True From f4eb3653530a6b1faa7014ae33f23719b2cc4a07 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:42:50 +0100 Subject: [PATCH 33/63] db: scope embedding identity by operator --- .../014_operator_scoped_embeddings.py | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 migrations/versions/014_operator_scoped_embeddings.py diff --git a/migrations/versions/014_operator_scoped_embeddings.py b/migrations/versions/014_operator_scoped_embeddings.py new file mode 100644 index 0000000..6ed1126 --- /dev/null +++ b/migrations/versions/014_operator_scoped_embeddings.py @@ -0,0 +1,43 @@ +"""M014: scope vector embedding identity by operator. + +Revision ID: 014 +Revises: 013 +""" +from typing import Sequence, Union + +from alembic import op + +revision: str = "014" +down_revision: Union[str, None] = "013" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.drop_index( + "ix_vector_embeddings_entity", + table_name="embeddings", + schema="vector", + ) + op.create_index( + "ix_vector_embeddings_entity", + "embeddings", + ["operator_id", "entity_kind", "entity_id"], + unique=True, + schema="vector", + ) + + +def downgrade() -> None: + op.drop_index( + "ix_vector_embeddings_entity", + table_name="embeddings", + schema="vector", + ) + op.create_index( + "ix_vector_embeddings_entity", + "embeddings", + ["entity_kind", "entity_id"], + unique=True, + schema="vector", + ) From 6389a0e8f556521e461c22ab312b9d90a15c9862 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:43:41 +0100 Subject: [PATCH 34/63] ci: verify migrations and tests --- .github/workflows/ci.yml | 53 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 .github/workflows/ci.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 0000000..03eec31 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,53 @@ +name: CI + +on: + push: + branches: + - main + - "cleanup/**" + pull_request: + +jobs: + test: + runs-on: ubuntu-latest + + services: + postgres: + image: pgvector/pgvector:pg16 + env: + POSTGRES_USER: utopia + POSTGRES_PASSWORD: utopia + POSTGRES_DB: utopia_test + ports: + - 5432:5432 + options: >- + --health-cmd "pg_isready -U utopia -d utopia_test" + --health-interval 5s + --health-timeout 3s + --health-retries 10 + + env: + DATABASE_URL: postgresql+asyncpg://utopia:utopia@localhost:5432/utopia_test + DATABASE_URL_SYNC: postgresql+psycopg://utopia:utopia@localhost:5432/utopia_test + ANTHROPIC_API_KEY: test + OPENAI_API_KEY: test + + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: pip + + - name: Install + run: python -m pip install -e ".[dev]" + + - name: Compile + run: python -m compileall -q src tests migrations + + - name: Apply migrations + run: alembic upgrade head + + - name: Test + run: pytest -q From f41cf386d9d2c12b07b92cfe45d7abe6f39a1f4b Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:43:47 +0100 Subject: [PATCH 35/63] docs: distinguish historical and cleanup migrations --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 4ba2dcb..9a885eb 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ The historical implementation includes: - FastAPI - PostgreSQL 16 + pgvector - SQLAlchemy + Alembic -- 13 database migrations +- 13 historical database migrations + 1 cleanup migration - typed ORM and Pydantic models - directional hierarchy: life arcs → seasons → missions → threads - subjective, behavioral, contextual, and derived evidence @@ -274,7 +274,7 @@ src/utopia/ ai/ # historical reasoning runtime migrations/ - versions/ # 13 historical migrations + versions/ # historical migrations + cleanup migrations docs/ architecture/ # cleaned architecture From 6d37b4100eebda2ead9499210429c11bc280b31d Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:51:46 +0100 Subject: [PATCH 36/63] fix: reuse predeclared PostgreSQL enums in migrations --- migrations/versions/003_integration_tables.py | 6 +++--- migrations/versions/004_vector_ctrl_tables.py | 16 ++++++++-------- migrations/versions/006_execution_tables.py | 14 +++++++------- migrations/versions/010_review_tables.py | 6 +++--- migrations/versions/011_system_audit_tables.py | 4 ++-- 5 files changed, 23 insertions(+), 23 deletions(-) diff --git a/migrations/versions/003_integration_tables.py b/migrations/versions/003_integration_tables.py index 3905bdc..97db0c8 100644 --- a/migrations/versions/003_integration_tables.py +++ b/migrations/versions/003_integration_tables.py @@ -12,7 +12,7 @@ from alembic import op import sqlalchemy as sa -from sqlalchemy.dialects.postgresql import ARRAY, BYTEA, JSONB +from sqlalchemy.dialects.postgresql import ARRAY, BYTEA, ENUM, JSONB revision: str = "003" down_revision: Union[str, None] = "002" @@ -31,7 +31,7 @@ def upgrade() -> None: sa.Column("scopes", ARRAY(sa.Text()), nullable=False, server_default="{}"), sa.Column( "status", - sa.Enum("active", "revoked", "expired", name="oauth_status", schema="core", create_type=False), + ENUM("active", "revoked", "expired", name="oauth_status", schema="core", create_type=False), nullable=False, ), sa.Column("encrypted_access_token", BYTEA(), nullable=True), @@ -81,7 +81,7 @@ def upgrade() -> None: sa.Column("processed_at", sa.DateTime(timezone=True), nullable=True), sa.Column( "processing_status", - sa.Enum("pending", "processed", "failed", name="processing_status", schema="core", create_type=False), + ENUM("pending", "processed", "failed", name="processing_status", schema="core", create_type=False), nullable=False, server_default="pending", ), diff --git a/migrations/versions/004_vector_ctrl_tables.py b/migrations/versions/004_vector_ctrl_tables.py index 662cf39..c5bac0a 100644 --- a/migrations/versions/004_vector_ctrl_tables.py +++ b/migrations/versions/004_vector_ctrl_tables.py @@ -13,7 +13,7 @@ from alembic import op import sqlalchemy as sa -from sqlalchemy.dialects.postgresql import ARRAY, JSONB +from sqlalchemy.dialects.postgresql import ARRAY, ENUM, JSONB revision: str = "004" down_revision: Union[str, None] = "003" @@ -31,7 +31,7 @@ def upgrade() -> None: sa.Column("description", sa.Text(), nullable=True), sa.Column( "status", - sa.Enum("active", "paused", "completed", "abandoned", "archived", "dormant", + ENUM("active", "paused", "completed", "abandoned", "archived", "dormant", name="status", schema="core", create_type=False), nullable=False, ), @@ -61,7 +61,7 @@ def upgrade() -> None: sa.Column("priority_stack", JSONB(), nullable=False, server_default="[]"), sa.Column( "status", - sa.Enum("planned", "active", "closed", + ENUM("planned", "active", "closed", name="season_status", schema="core", create_type=False), nullable=False, ), @@ -84,14 +84,14 @@ def upgrade() -> None: sa.Column("description", sa.Text(), nullable=True), sa.Column( "mission_kind", - sa.Enum("strategic", "technical", "personal", "recovery", "exploratory", + ENUM("strategic", "technical", "personal", "recovery", "exploratory", name="mission_kind", schema="core", create_type=False), nullable=False, ), sa.Column("priority_score", sa.Numeric(6, 3), nullable=True), sa.Column( "status", - sa.Enum("active", "paused", "completed", "abandoned", "archived", "dormant", + ENUM("active", "paused", "completed", "abandoned", "archived", "dormant", name="status", schema="core", create_type=False), nullable=False, ), @@ -122,13 +122,13 @@ def upgrade() -> None: sa.Column("description", sa.Text(), nullable=True), sa.Column( "thread_kind", - sa.Enum("build", "research", "decision", "admin", "recovery", + ENUM("build", "research", "decision", "admin", "recovery", name="thread_kind", schema="core", create_type=False), nullable=False, ), sa.Column( "status", - sa.Enum("active", "blocked", "paused", "closed", + ENUM("active", "blocked", "paused", "closed", name="thread_status", schema="core", create_type=False), nullable=False, ), @@ -163,7 +163,7 @@ def upgrade() -> None: sa.Column("description", sa.Text(), nullable=False), sa.Column( "hardness", - sa.Enum("hard", "soft", "assumed", + ENUM("hard", "soft", "assumed", name="constraint_hardness", schema="core", create_type=False), nullable=False, ), diff --git a/migrations/versions/006_execution_tables.py b/migrations/versions/006_execution_tables.py index 9287669..f3fb8f1 100644 --- a/migrations/versions/006_execution_tables.py +++ b/migrations/versions/006_execution_tables.py @@ -18,7 +18,7 @@ from alembic import op import sqlalchemy as sa -from sqlalchemy.dialects.postgresql import JSONB +from sqlalchemy.dialects.postgresql import ENUM, JSONB revision: str = "006" down_revision: Union[str, None] = "005" @@ -35,7 +35,7 @@ def upgrade() -> None: sa.Column("thread_id", sa.Uuid(), sa.ForeignKey("vector_ctrl.threads.id"), nullable=True), sa.Column( "state_kind", - sa.Enum( + ENUM( "recover", "preserve", "orient", "clarify", "reenter", "execute", "deep_work", "close_loop", "review", "drift", name="state_kind", schema="core", create_type=False, @@ -66,7 +66,7 @@ def upgrade() -> None: sa.Column("thread_id", sa.Uuid(), sa.ForeignKey("vector_ctrl.threads.id"), nullable=True), sa.Column( "blocker_kind", - sa.Enum( + ENUM( "ambiguity", "scope_overload", "physiological_depletion", "emotional_threat", "context_fracture", "vector_conflict", "environmental_friction", "stimulation_hijack", "narrative_distortion", @@ -138,7 +138,7 @@ def upgrade() -> None: sa.Column("blocker_estimate_id", sa.Uuid(), sa.ForeignKey("execution.blocker_estimates.id"), nullable=True), sa.Column( "mode", - sa.Enum( + ENUM( "recover", "preserve", "orient", "reenter", "clarify", "ask", "execute", "close_loop", "review", name="intervention_kind", schema="core", create_type=False, @@ -147,7 +147,7 @@ def upgrade() -> None: ), sa.Column( "intervention_kind", - sa.Enum( + ENUM( "recover", "preserve", "orient", "reenter", "clarify", "ask", "execute", "close_loop", "review", name="intervention_kind", schema="core", create_type=False, @@ -156,7 +156,7 @@ def upgrade() -> None: ), sa.Column( "action_depth", - sa.Enum( + ENUM( "tiny", "narrow", "moderate", "deep", name="action_depth", schema="core", create_type=False, ), @@ -189,7 +189,7 @@ def upgrade() -> None: sa.Column("policy_decision_id", sa.Uuid(), sa.ForeignKey("execution.policy_decisions.id"), nullable=True), sa.Column( "trace_kind", - sa.Enum( + ENUM( "action", "question", "preserve", "recovery", "closure", name="trace_kind", schema="core", create_type=False, ), diff --git a/migrations/versions/010_review_tables.py b/migrations/versions/010_review_tables.py index 65e5bc1..79de72c 100644 --- a/migrations/versions/010_review_tables.py +++ b/migrations/versions/010_review_tables.py @@ -16,7 +16,7 @@ from alembic import op import sqlalchemy as sa -from sqlalchemy.dialects.postgresql import JSONB +from sqlalchemy.dialects.postgresql import ENUM, JSONB revision: str = "010" down_revision: Union[str, None] = "009" @@ -35,7 +35,7 @@ def upgrade() -> None: sa.Column("operator_id", sa.Uuid(), sa.ForeignKey("core.operators.id"), nullable=False), sa.Column("thread_id", sa.Uuid(), sa.ForeignKey("vector_ctrl.threads.id"), nullable=True), sa.Column("mission_id", sa.Uuid(), sa.ForeignKey("vector_ctrl.missions.id"), nullable=True), - sa.Column("closure_type", sa.Enum("complete", "archive", "pause", "merge", name="closure_type", schema="core", create_type=False), nullable=False), + sa.Column("closure_type", ENUM("complete", "archive", "pause", "merge", name="closure_type", schema="core", create_type=False), nullable=False), sa.Column("outcome_summary", sa.Text(), nullable=True), sa.Column("lessons_learned", JSONB(), nullable=False, server_default="[]"), sa.Column("truth_revealed", sa.Text(), nullable=True), @@ -65,7 +65,7 @@ def upgrade() -> None: "review_sessions", sa.Column("id", sa.Uuid(), primary_key=True), sa.Column("operator_id", sa.Uuid(), sa.ForeignKey("core.operators.id"), nullable=False), - sa.Column("review_scope", sa.Enum("micro", "daily", "weekly", "monthly", name="review_scope", schema="core", create_type=False), nullable=False), + sa.Column("review_scope", ENUM("micro", "daily", "weekly", "monthly", name="review_scope", schema="core", create_type=False), nullable=False), sa.Column("window_start", sa.DateTime(timezone=True), nullable=False), sa.Column("window_end", sa.DateTime(timezone=True), nullable=False), sa.Column("summary", sa.Text(), nullable=True), diff --git a/migrations/versions/011_system_audit_tables.py b/migrations/versions/011_system_audit_tables.py index cafc3fe..4e2c9a8 100644 --- a/migrations/versions/011_system_audit_tables.py +++ b/migrations/versions/011_system_audit_tables.py @@ -15,7 +15,7 @@ from alembic import op import sqlalchemy as sa -from sqlalchemy.dialects.postgresql import JSONB +from sqlalchemy.dialects.postgresql import ENUM, JSONB revision: str = "011" down_revision: Union[str, None] = "010" @@ -152,7 +152,7 @@ def upgrade() -> None: sa.Column("event_type", sa.Text(), nullable=False), sa.Column("destination", sa.Text(), nullable=False), sa.Column("payload", JSONB(), nullable=False, server_default="{}"), - sa.Column("status", sa.Enum("pending", "processed", "failed", name="processing_status", schema="core", create_type=False), nullable=False), + sa.Column("status", ENUM("pending", "processed", "failed", name="processing_status", schema="core", create_type=False), nullable=False), sa.Column("attempts", sa.Integer(), nullable=False, server_default="0"), sa.Column("last_error", sa.Text(), nullable=True), sa.Column("created_at", sa.DateTime(timezone=True), nullable=False, server_default=sa.text("now()")), From b3ff0b3cdfd781ea7c2b9dd1f9dab342514eb483 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:54:52 +0100 Subject: [PATCH 37/63] fix: normalize UUIDv7 values at service boundaries --- src/utopia/ids.py | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 src/utopia/ids.py diff --git a/src/utopia/ids.py b/src/utopia/ids.py new file mode 100644 index 0000000..9dd6292 --- /dev/null +++ b/src/utopia/ids.py @@ -0,0 +1,16 @@ +"""UUID helpers for the historical Utopia implementation. + +The project originally used uuid-utils for UUIDv7 generation while the ORM +and Pydantic schemas were typed against the standard-library uuid.UUID. +Normalize at generation time so UUIDv7 semantics are preserved without +leaking a second UUID runtime type through service boundaries. +""" + +import uuid + +from uuid_utils import uuid7 as _uuid7 + + +def new_uuid7() -> uuid.UUID: + """Generate a UUIDv7 represented as a standard-library UUID.""" + return uuid.UUID(str(_uuid7())) From d499b02bd8f72a61402a3fa7965342458ed3d046 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:54:58 +0100 Subject: [PATCH 38/63] fix: normalize generated UUIDv7 values --- src/utopia/services/vector_service.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/utopia/services/vector_service.py b/src/utopia/services/vector_service.py index 7804428..693c703 100644 --- a/src/utopia/services/vector_service.py +++ b/src/utopia/services/vector_service.py @@ -14,7 +14,7 @@ from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.models.vector_ctrl import ( AntiGoal, @@ -53,7 +53,7 @@ async def commit(self) -> None: async def create_life_arc(self, data: LifeArcCreate) -> LifeArc: arc = LifeArc( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, title=data.title, description=data.description, @@ -76,7 +76,7 @@ async def get_life_arc(self, life_arc_id: _uuid.UUID) -> LifeArc | None: async def create_season(self, data: SeasonCreate) -> Season: season = Season( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, life_arc_id=data.life_arc_id, title=data.title, @@ -105,7 +105,7 @@ async def create_mission(self, data: MissionCreate) -> Mission: are the fields that give Vector its governance power. """ mission = Mission( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, season_id=data.season_id, title=data.title, @@ -146,7 +146,7 @@ async def create_thread(self, data: ThreadCreate) -> Thread: Blocker Classifier, Schrodinger — use for policy selection. """ thread = Thread( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, mission_id=data.mission_id, parent_thread_id=data.parent_thread_id, @@ -184,7 +184,7 @@ async def list_threads( async def add_thread_constraint(self, data: ThreadConstraintCreate) -> ThreadConstraint: constraint = ThreadConstraint( - id=uuid7(), + id=new_uuid7(), thread_id=data.thread_id, constraint_type=data.constraint_type, description=data.description, @@ -205,7 +205,7 @@ async def create_anti_goal(self, data: AntiGoalCreate) -> AntiGoal: to detect drift and block misaligned action proposals. """ anti_goal = AntiGoal( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, scope_type=data.scope_type, scope_id=data.scope_id, From ff313f80e15ebc5463e214b96ce3eec9ec8beaba Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:00 +0100 Subject: [PATCH 39/63] fix: normalize generated UUIDv7 values --- src/utopia/services/aether_service.py | 34 +++++++++++++-------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/src/utopia/services/aether_service.py b/src/utopia/services/aether_service.py index d5b471e..a1f49fe 100644 --- a/src/utopia/services/aether_service.py +++ b/src/utopia/services/aether_service.py @@ -13,7 +13,7 @@ from sqlalchemy import select from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.models.aether import ( Case, @@ -72,7 +72,7 @@ async def commit(self) -> None: async def ingest_source(self, data: SourceCreate) -> Source: source = Source( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, source_kind=data.source_kind, title=data.title, @@ -93,7 +93,7 @@ async def get_source(self, source_id: _uuid.UUID) -> Source | None: async def add_source_chunk(self, data: SourceChunkCreate) -> SourceChunk: chunk = SourceChunk( - id=uuid7(), + id=new_uuid7(), source_id=data.source_id, chunk_index=data.chunk_index, raw_text=data.raw_text, @@ -110,7 +110,7 @@ async def record_extraction(self, data: ExtractionCreate) -> Extraction: stmt = ( pg_insert(Extraction) .values( - id=uuid7(), + id=new_uuid7(), source_id=data.source_id, extraction_version=data.extraction_version, extraction_status=data.extraction_status, @@ -146,7 +146,7 @@ async def create_concept(self, data: ConceptCreate) -> Concept: stmt = ( pg_insert(Concept) .values( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, canonical_name=data.canonical_name, definition=data.definition, @@ -182,7 +182,7 @@ async def list_concepts( async def create_mechanism(self, data: MechanismCreate) -> Mechanism: mechanism = Mechanism( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, name=data.name, description=data.description, @@ -196,7 +196,7 @@ async def create_mechanism(self, data: MechanismCreate) -> Mechanism: async def create_tradeoff(self, data: TradeoffCreate) -> Tradeoff: tradeoff = Tradeoff( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, name=data.name, pole_a=data.pole_a, @@ -210,7 +210,7 @@ async def create_tradeoff(self, data: TradeoffCreate) -> Tradeoff: async def create_failure_mode(self, data: FailureModeCreate) -> FailureMode: failure_mode = FailureMode( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, name=data.name, description=data.description, @@ -224,7 +224,7 @@ async def create_failure_mode(self, data: FailureModeCreate) -> FailureMode: async def create_heuristic(self, data: HeuristicCreate) -> Heuristic: heuristic = Heuristic( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, statement=data.statement, domain=data.domain, @@ -240,7 +240,7 @@ async def create_diagnostic_question( self, data: DiagnosticQuestionCreate ) -> DiagnosticQuestion: question = DiagnosticQuestion( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, question_text=data.question_text, question_class=data.question_class, @@ -253,7 +253,7 @@ async def create_diagnostic_question( async def create_protocol(self, data: ProtocolCreate) -> Protocol: protocol = Protocol( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, protocol_name=data.protocol_name, domain=data.domain, @@ -271,7 +271,7 @@ async def create_protocol(self, data: ProtocolCreate) -> Protocol: async def create_lens_pack(self, data: LensPackCreate) -> LensPack: pack = LensPack( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, name=data.name, domain=data.domain, @@ -288,7 +288,7 @@ async def get_lens_pack(self, lens_pack_id: _uuid.UUID) -> LensPack | None: async def add_lens_pack_item(self, data: LensPackItemCreate) -> LensPackItem: item = LensPackItem( - id=uuid7(), + id=new_uuid7(), lens_pack_id=data.lens_pack_id, item_kind=data.item_kind, item_id=data.item_id, @@ -315,7 +315,7 @@ async def list_lens_pack_items( async def create_case(self, data: CaseCreate) -> Case: case = Case( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, title=data.title, case_kind=data.case_kind, @@ -333,7 +333,7 @@ async def create_case(self, data: CaseCreate) -> Case: async def create_rule(self, data: RuleCreate) -> Rule: rule = Rule( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, rule_text=data.rule_text, rule_kind=data.rule_kind, @@ -381,7 +381,7 @@ async def list_rules( async def create_pattern(self, data: PatternCreate) -> Pattern: pattern = Pattern( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, pattern_name=data.pattern_name, description=data.description, @@ -420,7 +420,7 @@ async def list_patterns( async def create_edge(self, data: EdgeCreate) -> Edge: edge = Edge( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, src_kind=data.src_kind, dst_kind=data.dst_kind, From 1a5906f5778ce82cd2fc29a03bbb729b332505c1 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:03 +0100 Subject: [PATCH 40/63] fix: normalize generated UUIDv7 values --- src/utopia/services/review_service.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/utopia/services/review_service.py b/src/utopia/services/review_service.py index 2ce9ee0..f2eef11 100644 --- a/src/utopia/services/review_service.py +++ b/src/utopia/services/review_service.py @@ -12,7 +12,7 @@ from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.enums import ReviewScope from utopia.models.review import ( @@ -50,7 +50,7 @@ async def commit(self) -> None: async def create_closure(self, data: ClosureCreate) -> Closure: closure = Closure( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, mission_id=data.mission_id, @@ -95,7 +95,7 @@ async def create_review_session( self, data: ReviewSessionCreate ) -> ReviewSession: session = ReviewSession( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, review_scope=data.review_scope, window_start=data.window_start, @@ -138,7 +138,7 @@ async def record_rule_promotion( self, data: RulePromotionCreate ) -> RulePromotion: promotion = RulePromotion( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, review_session_id=data.review_session_id, rule_id=data.rule_id, @@ -174,7 +174,7 @@ async def record_pattern_update( self, data: PatternUpdateCreate ) -> PatternUpdate: update = PatternUpdate( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, review_session_id=data.review_session_id, pattern_id=data.pattern_id, @@ -212,7 +212,7 @@ async def record_calibration( self, data: CalibrationRecordCreate ) -> CalibrationRecord: record = CalibrationRecord( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, review_session_id=data.review_session_id, estimate_kind=data.estimate_kind, From d78c63d17df2e20429b526ff87d6c0e80ed011d5 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:06 +0100 Subject: [PATCH 41/63] fix: normalize generated UUIDv7 values --- src/utopia/services/evidence_service.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/utopia/services/evidence_service.py b/src/utopia/services/evidence_service.py index e6fc77d..edd88df 100644 --- a/src/utopia/services/evidence_service.py +++ b/src/utopia/services/evidence_service.py @@ -10,7 +10,7 @@ from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.models.evidence import ( BehaviorEvent, @@ -50,7 +50,7 @@ async def record_checkin(self, data: SubjectiveCheckinCreate) -> SubjectiveCheck is meaningful evidence for the State Estimator. """ checkin = SubjectiveCheckin( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, energy=data.energy, @@ -98,7 +98,7 @@ async def record_behavior_event(self, data: BehaviorEventCreate) -> BehaviorEven aversion. The Blocker Classifier consumes these. """ event = BehaviorEvent( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, event_type=data.event_type, @@ -142,7 +142,7 @@ async def record_context_snapshot(self, data: ContextSnapshotCreate) -> ContextS the context it needs to weight feasibility of action depth. """ snapshot = ContextSnapshot( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, local_time=data.local_time, @@ -182,7 +182,7 @@ async def store_derived_feature(self, data: DerivedFeatureCreate) -> DerivedFeat thread_decay_hours, drift_probability, completion_aversion_score. """ feature = DerivedFeature( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, feature_name=data.feature_name, From 06792cc83aa5311e338e789d1a508773e4797f1d Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:21 +0100 Subject: [PATCH 42/63] fix: normalize generated UUIDv7 values --- src/utopia/services/reasoning_service.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/src/utopia/services/reasoning_service.py b/src/utopia/services/reasoning_service.py index 212000f..686e074 100644 --- a/src/utopia/services/reasoning_service.py +++ b/src/utopia/services/reasoning_service.py @@ -13,7 +13,7 @@ from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.models.reasoning import ( ContradictionReport, @@ -52,7 +52,7 @@ async def commit(self) -> None: async def create_problem(self, data: ProblemCreate) -> Problem: problem = Problem( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, title=data.title, raw_prompt=data.raw_prompt, @@ -92,7 +92,7 @@ async def create_problem_structure( self, data: ProblemStructureCreate ) -> ProblemStructure: structure = ProblemStructure( - id=uuid7(), + id=new_uuid7(), problem_id=data.problem_id, objective=data.objective, stakes=data.stakes, @@ -130,7 +130,7 @@ async def get_problem_structure( async def create_interrogation(self, data: InterrogationCreate) -> Interrogation: interrogation = Interrogation( - id=uuid7(), + id=new_uuid7(), problem_id=data.problem_id, interrogation_kind=data.interrogation_kind, questions=data.questions, @@ -157,7 +157,7 @@ async def list_interrogations( async def create_decision_brief(self, data: DecisionBriefCreate) -> DecisionBrief: brief = DecisionBrief( - id=uuid7(), + id=new_uuid7(), problem_id=data.problem_id, classification=data.classification, summary=data.summary, @@ -190,7 +190,7 @@ async def get_decision_brief( async def add_option_path(self, data: OptionPathCreate) -> OptionPath: option = OptionPath( - id=uuid7(), + id=new_uuid7(), decision_brief_id=data.decision_brief_id, option_label=data.option_label, description=data.description, @@ -223,7 +223,7 @@ async def record_contradiction( self, data: ContradictionReportCreate ) -> ContradictionReport: report = ContradictionReport( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, problem_id=data.problem_id, contradiction_kind=data.contradiction_kind, From fba28e33e9506a78ee8b716349ec83fc45db7526 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:23 +0100 Subject: [PATCH 43/63] fix: normalize generated UUIDv7 values --- src/utopia/services/execution_service.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/utopia/services/execution_service.py b/src/utopia/services/execution_service.py index 86ece44..6ef0ef7 100644 --- a/src/utopia/services/execution_service.py +++ b/src/utopia/services/execution_service.py @@ -15,7 +15,7 @@ from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.models.execution import ( BlockerEstimate, @@ -58,7 +58,7 @@ async def record_state_estimate(self, data: StateEstimateCreate) -> StateEstimat downstream policy selection. """ estimate = StateEstimate( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, state_kind=data.state_kind, @@ -96,7 +96,7 @@ async def record_blocker_estimate(self, data: BlockerEstimateCreate) -> BlockerE blocker typing, the system recommends the wrong thing. """ estimate = BlockerEstimate( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, blocker_kind=data.blocker_kind, @@ -134,7 +134,7 @@ async def create_reentry_artifact(self, data: ReentryArtifactCreate) -> ReentryA unresolved edge, smallest next move, trap to avoid. """ artifact = ReentryArtifact( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, last_completed_step=data.last_completed_step, @@ -193,7 +193,7 @@ async def record_policy_decision(self, data: PolicyDecisionCreate) -> PolicyDeci AI Fabric exists, this can be created manually or by deterministic rules. """ decision = PolicyDecision( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, problem_id=data.problem_id, @@ -237,7 +237,7 @@ async def record_trace(self, data: TraceCreate) -> Trace: Model. Completes the chain: evidence -> inference -> policy -> outcome. """ trace = Trace( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, thread_id=data.thread_id, policy_decision_id=data.policy_decision_id, From da7b7ddade01e2493ba99dd34f3ec3d22cf5b833 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:26 +0100 Subject: [PATCH 44/63] fix: normalize generated UUIDv7 values --- src/utopia/services/physiology_service.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/src/utopia/services/physiology_service.py b/src/utopia/services/physiology_service.py index 477cf7f..8f0cc72 100644 --- a/src/utopia/services/physiology_service.py +++ b/src/utopia/services/physiology_service.py @@ -14,7 +14,7 @@ from sqlalchemy import select from sqlalchemy.dialects.postgresql import insert as pg_insert from sqlalchemy.ext.asyncio import AsyncSession -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.models.physiology import ( BiomarkerPanel, @@ -58,7 +58,7 @@ async def commit(self) -> None: async def connect_whoop(self, data: WhoopConnectionCreate) -> WhoopConnection: conn = WhoopConnection( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, oauth_connection_id=data.oauth_connection_id, scope_granted=data.scope_granted, @@ -79,7 +79,7 @@ async def record_body_measurement( self, data: WhoopBodyMeasurementCreate ) -> WhoopBodyMeasurement: measurement = WhoopBodyMeasurement( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, measured_at=data.measured_at, height_cm=data.height_cm, @@ -100,7 +100,7 @@ async def upsert_cycle(self, data: WhoopCycleCreate) -> WhoopCycle: stmt = ( pg_insert(WhoopCycle) .values( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, provider_cycle_id=data.provider_cycle_id, whoop_user_id=data.whoop_user_id, @@ -141,7 +141,7 @@ async def upsert_sleep(self, data: WhoopSleepCreate) -> WhoopSleep: stmt = ( pg_insert(WhoopSleep) .values( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, provider_sleep_id=data.provider_sleep_id, provider_cycle_id=data.provider_cycle_id, @@ -185,7 +185,7 @@ async def upsert_recovery(self, data: WhoopRecoveryCreate) -> WhoopRecovery: stmt = ( pg_insert(WhoopRecovery) .values( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, provider_cycle_id=data.provider_cycle_id, provider_sleep_id=data.provider_sleep_id, @@ -230,7 +230,7 @@ async def upsert_workout(self, data: WhoopWorkoutCreate) -> WhoopWorkout: stmt = ( pg_insert(WhoopWorkout) .values( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, provider_workout_id=data.provider_workout_id, workout_type=data.workout_type, @@ -268,7 +268,7 @@ async def store_physiology_feature( stmt = ( pg_insert(PhysiologyFeature) .values( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, feature_date=data.feature_date, feature_name=data.feature_name, @@ -317,7 +317,7 @@ async def record_biomarker_panel( self, data: BiomarkerPanelCreate ) -> BiomarkerPanel: panel = BiomarkerPanel( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, panel_date=data.panel_date, provider=data.provider, From e13258be49db76b39e2ba948d6fba20f6862c1c9 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:29 +0100 Subject: [PATCH 45/63] fix: normalize generated UUIDv7 values --- src/utopia/services/system_audit_service.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/utopia/services/system_audit_service.py b/src/utopia/services/system_audit_service.py index 66e3b0f..9bfc425 100644 --- a/src/utopia/services/system_audit_service.py +++ b/src/utopia/services/system_audit_service.py @@ -9,7 +9,7 @@ from sqlalchemy import select from sqlalchemy.ext.asyncio import AsyncSession -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.enums import ProcessingStatus from utopia.models.system_audit import ( @@ -49,7 +49,7 @@ async def create_model_provider( self, data: ModelProviderCreate ) -> ModelProvider: provider = ModelProvider( - id=uuid7(), + id=new_uuid7(), name=data.name, provider_kind=data.provider_kind, api_base_url=data.api_base_url, @@ -82,7 +82,7 @@ async def list_model_providers( async def record_model_run(self, data: ModelRunCreate) -> ModelRun: run = ModelRun( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, provider_id=data.provider_id, model_name=data.model_name, @@ -127,7 +127,7 @@ async def record_retrieval_run( self, data: RetrievalRunCreate ) -> RetrievalRun: run = RetrievalRun( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, model_run_id=data.model_run_id, collection=data.collection, @@ -170,7 +170,7 @@ async def list_retrieval_runs( async def log_event(self, data: EventLogCreate) -> EventLog: event = EventLog( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, event_type=data.event_type, source=data.source, @@ -211,7 +211,7 @@ async def list_events( async def create_outbox_event(self, data: OutboxEventCreate) -> OutboxEvent: event = OutboxEvent( - id=uuid7(), + id=new_uuid7(), operator_id=data.operator_id, event_type=data.event_type, destination=data.destination, From f9814bde0fab24b22bf7e563fcb2dc90cac14d20 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:37 +0100 Subject: [PATCH 46/63] fix: normalize generated UUIDv7 values --- src/utopia/services/vector_search_service.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/utopia/services/vector_search_service.py b/src/utopia/services/vector_search_service.py index e567fcf..726c6e9 100644 --- a/src/utopia/services/vector_search_service.py +++ b/src/utopia/services/vector_search_service.py @@ -13,7 +13,7 @@ from sqlalchemy import delete, select, text from sqlalchemy.ext.asyncio import AsyncSession -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.ai.providers import openai_embeddings from utopia.config import settings @@ -105,7 +105,7 @@ async def embed_entity( # Create new emb = Embedding( - id=uuid7(), + id=new_uuid7(), operator_id=operator_id, entity_kind=entity_kind, entity_id=entity_id, @@ -169,7 +169,7 @@ async def embed_entities_batch( results.append(existing) else: emb = Embedding( - id=uuid7(), + id=new_uuid7(), operator_id=operator_id, entity_kind=entity_data["entity_kind"], entity_id=entity_data["entity_id"], From 5d3d980cfe77747b5a66a2e059101baa8a264cc3 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:41 +0100 Subject: [PATCH 47/63] fix: normalize generated UUIDv7 values --- tests/test_services.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/test_services.py b/tests/test_services.py index 6f587cd..80327b2 100644 --- a/tests/test_services.py +++ b/tests/test_services.py @@ -15,7 +15,7 @@ from decimal import Decimal import pytest -from uuid_utils import uuid7 +from utopia.ids import new_uuid7 from utopia.enums import ( ActionDepth, @@ -244,14 +244,14 @@ async def test_reentry_artifact_lifecycle(self, db_session, execution_service, o from utopia.models.vector_ctrl import Mission, Thread mission = Mission( - id=uuid7(), operator_id=operator_id, title="Test Mission", + id=new_uuid7(), operator_id=operator_id, title="Test Mission", mission_kind=MissionKind.technical, status=Status.active, ) db_session.add(mission) await db_session.flush() thread = Thread( - id=uuid7(), operator_id=operator_id, mission_id=mission.id, + id=new_uuid7(), operator_id=operator_id, mission_id=mission.id, title="Test Thread", thread_kind=ThreadKind.build, status=ThreadStatus.active, ) db_session.add(thread) From a8e703eb6b01516e281bbba1d1c341d89881ee43 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:55:43 +0100 Subject: [PATCH 48/63] fix: default evidence timestamps to current UTC --- src/utopia/schemas/evidence.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/utopia/schemas/evidence.py b/src/utopia/schemas/evidence.py index 45d5f40..31c293b 100644 --- a/src/utopia/schemas/evidence.py +++ b/src/utopia/schemas/evidence.py @@ -32,7 +32,9 @@ class SubjectiveCheckinCreate(BaseModel): emotional_load: int | None = Field(None, ge=0, le=100) perceived_urgency: int | None = Field(None, ge=0, le=100) free_text: str | None = None - recorded_at: datetime.datetime + recorded_at: datetime.datetime = Field( + default_factory=lambda: datetime.datetime.now(datetime.timezone.utc) + ) class SubjectiveCheckinRead(BaseModel): @@ -136,7 +138,9 @@ class DerivedFeatureCreate(BaseModel): feature_json: dict | None = None feature_window: str | None = None confidence: Decimal | None = Field(None, ge=0, le=1) - observed_at: datetime.datetime + observed_at: datetime.datetime = Field( + default_factory=lambda: datetime.datetime.now(datetime.timezone.utc) + ) class DerivedFeatureRead(BaseModel): From 182d65d25cd7105f2278c94da90cbfe9cf8f0bbe Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:58:57 +0100 Subject: [PATCH 49/63] ci: harden release verification --- .github/workflows/ci.yml | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 03eec31..eaf4338 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -7,8 +7,26 @@ on: - "cleanup/**" pull_request: +permissions: + contents: read + jobs: + secrets: + name: Secret history scan + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - uses: gitleaks/gitleaks-action@v3 + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + GITLEAKS_ENABLE_COMMENTS: "false" + GITLEAKS_ENABLE_UPLOAD_ARTIFACT: "false" + test: + name: Python / migrations / tests runs-on: ubuntu-latest services: @@ -33,9 +51,9 @@ jobs: OPENAI_API_KEY: test steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v6 - - uses: actions/setup-python@v5 + - uses: actions/setup-python@v7 with: python-version: "3.12" cache: pip @@ -46,8 +64,16 @@ jobs: - name: Compile run: python -m compileall -q src tests migrations - - name: Apply migrations + - name: Ruff + run: ruff check . + + - name: Apply migrations from zero run: alembic upgrade head + - name: Verify cleanup migration is reversible + run: | + alembic downgrade 013 + alembic upgrade head + - name: Test run: pytest -q From 578b2e8f3fe84e6858f7379a0d1e0fda971332ae Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:59:05 +0100 Subject: [PATCH 50/63] test: refuse unidentified non-test databases --- tests/conftest.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/tests/conftest.py b/tests/conftest.py index 033697f..dd61326 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -12,21 +12,25 @@ import pytest from httpx import ASGITransport, AsyncClient +from sqlalchemy.engine import make_url from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker, create_async_engine from utopia.api.app import app from utopia.api.deps import get_db from utopia.config import settings -from utopia.db import Base # --------------------------------------------------------------------------- # Database fixtures # --------------------------------------------------------------------------- -# Use the same database URL but with a test-aware approach: -# In CI, point DATABASE_URL to a test database. -# Locally, this uses the configured database. TEST_DATABASE_URL = settings.database_url +TEST_DATABASE_NAME = make_url(TEST_DATABASE_URL).database or "" + +if not TEST_DATABASE_NAME.endswith("_test"): + raise RuntimeError( + "Refusing to run tests unless DATABASE_URL names a database ending in '_test'. " + "Set DATABASE_URL and DATABASE_URL_SYNC to an isolated test database." + ) @pytest.fixture(scope="session") From 467b41eeb3eab8fabd13d6f18ce2f280fbd261ed Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:59:48 +0100 Subject: [PATCH 51/63] security: bind local postgres to loopback --- docker-compose.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker-compose.yml b/docker-compose.yml index 2edddb3..bd98590 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -6,7 +6,7 @@ services: POSTGRES_PASSWORD: utopia POSTGRES_DB: utopia ports: - - "5432:5432" + - "127.0.0.1:5432:5432" volumes: - pgdata:/var/lib/postgresql/data healthcheck: From c4d5918018dc7e358882dcf4bc8a6f84eeb232ea Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 16:59:50 +0100 Subject: [PATCH 52/63] security: ignore common local sensitive artifacts --- .gitignore | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.gitignore b/.gitignore index 87d1481..4b2347b 100644 --- a/.gitignore +++ b/.gitignore @@ -8,8 +8,14 @@ build/ .venv/ venv/ .env +.env.* +!.env.example *.db *.sqlite3 +*.log +*.dump +*.sql +*.parquet .ruff_cache/ .pytest_cache/ .mypy_cache/ From 6cefc931e817bb2d72484526748199326f691d6c Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:00:24 +0100 Subject: [PATCH 53/63] ci: lint active code without rewriting historical migrations --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index eaf4338..3acb5cd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -65,7 +65,7 @@ jobs: run: python -m compileall -q src tests migrations - name: Ruff - run: ruff check . + run: ruff check src tests migrations/env.py migrations/versions/014_operator_scoped_embeddings.py - name: Apply migrations from zero run: alembic upgrade head From 2c0a8ba11f19f7e47b0d4258dc3d8b955cf0c711 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:01:13 +0100 Subject: [PATCH 54/63] test: cover vector isolation and zero-valued evidence --- tests/test_services.py | 98 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/tests/test_services.py b/tests/test_services.py index 80327b2..724085e 100644 --- a/tests/test_services.py +++ b/tests/test_services.py @@ -351,3 +351,101 @@ async def test_contradiction_report(self, db_session, reasoning_service, operato reports = await reasoning_service.list_contradictions(operator_id) assert len(reports) >= 1 + + +# --------------------------------------------------------------------------- +# Cleanup regression coverage +# --------------------------------------------------------------------------- + +class TestCleanupRegressions: + """Regression tests for public-release correctness fixes.""" + + @pytest.mark.asyncio + async def test_zero_valued_derived_evidence_is_preserved( + self, db_session, evidence_service, operator_id + ): + from utopia.ai.assess import gather_evidence + + await _insert_operator(db_session, operator_id) + await evidence_service.store_derived_feature( + DerivedFeatureCreate( + operator_id=operator_id, + feature_name="zero_signal", + feature_value=Decimal("0"), + confidence=Decimal("0"), + ) + ) + + evidence = await gather_evidence(evidence_service, operator_id) + feature = evidence["derived_features"][0] + + assert Decimal(feature["feature_value"]) == 0 + assert Decimal(feature["confidence"]) == 0 + + @pytest.mark.asyncio + async def test_vector_retrieval_is_operator_scoped( + self, + db_session, + vector_search_service, + mock_embeddings, + operator_id, + ): + other_operator_id = uuid.uuid4() + shared_entity_id = uuid.uuid4() + + await _insert_operator(db_session, operator_id) + await _insert_operator(db_session, other_operator_id) + + await vector_search_service.embed_entity( + operator_id, + "concept", + shared_entity_id, + "operator one memory", + ) + await vector_search_service.embed_entity( + other_operator_id, + "concept", + shared_entity_id, + "operator two memory", + ) + + own_results = await vector_search_service.search( + "memory", + operator_id=operator_id, + top_k=10, + ) + + assert [result.content_text for result in own_results] == ["operator one memory"] + + @pytest.mark.asyncio + async def test_find_similar_is_operator_scoped( + self, + db_session, + vector_search_service, + mock_embeddings, + operator_id, + ): + other_operator_id = uuid.uuid4() + source_id = uuid.uuid4() + + await _insert_operator(db_session, operator_id) + await _insert_operator(db_session, other_operator_id) + + await vector_search_service.embed_entity( + operator_id, "concept", source_id, "source memory" + ) + await vector_search_service.embed_entity( + operator_id, "concept", uuid.uuid4(), "same operator neighbor" + ) + await vector_search_service.embed_entity( + other_operator_id, "concept", uuid.uuid4(), "other operator neighbor" + ) + + results = await vector_search_service.find_similar( + operator_id, + "concept", + source_id, + top_k=10, + ) + + assert [result.content_text for result in results] == ["same operator neighbor"] From d37d09111c90f02ed2a92410b842422e8c4d08e5 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:01:57 +0100 Subject: [PATCH 55/63] chore: remove stale imports --- src/utopia/ai/assess.py | 5 ----- 1 file changed, 5 deletions(-) diff --git a/src/utopia/ai/assess.py b/src/utopia/ai/assess.py index ada865e..415a8a5 100644 --- a/src/utopia/ai/assess.py +++ b/src/utopia/ai/assess.py @@ -15,11 +15,6 @@ from typing import Any from utopia.ai import blocker_classifier, policy_selector, state_estimator -from utopia.schemas.execution import ( - BlockerEstimateCreate, - PolicyDecisionCreate, - StateEstimateCreate, -) from utopia.services.evidence_service import EvidenceService from utopia.services.execution_service import ExecutionService From 2b5a9d42626f25173f4bcaa817917c895038ed62 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:01:59 +0100 Subject: [PATCH 56/63] chore: remove stale imports --- src/utopia/ai/council.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/utopia/ai/council.py b/src/utopia/ai/council.py index f13ec50..70cb090 100644 --- a/src/utopia/ai/council.py +++ b/src/utopia/ai/council.py @@ -11,7 +11,6 @@ import json import logging -import uuid from dataclasses import dataclass, field from utopia.ai.providers import claude From e793b30fe69a7ff8223acf5a2f68c7afa6232762 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:02:02 +0100 Subject: [PATCH 57/63] chore: remove stale imports --- src/utopia/api/routes/vector_search.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/utopia/api/routes/vector_search.py b/src/utopia/api/routes/vector_search.py index 9ee122f..5d5904e 100644 --- a/src/utopia/api/routes/vector_search.py +++ b/src/utopia/api/routes/vector_search.py @@ -1,9 +1,8 @@ """Vector search routes — embedding management and semantic retrieval.""" import uuid -from typing import Any -from fastapi import APIRouter, Depends, Query +from fastapi import APIRouter, Depends from pydantic import BaseModel, Field from utopia.api.deps import get_vector_search_service From 1ff203b8642a06eeb3a5df7307b68e95c701f1bf Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:02:06 +0100 Subject: [PATCH 58/63] chore: remove stale imports --- tests/test_ai_modules.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/test_ai_modules.py b/tests/test_ai_modules.py index 7e6da9c..9ac9806 100644 --- a/tests/test_ai_modules.py +++ b/tests/test_ai_modules.py @@ -10,7 +10,6 @@ import json import uuid from decimal import Decimal -from unittest.mock import AsyncMock, patch import pytest From 93f5bbab501011ba5df7f74bc366fcd68f1be5a6 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:02:08 +0100 Subject: [PATCH 59/63] chore: remove stale imports --- tests/test_routes.py | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/test_routes.py b/tests/test_routes.py index f177fda..facf8f2 100644 --- a/tests/test_routes.py +++ b/tests/test_routes.py @@ -9,7 +9,6 @@ import json import uuid -from unittest.mock import AsyncMock, patch import pytest From e5acc6bdc4fe10c31bbfff1a8b610c054c859f6b Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:02:17 +0100 Subject: [PATCH 60/63] chore: modernize cleanup migration typing --- migrations/versions/014_operator_scoped_embeddings.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/migrations/versions/014_operator_scoped_embeddings.py b/migrations/versions/014_operator_scoped_embeddings.py index 6ed1126..d13c6ca 100644 --- a/migrations/versions/014_operator_scoped_embeddings.py +++ b/migrations/versions/014_operator_scoped_embeddings.py @@ -3,14 +3,14 @@ Revision ID: 014 Revises: 013 """ -from typing import Sequence, Union +from collections.abc import Sequence from alembic import op revision: str = "014" -down_revision: Union[str, None] = "013" -branch_labels: Union[str, Sequence[str], None] = None -depends_on: Union[str, Sequence[str], None] = None +down_revision: str | None = "013" +branch_labels: str | Sequence[str] | None = None +depends_on: str | Sequence[str] | None = None def upgrade() -> None: From b16665ec7f790fbe55b8ddc994fe0225c53efd48 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:02:20 +0100 Subject: [PATCH 61/63] ci: enforce focused correctness lint rules --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 3acb5cd..8bca057 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -65,7 +65,7 @@ jobs: run: python -m compileall -q src tests migrations - name: Ruff - run: ruff check src tests migrations/env.py migrations/versions/014_operator_scoped_embeddings.py + run: ruff check --select E4,E7,E9,F src tests migrations/env.py migrations/versions/014_operator_scoped_embeddings.py - name: Apply migrations from zero run: alembic upgrade head From 46fc5f07271995a5e908b73e9589bbff6b292ad6 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:25:01 +0100 Subject: [PATCH 62/63] docs: remove unrelated visual prosthetic thesis reference --- README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/README.md b/README.md index 9a885eb..08dcc62 100644 --- a/README.md +++ b/README.md @@ -281,7 +281,6 @@ docs/ rfcs/ # redesign contracts audit/ # static audit record history/ # original Utopia design material - adhd-visual-prosthetic-thesis.md tests/ test_services.py From 85ef5b365c097b5c78a1b703a831e1d99a761e85 Mon Sep 17 00:00:00 2001 From: sineth madduma <149978246+sinethxyz@users.noreply.github.com> Date: Wed, 23 Sep 2026 17:25:05 +0100 Subject: [PATCH 63/63] docs: remove unrelated visual prosthetic thesis --- docs/adhd-visual-prosthetic-thesis.md | 256 -------------------------- 1 file changed, 256 deletions(-) delete mode 100644 docs/adhd-visual-prosthetic-thesis.md diff --git a/docs/adhd-visual-prosthetic-thesis.md b/docs/adhd-visual-prosthetic-thesis.md deleted file mode 100644 index 7ba12d5..0000000 --- a/docs/adhd-visual-prosthetic-thesis.md +++ /dev/null @@ -1,256 +0,0 @@ -# ADHD as a Continuity Problem - -## A thesis for AR as a visual prosthetic - -### TL;DR - -ADHD is often framed as a motivation problem, a discipline problem, or a willpower problem. - -That is not the most useful frame. - -A better frame is this: - -**ADHD is, in part, a continuity failure problem.** - -The issue is not always knowing what matters. -The issue is keeping what matters perceptually alive long enough for action to remain aligned with intention. - -In that sense, AR glasses may become more than a display layer or convenience device. -They may become a **visual prosthetic for priority persistence**. - ---- - -## The core idea - -For many people with ADHD, the main objective does not remain stably present in working consciousness. - -The mind does not only get distracted by trivial things. -It gets captured by whatever becomes most immediate, most stimulating, most novel, or most visible. - -This creates a recurring pattern: - -1. The person knows the real priority. -2. Attention drifts. -3. A lower-value but more immediate task captures the system. -4. Time passes under false importance. -5. The person "returns" and realizes they were not acting on the true objective. - -This is not merely forgetfulness. -It is a breakdown in **priority persistence**. - -If the problem is partly perceptual, then part of the solution may also be perceptual. - ---- - -## Why vision matters - -Attention is heavily shaped by what remains in the visual field. - -What stays visible has a structural advantage over what must be re-remembered. -A priority that must be mentally reloaded is fragile. -A priority that is continuously and intelligently present in the line of sight is more stable. - -This suggests a simple but powerful hypothesis: - -**If the most important task, constraint, or objective could remain perceptually present at the edge of awareness, some forms of ADHD-related drift may be reduced.** - -Not cured. -Reduced. - -The mechanism is not magical. -It is environmental scaffolding applied directly to the attentional bottleneck. - ---- - -## AR glasses as a prosthetic layer - -Most productivity tools assume the user will remember to check them. - -That assumption breaks under attentional volatility. - -Phones, task managers, notes apps, and calendar alerts all rely on re-entry: - -- open the app -- remember the context -- reconstruct the priority -- resume motion - -For people with ADHD, that reload cost is often exactly where continuity breaks. - -AR changes the interface model. - -Instead of requiring repeated re-entry, the system can place the correct cue directly into perception. - -That means AR glasses could function as a prosthetic layer for: - -- main objective persistence -- task hierarchy visibility -- interruption recovery -- anti-drift signaling -- contextual reminders at the moment of deviation - -The point is not to fill vision with noise. -The point is to create a **minimal visual anchor** that protects the top priority from disappearing. - ---- - -## Prosthetic, not enhancement - -The word *prosthetic* matters. - -A prosthetic does not make someone superhuman. -It compensates for a recurring functional gap. - -In this case, the gap is not intelligence. -It is not effort. -It is not moral weakness. - -The gap is the unstable persistence of the correct target inside consciousness. - -So the proposition is not: - -"AR glasses make people more productive." - -The proposition is: - -**For some ADHD users, AR glasses may serve as an attentional prosthetic by keeping the correct target visually available when internal continuity fails.** - -That is a much stronger idea. - ---- - -## A simple operating model - -An ADHD visual prosthetic could work through a few basic rules. - -### 1. One objective, always visible -A single line in the display: - -- current objective -- current highest-value action -- current anti-goal - -Not ten tasks. -Not a dashboard. -One live anchor. - -### 2. Drift detection -If the user opens unrelated apps, changes context too often, stalls, or loops, the system infers drift. - -### 3. Re-entry prompts -When drift is detected, the display surfaces a tiny corrective cue: - -- `Main objective: finish X` -- `This is not the priority` -- `Return to the next smallest move` - -### 4. State-aware adaptation -If the user is depleted, overwhelmed, or fragmented, the system changes the prompt depth. - -Not: -- "Finish the project" - -But: -- "Open the file" -- "Write one sentence" -- "Send one reply" - -### 5. Constraint visibility -The display can also keep a live constraint present: - -- do not switch tasks -- no new tabs -- finish before exploring -- stay with the bottleneck - -This is less about reminder technology and more about **real-time cognitive stabilization**. - ---- - -## What this solves - -This kind of system may help with: - -- losing the real priority after interruptions -- false urgency replacing real importance -- hyperfixating on low-output side quests -- context collapse between intention and action -- friction of reloading the task stack from memory - -In plain English: - -It may help the user keep the true thing visible long enough to act on it. - ---- - -## What it does not solve - -This thesis should not be exaggerated. - -A visual prosthetic would not solve every dimension of ADHD. -It would not fix sleep, emotional dysregulation, under-stimulation, shame loops, avoidance, or deeper executive dysfunction by itself. - -But that is not a weakness of the idea. -It is exactly why the frame is useful. - -A good prosthetic does not need to solve everything. -It only needs to reliably support one broken part of the loop. - -And if that broken part is **priority persistence**, then the leverage may be enormous. - ---- - -## Why this matters now - -AR hardware is often marketed through entertainment, novelty, or generic future-tech language. - -But one of the strongest near-term categories may be cognitive assistance. -Not in the abstract. -Not as hype. -But in practical, narrow, repeatable use cases. - -ADHD is one of the clearest candidates because: - -- the problem is frequent -- the problem is daily -- the problem is experiential and felt -- the problem is partly perceptual and contextual -- the intervention can be minimal but continuous - -This makes AR less like a toy and more like interface infrastructure for cognition. - ---- - -## Product direction - -A serious prototype would likely need to be: - -- minimal rather than immersive -- persistent rather than flashy -- state-aware rather than static -- corrective rather than distracting -- personalized rather than generic - -The wrong product would feel like more noise. -The right product would feel like **a calm second layer of executive function**. - -That is the design challenge. - ---- - -## Final claim - -The real promise of AR for ADHD may not be information access. -It may be **objective persistence**. - -Not more data. -Not more stimulation. -Not another app. - -A perceptual support layer that keeps the right thing alive in the line of sight when the mind would otherwise drop it. - -That is why AR glasses may become a genuine visual prosthetic for some people with ADHD. - -And if that is true, then the category is much bigger than smart glasses. - -It is cognitive infrastructure.