docs: Design Philosophy — "imperfection by design"

README.md

@@ -2,7 +2,7 @@
 
 <h1>CogniFold: Always-On Proactive Memory<br/>via Cognitive Folding</h1>
 
-<a href="https://opennorve.github.io/CogniFold/" target="_blank"><img alt="Live Demo" src="https://img.shields.io/badge/%F0%9F%A7%A0_Live_Demo-opennorve.github.io-7C3AED?style=flat-square" height="22px"></a>
+<a href="https://opennorve.github.io/CogniFold/" target="_blank"><img alt="Live progress track" src="https://img.shields.io/badge/%F0%9F%A7%A0_Live_progress_track-opennorve.github.io-7C3AED?style=flat-square" height="22px"></a>
 <a href="https://opennorve.github.io/CogniFold/" target="_blank"><img alt="Brain Memory Coverage" src="https://img.shields.io/badge/Brain_Memory_Coverage-~60%25-8B5CF6?style=flat-square" height="22px"></a>
 <a href="https://arxiv.org/abs/2605.13438" target="_blank"><img alt="Paper" src="https://img.shields.io/badge/Technical_Report-arXiv-B31B1B?logo=arxiv&logoColor=white&style=flat-square" height="22px"></a>
 <a href="https://huggingface.co/datasets/OpenNorve/CogEval-Bench" target="_blank"><img alt="CogEval-Bench Dataset" src="https://img.shields.io/badge/Dataset-CogEval--Bench-FFD21E?logo=huggingface&logoColor=black&style=flat-square" height="22px"></a>
@@ -20,7 +20,7 @@
 
 <div align="center">
 
-<a href="https://opennorve.github.io/CogniFold/"><b>🌐 Live Demo</b></a> &nbsp;·&nbsp; <a href="docs/ARCHITECTURE.md"><b>🏗️ Architecture</b></a> &nbsp;·&nbsp; <a href="docs/DEPLOYMENT.md"><b>🚀 Deploy</b></a> &nbsp;·&nbsp; <a href="docs/INTEGRATIONS.md"><b>🔌 Integrations</b></a> &nbsp;·&nbsp; <a href="docs/PROMPTS.md"><b>✍️ Prompt Profiles</b></a> &nbsp;·&nbsp; <a href="docs/NORTH_STAR_METHODOLOGY.md"><b>🧭 North Star</b></a>
+<a href="https://opennorve.github.io/CogniFold/"><b>🌐 Live progress track</b></a> &nbsp;·&nbsp; <a href="docs/ARCHITECTURE.md"><b>🏗️ Architecture</b></a> &nbsp;·&nbsp; <a href="docs/DEPLOYMENT.md"><b>🚀 Deploy</b></a> &nbsp;·&nbsp; <a href="docs/INTEGRATIONS.md"><b>🔌 Integrations</b></a> &nbsp;·&nbsp; <a href="docs/PROMPTS.md"><b>✍️ Prompt Profiles</b></a> &nbsp;·&nbsp; <a href="docs/PHILOSOPHY.md"><b>🧭 Philosophy</b></a> &nbsp;·&nbsp; <a href="docs/NORTH_STAR_METHODOLOGY.md"><b>🧭 North Star</b></a>
 
 </div>
 
@@ -34,6 +34,7 @@
 
 ## 📖 Table of Contents
 
+- [🧭 Design Philosophy: imperfection by design](#-design-philosophy-imperfection-by-design)
 - [🎯 Highlights](#-highlights)
 - [🧠 Concepts in 60 seconds](#-concepts-in-60-seconds)
 - [🎬 Demo](#-demo)
@@ -49,6 +50,19 @@
 
 ---
 
+## 🧭 Design Philosophy: imperfection by design
+
+CogniFold does **not** chase a perfect, omniscient, unbiased recall store. It models memory the way cognition actually works — *situated, lossy, and opinionated* — because that is precisely what makes proactivity possible. A system that stored everything with equal weight and perfect fidelity would be a database: faithful, and completely reactive. What lets memory *act ahead of you* is the same thing that makes it imperfect — it decides what matters, what fades, and what crystallizes into an intent. **The bias is not a defect we are slowly engineering away. The bias is the mechanism.** We optimize for useful proactive structure, not for maximal ground-truth fidelity.
+
+Four cognitive realities we model **on purpose** instead of designing around:
+
+1. **Situated cognition.** Cognition is embedded in the active context, goals, and history; the deeper into a problem, the more the frame gets *locked*. Retrieval is conditioned on the active intent and recent trace, so the graph reasons *from where it already is*.
+2. **Confirmation bias / reasoning inertia.** Once an understanding forms, contradicting signals get filtered and the view is reinforced — agents inherit this through the reasoning path accumulated in a context window. We bound that inertia structurally (decay, completion, re-linking) instead of pretending each turn is a clean slate.
+3. **Locality of working memory.** Human working memory activates only the currently relevant nodes; an LLM's attention weights decide *what is even seen*. We embrace locality — the hierarchical context window surfaces a focused, partial view rather than dumping the whole graph.
+4. **Metacognitive blind spots (unknown unknowns).** The most dangerous gap is the part you don't know you don't know — you can't feel you need help. A *proactive* substrate matters here: intents that crystallize from topology can surface what you never thought to ask, partially covering the blind spot pure on-demand retrieval can't reach.
+
+This is also why our reported benchmark numbers are the **proactive-substrate stack, not per-benchmark tuned ceilings**. Several older tasks are trivially inflatable with a task-specialized reader — but that path rewards *auto-loop hallucination*, the reader confabulating to satisfy a metric instead of reading memory. That's exactly the metacognitive failure above, so we don't optimize for it. **The flaw is the point.** → Full writeup: [docs/PHILOSOPHY.md](docs/PHILOSOPHY.md).
+
 ## 🎯 Highlights
 
 1. **🔮 Proactive Memory.** Proactivity is a property of the memory substrate, not the agent's policy — goals emerge from the topology that accumulates the conditions for them.

docs/PHILOSOPHY.md

@@ -0,0 +1,191 @@
+# Design Philosophy: Imperfection by Design
+
+> **This document is an extension of `CLAUDE.md` and the [README](../README.md). Read those first for the system overview.**
+
+CogniFold does **not** chase a perfect, omniscient, unbiased recall store. It models
+memory the way cognition actually works — *situated, lossy, and opinionated* — because
+that is precisely what makes proactivity possible.
+
+This page explains why the biases below are deliberate design commitments rather than
+defects we are slowly engineering away, and how each one maps to a concrete mechanism in
+the system.
+
+---
+
+## The thesis
+
+A memory that stored everything, weighted everything equally, and recalled it with
+perfect fidelity would be a **database**. It would be faithful — and completely
+**reactive**. It can only answer what you explicitly ask, in the framing you happen to
+ask it.
+
+What lets memory *act ahead of you* — surface a deadline you forgot, connect two
+conversations you never linked, raise an intent you didn't think to query — is the same
+machinery that makes it imperfect. To be proactive, a memory must **decide**: what
+matters, what fades, what gets reinforced, and what crystallizes into an intent. Every
+one of those decisions is a bias. **The bias is not a bug on the way to a perfect memory.
+The bias is the mechanism.**
+
+So CogniFold optimizes for **useful proactive structure**, not for maximal ground-truth
+fidelity. The flaw is the point.
+
+---
+
+## Why "a faithful memory" is the wrong target
+
+The dominant framing for agent memory is *retrieval accuracy*: store text, embed it,
+fetch the nearest neighbors, measure how often the right chunk came back. Under that
+framing, every departure from verbatim recall is an error to be minimized, and the ideal
+system is a lossless index.
+
+But human memory is not a lossless index, and it is not trying to be. It is **generative
+and reconstructive** — it abstracts, forgets, distorts, and completes. Those are not
+failures of the system; they are how it stays small, fast, and *forward-looking* enough to
+be useful in an open-ended world. CogniFold takes the same position: the interesting
+target is not "did we keep every token," but "did the structure that emerged let the
+system act on the user's behalf."
+
+This reframes what counts as a good memory and, consequently, what counts as a good
+benchmark (see [What this means for evaluation](#what-this-means-for-evaluation)).
+
+---
+
+## Four cognitive realities we model on purpose
+
+Each of the following is a well-studied feature of biological cognition that *also* shows
+up in LLM-based agents. Rather than design around them, CogniFold embodies them.
+
+### 1. Situated cognition
+
+**The phenomenon.** Cognition is never isolated. It is embedded in the current context,
+the active goal, and the history that led here. The deeper you go into a problem, the more
+your frame gets *locked* by that situation — and the harder it is to step outside it.
+
+**The agent analogue.** A reasoning agent reads from where it already is: the active task
+and recent trace condition everything that follows. There is no "view from nowhere."
+
+**CogniFold's stance.** We make situatedness explicit rather than pretending to a neutral
+global read. Retrieval is conditioned on the **active intent** and recent episodic trace,
+so the graph reasons *from its current standpoint*. This is a feature: a proactive system
+must be anchored in a situation to know what to push.
+
+**Mechanism.** Intent-conditioned retrieval; the hierarchical context window's
+`immediate / working / background` bands; edge-type weighting routed by query intent
+(`symbolic/intent_router`).
+
+### 2. Confirmation bias / reasoning inertia
+
+**The phenomenon.** Once an understanding forms in a particular direction, contradicting
+signals get filtered out and the existing interpretation is reinforced. Belief has
+momentum.
+
+**The agent analogue.** The reasoning path accumulated in a context window builds
+inertia — each step is anchored by the steps before it, so an early commitment propagates
+forward whether or not it was right.
+
+**CogniFold's stance.** We treat this inertia as **real and structural**, not something a
+"fresh" prompt erases. Instead of pretending each turn is a clean slate, we bound and
+correct accumulated belief through explicit graph rewrites over time.
+
+**Mechanism.** The four structural debts of a streaming log — **accumulation, compression,
+decay, completion** — resolved as transparent, auditable graph rewrites (test-time
+learning, no gradient updates, no surface-text rewriting). Decay and re-linking are how an
+over-reinforced path loses its grip.
+
+### 3. Locality of working memory
+
+**The phenomenon.** Human working memory is finite. Only the currently relevant knowledge
+nodes are activated at once; other perspectives are suppressed. You cannot hold the whole
+of what you know in view simultaneously.
+
+**The agent analogue.** An LLM's attention is similar: the weight distribution over the
+current tokens decides *what is even seen*. Context is finite and the salience profile is
+narrow by construction.
+
+**CogniFold's stance.** We embrace locality as a **feature**, not a limit to paper over by
+stuffing the window. The system deliberately surfaces a focused, partial view rather than
+dumping the whole graph — because a partial-but-relevant view is what enables fast,
+situated action.
+
+**Mechanism.** The `HierarchicalContextSelector` and its bounded bands; scored, capped
+context assembly instead of full-graph serialization.
+
+### 4. Metacognitive blind spots (unknown unknowns)
+
+**The phenomenon.** The most dangerous gap is the part you don't know you don't know. You
+can't feel its absence, so you never think to ask — and therefore can't be helped by any
+system that only answers questions.
+
+**The agent analogue.** A purely on-demand (query → retrieve → answer) memory is
+structurally incapable of covering this gap: it can only return what you knew enough to
+ask for.
+
+**CogniFold's stance.** This is the deepest reason the substrate is **proactive**. Intents
+that crystallize from graph *topology* — from conditions accumulating whether or not you
+noticed — can surface something you never thought to query. That is partial coverage of a
+blind spot that on-demand retrieval can never reach.
+
+**Mechanism.** Intent crystallization: when a concept cluster crosses a density threshold,
+an `intent` node forms and is surfaced through the proactive context window with no query
+asked. Prospective memory as a property of the topology, not the agent's policy.
+
+---
+
+## What this means for the architecture
+
+The philosophy is not decoration — it is why the tri-layer substrate looks the way it
+does:
+
+| Cognitive commitment | Mechanism in CogniFold |
+|---|---|
+| Situated cognition | Intent-conditioned retrieval; intent-routed edge weights; context bands |
+| Reasoning inertia is real | Accumulation / compression / **decay** / completion as graph rewrites |
+| Locality is a feature | Hierarchical, bounded context window (not full-graph dumps) |
+| Cover unknown unknowns | **Intent crystallization** from topology → proactive surfacing |
+
+A memory that refused these biases could not crystallize an intent, could not forget a
+stale one, and could not decide what to surface unprompted. It would be inert.
+
+---
+
+## What this means for evaluation
+
+If imperfection is the design, then **the highest score on a recall-style benchmark is not
+the goal**, and chasing it can actively harm the system.
+
+- We report the **proactive-substrate stack** — the configuration that keeps intent
+  generation working end-to-end — **not a per-benchmark tuned ceiling.**
+- Several older benchmarks (ToMi in particular) are **trivially inflatable** with a
+  task-specialized reader. But that path rewards **auto-loop hallucination**: the reader
+  confabulates an answer that satisfies the metric instead of reading from memory. That is
+  precisely the *metacognitive blind spot* failure above — the model not knowing that it
+  doesn't know, and filling the gap with a fabrication.
+- Optimizing for those points would trade the system's purpose for a leaderboard number.
+  So we don't. We'd rather publish an honest number from the substrate we actually ship.
+
+This is also why **CogEval-Bench** exists: it isolates *proactive emergence* (Purity,
+Proactivity, Compression) from retrieval accuracy, measuring the thing the philosophy
+actually commits to. See [BENCHMARK.md](BENCHMARK.md) for the full results and protocol.
+
+---
+
+## What we explicitly reject
+
+- **"Perfect recall is the goal."** No — perfect recall is a reactive database. Proactivity
+  requires deciding, and deciding is biasing.
+- **"Bias is a defect to minimize to zero."** No — bias is the selection pressure that makes
+  structure (and intent) emerge.
+- **"A bigger context window solves locality."** No — locality is a feature; relevance beats
+  volume.
+- **"Higher benchmark number = better memory."** No — not when the number comes from a
+  configuration we don't ship and that encourages confabulation.
+
+---
+
+## Closing
+
+CogniFold is a bet that the path to genuinely helpful, *proactive* memory runs through
+modeling cognition honestly — biases included — rather than approximating an omniscient
+oracle. We don't pursue a perfect ground truth. We pursue a substrate that is situated,
+forgets on purpose, sees locally, and occasionally tells you something you didn't know to
+ask. **The flaw is the point.**