Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ jobs:
- uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
- run: pip install -e ".[mcp,dev]"
- run: pip install -e ".[dev]"
- run: pytest -q
# the example graph is documentation — it must pass its own rules
- run: knoten validate
Expand Down
14 changes: 5 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ $ knoten query "self-consistency"
## Use it

```bash
pip install -e . # the CLI is the agent surface; add ".[mcp]" only for shell-less clients
pip install -e .
knoten init my-topic # a new graph (it's a folder)
```

Expand Down Expand Up @@ -269,14 +269,10 @@ hypothesis can be opened now and closed later. `update` appends and moves status
rewrite a result already recorded — that is what retraction is for.

`ops.py` holds the one implementation behind every read — index, query, frontier, gates,
show, validate, path — as a function returning a dict. The CLI renders it as prose or dumps
it with `--json`; the MCP server serialises the same dict. One behaviour to keep correct,
not two that can drift.

**Clients without a shell (MCP).** For a chat UI wired to MCP rather than a coding agent,
`pip install -e ".[mcp]"` (needs mcp 2.x). It costs ~2,340 tokens of schema and instructions
in every session whether the graph is touched or not, against ~304 for `knoten --help` and
only when asked. Use the CLI if the client can run one.
show, validate, path — as a function returning a dict, which the CLI renders as prose or
dumps with `--json`. There was a second surface for shell-less clients; it cost ~2,340 tokens of schema in
every session against ~304 for `knoten --help`, and it was a second thing to keep
correct. It was removed. The shell is the interface.

## Why bother

Expand Down
96 changes: 43 additions & 53 deletions SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ tool combines git-native markdown nodes + falsification-aware typed edges +
agent-native read/write**. The gap is *structural*: every candidate buys one axis by
foreclosing another.

| system | markdown + git | falsification edges | agent / MCP |
| system | markdown + git | falsification edges | agent-native surface |
|---|---|---|---|
| Basic Memory | markdown ✓ · **git ✗** | **✗** free-form wikilinks | ✓ |
| Graphiti | **✗** (Neo4j) | partial (bi-temporal) | ✓ |
Expand Down Expand Up @@ -94,7 +94,7 @@ open ──► alive ──────► superseded
└───► retracted (WE WERE WRONG — the most valuable node type)
```

Every arrow above is walkable from the agent surface via `knoten_update`, which appends
Every arrow above is walkable from the agent surface via `knoten update`, which appends
and moves the status but cannot rewrite a claim. Immutability protects **what was
claimed**, never the status field — the status field *is* the lifecycle, and git already
holds the before and after (§7). Without this an agent could open a hypothesis and never
Expand Down Expand Up @@ -315,7 +315,7 @@ my-graph/
attachments/<id>/* # the script that killed it, the plot that shows why
```

`knoten new` and `knoten_commit` stamp `created:`; `knoten_update` stamps `updated:`.
`knoten new` and `knoten commit` stamp `created:`; `knoten update` stamps `updated:`.
Both are plain ISO date strings, which the YAML 1.2 loader keeps as strings rather than
coercing to `datetime.date`. Git knows when the *file* changed, which is not when the
*claim* did — a typo fix and a status flip are the same event to git — and reading it
Expand All @@ -328,86 +328,76 @@ directions, **PRs-as-peer-review**, and hosting. We write none of it.

## 8. The agent surface — CLI first

This section used to be titled "The agent surface (MCP) — this is what makes it
compound," and meant it. That claim is now wrong, and this is the reversal, stated
plainly rather than slid past.
This section once named a tool-protocol server as the thing that made knoten compound,
and meant it. That claim was wrong, and this is the reversal, stated plainly rather than
slid past.

The previous attempt failed because the graph was a **byproduct of automation**: when
the orchestrator did not run, nothing was written. Meanwhile a plain chat session
produced 15 experiments whose knowledge would have evaporated without hand-written
memory files. The fix was still right — make the graph primary, and make writing to it
the path of least resistance — but MCP turned out to be the expensive way to deliver it
to an agent that already has a shell.
the path of least resistance — but a tool-protocol server turned out to be the expensive
way to deliver it to an agent that already has a shell.

**The measurement.** knoten's MCP surface loads ~2,340 tokens into every session whether
the agent touches the graph or not (1,928 of tool schema + 412 of instructions).
**The measurement.** That surface loaded ~2,340 tokens into every session whether the
agent touched the graph or not (1,928 of tool schema + 412 of instructions).
`knoten --help` costs ~304 tokens, and only when the agent asks for it — most sessions
never pay even that. And format compounds the gap: the same 55 nodes cost ~2,551 tokens
as MCP's JSON (46/node) against ~1,185 as the CLI's columnar prose (21/node) — 2.2x. That
as its JSON (46/node) against ~1,185 as the CLI's columnar prose (21/node) — 2.2x. That
is why prose is the CLI's default and `--json` is opt-in, not the other way round.

**So: the CLI is now the primary agent surface.** `ops.py` holds the one implementation
of every read — index, query, frontier, gates, show, validate, path — as a plain
function returning a dict; the CLI renders it as prose or dumps it with `--json`, and MCP
serialises the same dict as a tool result. `commit` and `update` are likewise shared
function returning a dict; the CLI renders it as prose or dumps it with `--json`.
`commit` and `update` are likewise shared
functions, not surface-specific code paths. `SKILL.md`, at the repo root, is how an agent
now learns the loop: `frontier` → `index`/`query` → `show` → `gates` → `commit`/`update`
→ `attach`.

**MCP is retained, not deleted.** It is the right surface for a client that has no
shell — a chat UI wired to MCP servers rather than a coding agent with Bash. Those tools
still exist, still delegate to the same `ops`/`commit`/`update` functions, and still carry
the loop as the server's `instructions`:

| tool | purpose |
|---|---|
| `knoten_frontier()` | *"what should I work on next?"* → open claims, standing offers, unused gates |
| `knoten_index(...)` | *"anything LIKE this?"* / *"what is still open?"* → the graph, one line per node |
| `knoten_query(q)` | *"has this been tried?"* → nodes + verdicts + causes of death, by keyword |
| `knoten_get(id)` | full node, including the post-mortem |
| `knoten_gates()` | *"what must this survive?"* → the gates, their rule, and their record |
| `knoten_commit(node)` | append a node (validates first; **rejects on rule violation**) |
| `knoten_update(id, …)` | move a node's status and append to it — the lifecycle in §3, walkable |
| `knoten_attach(id, files)` | the script that ran it and the plot that shows it |
| `knoten_path(start, end)` | show the research path — how did we get from A to B? |
| `knoten_validate()` | run the graph's own declared rules |

The gate is the point, on either surface: **a `status: alive` node with no
`kn:survivedGate` edge is refused**, whether it arrives via `knoten commit` or
`knoten_commit`. The system will not let an agent — or a human — record an unchallenged
claim as a finding.
**That surface has since been deleted.** It was retained for a while as the right answer
for a client with no shell — a chat UI rather than a coding agent with Bash — and that
argument was not wrong, only outweighed. It cost ~2,340 tokens in every
session against ~304 for `knoten --help`, and it was a second surface that had to be kept
in step with `ops.py` forever. Every cross-surface drift bug this project has recorded —
`update`'s refusal built twice with different keys, `--field` coercing `2` to `2.0` on one
side and not the other, `ops.index(query=...)` reachable from one surface with no flag on
the other — came from having two. The shell is the interface.

What the server taught, which outlived it:

- **Derive the schema, never write one.** Its tools were plain functions: the name was the
tool name, the docstring was the description the agent read, and the annotated signature
*was* the input schema. A hand-written schema quietly stops matching the code it
documents; a derived one cannot.
- **State the loop once.** It lived in the server's `instructions`, not re-asserted by
every tool description — four tools each shouting "CALL THIS FIRST" give an agent the
same ordering signal as none. `SKILL.md` inherited that job.
- **The gate belongs under both doors.** A `status: alive` node with no `kn:survivedGate`
edge was refused on either surface, because the check lived in `commit`, not in the
transport. That is why removing the transport removed no enforcement.
- **An id authored by an LLM is not a path.** It becomes a filename, so it is constrained
to kebab-case. The CLI did not guard this until the day the other surface's guard was
the only one.

The candidate node is parsed and checked **in memory**; nothing reaches the filesystem
until it is clean.

Each MCP tool is a plain function: its name is the tool name, its docstring is the
description the agent reads, and its annotated signature IS the input schema. There is no
second copy of any of that to drift — the failure mode being avoided is a hand-written
schema that quietly stops matching the code it documents. The loop above lives once, in
the server's `instructions`, rather than being re-asserted by every tool description.

And because the `id` becomes a filename and is authored by an LLM, it is constrained to
kebab-case — an id is not a path.

`knoten_path` takes `start` / `end`. It took `from` / `to` until the 2.0 migration, where
the schema is derived from the function signature and `from` is a Python keyword.

---

### 8.1 Retrieval — two questions, two mechanisms

*"Has this been tried?"* and *"have we done anything **like** this?"* are different
questions, and one mechanism cannot answer both.

`knoten_query` is keyword retrieval: tokens weighted by idf, ranked, capped. It was
`knoten query` is keyword retrieval: tokens weighted by idf, ranked, capped. It was
originally an **AND** over tokens, which made the tool's headline question fail on its own
README example — `"has anyone tried self-consistency?"` matched nothing, because the node
contains no "has", "anyone" or "tried", and the agent was told the work was untested. A
false negative is the only failure mode of this system that causes real work to be redone.
So: OR with ranking, the full frontmatter in the haystack (`repro.model` was unsearchable),
and **a miss now says so honestly** — "no keyword match, this is NOT proof it is untested."

`knoten_index` answers the second question, and it does so by **not being a search engine
`knoten index` answers the second question, and it does so by **not being a search engine
at all**. It emits the whole graph as one line per node — id, verdict, tags, claim — and
lets the reader judge relatedness. The reader is an LLM; it is a better semantic matcher
than any similarity metric we could ship, and it costs nothing to ship. On a 500-node
Expand All @@ -431,8 +421,8 @@ index, which the 1k–5k node case does not.
|---|---|---|
| **0** | **Dogfood** — encode a real investigation by hand | ✅ done |
| **1** | `knoten` CLI (`init/new/validate/query/index/frontier/gates/path/show/attach`) + rule engine | ✅ done |
| **2** | **MCP server** (the tools above) | ✅ done — later demoted to the shell-less fallback (§8) |
| **2.5** | CLI becomes the primary agent surface: `ops.py` as the one implementation behind CLI/MCP, `--json` on every read, `commit`/`update` on the CLI, `SKILL.md` | ✅ done |
| **2** | **Tool-protocol server** | ✅ done — later demoted to a fallback, then removed (§8) |
| **2.5** | CLI becomes the primary agent surface: `ops.py` as the one implementation behind every read, `--json` on every read, `commit`/`update` on the CLI, `SKILL.md` | ✅ done |
| 3 | Static-site graph viewer → GitHub Pages | free hosting |
| 4 | Hosted multi-graph service | probably never needed |

Expand All @@ -443,7 +433,7 @@ blockers, and prose that no JSON schema could hold.

## 10. Licensing note

**Basic Memory is AGPL-3.0** (strong copyleft). Its MCP tool design is excellent and
**Basic Memory is AGPL-3.0** (strong copyleft). Its tool design is excellent and
worth studying, but vendoring its code would make this AGPL too. **Reimplement, do
not fork**, if this may ever be distributed or hosted.

Expand All @@ -455,7 +445,7 @@ Micropublications, nanopublications, PROV-O and LinkML are all open and safe to

1. **Do we emit RDF?** A `graph.ttl` export would make the graph interoperable with
the nanopub ecosystem for ~nothing. Probably yes, phase 3.
2. ~~**Embeddings for `knoten_query`?**~~ **Answered (§8.1): no, and probably never.**
2. ~~**Embeddings for `knoten query`?**~~ **Answered (§8.1): no, and probably never.**
The consumer is an LLM, so the whole graph as a one-line-per-node index beats vector
similarity at the question that matters ("anything like this?") for zero dependencies.
Revisit only when a tag-filtered index stops fitting in a context window.
Expand Down
7 changes: 1 addition & 6 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,14 @@ classifiers = [
dependencies = ["pyyaml>=6"]

[project.optional-dependencies]
# The upper bound is load-bearing, and its absence has already broken CI once: mcp 2.0
# removed the low-level Server decorators this file used to be built on. Assume 3.0
# renames something too, and find out on our own schedule rather than on release day.
mcp = ["mcp>=2,<3"]
dev = ["pytest>=8", "anyio>=4"]
dev = ["pytest>=8"]

[project.urls]
Homepage = "https://github.com/BY571/knoten"
Source = "https://github.com/BY571/knoten"

[project.scripts]
knoten = "knoten.cli:cli"
knoten-mcp = "knoten.mcp_server:main"

[build-system]
requires = ["setuptools>=61"]
Expand Down
4 changes: 2 additions & 2 deletions src/knoten/attachments.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""Attaching files to a node — the code that killed it, the plot that shows why.

Print-free on purpose: the MCP server speaks JSON-RPC over STDOUT, so a stray `print`
here would corrupt every response. This returns data; the CLI does the printing.
Print-free on purpose. It returns data and the CLI does the printing, so `--json`
emits exactly what the function produced and nothing writes to stdout behind its back.
"""
from __future__ import annotations

Expand Down
6 changes: 3 additions & 3 deletions src/knoten/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ def render_update(payload: dict) -> None:

def update_cmd(root, nid, status, append, results, links, fields, as_json) -> int:
# ops.update() is the ONE shape for both outcomes — this used to build its own
# dict here, and a different one in mcp_server.py, and the two shapes drifted.
# dict here, and a different one on a second surface since removed, and they drifted.
payload = ops.update(root, nid, status=status, append=_read(append) if append else None,
results=_kv(results), links=_links(links), fields=_fields(fields))
if payload["status"] == "REJECTED":
Expand Down Expand Up @@ -432,7 +432,7 @@ def new(root, ntype, nid, status) -> int:
f"are nodes, not edits.")

cfg = load_config(root)
# `new` used to skip this while knoten_commit enforced it: the same node was accepted
# `new` used to skip this while knoten commit enforced it: the same node was accepted
# by one entry point and rejected by the other.
for field, declared in [("type", cfg.get("node_types")), ("status", cfg.get("statuses"))]:
value = ntype if field == "type" else status
Expand Down Expand Up @@ -617,7 +617,7 @@ def main(argv=None) -> int:
}[args.cmd]()
except (GraphError, OSError) as e:
# OSError: a typo'd --frontmatter/--body/--append path is ordinary user error,
# not a traceback — mcp_server.tool already guards this for the same reason.
# not a traceback. Every entry point owes the user one line, not a stack.
return _fail({"error": str(e)}, e, getattr(args, "json", False))


Expand Down
4 changes: 2 additions & 2 deletions src/knoten/commit.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
reaches the filesystem until it is clean. An agent cannot record a shiny result that cites
no test it survived.

This lived inside the MCP server, which put the domain logic in the transport layer and —
worse — made writing a node from Python require the `mcp` SDK, an optional dependency for
This lived inside a transport layer, which put domain logic in the wrong place and — worse
— made writing a node from Python require that transport's SDK, an optional dependency for
a transport you may not be using. `attach` and `update` never had that problem.
"""
from __future__ import annotations
Expand Down
11 changes: 6 additions & 5 deletions src/knoten/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,8 @@ class GraphError(Exception):

# An id becomes a filename, so anything else is a path traversal. Go through node_path()
# for EVERY id -> file conversion: `knoten detach ../../x f` used to delete a file outside
# the graph, because only the MCP surface (where the id comes from an LLM) was guarded.
# the graph: an id authored by a model is not a path, and only one entry point
# checked that — not the one people used.
ID_RE = re.compile(r"^[a-z0-9][a-z0-9_-]*$")


Expand Down Expand Up @@ -227,7 +228,7 @@ def read_frontmatter(path: Path) -> tuple[dict, str]:


def parse_text(text: str, nid: str, label: str | None = None) -> Node:
"""Build a Node from a string. Used by `knoten_commit` to validate a candidate
"""Build a Node from a string. Used by `knoten commit` to validate a candidate
node in memory, so an invalid node never reaches the filesystem at all."""
label = label or f"{nid}.md"
fm, body = split(text, label)
Expand Down Expand Up @@ -272,8 +273,8 @@ def backlink(nodes: dict[str, Node]) -> dict[str, Node]:


# ---------------------------------------------------------------------------------
# Shared by BOTH surfaces. These lived twice — once in cli.py, once in mcp_server.py
# and drifted: the CLI's path printed relation labels while the MCP's did not, and a
# Shared. These lived twice — once in cli.py, once in a second surface since removed
# and drifted: one path printed relation labels and the other did not, and a
# search fix landed in one copy and not the other.

def find_root(start: Path | None = None) -> Path:
Expand Down Expand Up @@ -400,7 +401,7 @@ def retrieve(nodes: dict[str, Node], query: str | None = None, tags=None,
Tokens are weighted by idf, so a word in every node counts for nothing without
anybody having to list it, and a rare one dominates.

This is the ONE retrieval seam: `query`, `index` and the MCP tools all come through
This is the ONE retrieval seam: `query` and `index` both come through
here, so a semantic backend replaces this body and nothing above it changes.
"""
pool = [n for n in nodes.values() if _passes(n, tags, status, type, where, since)]
Expand Down
Loading
Loading