Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions docs/concepts/memories.md
Original file line number Diff line number Diff line change
Expand Up @@ -173,13 +173,16 @@ For retrieval, see [`search`](search.md) (MCP `search`, REST `POST

## Storage layout

YAML stores one Markdown file per memory at `memories/<id>.md` — YAML
frontmatter for the structured fields (`description`, `entities`,
`query`, `created_at`, `version`) and the Markdown body as the
`learning`. The id is the filename (not repeated in the frontmatter).
A legacy flat `memories.yaml` is migrated into per-file `.md` on first
open (and then deleted). SQLite uses a `memories` table plus a
`memory_entities` index table for the entity-overlap filter.
YAML stores one Markdown file per memory. Portable lowercase ASCII ids
use `memories/<id>.md`; other ids use
`memories/.encoded/<utf8-hex-id>.md`, which prevents filesystem case folding
or Unicode normalization from merging distinct ids. YAML frontmatter holds
the structured fields (`description`, `entities`, `query`, `created_at`,
`version`) and the Markdown body is the `learning`. The id is derived from the
reversible filename (not repeated in the frontmatter). A legacy flat
`memories.yaml` is migrated into per-file `.md` on first open (and then
deleted). SQLite uses a `memories` table plus a `memory_entities` index table
for the entity-overlap filter.

IDs are non-empty strings (DEV-1428). The auto-allocator walks
`max(int-shaped id) + 1` over the existing corpus where "int-shaped"
Expand Down
2 changes: 1 addition & 1 deletion docs/concepts/models.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Models

A model is SLayer's view of a database table or an underlying SQL query. It declares the columns, named metric formulas, joins, and always-applied filters that queries can build on. Models are defined as YAML (one file per model under `models/<data_source>/`) or created via API/MCP — the two paths produce the same persisted object.
A model is SLayer's view of a database table or an underlying SQL query. It declares the columns, named metric formulas, joins, and always-applied filters that queries can build on. Models are defined as YAML (one file per model under `models/<data_source>/`) or created via API/MCP — the two paths produce the same persisted object. Non-portable ids use reversible UTF-8 hex path components under `models/.encoded/` so filesystem normalization cannot merge distinct model identities.

A tiny example to anchor what follows:

Expand Down
6 changes: 5 additions & 1 deletion docs/configuration/datasources.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# Datasources

Datasources configure database connections. They are stored as individual YAML files in the `datasources/` directory.
Datasources configure database connections. They are stored as individual YAML
files in the `datasources/` directory. Human-authored, portable lowercase
filenames keep the layout shown below; ids that are not portable across
filesystems use a reversible UTF-8 hex filename under
`datasources/.encoded/`.

## YAML Format

Expand Down
11 changes: 11 additions & 0 deletions docs/configuration/storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,17 @@ slayer_data/

**Layout note:** Models live under `models/<data_source>/<name>.yaml` so two datasources sharing a table name don't collide. Opening a `YAMLStorage` on a legacy flat directory migrates `models/<name>.yaml` files into the nested layout automatically. If a flat file has an empty `data_source` and exactly one datasource is registered, the migrator auto-fills it; otherwise it hard-fails so the user can edit `data_source` by hand before reopening.

Lowercase ASCII ids containing only letters, digits, `.`, `_`, or `-` keep the
readable paths shown above, except Windows-reserved device names. Other
ids—and a portable id that would collide with an existing legacy filename—are
stored under a reserved `.encoded/` directory using reversible UTF-8 hex.
Models use
`models/.encoded/<data-source-hex>/<model-hex>.yaml`; datasource configs and
memories use `datasources/.encoded/<id-hex>.yaml` and
`memories/.encoded/<id-hex>.md`. This keeps distinct ids such as `X` and `x`
distinct on case-insensitive filesystems. Existing human-authored paths remain
readable, but legacy lookup requires an exact spelling match.

**Embeddings sidecar (DEV-1405):** Embedding rows used by the optional dense-search channel live in a SQLite file at `<base_dir>/embeddings.db`, **not** in `embeddings.yaml`. Embeddings are derived artifacts (regeneratable by `slayer ingest` / `--ingest-on-startup`), not user-authored config, so the diffable-in-git property that drives the YAML choice for models doesn't apply. A pre-DEV-1405 `embeddings.yaml` or `counters.yaml` is silently renamed to `<name>.yaml.legacy` on first open and ignored thereafter; re-run `slayer ingest` to repopulate `embeddings.db`. The schema is identical to the `SQLiteStorage` embedding table — both backends delegate to a shared `SidecarEmbeddingStore` helper.

**Memory id allocation (DEV-1658):** The next int-shaped memory id is derived by scanning the `memories/` directory of per-id `.md` files — `max(int-shaped id) + 1` — rather than a separate counter file. Non-int ids (e.g. `help.intro`, `kb.policy.42`) are ignored by the allocator. Ids of deleted memories may be reused by future saves; cascade-on-delete in `delete_memory` already removes the matching embedding row.
Expand Down
Loading
Loading