Reject case-colliding ids when using YAMLStorage#252
Conversation
YAMLStorage uses raw ids as filenames, so ids differing only by case (X vs x) address the same file on case-insensitive filesystems (macOS, Windows) and silently overwrite each other — datasources, models, and memories alike. SQLiteStorage kept them distinct, so the backends silently diverged. Saving an id that differs only by case from an existing one now raises IdCollisionError(SlayerError, ValueError), enforced in the StorageBackend base templates so every backend rejects on every platform and stores stay portable. Exact-id re-saves remain upserts; a legacy store already holding a colliding pair raises on a save of either spelling. - save_datasource becomes a template method delegating to a new abstract _save_datasource_impl (same pattern as save_model) - save_memory checks via a new cheap _list_memory_ids() primitive; SQLiteStorage's atomic save_memory override duplicates the check inside its transaction - model saves check both identity components: data_source against datasource names and other models' data_sources, name within the same datasource - YAML reads/deletes verify the exact directory entry via os.listdir comparison, so a case-variant lookup reads as not-found and a delete is a no-op instead of hitting the wrong file - migrate_memories_layout and migrate_yaml_layout pre-check collisions before writing anything, preserving legacy files on failure - surfacing: REST POST /datasources maps to 400; MCP create_datasource returns a friendly message and isolates per-model ingest-save failures; CLI prints and exits or skips-and-continues in ingest loops Closes #249 Claude-Session: https://claude.ai/code/session_011sbQKNXcUtGAsCXtmJXHvR
📝 WalkthroughWalkthroughYAML storage now rejects case-only identifier collisions for datasources, models, and memories, enforces exact-case lookups, prechecks migrations, and surfaces failures through REST, CLI, and MCP flows. SQLite retains distinct case-variant identifiers. Tests and documentation cover the updated behavior. ChangesCase-collision persistence safeguards
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant Interface
participant StorageBackend
participant YAMLStorage
Client->>Interface: create datasource or ingest models
Interface->>StorageBackend: save entity
StorageBackend->>YAMLStorage: validate case-folded identity
YAMLStorage-->>StorageBackend: raise IdCollisionError
StorageBackend-->>Interface: return ValueError-compatible error
Interface-->>Client: HTTP 400 or per-model failure
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
slayer/storage/join_sync.py (1)
259-282: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winDelegate
save_memoryto the inner backend
JoinSyncStoragestill needs a publicsave_memoryoverride that forwards toself._inner.save_memory(...). The inherited template splits the collision check from the write and bypassesSQLiteStorage.save_memory’s transaction, even though the low-level SQLite helpers keepmemory_entitiesin sync.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/join_sync.py` around lines 259 - 282, Add a public save_memory override to JoinSyncStorage that forwards the memory and any supported arguments directly to self._inner.save_memory(...). Place it alongside the existing memory delegation methods, ensuring calls use SQLiteStorage.save_memory’s transactional implementation instead of the inherited template.
🧹 Nitpick comments (3)
slayer/storage/sqlite_storage.py (1)
439-452: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDuplicate
"SELECT id FROM memories"literal (SonarCloud S1192).The same literal appears 3 times (here,
_next_memory_seq_sync_from_conn,_list_memory_ids_sync). Extract a module-level constant to avoid drift between the three query sites.♻️ Proposed fix
+_SELECT_MEMORY_IDS_SQL = "SELECT id FROM memories" + ... - ids = [ - row[0] if isinstance(row[0], str) else str(row[0]) - for row in conn.execute("SELECT id FROM memories") - ] + ids = [ + row[0] if isinstance(row[0], str) else str(row[0]) + for row in conn.execute(_SELECT_MEMORY_IDS_SQL) + ]Also applies to: 494-505, 562-569
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/sqlite_storage.py` around lines 439 - 452, Extract the repeated "SELECT id FROM memories" SQL into a module-level constant, then update the query in the transaction case-collision check and the corresponding queries in _next_memory_seq_sync_from_conn and _list_memory_ids_sync to use that constant.Source: Linters/SAST tools
slayer/storage/base.py (1)
126-141: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
_find_case_colliding_idis called positionally everywhere; guideline requires keyword args for multi-param functions.Every call site (
_check_model_identity_collisionlines 236/242,save_datasourceline 465,save_memoryline 702, andsqlite_storage.pyline 446) passescandidate/existingpositionally. As per coding guidelines, "Use keyword arguments for functions with more than one parameter." Consider marking the params keyword-only (*) to enforce this at every call site.♻️ Proposed fix
def _find_case_colliding_id( - candidate: str, existing: Iterable[str], + *, candidate: str, existing: Iterable[str], ) -> str | None:Then update each call site to
_find_case_colliding_id(candidate=..., existing=...).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/base.py` around lines 126 - 141, Make _find_case_colliding_id parameters keyword-only by adding the separator before candidate, then update every call in _check_model_identity_collision, save_datasource, save_memory, and sqlite_storage.py to pass candidate= and existing= explicitly. Preserve the function’s collision behavior unchanged.Source: Path instructions
docs/configuration/storage.md (1)
85-86: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMinor: "enforced in the shared base class" is imprecise for the SQLite memory path.
For SQLite, memory-id collision checking is actually reimplemented locally in
_save_memory_atomic_sync(for transactional atomicity), not routed through the shared base-class check used by models/datasources. Worth a one-clause caveat if you want full precision, but this doesn't materially mislead.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/configuration/storage.md` around lines 85 - 86, Update the SQLite case-sensitivity documentation to clarify that collision enforcement uses the shared base class for models and datasources, while the SQLite memory path reimplements the check locally in _save_memory_atomic_sync for transactional atomicity. Preserve the existing export and cross-platform migration guidance.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@slayer/storage/base.py`:
- Around line 453-470: Make the collision check and persistence in
save_datasource and save_model atomic for SQLite by executing both under the
backend’s existing BEGIN IMMEDIATE transaction pattern. Keep the list/identity
checks and corresponding _save_datasource_impl or model-save implementation
within the same transaction, so concurrent case-variant saves cannot both pass
validation; preserve the existing memory-path behavior.
---
Outside diff comments:
In `@slayer/storage/join_sync.py`:
- Around line 259-282: Add a public save_memory override to JoinSyncStorage that
forwards the memory and any supported arguments directly to
self._inner.save_memory(...). Place it alongside the existing memory delegation
methods, ensuring calls use SQLiteStorage.save_memory’s transactional
implementation instead of the inherited template.
---
Nitpick comments:
In `@docs/configuration/storage.md`:
- Around line 85-86: Update the SQLite case-sensitivity documentation to clarify
that collision enforcement uses the shared base class for models and
datasources, while the SQLite memory path reimplements the check locally in
_save_memory_atomic_sync for transactional atomicity. Preserve the existing
export and cross-platform migration guidance.
In `@slayer/storage/base.py`:
- Around line 126-141: Make _find_case_colliding_id parameters keyword-only by
adding the separator before candidate, then update every call in
_check_model_identity_collision, save_datasource, save_memory, and
sqlite_storage.py to pass candidate= and existing= explicitly. Preserve the
function’s collision behavior unchanged.
In `@slayer/storage/sqlite_storage.py`:
- Around line 439-452: Extract the repeated "SELECT id FROM memories" SQL into a
module-level constant, then update the query in the transaction case-collision
check and the corresponding queries in _next_memory_seq_sync_from_conn and
_list_memory_ids_sync to use that constant.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0af37df7-0ea6-446f-9710-556c7866a3a0
📒 Files selected for processing (18)
CLAUDE.mddocs/concepts/ingestion.mddocs/concepts/memories.mddocs/concepts/models.mddocs/configuration/storage.mdslayer/api/server.pyslayer/cli.pyslayer/core/errors.pyslayer/mcp/server.pyslayer/storage/base.pyslayer/storage/join_sync.pyslayer/storage/sqlite_storage.pyslayer/storage/v4_migration.pyslayer/storage/yaml_storage.pytests/test_api_server.pytests/test_id_collision.pytests/test_memory_string_ids.pytests/test_yaml_exact_entry.py
| async def save_datasource(self, datasource: DatasourceConfig) -> None: | ||
| """Persist a datasource config. | ||
|
|
||
| Rejects a name that differs only by case from an existing | ||
| datasource name — or from a saved model's ``data_source`` (they | ||
| share the ``models/<ds>/`` directory namespace in the YAML | ||
| backend) — then delegates to the backend-specific | ||
| :meth:`_save_datasource_impl`. Same-name saves are upserts. | ||
| Concrete backends must NOT override this method. | ||
| """ | ||
| existing = set(await self.list_datasources()) | ||
| existing.update(ds for ds, _ in await self._list_all_model_identities()) | ||
| collide = _find_case_colliding_id(datasource.name, existing) | ||
| if collide is not None: | ||
| raise IdCollisionError( | ||
| kind="datasource", new_id=datasource.name, existing_id=collide, | ||
| ) | ||
| await self._save_datasource_impl(datasource) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🌐 Web query:
Does asyncio.to_thread yield control to the event loop, allowing interleaving between two separate to_thread calls?
💡 Result:
Yes, asyncio.to_thread yields control to the event loop, allowing for interleaving [1][2]. When you call await asyncio.to_thread(...), the current coroutine suspends and yields control back to the event loop [1]. While the function submitted to to_thread is executing in a separate thread (via a ThreadPoolExecutor), the event loop remains free to run other tasks, callbacks, or perform I/O operations [1][2]. Regarding the interleaving of two separate to_thread calls: 1. If you await each to_thread call sequentially (e.g., await asyncio.to_thread(func1); await asyncio.to_thread(func2)), the second call will not start until the first has finished because the event loop is blocked at the first await expression [3]. 2. If you want two separate to_thread calls to run concurrently and interleave, you must schedule them as concurrent tasks, such as by using asyncio.gather or by creating tasks with asyncio.create_task [3]. This allows the event loop to manage both operations simultaneously while waiting for their completion in separate threads [3]. In essence, asyncio.to_thread is a non-blocking awaitable that allows you to offload blocking synchronous code to a separate thread without stalling the event loop, provided that you properly schedule the calls to run concurrently [1][2][3].
Citations:
- 1: https://stackoverflow.com/questions/65316863/is-asyncio-to-thread-method-different-to-threadpoolexecutor
- 2: https://docs.python.org/3/library/asyncio-task.html
- 3: https://stackoverflow.com/questions/75392320/blocking-function-sent-to-thread-not-executing-if-awaited-secondly
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== base.py outline ==\n'
ast-grep outline slayer/storage/base.py --view expanded | sed -n '1,220p'
printf '\n== sqlite_storage.py outline ==\n'
ast-grep outline slayer/storage/sqlite_storage.py --view expanded | sed -n '1,260p'
printf '\n== relevant save/collision methods ==\n'
rg -n "async def (save_datasource|save_model|_save_datasource_impl|_save_model_impl|_list_all_model_identities|list_datasources|_check_model_identity_collision|_save_memory_atomic_sync|BEGIN IMMEDIATE)" slayer/storage/base.py slayer/storage/sqlite_storage.pyRepository: MotleyAI/slayer
Length of output: 7188
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
for p in [Path('slayer/storage/base.py'), Path('slayer/storage/sqlite_storage.py')]:
print(f"\n### {p} ###")
text = p.read_text()
for needle in [
'async def save_datasource',
'async def save_model',
'async def _save_datasource_impl',
'async def _save_model_impl',
'async def _list_all_model_identities',
'async def list_datasources',
'async def _check_model_identity_collision',
'BEGIN IMMEDIATE',
'asyncio.to_thread',
'asyncio.Lock',
]:
if needle in text:
print(f"FOUND {needle}")
PYRepository: MotleyAI/slayer
Length of output: 674
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,260p' slayer/storage/base.py | nl -ba | sed -n '1,260p'
printf '\n----\n'
sed -n '1,260p' slayer/storage/sqlite_storage.py | nl -ba | sed -n '1,260p'Repository: MotleyAI/slayer
Length of output: 193
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== base.py relevant excerpts ==\n'
sed -n '200,280p' slayer/storage/base.py
printf '\n----\n'
sed -n '453,490p' slayer/storage/base.py
printf '\n====\n'
printf '\n== sqlite_storage.py relevant excerpts ==\n'
sed -n '186,380p' slayer/storage/sqlite_storage.py
printf '\n----\n'
sed -n '404,540p' slayer/storage/sqlite_storage.py
printf '\n----\n'
sed -n '540,640p' slayer/storage/sqlite_storage.pyRepository: MotleyAI/slayer
Length of output: 22447
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== sqlite_storage init/schema excerpts ==\n'
sed -n '133,185p' slayer/storage/sqlite_storage.py
printf '\n== schema-related searches ==\n'
rg -n "CREATE TABLE|CREATE UNIQUE|UNIQUE INDEX|COLLATE NOCASE|PRIMARY KEY|models \\(|datasources \\(" slayer/storage/sqlite_storage.py slayer/storage/*.pyRepository: MotleyAI/slayer
Length of output: 5760
Make the case-collision guard atomic on SQLite
save_model and save_datasource can still race across the asyncio.to_thread boundaries: SQLite’s PRIMARY KEY here is case-sensitive, so concurrent saves like Foo/foo can both pass the check and persist. The memory path already uses BEGIN IMMEDIATE; models/datasources need the same protection.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@slayer/storage/base.py` around lines 453 - 470, Make the collision check and
persistence in save_datasource and save_model atomic for SQLite by executing
both under the backend’s existing BEGIN IMMEDIATE transaction pattern. Keep the
list/identity checks and corresponding _save_datasource_impl or model-save
implementation within the same transaction, so concurrent case-variant saves
cannot both pass validation; preserve the existing memory-path behavior.
…c helper The template-method split (save_datasource delegating to a new abstract _save_datasource_impl) forced third-party backends to rename their overrides on upgrade. Revert it: save_datasource stays the abstract method backends implement directly, and the case-collision check moves into a public base helper, check_datasource_id_collision(name), which the built-in YAML/SQLite implementations call before writing. Custom backends opt into the same protection by calling it from their own save_datasource. Claude-Session: https://claude.ai/code/session_011sbQKNXcUtGAsCXtmJXHvR
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
slayer/storage/yaml_storage.py (2)
108-119: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winHandle unreadable directories consistently with the helper contract.
The docstring says missing or unreadable directories return
False, butPermissionErrorfromos.listdir()is not caught. Either catchPermissionErroror update the contract and tests to propagate permission failures.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/yaml_storage.py` around lines 108 - 119, Update _exact_entry_exists to catch PermissionError from os.listdir and return False, preserving the helper’s documented behavior for missing, invalid, and unreadable directories.
201-220: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftReject collisions among already-existing memory files.
The
id_by_keyconstruction silently overwrites an earlier stem when two existing files differ only by case. With bothX.mdandx.mdpresent, migration behavior depends onos.listdir()order and may allow a later write to clobber data. Detect duplicate casefold keys while indexing existing files and fail before processing or writing legacy rows.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/yaml_storage.py` around lines 201 - 220, Update the existing-file indexing in the id_by_key construction to detect when two .md stems share the same casefold key but differ in original casing, and raise the same collision ValueError immediately. Do this while iterating os.listdir(mem_dir), before processing normalized legacy rows or performing any writes; retain the existing normalized-row collision checks.slayer/storage/base.py (2)
236-236: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUse keyword arguments for the collision helper.
These
_find_case_colliding_idcalls pass multiple parameters positionally. As per coding guidelines, Python functions with more than one parameter must use keyword arguments; update all four call sites using the helper’s declared parameter names.Also applies to: 242-242, 474-474, 705-705
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/base.py` at line 236, Update all four call sites of _find_case_colliding_id in the affected storage flows to pass every argument by keyword, using the helper’s declared parameter names. Preserve the existing argument values and behavior while removing positional argument usage.Source: Coding guidelines
217-221:⚠️ Potential issue | 🟠 MajorKeep collision validation and persistence atomic.
save_modelperforms the collision check and backend write in separate awaits, whilecheck_datasource_id_collisionis also only a preflight helper. The supplied SQLite and YAML implementations still check first and write afterward, so concurrentFoo/foosaves can both pass validation. Keep each backend’s check-and-write sequence under the same transaction or lock.Also applies to: 462-478
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/base.py` around lines 217 - 221, Make the collision validation and persistence atomic in save_model and the SQLite/YAML backend implementations: execute _check_model_identity_collision (or check_datasource_id_collision) and the corresponding _save_model_impl write within the same transaction or lock. Ensure concurrent Foo/foo saves cannot both pass validation before either write commits, while preserving the existing validation and column-cycle checks.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@slayer/storage/sqlite_storage.py`:
- Around line 352-354: Make datasource collision validation atomic with
persistence: in slayer/storage/sqlite_storage.py lines 352-354, move
check_datasource_id_collision into the same SQLite write transaction used by
_save_datasource_sync; in slayer/storage/yaml_storage.py lines 410-415,
serialize collision validation and file creation using the datasource lock or
equivalent atomic operation. Preserve case-insensitive collision handling for
concurrent saves.
---
Outside diff comments:
In `@slayer/storage/base.py`:
- Line 236: Update all four call sites of _find_case_colliding_id in the
affected storage flows to pass every argument by keyword, using the helper’s
declared parameter names. Preserve the existing argument values and behavior
while removing positional argument usage.
- Around line 217-221: Make the collision validation and persistence atomic in
save_model and the SQLite/YAML backend implementations: execute
_check_model_identity_collision (or check_datasource_id_collision) and the
corresponding _save_model_impl write within the same transaction or lock. Ensure
concurrent Foo/foo saves cannot both pass validation before either write
commits, while preserving the existing validation and column-cycle checks.
In `@slayer/storage/yaml_storage.py`:
- Around line 108-119: Update _exact_entry_exists to catch PermissionError from
os.listdir and return False, preserving the helper’s documented behavior for
missing, invalid, and unreadable directories.
- Around line 201-220: Update the existing-file indexing in the id_by_key
construction to detect when two .md stems share the same casefold key but differ
in original casing, and raise the same collision ValueError immediately. Do this
while iterating os.listdir(mem_dir), before processing normalized legacy rows or
performing any writes; retain the existing normalized-row collision checks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 911612e1-ee36-4025-a8b3-b3e2b6b8e0b5
📒 Files selected for processing (6)
CLAUDE.mddocs/configuration/storage.mdslayer/storage/base.pyslayer/storage/join_sync.pyslayer/storage/sqlite_storage.pyslayer/storage/yaml_storage.py
🚧 Files skipped from review as they are similar to previous changes (2)
- docs/configuration/storage.md
- CLAUDE.md
| async def save_datasource(self, datasource: DatasourceConfig) -> None: | ||
| await self.check_datasource_id_collision(datasource.name) | ||
| await asyncio.to_thread(self._save_datasource_sync, datasource) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Datasource collision checks are not atomic with persistence.
Both backends can accept concurrent case-variant saves because validation completes before the write begins.
slayer/storage/sqlite_storage.py#L352-L354: perform collision validation inside the same SQLite write transaction as the upsert.slayer/storage/yaml_storage.py#L410-L415: serialize collision validation and file creation with a datasource lock or equivalent atomic operation.
📍 Affects 2 files
slayer/storage/sqlite_storage.py#L352-L354(this comment)slayer/storage/yaml_storage.py#L410-L415
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@slayer/storage/sqlite_storage.py` around lines 352 - 354, Make datasource
collision validation atomic with persistence: in
slayer/storage/sqlite_storage.py lines 352-354, move
check_datasource_id_collision into the same SQLite write transaction used by
_save_datasource_sync; in slayer/storage/yaml_storage.py lines 410-415,
serialize collision validation and file creation using the datasource lock or
equivalent atomic operation. Preserve case-insensitive collision handling for
concurrent saves.
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (3)
slayer/storage/yaml_storage.py (3)
197-213: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winReject collisions already present on disk.
The existing-stem index overwrites a prior entry with the same
casefold()key. If bothFoo.mdandfoo.mdalready exist, migration proceeds and deletesmemories.yamlinstead of failing before data changes. Detect an existingpriorcollision while building the index and raise before any writes or removal.Proposed fix
for fname in os.listdir(mem_dir): if fname.endswith(".md"): stem = fname[: -len(".md")] + prior = id_by_key.get(stem.casefold()) + if prior is not None and prior != stem: + raise ValueError( + f"{base_dir}: memory ids {prior!r} and {stem!r} " + "differ only by case. Rename one, then reopen." + ) id_by_key[stem.casefold()] = stem🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/yaml_storage.py` around lines 197 - 213, Update the existing-stem index construction in the migration flow to detect when a new .md filename’s casefolded key already maps to a different stem, rather than overwriting id_by_key. Raise the same collision ValueError during directory scanning, before processing normalized records or performing any writes or memories.yaml removal; retain the current normalized-ID collision checks.
297-304: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUse keyword arguments for the new multi-parameter helper calls.
The added calls to
_exact_entry_existsand_model_entry_existspass multiple arguments positionally, contrary to the repository’s Python guideline. Use explicit keywords at these call sites.As per coding guidelines, use keyword arguments for functions with more than one parameter.
Also applies to: 337-338, 357-359, 374-378, 412-413, 436-439, 574-575, 605-606
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/yaml_storage.py` around lines 297 - 304, Update every multi-parameter call to _exact_entry_exists and _model_entry_exists in the affected storage methods to pass arguments by explicit keyword, including _model_entry_exists itself and the additional call sites noted in the review. Preserve the existing argument values and behavior while following the repository’s keyword-argument convention.Source: Coding guidelines
401-406: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftMake datasource save/check atomic.
check_datasource_id_collision()still races with the write here, so two concurrent saves can both pass and then alias/overwrite on a case-insensitive filesystem. Hold the check and persistence under the same lock, or use an atomic reservation path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/yaml_storage.py` around lines 401 - 406, Update save_datasource to make the collision check and YAML persistence one atomic operation: acquire the same async lock around check_datasource_id_collision, path construction, serialization, and file write, ensuring concurrent saves cannot both pass the check and overwrite or alias the datasource file.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@slayer/storage/yaml_storage.py`:
- Around line 197-213: Update the existing-stem index construction in the
migration flow to detect when a new .md filename’s casefolded key already maps
to a different stem, rather than overwriting id_by_key. Raise the same collision
ValueError during directory scanning, before processing normalized records or
performing any writes or memories.yaml removal; retain the current normalized-ID
collision checks.
- Around line 297-304: Update every multi-parameter call to _exact_entry_exists
and _model_entry_exists in the affected storage methods to pass arguments by
explicit keyword, including _model_entry_exists itself and the additional call
sites noted in the review. Preserve the existing argument values and behavior
while following the repository’s keyword-argument convention.
- Around line 401-406: Update save_datasource to make the collision check and
YAML persistence one atomic operation: acquire the same async lock around
check_datasource_id_collision, path construction, serialization, and file write,
ensuring concurrent saves cannot both pass the check and overwrite or alias the
datasource file.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: cea1a277-50b6-4a93-9a41-0b8b999e905b
📒 Files selected for processing (10)
slayer/core/errors.pyslayer/mcp/server.pyslayer/storage/base.pyslayer/storage/join_sync.pyslayer/storage/sqlite_storage.pyslayer/storage/v4_migration.pyslayer/storage/yaml_storage.pytests/test_id_collision.pytests/test_memory_string_ids.pytests/test_yaml_exact_entry.py
💤 Files with no reviewable changes (2)
- tests/test_memory_string_ids.py
- slayer/storage/join_sync.py
🚧 Files skipped from review as they are similar to previous changes (7)
- slayer/core/errors.py
- slayer/mcp/server.py
- slayer/storage/sqlite_storage.py
- tests/test_yaml_exact_entry.py
- slayer/storage/v4_migration.py
- slayer/storage/base.py
- tests/test_id_collision.py
SQLite keys are case-sensitive TEXT PKs and handle case-variant ids fine; enforcing the rule there added complexity (a duplicated transaction check, the _list_memory_ids primitive with per-backend overrides) for no real protection. Revert SQLite to its original behavior: case-variant ids are distinct rows. The checks stay in the base save templates — preserving the _validate=False migration bypass and coverage through JoinSyncStorage — gated on a class flag, _ids_collide_as_filenames (True on YAMLStorage, copied from the inner backend by the wrapper). Custom backends with a case-insensitive keyspace can opt in via the same flag plus check_datasource_id_collision. Claude-Session: https://claude.ai/code/session_011sbQKNXcUtGAsCXtmJXHvR
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/configuration/storage.md (1)
114-131: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMake the custom-backend example asynchronous.
The
StorageBackendmethods are async, but the example declares them withdef; Line 131 then requiresawait. A copied backend will not satisfy the runtime contract and will fail when the base template awaits its implementation.As per coding guidelines, storage operations are async-first.
Suggested documentation fix
- def _save_model_impl(self, model: SlayerModel) -> None: ... - def _list_all_model_identities(self) -> list[tuple[str, str]]: ... - def get_model(self, name: str, data_source: str | None = None) -> SlayerModel | None: ... - def _delete_model_row(self, *, data_source: str, name: str) -> bool: ... + async def _save_model_impl(self, model: SlayerModel) -> None: ... + async def _list_all_model_identities(self) -> list[tuple[str, str]]: ... + async def get_model(self, name: str, data_source: str | None = None) -> SlayerModel | None: ... + async def _delete_model_row(self, *, data_source: str, name: str) -> bool: ... - def save_datasource(self, datasource: DatasourceConfig) -> None: ... - def get_datasource(self, name: str) -> DatasourceConfig | None: ... - def list_datasources(self) -> list[str]: ... - def _delete_datasource_row(self, name: str) -> bool: ... + async def save_datasource(self, datasource: DatasourceConfig) -> None: ... + async def get_datasource(self, name: str) -> DatasourceConfig | None: ... + async def list_datasources(self) -> list[str]: ... + async def _delete_datasource_row(self, name: str) -> bool: ...🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/configuration/storage.md` around lines 114 - 131, Update the custom-backend example methods to use async definitions consistent with the StorageBackend contract, including _save_model_impl, _list_all_model_identities, get_model, _delete_model_row, save_datasource, get_datasource, list_datasources, and _delete_datasource_row. Preserve the existing method signatures and ensure the save_datasource example can await check_datasource_id_collision.Source: Coding guidelines
🧹 Nitpick comments (1)
slayer/storage/base.py (1)
233-239: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse keyword arguments for the collision helper.
_find_case_colliding_idreceives more than one argument positionally in both changed paths. Change these calls tocandidate=...andexisting=....As per coding guidelines, use keyword arguments for functions with more than one parameter.
Also applies to: 690-695
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/storage/base.py` around lines 233 - 239, Update both _find_case_colliding_id calls in the affected collision-check paths to pass the candidate and existing collections using the keyword arguments candidate= and existing=. Preserve the current values and collision handling behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_id_collision.py`:
- Around line 3-6: Update tests/test_id_collision.py lines 3-6 to document
uniform case-insensitive collision rejection across backends; update lines
191-220 so both direct SQLite and resolve_storage() saves expect
IdCollisionError for case-variant IDs while preserving exact-ID upserts; update
tests/test_memory_string_ids.py lines 183-198 to remove the YAML-only branch and
assert collision rejection for every backend.
---
Outside diff comments:
In `@docs/configuration/storage.md`:
- Around line 114-131: Update the custom-backend example methods to use async
definitions consistent with the StorageBackend contract, including
_save_model_impl, _list_all_model_identities, get_model, _delete_model_row,
save_datasource, get_datasource, list_datasources, and _delete_datasource_row.
Preserve the existing method signatures and ensure the save_datasource example
can await check_datasource_id_collision.
---
Nitpick comments:
In `@slayer/storage/base.py`:
- Around line 233-239: Update both _find_case_colliding_id calls in the affected
collision-check paths to pass the candidate and existing collections using the
keyword arguments candidate= and existing=. Preserve the current values and
collision handling behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: f718a0fc-cbcf-4a6b-927f-a22b8f241fff
📒 Files selected for processing (11)
CLAUDE.mddocs/concepts/ingestion.mddocs/concepts/memories.mddocs/concepts/models.mddocs/configuration/storage.mdslayer/core/errors.pyslayer/storage/base.pyslayer/storage/join_sync.pyslayer/storage/yaml_storage.pytests/test_id_collision.pytests/test_memory_string_ids.py
🚧 Files skipped from review as they are similar to previous changes (5)
- docs/concepts/models.md
- docs/concepts/ingestion.md
- slayer/core/errors.py
- CLAUDE.md
- slayer/storage/yaml_storage.py
| Ids are filenames in the YAML backend, so saving an id that differs only | ||
| by case from an existing one raises IdCollisionError there (exact-id | ||
| re-saves remain upserts). SQLite keys are case-sensitive and stores | ||
| case-variant ids distinctly. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Align regression expectations with the cross-backend collision contract.
These tests codify SQLite-specific acceptance of casefold-colliding IDs, contrary to the PR objective that collisions raise IdCollisionError across storage backends.
tests/test_id_collision.py#L3-L6: document uniform collision rejection rather than SQLite-specific distinct identities.tests/test_id_collision.py#L191-L220: expectIdCollisionErrorfor SQLite direct andresolve_storage()saves.tests/test_memory_string_ids.py#L183-L198: remove the YAML-only branch and assert collision rejection for every backend.
📍 Affects 2 files
tests/test_id_collision.py#L3-L6(this comment)tests/test_id_collision.py#L191-L220tests/test_memory_string_ids.py#L183-L198
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/test_id_collision.py` around lines 3 - 6, Update
tests/test_id_collision.py lines 3-6 to document uniform case-insensitive
collision rejection across backends; update lines 191-220 so both direct SQLite
and resolve_storage() saves expect IdCollisionError for case-variant IDs while
preserving exact-ID upserts; update tests/test_memory_string_ids.py lines
183-198 to remove the YAML-only branch and assert collision rejection for every
backend.
- migrate_memories_layout now also rejects a case-colliding pair already present among memories/ stems (possible on case-sensitive filesystems), instead of letting listdir order decide which spelling wins the index - docs custom-backend example methods are async, matching the contract - multi-parameter helper calls use keyword arguments per convention Claude-Session: https://claude.ai/code/session_011sbQKNXcUtGAsCXtmJXHvR
|
Review triage (@coderabbitai), addressed in 8ad935f: Fixed
Skipped
|
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '=== Relevant files ==='
fd -a -t f -E .git '^(yaml_storage|base|v4_migration|storage)\.py$|^test_id_collision\.py$|^storage\.md$' . \
| sed -n '1,160p'
printf '%s\n' '=== YAML migration and collision-related symbols ==='
ast-grep outline slayer/storage/yaml_storage.py --items all --match 'migrate_memories_layout|_find_case_colliding_id|_exact_entry_exists|_model_entry_exists' --view expanded || true
ast-grep outline slayer/storage/v4_migration.py --items all --match '_check_layout_case_collisions|migrate_yaml_layout' --view expanded || true
printf '%s\n' '=== Implementation excerpts ==='
rg -n -A55 -B12 'def migrate_memories_layout|def _exact_entry_exists|def _model_entry_exists|_find_case_colliding_id\(' slayer/storage/yaml_storage.py
rg -n -A65 -B10 'def _check_layout_case_collisions|def migrate_yaml_layout' slayer/storage/v4_migration.py
printf '%s\n' '=== Deterministic migration regression test ==='
rg -n -A75 -B10 'listdir|memories.*case|case.*memories|migrate_memories_layout' tests/test_id_collision.py tests/test_yaml_exact_entry.py
printf '%s\n' '=== Custom backend documentation contract ==='
rg -n -A100 -B10 'class MyCustomStorage|_save_model_impl|_delete_model_row|_save_datasource_impl|_delete_datasource_row' docs/configuration/storage.md
printf '%s\n' '=== Current collision flags and SQLite behavior ==='
rg -n -A20 -B8 '_ids_collide_as_filenames|check_datasource_id_collision|_check_model_identity_collision' slayer/storage/base.py slayer/storage/yaml_storage.py slayer/storage/sqlite_storage.py slayer/storage/join_sync.pyLength of output: 40436
The SQLite atomicity and uniform-rejection findings are stale under that revised contract. The datasource write-lock and ✏️ Learnings added
|
|
whimo
changed the title
1 participant
Closes #249.
Problem
YAMLStorageuses raw logical ids as filenames, so ids differing only by case (Xvsx) address the same file on case-insensitive filesystems (macOS / Windows defaults) and silently overwrite each other — datasources, models, and memories alike. Reads and deletes were equally unsafe (get_model("X")returned modelx;delete_model("X")removedx.yaml), and the legacy memory migration could collapse colliding records and then deletememories.yaml.Fix: fail loudly in the YAML backend
Instead of the reversible filename encoding suggested in the issue, saving an id that differs only by case (
casefold()-equal, raw-different) from an existing one raises a newIdCollisionError(SlayerError, ValueError)in the YAML backend — deterministically on every platform, so a YAML store created on Linux stays portable to macOS/Windows. Exact-id re-saves remain upserts. SQLite is unchanged: its keys are case-sensitive TEXT PKs and case-variant ids remain distinct rows (a SQLite store holding such a pair can't be exported to the YAML layout without renaming — documented).Mechanism: the checks live in the
StorageBackendbase save templates — which preserves thesave_model(_validate=False)migration-write-back bypass and coverage through theJoinSyncStoragewrapper — gated on a class flag,_ids_collide_as_filenames(False on the base, True onYAMLStorage, copied from the inner backend by the wrapper). Model saves check both identity components:data_sourceagainst registered datasource names ∪ other models' data_sources (they share themodels/<ds>/directory namespace), andnamewithin the same datasource. The datasource check is a public base helper,check_datasource_id_collision(name), called byYAMLStorage.save_datasource;save_datasourceitself stays a backend-implemented abstract method, so third-party overrides keep working. A custom backend with a case-insensitive keyspace opts in via the flag + helper.YAML read/delete exact-name verification
YAML lookups and deletes now compare the exact entry against
os.listdir(_exact_entry_exists) instead ofos.path.exists/bareopen, so on macOS a case-variantget_model/get_datasource/get_memoryreturns not-found and a delete is a no-op — instead of silently reading or removing the wrong entity in a store copied over from a case-sensitive system.Migration pre-checks
migrate_memories_layoutandmigrate_yaml_layout(restructured to plan-then-move) refuse to run when targets would collide by case, before writing anything — legacy files preserved for manual repair.Surfacing
REST
POST /datasources→ 400 (model/memory endpoints already mappedValueError→ 400); MCPcreate_datasourcereturns a friendly message and its auto-ingest loop reports per-model save failures without aborting (quoted case-variant tables like"Orders"/orders: first wins, second reported); CLI prints-and-exits on datasource create, skips-and-continues in ingest loops.ingest_datasource_idempotentalready isolated per-model errors intoIngestionResult.errors.Tests
test_case_sensitive_ids(which only passed for the yaml param because CI runs on a case-sensitive FS) becomestest_case_variant_ids: YAML rejects, SQLite stores both distinctly.tests/test_id_collision.py: helpers; YAML rejection for all three entity kinds incl. cross-namespace datasource/model checks and the_validate=Falsebypass; SQLite-allows coverage (direct and wrapped); theresolve_storage(JoinSync-wrapped YAML) path; migration refusals.tests/test_yaml_exact_entry.py: read/delete not-found semantics for case-variant lookups (assertions hold on both case-sensitive and case-insensitive dev machines).Known scope choice: the key is
casefold()only — slightly broader than ASCII case (ß≡ss); Unicode-normalization aliasing (NFC vs NFD) is deliberately out of scope.https://claude.ai/code/session_011sbQKNXcUtGAsCXtmJXHvR
Summary by CodeRabbit
IdCollisionErrorwith collision details.