feat(v3.9.0): flock on batch maintenance (embed-backfill, bootstrap) - #6
Open
nicoechaniz wants to merge 6 commits into
Open
feat(v3.9.0): flock on batch maintenance (embed-backfill, bootstrap)#6nicoechaniz wants to merge 6 commits into
nicoechaniz wants to merge 6 commits into
Conversation
…te_chapter - memoryctl: update_chapter (in-place edit, FTS5 delete+insert, drops stale embeddings on content/title change, book title/slug sync with collision guard) and delete_chapter (FK cascade for embeddings/links, FTS row removal, empty-book pruning, raw_sha256 report for archival) - CLI: memoryctl.py update / delete subcommands - hmk-memory plugin: librarian tool gains update + delete actions (1.1.0) - tests: 11 new tests against a real temp library.db; replace stale test_get_tool_schemas_empty with an enum-invariant test - also commits the previously deployed-but-uncommitted librarian tool (v3.8.0) and integrates origin's v3.7.3 memoryctl path-resolution fix
- plugin cli.py: update/delete subcommands matching the librarian tool and memoryctl semantics (no-field update exits 2, --keep-book flag) - plugin README: document the librarian tool action table and the embedding-drop / raw_sha256 contracts
Consistent with upsert_book() bumping it on every add_text; covered by test_update_bumps_book_updated_at with a pinned now_ts.
- scripts/corpus_policy.py: file-level blocking (never-touch names/globs) + content-level secret scan (private keys, API tokens, JWTs, Bearer) - Selective embedding: code (.py/.js/.sh) and config (.yaml/.json/.toml) files get embed_disabled=1 at ingest time — FTS5-only, no cloud API - chapters.embed_disabled + embed_disable_reason columns (auto-migrated) - embedding_candidates() / embed-backfill skip disabled chapters - stats() reports embed_disabled breakdown by reason - add_file blocks protected files; ingest_any enforces corpus policy - update_chapter re-scans content for secrets - 30 new tests in tests/test_corpus_policy.py (83 total, all pass) Cards: t_116003b9 (P1 corpus policy) + t_c2f96fa4 (P1 selective embedding)
- New link_suggestions table (status candidate/accepted/rejected) - suggest_links(): K nearest neighbors via cosine similarity on embeddings Filters: no self-links, already-linked, same-book - list_link_suggestions(): list with both chapters' context - review_link_suggestion(): accept → creates chapter_links edge, reject → marks blocked for re-proposal - CLI: suggest-links, review-links (accept/reject/list) - Librarian tool: suggest_links action (read-only — guard against self-approval) - stats() reports suggestions + suggestions_total counts Card: t_9cc06fde (P2)
…tstrap) - _lock_maintenance(): POSIX fcntl.flock non-blocking exclusive lock on $HMK_AGENT_MEMORY_BASE/.maintenance.lock - Fails fast with exit code 3 on conflict; locked commands: backfill_embeddings(), bootstrap() - Read-only commands and single-chapter mutations stay lock-free - No-op when BASE_DIR is None (tests pass unaffected) Card: t_f8d8b335 (P3)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements card t_f8d8b335 (P3): POSIX flock on batch maintenance commands to prevent interleaved runs.
Design
_lock_maintenance(): non-blocking exclusive fcntl.flock on/home/nicolas/.hermes/agent-memory/.maintenance.lockbackfill_embeddings(),bootstrap()Tests
83 pass. No new tests needed — lock is a no-op in test context (BASE_DIR is None for temp DBs). Two overlapping embed-backfill processes would need subprocess orchestration; verified manually that lock acquisition succeeds when no other process holds it.
Building on v3.9.0 (PRs #4 corpus policy, #5 suggest-links).