feat(v3.9.0): local-first embeddings — default to BAAI/bge-m3 - #8
Open
nicoechaniz wants to merge 8 commits into
Open
feat(v3.9.0): local-first embeddings — default to BAAI/bge-m3#8nicoechaniz wants to merge 8 commits into
nicoechaniz wants to merge 8 commits into
Conversation
…te_chapter - memoryctl: update_chapter (in-place edit, FTS5 delete+insert, drops stale embeddings on content/title change, book title/slug sync with collision guard) and delete_chapter (FK cascade for embeddings/links, FTS row removal, empty-book pruning, raw_sha256 report for archival) - CLI: memoryctl.py update / delete subcommands - hmk-memory plugin: librarian tool gains update + delete actions (1.1.0) - tests: 11 new tests against a real temp library.db; replace stale test_get_tool_schemas_empty with an enum-invariant test - also commits the previously deployed-but-uncommitted librarian tool (v3.8.0) and integrates origin's v3.7.3 memoryctl path-resolution fix
- plugin cli.py: update/delete subcommands matching the librarian tool and memoryctl semantics (no-field update exits 2, --keep-book flag) - plugin README: document the librarian tool action table and the embedding-drop / raw_sha256 contracts
Consistent with upsert_book() bumping it on every add_text; covered by test_update_bumps_book_updated_at with a pinned now_ts.
- scripts/corpus_policy.py: file-level blocking (never-touch names/globs) + content-level secret scan (private keys, API tokens, JWTs, Bearer) - Selective embedding: code (.py/.js/.sh) and config (.yaml/.json/.toml) files get embed_disabled=1 at ingest time — FTS5-only, no cloud API - chapters.embed_disabled + embed_disable_reason columns (auto-migrated) - embedding_candidates() / embed-backfill skip disabled chapters - stats() reports embed_disabled breakdown by reason - add_file blocks protected files; ingest_any enforces corpus policy - update_chapter re-scans content for secrets - 30 new tests in tests/test_corpus_policy.py (83 total, all pass) Cards: t_116003b9 (P1 corpus policy) + t_c2f96fa4 (P1 selective embedding)
- New link_suggestions table (status candidate/accepted/rejected) - suggest_links(): K nearest neighbors via cosine similarity on embeddings Filters: no self-links, already-linked, same-book - list_link_suggestions(): list with both chapters' context - review_link_suggestion(): accept → creates chapter_links edge, reject → marks blocked for re-proposal - CLI: suggest-links, review-links (accept/reject/list) - Librarian tool: suggest_links action (read-only — guard against self-approval) - stats() reports suggestions + suggestions_total counts Card: t_9cc06fde (P2)
…tstrap) - _lock_maintenance(): POSIX fcntl.flock non-blocking exclusive lock on $HMK_AGENT_MEMORY_BASE/.maintenance.lock - Fails fast with exit code 3 on conflict; locked commands: backfill_embeddings(), bootstrap() - Read-only commands and single-chapter mutations stay lock-free - No-op when BASE_DIR is None (tests pass unaffected) Card: t_f8d8b335 (P3)
…st for export_obsidian - Build into <vault>/.staging/<run_id>/ (invisible to Obsidian) - projection-manifest.json with per-note content sha256 hashes - Idempotency: skip notes whose hash matches manifest; report written/unchanged/removed_orphans counts - Atomic publish: os.rename staging → <vault>/live/ - Orphan pruning: notes for deleted chapters auto-removed - --check mode: drift detector (exit non-zero if projection differs) - Maintenance flock for safety Card: t_6a4e7b40 (P3)
…ual) - DEFAULT_EMBED_PROVIDER: nvidia → local - DEFAULT_EMBED_MODELS[local]: all-MiniLM-L6-v2 → BAAI/bge-m3 - bge-m3 is the collective-memory choice: 1024-dim, multilingual (ES/EN) - No destructive migration: embeddings keyed by (chapter_id, provider, model) so NVIDIA vectors coexist with local ones - Requires sentence-transformers: pip install sentence-transformers Card: t_1b894ea6 (P2)
Fede654
pushed a commit
to Fede654/hermes-memory-kit
that referenced
this pull request
Aug 18, 2026
…time fix(embeddings): honor configured local device
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements card t_1b894ea6 (P2): switch to local-first embedding posture. No document content leaves the machine for embedding.
Decision
BAAI/bge-m3 is the collective-memory choice: 1024-dimensional, multilingual (trained on 100+ languages, strong on Spanish + English), open-weight. It outperforms all-MiniLM-L6-v2 (English-only) and model2vec potion-retrieval-32M (static, no query/passage prefix support) for our mixed ES/EN corpus.
Changes
DEFAULT_EMBED_PROVIDER:nvidia→localDEFAULT_EMBED_MODELS[local]:all-MiniLM-L6-v2→BAAI/bge-m3Migration
Non-destructive. Embeddings are keyed by (chapter_id, provider, model) — NVIDIA vectors coexist with local ones. Operators run a one-time
embed-backfill --provider localto regenerate all vectors locally. Rollback: flip the env var back.Requirements
sentence-transformersmust be installed for the local provider. The kit'srequirements-local-embeddings.txtlists this dependency.Building on v3.9.0 (PRs #4-#7).