feat(memory): timestamp persistent memories so stale != current (ga-5js1) - #29
Open
Wldc4rd wants to merge 23 commits into
Open
feat(memory): timestamp persistent memories so stale != current (ga-5js1)#29Wldc4rd wants to merge 23 commits into
Wldc4rd wants to merge 23 commits into
Conversation
…3/1205) ClaimIssue, ClaimReadyIssue, UpdateIssue and CloseIssue each hand-rolled their own tx (BeginTx + DOLT_ADD/DOLT_COMMIT + Commit) outside withRetryTx, so a concurrent writer that lost Dolt's optimistic commit-time merge surfaced MySQL 1213/1205 as a hard failure instead of retrying. Dolt has no real row locking — FOR UPDATE / SKIP LOCKED are parse-only no-ops (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/) — so retry is the only safety net under N concurrent workers. These four are the work-queue hot paths: `bd ready --claim`, claim, update, close. Move each body into a withRetryTx closure (which owns BeginTx and the final Commit and retries serialization failures with backoff), preserving the single-tx CAS/mutation + DOLT_COMMIT atomicity exactly. On retry, ClaimReadyIssue re-scans the ready front from a fresh snapshot and claims the next available issue. Tested: TestEmbeddedReadyClaimConcurrent, TestEmbeddedReadyConcurrent, TestEmbeddedCloseConcurrent, TestEmbeddedUpdateConcurrent and the non-concurrent embedded Update/Close/Ready tests pass; build + vet clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nvariant) N workers concurrently dequeue one shared ready-front via ClaimReadyIssue until empty; asserts every issue claimed exactly once, none stranded, no double-claims, and no serialization error ever surfaces (withRetryTx absorbs the 1213/1205 from the claim-CAS cell collision). This is the regression guard for the multi-agent claim queue that #4491 makes safe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gas Station v1.1 (wy-5r9j). A claim was previously permanent: a worker that
died mid-task stranded its issue in_progress forever. This adds a lease so a
claim can expire and be recovered.
Schema (migration 0054): add lease_expires_at, heartbeat_at (DATETIME) and
row_lock (BIGINT) to issues (and wisps, for uniform issueops routing), plus
idx_issues_lease for the reaper scan. assignee doubles as the lease owner.
Lifecycle (shared issueops, so both server-mode DoltStore and EmbeddedDoltStore
get it):
- Claim stamps lease_expires_at = now + TTL (default 5m, WithLeaseTTL to
override), heartbeat_at = now, and a fresh row_lock.
- bd heartbeat <id> — owner-only; pushes the lease forward and rewrites
row_lock. Fails (ErrNotClaimable/ErrAlreadyClaimed) once the lease is gone.
- bd reclaim --older-than <dur> — reverts in_progress issues whose lease
expired more than <dur> ago back to ready (clears assignee/started_at,
records a lease_reclaimed event). Default grace 2×TTL.
The row_lock trick: Dolt has no row locking and merges concurrent commits
cell-by-cell, so a heartbeat (writes heartbeat_at) racing a reclaim (writes
status) would silently cell-merge into a zombie — an open/unassigned issue that
still carries the worker's fresh heartbeat. Every mutating path (claim, close,
update, heartbeat, reclaim) now rewrites the shared row_lock cell, forcing the
loser to a 1213/1205 serialization conflict that withRetryTx replays. All lease
paths are withRetryTx/withConn wrapped.
Tests (real Dolt): TestRowLockForcesConflictOnDisjointCellWrites proves the bug
without row_lock (silent merge to zombie) and correctness with it (conflict);
TestConcurrentHeartbeatReclaimClose is the integration race (live workers
heartbeat+close vs a continuous reaper vs dead workers) asserting exactly-once,
no lost close, no zombie; plus claim/heartbeat/reclaim unit coverage on both
backends.
Note: lease columns are second-granular DATETIME (Dolt rounds), so sub-second
TTLs are meaningless — production TTL is minutes. Heartbeat writes a Dolt
commit, so cadence must stay ≪ claim cadence to avoid history bloat.
Fixes the test coupling in TestMigration0053PromotesRigWisps, which assumed the
rig-wisp repair was always LatestVersion().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a cgo-backed DoltLite storage implementation under internal/storage/doltlite. The backend opens .beads/doltlite/<database>.db through the DoltLite SQLite driver, implements the Beads DoltStorage surface, delegates issue semantics to shared issueops helpers, and maps version-control operations to native DoltLite SQL functions. Include the backend bootstrap pieces needed for a usable first PR: schema initialization, local file locking, custom status/type backfill, precreated database files for DoltLite open compatibility, direct SQLite driver dependency wiring, smoke and multiprocess coverage, and portable schema cursor metadata detection for DoltLite's SQLite-compatible catalog.
Remove the CLI restriction that limited bd sql to server-backed stores and expose DoltLite's persistent SQL connection through the RawDBAccessor interface. This gives embedded DoltLite workspaces the same diagnostic SQL path as the existing backend while keeping normal operations on typed storage APIs.
Allow context discovery to use an existing workspace-local .beads directory when no Git repository root is available. This keeps bd context and embedded-backend diagnostics usable in DoltLite-only workspaces while preserving Git root discovery when a repository is present.
Distinguish failures that happen after the SQL transaction body commits from failures inside the transaction itself. Graph apply can then recover generated IDs when DoltLite reports a retryable native commit error after rows are already committed, instead of treating successful SQL writes as a failed graph application. Add a storage.PostTransactionCommitError wrapper and coverage for the recovery path so backends with separate SQL and version-control commit phases can report this state precisely.
Align DoltLite commit behavior with the existing Dolt backend by committing permanent issue-table changes while leaving ignored runtime/wisp-only tables out of normal native commits. Keep CommitWithConfig available for operations that intentionally snapshot all working-set changes. Refresh stale persistent connections after retryable concurrent commits and add smoke coverage for commit-pending and native version-control behavior.
Bring the DoltLite backend up to the upstream storage API surface introduced by the lease-primitive work. Add ClaimIssue, ClaimReadyIssue, HeartbeatIssue, and ReclaimExpiredLeases by delegating to the shared issueops implementations, and align DoltLite CRUD/query helpers with the current upstream helper signatures. Also add missing counts and iterator support, current domain config defaults, dependency prefix rename support, compaction snapshot helpers, merge-resolution commit handling, and cleanup for stale duplicate migration files that are already represented in the squashed base schema.
`bd config set types.custom "..."` warned "not a recognized config key. Use
'custom.*' for user-defined keys." even though types.custom is a first-class
key — it's read via getConfigList("types.custom"), validated on import, and
used throughout internal/types. The warning fired only because
recognizedConfigPrefixes listed "status." but not "types.", an asymmetry given
both namespaces expose a .custom key that's consumed identically.
Add "types." to recognizedConfigPrefixes (mirroring "status."). This gates only
the advisory warning in `bd config set`; no behavioral change. Extend
TestIsRecognizedConfigKey to cover types.custom.
Surfaced migrating a Gas City's beads stores to the embedded DoltLite backend,
where `bd config set types.custom <gc-extended-types>` is a required migration
step and the spurious warning appeared on every scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fig-prefix fix(config): recognize the types.* config namespace
Memories live in the shared config key/value table (kv.memory.* rows), which
has only (key, value) columns and no timestamps. `bd memories` and the `bd
prime` injection therefore surfaced every memory undated and truncated, so a
weeks-old memory was indistinguishable from a fresh one, and prime's
alphabetical, uncapped dump silently dropped the most recent hand-offs on
hosts that cap tool-result size (ga-5js1; the house-staff bd-prime-truncation
playbook).
Carry created/updated timestamps inside the memory value as a versioned JSON
envelope ({"_bdmem":1,"content":..,"created_at":..,"updated_at":..}). This is
backend-agnostic (identical under Dolt and DoltLite), sidesteps the DoltLite
CURRENT_TIMESTAMP gap, and avoids widening the shared config table across every
SetConfig path. Legacy bare-string memories remain readable (reported undated)
and gain timestamps on their next rewrite.
- bd remember: writes an envelope; preserves created_at across updates and
refreshes updated_at (remember/recall --json now expose both).
- bd memories: shows an "(updated YYYY-MM-DD)" suffix per memory, searches
unwrapped content, and prints a footer pointing at the existing
recall / remember --key / forget verbs.
- bd recall / forget: unwrap for display.
- bd prime (formatMemoriesForPrime): unwraps envelopes (content, not raw JSON),
orders newest-first by updated_at, caps the full listing to 25, and emits an
explicit "+N older not shown" marker instead of the alphabetical uncapped dump.
Pure envelope helpers unit-tested; verified end-to-end against an embedded
DoltLite store (remember -> memories shows the date; recall shows content; an
update preserves created_at while updated_at advances; prime injects unwrapped,
newest-first, capped).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
duncan4123
force-pushed
the
main
branch
5 times, most recently
from
July 9, 2026 08:07
206590d to
f128a31
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (ga-5js1, Charlie report)
Persistent memories live in the shared config key/value table (
kv.memory.*rows), which has only(key, value)columns and no timestamps. As a result:bd memories/bd recallshow every memory undated, so a weeks-old memory is indistinguishable from a fresh one. (Concrete failure: a staledeacon-wake-stall-knownmemory read as authoritative alongside its…root-causedsuccessor.)bd prime(the SessionStart context-recovery injection) dumps all memories alphabetically by key, uncapped, with no recency — so on hosts that cap tool-result size the block truncates and silently drops the freshest hand-offs (~80% of the time per the house-staffbd-prime-truncationplaybook).Fix
Carry
created_at/updated_atinside the memory value as a versioned JSON envelope:This is deliberately backend-agnostic (just string content, identical under Dolt and DoltLite) — it sidesteps the DoltLite
CURRENT_TIMESTAMP/ON UPDATEgap and avoids widening the sharedconfigtable across everySetConfigpath in both backends. Legacy bare-string memories remain readable (reported "undated") and gain timestamps on next rewrite.bd rememberwrites an envelope, preservingcreated_atacross updates and refreshingupdated_at;remember/recall --jsonexpose both.bd memoriesshows an(updated YYYY-MM-DD)suffix, searches unwrapped content, and prints a footer surfacing the existingrecall/remember --key/forgetverbs (previously undiscoverable frombd memories).bd recall/bd forgetunwrap for display.bd prime(formatMemoriesForPrime) unwraps envelopes (content, not raw JSON), orders newest-first byupdated_at, caps the full listing to 25, and emits an explicit+N older not shownmarker instead of the alphabetical uncapped dump.Verification
memory_envelope_test.go): round-trip, legacy bare-string passthrough, JSON-without-marker treated as legacy, empty, marker presence.go vetclean;cmd/bdcompiles under-tags gms_pure_go.bd remember→bd memoriesshows(updated 2026-07-04);bd recallprints content, not envelope JSON.created_at(20:27:32Z) whileupdated_atadvances (20:28:01Z).bd prime --memories-onlyinjects unwrapped content, newest-first, with the guidance/marker.Notes
A dedicated
memoriestable with real columns is the cleaner long-term home if maintainers prefer columns over an envelope; the envelope was chosen to minimize blast radius and stay backend-identical. Happy to pivot.🤖 Generated with Claude Code