fix(doltlite): guard unsupported git remotes - #28
Open
Wldc4rd wants to merge 23 commits into
Open
Conversation
…3/1205) ClaimIssue, ClaimReadyIssue, UpdateIssue and CloseIssue each hand-rolled their own tx (BeginTx + DOLT_ADD/DOLT_COMMIT + Commit) outside withRetryTx, so a concurrent writer that lost Dolt's optimistic commit-time merge surfaced MySQL 1213/1205 as a hard failure instead of retrying. Dolt has no real row locking — FOR UPDATE / SKIP LOCKED are parse-only no-ops (https://www.dolthub.com/blog/2023-10-23-hold-my-beer/) — so retry is the only safety net under N concurrent workers. These four are the work-queue hot paths: `bd ready --claim`, claim, update, close. Move each body into a withRetryTx closure (which owns BeginTx and the final Commit and retries serialization failures with backoff), preserving the single-tx CAS/mutation + DOLT_COMMIT atomicity exactly. On retry, ClaimReadyIssue re-scans the ready front from a fresh snapshot and claims the next available issue. Tested: TestEmbeddedReadyClaimConcurrent, TestEmbeddedReadyConcurrent, TestEmbeddedCloseConcurrent, TestEmbeddedUpdateConcurrent and the non-concurrent embedded Update/Close/Ready tests pass; build + vet clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nvariant) N workers concurrently dequeue one shared ready-front via ClaimReadyIssue until empty; asserts every issue claimed exactly once, none stranded, no double-claims, and no serialization error ever surfaces (withRetryTx absorbs the 1213/1205 from the claim-CAS cell collision). This is the regression guard for the multi-agent claim queue that #4491 makes safe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Gas Station v1.1 (wy-5r9j). A claim was previously permanent: a worker that
died mid-task stranded its issue in_progress forever. This adds a lease so a
claim can expire and be recovered.
Schema (migration 0054): add lease_expires_at, heartbeat_at (DATETIME) and
row_lock (BIGINT) to issues (and wisps, for uniform issueops routing), plus
idx_issues_lease for the reaper scan. assignee doubles as the lease owner.
Lifecycle (shared issueops, so both server-mode DoltStore and EmbeddedDoltStore
get it):
- Claim stamps lease_expires_at = now + TTL (default 5m, WithLeaseTTL to
override), heartbeat_at = now, and a fresh row_lock.
- bd heartbeat <id> — owner-only; pushes the lease forward and rewrites
row_lock. Fails (ErrNotClaimable/ErrAlreadyClaimed) once the lease is gone.
- bd reclaim --older-than <dur> — reverts in_progress issues whose lease
expired more than <dur> ago back to ready (clears assignee/started_at,
records a lease_reclaimed event). Default grace 2×TTL.
The row_lock trick: Dolt has no row locking and merges concurrent commits
cell-by-cell, so a heartbeat (writes heartbeat_at) racing a reclaim (writes
status) would silently cell-merge into a zombie — an open/unassigned issue that
still carries the worker's fresh heartbeat. Every mutating path (claim, close,
update, heartbeat, reclaim) now rewrites the shared row_lock cell, forcing the
loser to a 1213/1205 serialization conflict that withRetryTx replays. All lease
paths are withRetryTx/withConn wrapped.
Tests (real Dolt): TestRowLockForcesConflictOnDisjointCellWrites proves the bug
without row_lock (silent merge to zombie) and correctness with it (conflict);
TestConcurrentHeartbeatReclaimClose is the integration race (live workers
heartbeat+close vs a continuous reaper vs dead workers) asserting exactly-once,
no lost close, no zombie; plus claim/heartbeat/reclaim unit coverage on both
backends.
Note: lease columns are second-granular DATETIME (Dolt rounds), so sub-second
TTLs are meaningless — production TTL is minutes. Heartbeat writes a Dolt
commit, so cadence must stay ≪ claim cadence to avoid history bloat.
Fixes the test coupling in TestMigration0053PromotesRigWisps, which assumed the
rig-wisp repair was always LatestVersion().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a cgo-backed DoltLite storage implementation under internal/storage/doltlite. The backend opens .beads/doltlite/<database>.db through the DoltLite SQLite driver, implements the Beads DoltStorage surface, delegates issue semantics to shared issueops helpers, and maps version-control operations to native DoltLite SQL functions. Include the backend bootstrap pieces needed for a usable first PR: schema initialization, local file locking, custom status/type backfill, precreated database files for DoltLite open compatibility, direct SQLite driver dependency wiring, smoke and multiprocess coverage, and portable schema cursor metadata detection for DoltLite's SQLite-compatible catalog.
Remove the CLI restriction that limited bd sql to server-backed stores and expose DoltLite's persistent SQL connection through the RawDBAccessor interface. This gives embedded DoltLite workspaces the same diagnostic SQL path as the existing backend while keeping normal operations on typed storage APIs.
Allow context discovery to use an existing workspace-local .beads directory when no Git repository root is available. This keeps bd context and embedded-backend diagnostics usable in DoltLite-only workspaces while preserving Git root discovery when a repository is present.
Distinguish failures that happen after the SQL transaction body commits from failures inside the transaction itself. Graph apply can then recover generated IDs when DoltLite reports a retryable native commit error after rows are already committed, instead of treating successful SQL writes as a failed graph application. Add a storage.PostTransactionCommitError wrapper and coverage for the recovery path so backends with separate SQL and version-control commit phases can report this state precisely.
Align DoltLite commit behavior with the existing Dolt backend by committing permanent issue-table changes while leaving ignored runtime/wisp-only tables out of normal native commits. Keep CommitWithConfig available for operations that intentionally snapshot all working-set changes. Refresh stale persistent connections after retryable concurrent commits and add smoke coverage for commit-pending and native version-control behavior.
Bring the DoltLite backend up to the upstream storage API surface introduced by the lease-primitive work. Add ClaimIssue, ClaimReadyIssue, HeartbeatIssue, and ReclaimExpiredLeases by delegating to the shared issueops implementations, and align DoltLite CRUD/query helpers with the current upstream helper signatures. Also add missing counts and iterator support, current domain config defaults, dependency prefix rename support, compaction snapshot helpers, merge-resolution commit handling, and cleanup for stale duplicate migration files that are already represented in the squashed base schema.
`bd config set types.custom "..."` warned "not a recognized config key. Use
'custom.*' for user-defined keys." even though types.custom is a first-class
key — it's read via getConfigList("types.custom"), validated on import, and
used throughout internal/types. The warning fired only because
recognizedConfigPrefixes listed "status." but not "types.", an asymmetry given
both namespaces expose a .custom key that's consumed identically.
Add "types." to recognizedConfigPrefixes (mirroring "status."). This gates only
the advisory warning in `bd config set`; no behavioral change. Extend
TestIsRecognizedConfigKey to cover types.custom.
Surfaced migrating a Gas City's beads stores to the embedded DoltLite backend,
where `bd config set types.custom <gc-extended-types>` is a required migration
step and the spurious warning appeared on every scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…fig-prefix fix(config): recognize the types.* config namespace
Wldc4rd
force-pushed
the
fix/doltlite-git-remote-guard
branch
4 times, most recently
from
July 4, 2026 03:09
a4aebaf to
a15dd00
Compare
DoltLite remote sync only supports file:// and http:// URLs, while the previous store path allowed git-protocol remotes to be configured and then failed late inside libdoltlite during push/pull. Reject unsupported DoltLite remote URLs at AddRemote time and guard existing bad remotes before push, pull, fetch, and peer sync operations so users get a backend-specific recovery path. Agent-Signature: codex-unknown-model-unknown-reasoning on behalf of Charlie Coutts
Wldc4rd
force-pushed
the
fix/doltlite-git-remote-guard
branch
from
July 4, 2026 03:27
a15dd00 to
501aa3f
Compare
duncan4123
force-pushed
the
main
branch
2 times, most recently
from
July 7, 2026 13:19
5d49b56 to
8f055a9
Compare
duncan4123
force-pushed
the
main
branch
3 times, most recently
from
July 9, 2026 08:07
206590d to
f128a31
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
bd dolt remote addtimenewDoltliteStorestub so pure-Gocmd/bdbuilds report the intended CGO/libdoltlite error instead of failing to compiletesting.Short()skips, documenting constrained gosec cases, and handlingRows.Close()errorsvendorHashto match the current Go module graphContext
DoltLite remote transfer currently fails late inside libdoltlite for
git+ssh://...remotes withURL must start with file:// or http://. The normal Dolt backend has a CLI fallback for git-protocol remotes, but DoltLite's CLI path is a SQLite database file rather than a.doltrepository, so that fallback is not available in this backend.This change keeps DoltLite's backend boundary explicit: new unsupported remotes are rejected before they are persisted, and existing bad remotes fail before transfer with guidance to use the Dolt backend for GitHub/git+ssh sync or replace the remote with a DoltLite-supported
file://orhttp://URL.Verification
PATH=/usr/local/go/bin:$PATH .githooks/pre-commitPATH=/usr/local/go/bin:$PATH make ci-pr-policyPATH=/usr/local/go/bin:$PATH make ci-pr-lintPATH=/usr/local/go/bin:$PATH golangci-lint run --timeout=10m --build-tags=gms_pure_gotmpbd=$(mktemp /tmp/bd-docs.XXXXXX); CGO_ENABLED=0 /usr/local/go/bin/go build -tags gms_pure_go -o "$tmpbd" ./cmd/bd/; ./scripts/check-doc-flags.sh "$tmpbd"; ./scripts/check-doc-freshness.sh; rm -f "$tmpbd"CGO_ENABLED=0 /usr/local/go/bin/go test -tags gms_pure_go ./cmd/bd -run 'TestNocgoNewDoltliteStore_ErrorRequiresCGO|TestNocgoNewDoltStore_ErrorSuggestsCorrectFlag|TestNocgoNewDoltStoreFromConfig_ErrorSuggestsCorrectFlag|TestNocgoNewReadOnlyStoreFromConfig_ErrorSuggestsCorrectFlag' -count=1CGO_ENABLED=0 /usr/local/go/bin/go test -tags gms_pure_go ./internal/storage/dolt -run 'TestConcurrentHeartbeatReclaimClose|TestConcurrentWorkQueueDrain' -count=1CGO_ENABLED=0 /usr/local/go/bin/go test -tags gms_pure_go ./internal/storage/issueops -count=1CGO_ENABLED=1 CGO_CFLAGS='-I/home/charlie/.local/gascity-doltlite/v0.11.24/linux-x64/include' CGO_LDFLAGS='-L/home/charlie/.local/lib -Wl,-rpath,/home/charlie/.local/lib -ldoltlite' /usr/local/go/bin/go test -tags libsqlite3 ./internal/storage/doltlite -count=1CGO_ENABLED=1 CGO_CFLAGS='-I/home/charlie/.local/gascity-doltlite/v0.11.24/linux-x64/include' CGO_LDFLAGS='-L/home/charlie/.local/lib -Wl,-rpath,/home/charlie/.local/lib -ldoltlite' /usr/local/go/bin/go vet -tags libsqlite3 ./internal/storage/doltlitebd:bd dolt remote add origin git+ssh://git@github.com/org/repo.gitnow fails with the DoltLite-specific unsupported remote guidancebd, then rebuiltbd dolt pushfails before transfer with the same guidanceLocal note:
nixis not installed on this host, so the NixvendorHashrefresh is verified by the PR Risk workflow.