Extend Local-first Durable Job Queue

.github/workflows/ci.yml

@@ -39,4 +39,6 @@ jobs:
       - name: Build
         run: go build ./...
       - name: Benchmark queue paths
-        run: go test ./internal/queue -run '^$' -bench Benchmark -benchmem -count=1
+        run: go test ./internal/queue -run '^$' -bench Benchmark -benchmem -count=1
+      - name: Run demo smoke test
+        run: go run . demo -run 5s

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 DanielCuevas1208
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -2,7 +2,9 @@
 
 A small durable background queue built with Go and SQLite.
 
-The project shows leases, retries, idempotency, crash recovery, priority dispatch, priority aging, a dead-letter queue, Prometheus metrics, and an append-only event log.
+The project shows leases, retries, idempotency, crash recovery, priority dispatch, priority aging, a dead-letter queue, Prometheus metrics, an append-only event log, and shared-file scaling.
+
+Two or more worker processes can drain one SQLite file.
 
 ## Value
 
@@ -27,6 +29,8 @@ A job starts as `pending`.
 
 A worker claims it with a time-limited lease.
 
+The lease claim is one atomic SQL statement.
+
 The worker acknowledges success or records failure.
 
 An expired lease returns to the queue during recovery.
@@ -74,6 +78,17 @@ go run . history <id> -db queue.db
 go run . requeue <id> -db queue.db
 ```
 
+Run two worker processes against the same file.
+
+Each worker claims a disjoint set of jobs.
+
+```text
+jobqueue work -kind email -db queue.db &
+jobqueue work -kind email -db queue.db &
+```
+
+Leases keep the two workers from processing the same job twice.
+
 ## Commands
 
 ### `enqueue`
@@ -94,6 +109,10 @@ jobqueue work -kind <type> [-concurrency <n>] [-lease <duration>] [-poll <durati
 
 The worker recovers expired leases when it starts.
 
+You may run many worker processes against one file.
+
+Each process leases distinct jobs, so no job runs twice.
+
 Priority aging is enabled by default with a 30-second interval.
 
 A job gains one priority point per interval it waits.
@@ -170,6 +189,10 @@ A lease gives one worker temporary ownership of a job.
 
 An expired lease becomes recoverable.
 
+A worker claims a lease with one atomic SQL statement.
+
+Two workers cannot lease the same job, even on one file.
+
 ### Priority dispatch
 
 Ready jobs with higher priority values lease first.
@@ -230,6 +253,26 @@ Recovery consumes an attempt and records a `recovered` event.
 
 A recovered job with no attempts left enters the dead-letter queue.
 
+### Shared-file scaling
+
+Many worker processes can drain one SQLite file.
+
+Each process opens its own store connection.
+
+WAL mode lets readers run in parallel.
+
+The lease claim is a single atomic statement.
+
+The state guard stops two processes from claiming one job.
+
+An acknowledgement carries the lease deadline.
+
+A stale worker cannot finish a job another process owns.
+
+Two processes cannot recover the same lease twice.
+
+The tests run this path with two stores on one file.
+
 ### Event log
 
 Every state change appends one event row.
@@ -364,18 +407,22 @@ Verification status: tests, vet, build, and benchmarks pass locally and in CI.
 
 Race tests run in CI on Ubuntu.
 
+The shared-file tests cover leases, recovery, and acknowledgement across two stores.
+
 ## Limitations
 
 SQLite serializes writes through one store connection.
 
 A sustained backlog can exceed the writer's capacity.
 
+The shared file works best on one host.
+
+Network filesystems can weaken SQLite file locking.
+
 Jobs and events remain until an operator removes them.
 
 A high-priority stream can delay lower-priority jobs until aging lifts them.
 
-The worker is one process and does not coordinate across hosts.
-
 The project does not provide a web interface.
 
 ## Roadmap
@@ -386,12 +433,26 @@ The project does not provide a web interface.
 - [x] Priority aging to prevent starvation.
 - [x] Dead-letter queue with requeue of permanently failed jobs.
 - [x] Prometheus metrics for queue inspection.
+- [x] Horizontal scaling with a shared SQLite file.
 - [ ] Web UI for queue inspection.
-- [ ] Horizontal scaling with a shared SQLite file.
 
 ### Release notes
 
-This release adds Prometheus metrics.
+This release makes the queue safe for shared-file scaling.
+
+The lease claim is now one atomic SQL statement.
+
+A second worker cannot lease a job the first worker holds.
+
+Recovery checks the row count, so two processes cannot double-recover.
+
+An acknowledgement carries the lease deadline.
+
+A stale worker cannot finish a job another process owns.
+
+The tests open two stores on one file to prove the behavior.
+
+The previous release added Prometheus metrics.
 
 The new `metrics` command serves the exposition format over HTTP.
 

internal/queue/queue.go

@@ -207,8 +207,19 @@ func (q *Queue) Lease(ctx context.Context, kind string, leaseDuration time.Durat
 	return job, nil
 }
 
+// Acknowledge marks a leased job completed. The method is kept for callers that
+// hold the lease token separately; see AcknowledgeLease for the lease-aware
+// form used by workers.
 func (q *Queue) Acknowledge(jobID string) error {
-	if err := q.store.CompleteJob(jobID); err != nil {
+	return q.AcknowledgeLease(jobID, nil)
+}
+
+// AcknowledgeLease marks a leased job completed. When leasedUntil is non-nil,
+// the completion only applies while the job still holds that lease deadline. A
+// worker that finished after its lease was recovered by another process gets an
+// error instead of completing a job it no longer owns.
+func (q *Queue) AcknowledgeLease(jobID string, leasedUntil *time.Time) error {
+	if err := q.store.CompleteJob(jobID, leasedUntil); err != nil {
 		return fmt.Errorf("complete job: %w", err)
 	}
 	ev := Event{
@@ -222,7 +233,17 @@ func (q *Queue) Acknowledge(jobID string) error {
 	return nil
 }
 
+// Fail records a failure for a leased job. The method is kept for callers that
+// hold the lease token separately; see FailLease for the lease-aware form used
+// by workers.
 func (q *Queue) Fail(jobID string, errMsg string) error {
+	return q.FailLease(jobID, nil, errMsg)
+}
+
+// FailLease records a failure for a leased job. When leasedUntil is non-nil,
+// the update only applies while the job still holds that lease deadline, so a
+// stale failure cannot corrupt a lease another worker now owns.
+func (q *Queue) FailLease(jobID string, leasedUntil *time.Time, errMsg string) error {
 	job, err := q.store.GetJob(jobID)
 	if err != nil {
 		return fmt.Errorf("get job: %w", err)
@@ -232,7 +253,7 @@ func (q *Queue) Fail(jobID string, errMsg string) error {
 	// The current attempt is RetryCount plus one. Retry only when an additional
 	// attempt still fits inside the limit.
 	shouldRetry := job.RetryCount+1 < job.MaxAttempts
-	if err := q.store.FailJob(jobID, shouldRetry); err != nil {
+	if err := q.store.FailJob(jobID, shouldRetry, leasedUntil); err != nil {
 		return fmt.Errorf("fail job: %w", err)
 	}