Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Contributing

Thanks for improving the Local-first Durable Job Queue.

## Development

Require Go 1.25 or newer.

Format your code before you commit.

```text
gofmt -l .
```

Build every package with this command.

```text
go build ./...
```

Run static checks with this command.

```text
go vet ./...
```

Run the full test suite with this command.

```text
go test -count=1 -race ./...
```

Run queue benchmarks with this command.

```text
go test ./internal/queue -run '^$' -bench Benchmark -benchmem -count=1
```

## Changes

Keep one release slice per pull request.

Add deterministic tests for every behavior you change.

Update the README roadmap and release notes when you add a feature.

Keep the public queue API stable unless a tested correction needs a change.

## Documentation

Write public documentation with ASD-STE100 Issue 9 principles.

Use active voice and short paragraphs.

Use no more than 20 words in instructions.

Use no more than 25 words in descriptive sentences.

Do not use emojis in public documentation.
134 changes: 130 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@

A small durable background queue built with Go and SQLite.

The project shows leases, retries, idempotency, crash recovery, priority dispatch, priority aging, a dead-letter queue, Prometheus metrics, and an append-only event log.
The project shows leases, retries, idempotency, crash recovery, and priority dispatch.

It also shows priority aging, a dead-letter queue, Prometheus metrics, a web inspection UI, and an append-only event log.

## Value

Expand All @@ -22,6 +24,7 @@ The queue separates durable state from worker execution.
- `internal/cli` renders commands, snapshots, history, and the demo.
- `internal/metrics` renders queue state in the Prometheus text format.
- `internal/fixture` provides repeatable sample workloads.
- `internal/web` renders a browser interface and a read-only JSON API.

A job starts as `pending`.

Expand Down Expand Up @@ -74,6 +77,14 @@ go run . history <id> -db queue.db
go run . requeue <id> -db queue.db
```

Browse the queue in a browser with this command.

```text
go run . web -addr :8080 -db queue.db
```

Open `http://localhost:8080` to view the dashboard.

## Commands

### `enqueue`
Expand Down Expand Up @@ -154,6 +165,36 @@ Scrape the endpoint with a Prometheus server.

Use `-once` to print one snapshot and exit.

### `web`

```text
jobqueue web [-addr <addr>] [-db <path>]
```

The command serves a read-only browser interface.

The default address is `:8080`.

The dashboard shows state counts, per-kind breakdowns, recent events, and recent jobs.

The dashboard auto-refreshes every two seconds.

The jobs page filters the job table by kind and state.

The job page shows one job and its complete event timeline.

The interface never mutates the queue.

Use the `requeue` command to change job state from the terminal.

The JSON API exposes the same views for scripts.

The endpoint `/api/overview` returns the dashboard snapshot.

The endpoint `/api/jobs` returns the filtered job list.

The endpoint `/api/jobs/<id>` returns one job and its events.

### `demo`

```text
Expand Down Expand Up @@ -260,6 +301,28 @@ Use the `metrics` command for one snapshot or a live endpoint.

Use `work -metrics-addr` to serve the same endpoint beside a worker.

### Web UI

The `web` command serves a read-only browser interface.

Each page reads the SQLite store directly, so no collection pipeline exists.

The dashboard shows one live view of the queue.

The dashboard auto-refreshes every two seconds.

The jobs page filters by kind and state.

The job page shows the payload and the full event timeline.

The interface is read-only.

Operators keep using the `requeue` command to change job state.

The JSON API supports scripts and other inspection tools.

HTML templates are embedded in the binary, so no asset directory is needed.

### Scheduling

A scheduled job stores its earliest lease time in `run_at`.
Expand Down Expand Up @@ -333,12 +396,50 @@ jobqueue_events_total{type="enqueued"} 5
jobqueue_events_total{type="retried"} 5
jobqueue_events_total{type="dead_lettered"} 1
jobqueue_events_total{type="requeued"} 1

inspect again with: jobqueue inspect -db "..."
inspect a job with: jobqueue history <id> -db "..."
requeue a dead letter with: jobqueue requeue <id> -db "..."
browse the queue with: jobqueue web -db "..."
```

The demo uses generated job IDs and current timestamps.

The final counts depend on the scenario and run deadline.

### Web interface

Start the browser view with this command.

```text
jobqueue web -addr :8080 -db queue.db
```

```text
web interface listening on :8080 (db=queue.db)
open with: http://localhost:8080
```

Open `http://localhost:8080` in a browser.

The dashboard shows the queue state.

The jobs page filters the job list.

The job page shows one event timeline.

Scrape the JSON API for scripts.

```text
curl -s http://localhost:8080/api/overview
```

```text
{"total":4,"stats":[{"state":"pending","count":2},{"state":"leased","count":0},{"state":"completed","count":1},{"state":"dead_letter","count":1},{"state":"failed","count":0}],"kinds":[...],"events":[...],"jobs":[...]}
```

The counts in the example come from a seeded database.

## Verification

Run the full test suite with this command.
Expand Down Expand Up @@ -376,7 +477,13 @@ A high-priority stream can delay lower-priority jobs until aging lifts them.

The worker is one process and does not coordinate across hosts.

The project does not provide a web interface.
The web interface is read-only.

Use the `requeue` command to change job state.

The web interface has no authentication.

Bind it to localhost when the database is sensitive.

## Roadmap

Expand All @@ -386,12 +493,31 @@ The project does not provide a web interface.
- [x] Priority aging to prevent starvation.
- [x] Dead-letter queue with requeue of permanently failed jobs.
- [x] Prometheus metrics for queue inspection.
- [ ] Web UI for queue inspection.
- [x] Read-only web UI for queue inspection.
- [ ] Browser-based requeue of dead-lettered jobs.
- [ ] Horizontal scaling with a shared SQLite file.

### Release notes

This release adds Prometheus metrics.
This release adds a read-only web UI for queue inspection.

The new `web` command serves a browser interface on `:8080` by default.

The dashboard shows state counts, per-kind breakdowns, recent events, and recent jobs.

The dashboard auto-refreshes every two seconds.

The jobs page filters by kind and state.

The job page shows the payload and the full event timeline.

The JSON API exposes the same views for scripts.

The `queue` package now shares one canonical state and event order.

The metrics exporter and the web UI both use the shared order.

The previous release added Prometheus metrics.

The new `metrics` command serves the exposition format over HTTP.

Expand Down
28 changes: 28 additions & 0 deletions SECURITY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Security

The Local-first Durable Job Queue is a local-first tool.

It has no authentication or authorization layer.

The web interface is read-only.

It does not mutate the queue.

Run the web interface on localhost when the database is sensitive.

```text
jobqueue web -addr 127.0.0.1:8080 -db queue.db
```

The database may contain private job payloads.

Protect the database file with the same rules as your other secrets.

## Reporting

Report a security issue in the GitHub issue tracker.

Do not include secret material in the report.

Describe the problem and the version you found it in.

16 changes: 16 additions & 0 deletions cmd/web/main.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package main

import (
"log"
"os"

"github.com/local-first-job-queue/internal/cli"
)

func main() {
log.SetFlags(0)
if err := cli.Web(os.Args[1:]); err != nil {
log.Fatal(err)
os.Exit(1)
}
}
1 change: 1 addition & 0 deletions internal/cli/demo.go
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@ func Demo(args []string) error {
fmt.Printf("\ninspect again with: jobqueue inspect -db %q\n", path)
fmt.Printf("inspect a job with: jobqueue history <id> -db %q\n", path)
fmt.Printf("requeue a dead letter with: jobqueue requeue <id> -db %q\n", path)
fmt.Printf("browse the queue with: jobqueue web -db %q\n", path)
return nil
}

Expand Down
37 changes: 37 additions & 0 deletions internal/cli/web.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
package cli

import (
"flag"
"fmt"
"log"
"net/http"

"github.com/local-first-job-queue/internal/queue"
"github.com/local-first-job-queue/internal/web"
)

// Web serves the read-only browser interface for a queue database. The pages
// render fresh snapshots from SQLite, so an operator can watch a live worker
// from the dashboard without an extra collection pipeline. The interface never
// mutates the queue; mutations stay in the other commands.
func Web(args []string) error {
fs := flag.NewFlagSet("web", flag.ExitOnError)
addr := fs.String("addr", ":8080", "listen address for the web interface")
dbPath := fs.String("db", "queue.db", "database path")
fs.Parse(args)

store, err := queue.NewSQLiteStore(*dbPath)
if err != nil {
return fmt.Errorf("open store: %w", err)
}
defer store.Close()

srv, err := web.New(store)
if err != nil {
return fmt.Errorf("prepare web interface: %w", err)
}

log.Printf("web interface listening on %s (db=%s)", *addr, *dbPath)
log.Printf("open with: http://localhost%s", *addr)
return http.ListenAndServe(*addr, srv.Handler())
}
27 changes: 2 additions & 25 deletions internal/metrics/metrics.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,29 +13,6 @@ import (
"github.com/local-first-job-queue/internal/queue"
)

// The canonical state order keeps the exposition stable across scrapes. The
// failed state appears for legacy databases that predate the dead-letter
// queue, so exporters still report it.
var stateOrder = []queue.JobState{
queue.StatePending,
queue.StateLeased,
queue.StateCompleted,
queue.StateDeadLetter,
queue.StateFailed,
}

var eventTypeOrder = []queue.EventType{
queue.EventEnqueued,
queue.EventScheduled,
queue.EventLeased,
queue.EventAcknowledged,
queue.EventFailed,
queue.EventRetried,
queue.EventRecovered,
queue.EventDeadLettered,
queue.EventRequeued,
}

// Option configures a Collector.
type Option func(*Collector)

Expand Down Expand Up @@ -100,7 +77,7 @@ func (c *Collector) writeStateGauge(w io.Writer, stats map[queue.JobState]int) e
if _, err := fmt.Fprintln(w, "# TYPE jobqueue_jobs gauge"); err != nil {
return err
}
for _, state := range stateOrder {
for _, state := range queue.StateOrder {
if _, err := fmt.Fprintf(w, "jobqueue_jobs{state=%q} %d\n", state, stats[state]); err != nil {
return err
}
Expand Down Expand Up @@ -134,7 +111,7 @@ func (c *Collector) writeEventCounters(w io.Writer, evCounts []queue.EventTypeCo
if _, err := fmt.Fprintln(w, "# TYPE jobqueue_events_total counter"); err != nil {
return err
}
for _, et := range eventTypeOrder {
for _, et := range queue.EventTypeOrder {
if _, err := fmt.Fprintf(w, "jobqueue_events_total{type=%q} %d\n", et, byType[et]); err != nil {
return err
}
Expand Down
Loading
Loading