Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 67 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,9 @@

A small durable background queue built with Go and SQLite.

The project shows leases, retries, idempotency, crash recovery, priority dispatch, priority aging, a dead-letter queue, Prometheus metrics, and an append-only event log.
The project shows leases, retries, idempotency, crash recovery, priority dispatch, and priority aging.

It also shows a dead-letter queue, Prometheus metrics, a web dashboard, and an append-only event log.

## Value

Expand All @@ -21,6 +23,7 @@ The queue separates durable state from worker execution.
- `internal/fault` injects deterministic errors, panics, delays, and stalls.
- `internal/cli` renders commands, snapshots, history, and the demo.
- `internal/metrics` renders queue state in the Prometheus text format.
- `internal/web` serves a read-only HTML dashboard for queue inspection.
- `internal/fixture` provides repeatable sample workloads.

A job starts as `pending`.
Expand All @@ -44,6 +47,7 @@ go build -o jobqueue .
./jobqueue enqueue -kind email -payload '{"to":"user@example.com"}' -priority 20
./jobqueue work -kind email
./jobqueue inspect
./jobqueue web
```

Use `-priority` to place urgent work ahead of normal work.
Expand Down Expand Up @@ -89,7 +93,7 @@ jobqueue enqueue -kind <type> -payload <json> [-priority <n>] [-idempotency-key
### `work`

```text
jobqueue work -kind <type> [-concurrency <n>] [-lease <duration>] [-poll <duration>] [-aging <duration>] [-metrics-addr <addr>] [-db <path>]
jobqueue work -kind <type> [-concurrency <n>] [-lease <duration>] [-poll <duration>] [-aging <duration>] [-metrics-addr <addr>] [-web-addr <addr>] [-db <path>]
```

The worker recovers expired leases when it starts.
Expand All @@ -102,6 +106,26 @@ Use `-aging 0` to disable aging.

Use `-metrics-addr` to serve Prometheus metrics beside the worker.

Use `-web-addr` to serve the web dashboard beside the worker.

### `web`

```text
jobqueue web [-addr <addr>] [-db <path>]
```

The command serves a read-only HTML dashboard.

Open the dashboard in a browser to inspect queue state.

The dashboard shows state counts, jobs, and recent events.

Each job page shows one job and its full event timeline.

The default address is `:8080`.

Every page reads the store and shows current data.

### `inspect`

```text
Expand Down Expand Up @@ -260,6 +284,20 @@ Use the `metrics` command for one snapshot or a live endpoint.

Use `work -metrics-addr` to serve the same endpoint beside a worker.

### Web dashboard

The `web` command serves a read-only HTML dashboard.

The dashboard shows state counts, jobs, and recent events.

Each job page shows one job and its event timeline.

The pages read the store on every request.

The dashboard refreshes every five seconds.

Use `work -web-addr` to serve the dashboard beside a worker.

### Scheduling

A scheduled job stores its earliest lease time in `run_at`.
Expand Down Expand Up @@ -339,6 +377,16 @@ The demo uses generated job IDs and current timestamps.

The final counts depend on the scenario and run deadline.

Serve the web dashboard with this command.

```text
jobqueue web -db queue.db
2026/08/04 12:00:00 dashboard listening on :8080 (db=queue.db)
2026/08/04 12:00:00 open the dashboard at http://localhost:8080
```

The dashboard shows the same data as `jobqueue inspect`.

## Verification

Run the full test suite with this command.
Expand Down Expand Up @@ -376,7 +424,7 @@ A high-priority stream can delay lower-priority jobs until aging lifts them.

The worker is one process and does not coordinate across hosts.

The project does not provide a web interface.
The dashboard is read-only and cannot change queue state.

## Roadmap

Expand All @@ -386,12 +434,26 @@ The project does not provide a web interface.
- [x] Priority aging to prevent starvation.
- [x] Dead-letter queue with requeue of permanently failed jobs.
- [x] Prometheus metrics for queue inspection.
- [ ] Web UI for queue inspection.
- [x] Web UI for queue inspection.
- [ ] Horizontal scaling with a shared SQLite file.

### Release notes

This release adds Prometheus metrics.
This release adds a web dashboard for queue inspection.

The new `web` command serves a read-only HTML interface.

The dashboard shows state counts, jobs, and recent events.

Each job page shows one job and its full event timeline.

The pages read the SQLite store on every request.

Use `work -web-addr` to serve the dashboard beside a worker.

The dashboard is read-only; the queue API keeps full control.

The previous release added Prometheus metrics.

The new `metrics` command serves the exposition format over HTTP.

Expand Down
1 change: 1 addition & 0 deletions internal/cli/demo.go
Original file line number Diff line number Diff line change
Expand Up @@ -208,6 +208,7 @@ func Demo(args []string) error {
fmt.Printf("\ninspect again with: jobqueue inspect -db %q\n", path)
fmt.Printf("inspect a job with: jobqueue history <id> -db %q\n", path)
fmt.Printf("requeue a dead letter with: jobqueue requeue <id> -db %q\n", path)
fmt.Printf("view in a browser with: jobqueue web -db %q\n", path)
return nil
}

Expand Down
31 changes: 31 additions & 0 deletions internal/cli/web.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
package cli

import (
"flag"
"fmt"
"log"
"net/http"

"github.com/local-first-job-queue/internal/queue"
"github.com/local-first-job-queue/internal/web"
)

// Web serves a read-only HTML dashboard for queue inspection. The dashboard
// shows state counts, jobs, and recent events, and each job page shows its full
// event timeline. Open the printed address in a browser.
func Web(args []string) error {
fs := flag.NewFlagSet("web", flag.ExitOnError)
addr := fs.String("addr", ":8080", "listen address for the dashboard")
dbPath := fs.String("db", "queue.db", "database path")
fs.Parse(args)

store, err := queue.NewSQLiteStore(*dbPath)
if err != nil {
return fmt.Errorf("open store: %w", err)
}
defer store.Close()

log.Printf("dashboard listening on %s (db=%s)", *addr, *dbPath)
log.Printf("open the dashboard at http://localhost%s", *addr)
return http.ListenAndServe(*addr, web.Handler(store))
}
26 changes: 21 additions & 5 deletions internal/cli/work.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (
"github.com/local-first-job-queue/internal/fault"
"github.com/local-first-job-queue/internal/metrics"
"github.com/local-first-job-queue/internal/queue"
"github.com/local-first-job-queue/internal/web"
"github.com/local-first-job-queue/internal/worker"
)

Expand All @@ -37,6 +38,7 @@ func Work(args []string) error {
pollInterval := fs.Duration("poll", time.Second, "time between lease attempts")
aging := fs.Duration("aging", queue.DefaultAgingInterval, "priority aging interval; a job gains one priority point per interval it waits (0 disables)")
metricsAddr := fs.String("metrics-addr", "", "address to serve Prometheus metrics on, e.g. :9090 (empty disables)")
webAddr := fs.String("web-addr", "", "address to serve the web dashboard on, e.g. :8080 (empty disables)")
fs.Parse(args)

store, err := queue.NewSQLiteStore(*dbPath, queue.WithAgingInterval(*aging))
Expand All @@ -53,16 +55,25 @@ func Work(args []string) error {
worker.WithPollInterval(*pollInterval),
)

var srv *http.Server
var metricsSrv, webSrv *http.Server
if *metricsAddr != "" {
srv = &http.Server{Addr: *metricsAddr, Handler: metrics.Handler(store)}
metricsSrv = &http.Server{Addr: *metricsAddr, Handler: metrics.Handler(store)}
go func() {
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
if err := metricsSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Printf("metrics server: %v", err)
}
}()
log.Printf("metrics listening on %s", *metricsAddr)
}
if *webAddr != "" {
webSrv = &http.Server{Addr: *webAddr, Handler: web.Handler(store)}
go func() {
if err := webSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
log.Printf("web dashboard: %v", err)
}
}()
log.Printf("web dashboard listening on %s", *webAddr)
}

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
Expand All @@ -71,10 +82,15 @@ func Work(args []string) error {
*kind, *concurrency, *leaseDuration, *pollInterval, *aging)
err = w.Run(ctx)

if srv != nil {
if metricsSrv != nil {
shutdownCtx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
_ = metricsSrv.Shutdown(shutdownCtx)
}
if webSrv != nil {
shutdownCtx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx)
_ = webSrv.Shutdown(shutdownCtx)
}
if errors.Is(err, context.Canceled) {
// A signal cancelled the run. This is a normal shutdown, not an error.
Expand Down
84 changes: 84 additions & 0 deletions internal/web/dashboard.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
{{define "title"}}Dashboard{{end}}
{{define "content"}}
<h1>Dashboard</h1>

<div class="grid">
{{range .Stats}}
<div class="stat {{stateClass .State}}">
<div class="num">{{.Count}}</div>
<div class="lbl">{{.State}}</div>
</div>
{{end}}
</div>

<section class="card">
<h2>Jobs by kind and state</h2>
{{if .Kinds}}
<table>
<thead>
<tr><th>Kind</th><th>State</th><th>Count</th></tr>
</thead>
<tbody>
{{range .Kinds}}
<tr>
<td>{{.Kind}}</td>
<td><span class="badge {{stateClass .State}}">{{.State}}</span></td>
<td>{{.Count}}</td>
</tr>
{{end}}
</tbody>
</table>
{{else}}
<p>No jobs yet.</p>
{{end}}
</section>

<section class="card">
<h2>Recent events ({{len .Events}})</h2>
{{if .Events}}
<table>
<thead>
<tr><th>Time</th><th>Job</th><th>Event</th><th>Detail</th></tr>
</thead>
<tbody>
{{range .Events}}
<tr>
<td>{{fmtTime .Timestamp}}</td>
<td><a href="/jobs/{{.JobID}}">{{shortID .JobID}}</a></td>
<td><span class="badge event">{{.EventType}}</span></td>
<td>{{if .Metadata}}{{.Metadata}}{{else}}<span class="muted">none</span>{{end}}</td>
</tr>
{{end}}
</tbody>
</table>
{{else}}
<p>No events recorded yet.</p>
{{end}}
</section>

<section class="card">
<h2>Jobs ({{len .Jobs}})</h2>
{{if .Jobs}}
<table>
<thead>
<tr><th>ID</th><th>Kind</th><th>State</th><th>Priority</th><th>Attempts</th><th>Created</th><th>Payload</th></tr>
</thead>
<tbody>
{{range .Jobs}}
<tr>
<td><a href="/jobs/{{.ID}}">{{shortID .ID}}</a></td>
<td>{{.Kind}}</td>
<td><span class="badge {{stateClass .State}}">{{.State}}</span></td>
<td>{{.Priority}}</td>
<td>{{.RetryCount}}/{{.MaxAttempts}}</td>
<td>{{fmtTime .CreatedAt}}</td>
<td><code>{{.Payload}}</code></td>
</tr>
{{end}}
</tbody>
</table>
{{else}}
<p>No jobs in the database.</p>
{{end}}
</section>
{{end}}
47 changes: 47 additions & 0 deletions internal/web/job.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
{{define "title"}}Job {{.Job.ID}}{{end}}
{{define "content"}}
<h1>Job {{.Job.ID}}</h1>
<p class="muted"><a href="/">&larr; back to dashboard</a></p>

<section class="card">
<h2>Details</h2>
<table class="kv">
<tr><th>State</th><td><span class="badge {{stateClass .Job.State}}">{{.Job.State}}</span></td></tr>
<tr><th>Kind</th><td>{{.Job.Kind}}</td></tr>
<tr><th>Priority</th><td>{{.Job.Priority}}</td></tr>
<tr><th>Attempts</th><td>{{.Job.RetryCount}} / {{.Job.MaxAttempts}}</td></tr>
<tr><th>Created</th><td>{{fmtTime .Job.CreatedAt}}</td></tr>
<tr><th>Updated</th><td>{{fmtTime .Job.UpdatedAt}}</td></tr>
{{if .Job.IdempotencyKey}}<tr><th>Idempotency key</th><td>{{.Job.IdempotencyKey}}</td></tr>{{end}}
{{if .Job.RunAt}}<tr><th>Run at</th><td>{{fmtTime .Job.RunAt}}</td></tr>{{end}}
{{if .Job.LeasedUntil}}<tr><th>Leased until</th><td>{{fmtTime .Job.LeasedUntil}}</td></tr>{{end}}
</table>
</section>

<section class="card">
<h2>Payload</h2>
<pre><code>{{.Job.Payload}}</code></pre>
</section>

<section class="card">
<h2>Event timeline ({{len .Events}})</h2>
{{if .Events}}
<table>
<thead>
<tr><th>Time</th><th>Event</th><th>Detail</th></tr>
</thead>
<tbody>
{{range .Events}}
<tr>
<td>{{fmtTime .Timestamp}}</td>
<td><span class="badge event">{{.EventType}}</span></td>
<td>{{if .Metadata}}{{.Metadata}}{{else}}<span class="muted">none</span>{{end}}</td>
</tr>
{{end}}
</tbody>
</table>
{{else}}
<p>No events for this job.</p>
{{end}}
</section>
{{end}}
Loading
Loading