Idempotent jobs and cancel

.github/codeql/codeql-config.yml

@@ -0,0 +1,13 @@
+name: "mediary CodeQL config"
+
+# Query exclusions.
+# See: https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#using-queries-in-your-own-repository
+query-filters:
+  # go/request-forgery ("Uncontrolled data used in network request"):
+  # mediary uploads a job's result to a *caller-provided presigned upload URL*.
+  # The destination being caller-controlled is the intended capability, not an
+  # SSRF bug — the URL comes from the trusted client/operator that submitted the
+  # job, and there is no fixed host to allowlist against (any S3-compatible
+  # endpoint, plus localhost in tests). Excluded repo-wide.
+  - exclude:
+      id: go/request-forgery

.github/dependabot.yml

@@ -2,6 +2,10 @@
 # package ecosystems to update and where the package manifests are located.
 # Please see the documentation for all configuration options:
 # https://docs.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
+#
+# Each ecosystem has two groups: one for scheduled version updates and one for
+# security updates. Groups default to applies-to: version-updates, so the
+# security group is needed to bundle vulnerability fixes into a single PR too.
 version: 2
 updates:
   - package-ecosystem: "gomod"
@@ -12,6 +16,11 @@ updates:
       - dependency-type: "direct"
     groups:
       go-dependencies:
+        applies-to: version-updates
+        patterns:
+          - "*"
+      go-security:
+        applies-to: security-updates
         patterns:
           - "*"
   - package-ecosystem: "docker"
@@ -20,6 +29,11 @@ updates:
       interval: "monthly"
     groups:
       docker-dependencies:
+        applies-to: version-updates
+        patterns:
+          - "*"
+      docker-security:
+        applies-to: security-updates
         patterns:
           - "*"
   - package-ecosystem: "github-actions"
@@ -28,5 +42,10 @@ updates:
       interval: "monthly"
     groups:
       github-actions:
+        applies-to: version-updates
         patterns:
-          - "*"
+          - "*"
+      github-actions-security:
+        applies-to: security-updates
+        patterns:
+          - "*"

.github/workflows/codeql.yml

@@ -47,6 +47,7 @@ jobs:
       uses: github/codeql-action/init@v4
       with:
         languages: ${{ matrix.language }}
+        config-file: ./.github/codeql/codeql-config.yml
         # If you wish to specify custom queries, you can do so here or in a config file.
         # By default, queries listed here will override any specified in a config file.
         # Prefix the list here with "+" to use these queries and those in the config file.

CLAUDE.md

@@ -0,0 +1,114 @@
+# mediary
+
+Go service that downloads, converts, and uploads media (magnet/YouTube → mp3 → S3),
+exposed over an HTTP API. Remote: `git@github.com:dir01/mediary.git`.
+
+## Dependency upgrades — do NOT run `go get -u ./...`
+
+A blanket `go get -u ./...` **breaks the build**. It drags the `anacrolix/*` transitive
+dependencies (generics, missinggo, dht, etc.) forward past what `github.com/anacrolix/torrent`
+supports. Those newer anacrolix versions have moved to `github.com/RoaringBitmap/roaring/v2`,
+while `torrent` still uses roaring **v1**, so the two collide inside torrent's own code:
+
+```
+github.com/anacrolix/torrent/peerconn.go: cannot use cn.t._completedPieces.Clone()
+(value of type *roaring.Bitmap) as *roaring/v2.Bitmap value in struct literal
+```
+
+`anacrolix/torrent` is a **direct** dependency (`downloader/torrent`, `tests/local_seeder.go`),
+and `v1.61.0` is already the latest release — there is nothing newer to bump *to*. The whole
+`anacrolix/*` + `RoaringBitmap/roaring` + `pion/*` cluster is tightly coupled and must move in
+lockstep with `torrent`, which can't move. Treat that cluster as **pinned**.
+
+**To upgrade dependencies:** upgrade everything *except* the torrent cluster with a targeted
+list, then tidy. Leave `github.com/anacrolix/*`, `github.com/RoaringBitmap/roaring*`, and
+`github.com/pion/*` untouched. Example (the deps worth refreshing — otel, aws, testcontainers,
+sqlite, and their trees; `golang.org/x/*` come along transitively and torrent tolerates them):
+
+```
+go get -u \
+  github.com/aws/aws-sdk-go-v2 github.com/aws/aws-sdk-go-v2/config \
+  github.com/aws/aws-sdk-go-v2/credentials github.com/aws/aws-sdk-go-v2/service/s3 \
+  github.com/testcontainers/testcontainers-go modernc.org/sqlite \
+  github.com/bogem/id3v2/v2 github.com/gojuno/minimock/v3 github.com/google/uuid \
+  github.com/joho/godotenv github.com/samber/oops github.com/dir01/sqlq \
+  go.opentelemetry.io/otel go.opentelemetry.io/otel/... \
+  go.opentelemetry.io/contrib/bridges/otelslog \
+  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
+go mod tidy
+```
+
+After any upgrade, verify with `go build ./...` before committing. If the build fails on
+`peerconn.go`/roaring, an anacrolix module slipped through — `git checkout -- go.mod go.sum`
+and redo the upgrade without it.
+
+## Go toolchain
+
+`go.mod` targets `go 1.26.4` with **no `toolchain` line**. CI sets up Go via
+`actions/setup-go` with `go-version: 1.26.x` (currently resolves to 1.26.4) and runs under
+`GOTOOLCHAIN=local`, so the `go` directive must not exceed the installed patch — bumping to
+1.26.5 made CI (and any `GOTOOLCHAIN=local` build) fail with "go.mod requires go >= 1.26.5".
+Don't raise it past what `setup-go`'s `1.26.x` resolves to.
+
+## Build & test
+
+- `go build ./...`, `go vet ./...`
+- Tests need **`CGO_ENABLED=1`** (sqlite driver).
+- The `tests/` e2e suite is behind a build tag: `CGO_ENABLED=1 go test -tags gen_docs ./tests/...`.
+  It spins up containers via **testcontainers**, so Docker must be running. It drives the real
+  HTTP mux (`mux.ServeHTTP`), so it's the end-to-end check for routing changes.
+
+## README is autogenerated — regenerate & commit it after any API change
+
+The `## Examples` block of `README.md`, between the markers
+`<!-- start autogenerated samples -->` and `<!-- stop autogenerated samples -->`, is
+**generated by the `tests/e2e_test.go` (`gen_docs`) run**, not hand-written. Each `t.Run`
+subtest emits its doc section via `docs.InsertText(...)` / `docs.PerformRequestForDocs(...)`
+(the `docsHelper` uses `'''` as a stand-in for backticks). `docs.Finish()` **rewrites the
+whole block from only the subtests that ran** — so generation is all-or-nothing:
+
+- If a dependency is missing, that subtest `t.Skip`s and **its section silently disappears
+  from the README**. The `/jobs`, `/jobs/:id`, `DELETE /jobs/:id`, and YouTube-download
+  sections require **localstack S3** (they skip when `s3Client == nil`); the YouTube sections
+  also need **`yt-dlp` + reachable YouTube**. So a regen in an environment missing S3 or
+  YouTube will *delete* real docs.
+- Running the suite also **overwrites `README.md`** as a side effect. Always `git diff README.md`
+  after; if sections vanished, your env was incomplete — do **not** commit that truncation
+  (`git checkout -- README.md` and fix the env instead).
+
+**To document a new endpoint:** add a `t.Run` subtest in `tests/e2e_test.go` that calls
+`InsertText` + `PerformRequestForDocs`, run `CGO_ENABLED=1 go test -tags gen_docs -run TestApplication ./tests/...`
+with Docker + localstack + YouTube all available, confirm the full README regenerated
+(all sections present), and commit `README.md` together with the test change.
+
+## HTTP routing
+
+The router (`http/http.go`) uses Go 1.22+ method-based `ServeMux` patterns
+(`POST /jobs`, `GET /jobs/{id}`, `DELETE /jobs/{id}`, `GET|POST /metadata`, …) with
+`req.PathValue("id")`. Span names come from `req.Pattern` (the low-cardinality route
+template like `GET /jobs/{id}`) — never bake concrete IDs into span names.
+
+## Job lifecycle: cancel vs. purge (and why retry needs purge)
+
+Jobs are **content-addressed**: `calculateJobId` hashes the params (upload URL normalized to
+path-only), and `CreateJob` dedupes on **row existence, not status** — if a row with that id
+exists in *any* state, it's returned as-is (`ErrJobAlreadyExists`, 202) and no work is
+re-enqueued. That's what kills the cost loop, but it also means a failed/cancelled job can't be
+retried by re-creating it — the identical params resolve straight back to the dead record.
+(The one field a duplicate *does* refresh is the stored `uploadUrl`: it carries a fresh presigned
+signature for the same destination, so a not-yet-uploaded run doesn't upload with an expired URL.)
+
+Two `DELETE /jobs/{id}` modes, and both **stop a running flow** first:
+- **default = cancel** (`CancelJob`): stops the flow if running, flips `DisplayStatus` to
+  `cancelled`, and **keeps the row** (an observable terminal state); does *not* free the id.
+- **`?purge=true` = hard delete** (`DeleteJob` → `storage.DeleteJob`): stops the flow if running,
+  then removes the row, freeing the id so an identical `POST /jobs` re-runs the work. This is the
+  retry primitive.
+
+Each flow runs under a cancelable context registered by job id (`Service.runningCancels`);
+cancel/purge cancel it. That also closes a resurrection race: `SaveJob` is an upsert
+(`INSERT OR REPLACE`), so an *un-cancelled* flow's next `updateJobStatus` would re-create a purged
+row — but a cancelled flow's status writes fail on the cancelled context instead. The
+queued-but-not-yet-running cases are handled in `onPublishedJob`, which no-ops when the job is
+missing (purged — also avoids a nil-record panic) or already `cancelled` (so a cancel that landed
+while the job sat in the queue isn't overwritten by a flow that runs anyway).