Skip to content

delivery-kid: port Coconut submit to V2 schema (preview path only)#90

Merged
jMyles merged 1 commit into
cryptograss:productionfrom
magent-cryptograss:fix-coconut-v2-preview
May 31, 2026
Merged

delivery-kid: port Coconut submit to V2 schema (preview path only)#90
jMyles merged 1 commit into
cryptograss:productionfrom
magent-cryptograss:fix-coconut-v2-preview

Conversation

@magent-cryptograss

Copy link
Copy Markdown
Contributor

What

Replace submit_to_coconut body with the minimal V2 schema we just verified end-to-end against api.coconut.co/v2/jobs (HTTP 201 on a real POST).

Also: capture Coconut's 4xx response body in the raised exception so the diagnostics panel surfaces the real reason instead of httpx's generic Client error '400 Bad Request' for url ....

Discovery summary

V2 rejected V1's shape with a chain of 400s, each pointing at one rename/restructure:

V2 error Lesson
The job 'notification' is not a valid object. webhooknotification: {type, url}
[mp4_preview] Format Container 'mp4_preview' is not valid. outputs dict key is the format-spec (mp4, mp4:480p, httpstream), not a free-form label
You must define the 'storage' URL if you specify an output with a 'path'. V2 requires explicit storage — no default hosting
The storage service is not valid. Valid services are s3, gcs, wasabi, dospaces, …, coconut, ipfs_pinata, ipfs_infura. The most helpful error of the bunch — confirms coconut is a valid service
Output param key 'video' is not valid. Output values don't accept nested video/audio blocks anymore

Minimal V2 shape that gets 201:

```json
{
"input": {"url": "..."},
"storage": {"service": "coconut"},
"outputs": {"mp4": {"path": "/preview.mp4"}},
"notification": {"type": "http", "url": "..."}
}
```

Scope

  • Preview path (include_preview=True): fully V2, single 480p MP4 output, no nested codec blocks. Verified against the real API.

  • HLS / finalize path (qualities, trim_start, trim_end): raises NotImplementedError. V2 uses an httpstream output block with HLS+DASH variants whose exact codec/variant shape needs more discovery. Punted to a follow-up PR rather than guessing and re-introducing the silent-400 hell we just escaped.

    Today's failure mode: NotImplementedError → upload_log gets a clear "V2 HLS not yet ported" entry → diagnostics panel surfaces it cleanly. No worse than the silent 400 it produced before, much louder to debug.

What may still need to change after deploy

The V2 job creation shape is nailed down. The webhook callback shape from V2 is not yet observed_update_draft_preview may need updates to handle a different payload structure (e.g. outputs arrived as an array in the creation response, not a dict; the callback might do the same).

Recommendation: deploy this, retry the upload, and observe the next failure (if any). The improved error capture should make the next iteration ~30 seconds rather than another curl-discovery session.

Why now

Pre-existing breakage of the entire preview transcoding path — any video upload reaches preview_status: failed immediately. Surfaced today by Sky's diagnostics-panel PR #82 finally rendering the underlying httpx 401/400 messages, which let us see exactly what Coconut was rejecting.

Test plan

  • Deploy delivery-kid (no --rebuild needed — Python change only)
  • Re-upload a small video into ReleaseDraft:1cdc44a0-... (the test draft from this morning)
  • Confirm preview_status advances past failed and a Coconut job ID appears in the preview_log
  • If the webhook callback handler works → preview_status: ready, side-effect: bot snapshot fires, ReleaseDraft:{id}/diagnostics sub-page should materialize (validates the delivery-kid: snapshot draft diagnostics to wiki sub-page #87 bot path too)
  • If the webhook callback handler doesn't recognise V2's payload shape → another iteration, but with clear error in the diagnostics panel

Coconut V2 rejects the V1 job shape with a chain of 400s — verified
iteratively against api.coconut.co/v2/jobs:

  - "The job 'notification' is not a valid object."
        → V2 renamed webhook → notification: {type, url}
  - "[mp4_preview] Format Container 'mp4_preview' is not valid."
        → outputs dict key is the format-spec (e.g. mp4, mp4:480p,
          httpstream), not a free-form label
  - "You must define the 'storage' URL if you specify an output with a 'path'."
        → V2 requires explicit storage; defaults are gone
  - "The storage service is not valid."  (the helpful one — lists every
        valid service: s3, gcs, wasabi, dospaces, coconut, ipfs_pinata, …)
  - "Output param key 'video' is not valid."
        → V2 outputs don't accept nested video/audio blocks; codec/bitrate
          customization goes into the format-spec key instead

The minimal V2 shape that gets 201 from Coconut:

    {
      "input": {"url": "..."},
      "storage": {"service": "coconut"},
      "outputs": {"mp4": {"path": "/preview.mp4"}},
      "notification": {"type": "http", "url": "..."}
    }

This PR ports only the preview path (include_preview=True). The HLS /
finalize path uses an httpstream output block whose exact shape needs
more discovery work — left raising NotImplementedError so failure is
loud rather than silent.

Also: rewrap the 4xx error so Coconut's response body is included in
the exception message. The previous resp.raise_for_status() lost the
body and the diagnostics panel only saw a generic httpx message —
which is exactly what made today's iterative debugging painful.

Tested with a real POST to V2 (HTTP 201, job id returned). Webhook
callback shape on the V2 side may or may not match what
_update_draft_preview expects today — to be discovered on the next
real upload + observed.
@jMyles
jMyles merged commit 9077586 into cryptograss:production May 31, 2026
jMyles added a commit that referenced this pull request Jun 1, 2026
…fast path

Three related fixes for the regression that produced empty Release
pages (the Nine Pound Hammer case: Release CID points at an IPFS dir
containing only metadata.json, no playable content).

## 1. submit_to_coconut: V2 httpstream for finalize

Replace #90's NotImplementedError. Verified against api.coconut.co/v2/jobs
that the minimal V2 shape returns HTTP 201:

    {
      "input": {"url": "..."},
      "storage": {"service": "coconut"},
      "outputs": {"httpstream": {"hls": {"path": "/hls"}}},
      "notification": {"type": "http", "url": "..."}
    }

Quality variants / trim / codec customization aren't yet exposed
because the V2 schema for those inside httpstream isn't documented
and we get 201 on the minimal shape. Falls back to Coconut V2 defaults
for now; warning logged when callers pass qualities/trim that won't
land. Follow-up to expose those.

## 2. Disable the finalize fast path

The fast path at finalize_sse_generator was reusing state.preview_cid
as the Release CID when wants_transcode + preview_cid + no trim. Safe
pre-V2 (preview produced full HLS), broken post-V2 (preview produces
a single mp4 — different artifact than finalize). Worse: when the V2
webhook download wasn't ported yet, preview_cid pointed at an IPFS
dir containing only metadata.json, and finalize blindly reused that.
That's the chain that produced the empty Release page.

Always go through the finalize transcode path until preview_cid can
be trusted to point at finalize-grade HLS.

## 3. download_hls_outputs: handle V2 callback shape

Coconut V2's webhook callback returns ``outputs`` as a list of dicts
(``[{"key": "httpstream", "url": ..., ...}]``), not the V1 dict shape
(``{"hls_av1_720p": {"url": ...}}``). Normalise both into a uniform
list, recognise the httpstream key, and chase the master playlist →
variant playlists → segments. Drops V1 keys silently.

Added _download_hls_variants_from_master that parses EXT-X-STREAM-INF
entries from the master playlist and fetches each variant + segments.

## Why this is the right scope right now

This is the minimum to get finalize producing real Release pages with
real HLS content again, in time for tour shows. The variant/codec/trim
customization is a clear follow-up but doesn't block uploads working.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants