Skip to content

feat(publish): add upload_post_publisher (narrow follow-up to #7) - #687

Open
mutonby wants to merge 5 commits into
OpenMontageInc:mainfrom
mutonby:feat/upload-post-publisher
Open

mutonby wants to merge 5 commits into
OpenMontageInc:mainfrom
mutonby:feat/upload-post-publisher

Conversation

@mutonby

@mutonby mutonby commented Sep 29, 2026

Copy link
Copy Markdown

This is the narrower follow-up to #7 that you asked for when closing it in triage:

"please reopen as a narrower PR with a clear scope, tests where applicable, and docs/provider-viability evidence for new integrations"

It adds one networked provider under the publish capability, next to export_bundle, following the extension point in export_bundle's own docstring ("A networked publisher ... can be added later as a separate provider under the same publish capability").

Disclosure: I'm on the Upload-Post team. Upload-Post is a hosted service with a free plan: 10 uploads/month on every platform here except TikTok, which needs a paid plan. Nothing in this PR runs unless a user sets UPLOAD_POST_API_KEY, and it adds no dependencies (requests is already in requirements.txt).

How this addresses each point from #7

You asked for In this PR
Narrower, clear scope One tool, tools/publishers/upload_post_publisher.py (tier PUBLISH, capability publish, provider upload_post). Video only, publish now. #7 covered photos, documents, text posts and 11 platforms; none of that is here. No scheduling, comments or DMs.
Tests tests/tools/test_upload_post_publisher.py: 19 mocked tests, no network (your session guard stays on). They cover the contract, registry discovery, status with and without the key, validation, request identity, a schema-valid publish_log, per-platform outcomes, and the interruption cases below. The full suite is green: 1848 passed, against 1829 on main + the 19 new ones, with no other changes. make lint passes.
Docs docs/PROVIDERS.md gets a provider section (setup, scope, pricing) plus rows in the env summary, the mapping and the coverage tables. Also .env.example, docs/ARCHITECTURE.md (the publishers/ line was "(Reserved)"), a Step 5b in publish-director.md, and animated-explainer.yaml tools_available, so the manifest and the skill agree.
Provider-viability evidence See below: real end-to-end output from this branch, plus public API docs, SDKs and adoption.

Design: publishing is irreversible, so never post twice or post stale inputs

Most platforms can't unpublish, or swap a cover, through their APIs once a post is live. A publisher that retries blindly is dangerous. That includes the failure mode seen in this repo in #410: an interrupted create was retried with stale inputs, and the wrong cover went live with no way to correct it. The tool is built around preventing that:

  1. The request id is a hash of what goes live. It covers the SHA-256 of the video and thumbnail, the caption, description, tags, platforms, profile and visibility, plus an attempt counter. It's sent as both request_id and Idempotency-Key. The same publish always maps to the same id, and a changed cover or caption is by construction a different publish.
  2. Resume before upload. Before sending a byte, the tool asks Upload-Post whether that id already exists. If it does, the tool resumes polling instead of uploading again. This works even outside the API's own 24 h idempotency window.
  3. No re-send on a dropped connection. On a network error the tool polls the same id. If the upload never arrived, it says so ("Nothing was published; re-running is safe").
  4. Stale-publish guard. A small ledger next to the export bundle (exports/upload_post_ledger.json) records every submission. If a different version of the same render (another cover or caption) is live or still in flight on an overlapping platform, the tool refuses to publish and names the earlier request. allow_additional_post: true is the explicit override.
  5. Auditable log. It returns a schema-valid publish_log with one entry per platform: published / failed / draft (TikTok inbox) / pending_review (still processing), plus URL or post id and visibility. publish_log.metadata records the request id and the hashes of the video, thumbnail and caption that were actually sent. No schema changes.
  6. Approval first. dry_run: true validates, fingerprints, and reports connected platforms, missing_platforms and conflicts without uploading. The publish-director calls the dry run before the stage's human-approval gate and publishes only after approval, with the same inputs.
  7. Safe defaults. YouTube is private, and TikTok uses the account's own privacy. visibility: "private" is rejected for platforms with no private mode rather than posting publicly. retry_policy.max_retries = 0, because resumption goes through the request id instead.

Registry check (no key set): publish: configured 1/2, available ["local"], unavailable ["upload_post"], and get_status() is unavailable with setup instructions.

Provider-viability evidence

Real end-to-end run from this branch (test profile, a synthetic 1080×1920 render, TikTok SELF_ONLY + YouTube private; the posts are private):

1. dry_run: validates, fingerprints, checks the account, uploads nothing
{
  "tool": "upload_post_publisher",
  "status": "available",
  "estimated_cost_usd": 0.0,
  "would_execute": true,
  "request_id": "om-8f6606699b5b7e8af0874b62d93ac3f5",
  "platforms": ["tiktok", "youtube"],
  "caption": "OpenMontage x Upload-Post provider test #openmontage",
  "input_hashes": {
    "video_sha256": "11d1e4610bd182b7aeedd345a6b70410730b8717ff51c238def672cc98225c03",
    "thumbnail_sha256": "87b31cba93ae1c2be1431858d0fababc32c7a4414ce815938689ef14c453fc5c",
    "caption_sha256": "536027b1bdeda5942a894cf976025a244db57d093c6b00307a674827e101a56f"
  },
  "missing_platforms": [],
  "already_submitted": false,
  "conflicts": []
}
2. Publish: success=True in 22 s, schema-valid publish_log
{
  "version": "1.0",
  "entries": [
    {"platform": "tiktok", "status": "published", "video_id": "v_pub_file~v2-1.7691061209349343254", "visibility": "private",
     "metadata_used": {"title": "OpenMontage x Upload-Post provider test #openmontage", "description": "Provider-viability test of tools/publishers/upload_post_publisher.py", "hashtags": ["openmontage"]},
     "timestamp": "2026-09-29T20:46:38.512288+00:00"},
    {"platform": "youtube", "status": "published", "url": "https://www.youtube.com/watch?v=uuawQDIRsM4", "video_id": "uuawQDIRsM4", "visibility": "private",
     "metadata_used": {"title": "OpenMontage x Upload-Post provider test #openmontage", "description": "Provider-viability test of tools/publishers/upload_post_publisher.py", "hashtags": ["openmontage"]},
     "timestamp": "2026-09-29T20:46:38.512288+00:00"}
  ],
  "metadata": {
    "provider": "upload_post",
    "request_id": "om-8f6606699b5b7e8af0874b62d93ac3f5",
    "attempt": 1,
    "resumed_existing_request": false,
    "upload_post_status": "completed",
    "caption_sent": "OpenMontage x Upload-Post provider test #openmontage",
    "input_hashes": {"video_sha256": "11d1e461…", "thumbnail_sha256": "87b31cba…", "caption_sha256": "536027b1…"},
    "notes": {"youtube": "Post uploaded as Private. No public URL available."}
  }
}
3. Re-running the same publish resumes it: no second upload
{"resumed": true, "request_id": "om-8f6606699b5b7e8af0874b62d93ac3f5",
 "statuses": [["tiktok", "published"], ["youtube", "published"]]}
4. The same render with a corrected cover is blocked while the first version is live
{"success": false,
 "error": "A different version of this render was already submitted and is live or in flight: om-8f6606699b5b7e8af0874b62d93ac3f5 (completed) on tiktok, youtube. Publishing now would create a second post. Check those posts first; set allow_additional_post=true only if a second post is intended.",
 "conflicts": [{"request_id": "om-8f6606699b5b7e8af0874b62d93ac3f5", "state": "completed", "platforms": ["tiktok", "youtube"]}]}
5. A real mid-upload disconnect: one upload, no re-send, both platforms published once

The read timeout was forced to 50 ms so the client hung up while the API was still answering:

### POST raised: ReadTimeout HTTPSConnectionPool(host='api.upload-post.com', port=443): Read timed out. (read timeout=0.05)
### interrupted run: {"success": true, "uploads_sent": 1, "resumed": false,
  "statuses": [["tiktok", "published", "v_pub_file~v2-1.7691061445253351446"],
               ["youtube", "published", "https://www.youtube.com/watch?v=7SD7WYP_zhE"]]}
### re-run: {"success": true, "uploads_sent_total": 1, "resumed": true}

Provider facts:

Platforms TikTok, Instagram (Reels), YouTube (incl. Shorts), LinkedIn, Facebook, X, Threads, Pinterest, Bluesky
Auth One API key (Authorization: Apikey …). Social accounts are connected once in the Upload-Post dashboard, with no developer app or OAuth client per platform
Cost Free plan: 10 uploads/month, all platforms except TikTok. Basic $24/mo ($16/mo yearly) unlimited uploads incl. TikTok; Professional $50/mo. No per-call charge, so estimate_cost() returns 0.0 and PROVIDERS.md states the plans
Limits Plan upload limits, plus each platform's own rules (e.g. YouTube titles ≤ 100 chars, which the tool validates)
API docs Public and maintained: docs.upload-post.com
SDKs & integrations Official Python SDK upload-post (2.13.0), JS SDK upload-post (2.14.1, ~5.1k downloads last month), MCP server @upload-post/mcp, n8n community node
Adoption in OSS video tools Merged in MoneyPrinterTurbo (app/services/upload_post.py) and autoclip (backend/services/upload_post_publisher.py, v1.3.2)

Files

  • tools/publishers/upload_post_publisher.py: the tool
  • tests/tools/test_upload_post_publisher.py: 19 mocked tests
  • docs/PROVIDERS.md, docs/ARCHITECTURE.md, .env.example
  • skills/pipelines/explainer/publish-director.md (Step 5b, and Step 6 on merging log entries)
  • pipeline_defs/animated-explainer.yaml (tools_available)

Happy to trim further. For example, I can drop the ledger guard or the publish-director step if you'd rather land the tool alone first.

…ovider

Narrow follow-up to OpenMontageInc#7 as requested in its triage: one video-only tool under
the publish capability (provider upload_post), next to export_bundle.

- Publishes a finished render to TikTok, Instagram Reels, YouTube, LinkedIn,
  Facebook, X, Threads, Pinterest and Bluesky and returns a schema-valid
  publish_log with one entry per platform.
- The request id is a hash of the video, cover and caption being published and
  is sent as the Idempotency-Key. Before uploading, the tool checks whether
  that id already exists and resumes it, so re-running an interrupted publish
  never posts twice.
- A dropped connection is never answered with a re-send; the tool polls the
  same id.
- A ledger next to the export bundle blocks publishing a different version of
  the same render (e.g. a corrected cover) while an earlier one is live or in
  flight, unless allow_additional_post is set.
- publish_log.metadata records the request id and the SHA-256 of the video,
  thumbnail and caption actually sent.
- dry_run validates, fingerprints and reports connected platforms and
  conflicts without uploading; the publish-director only publishes after the
  stage's approval gate.

Docs: PROVIDERS.md section + tables, .env.example, ARCHITECTURE.md,
publish-director Step 5b, animated-explainer tools_available.
Tests: 19 mocked tests (no network), including the interrupted-create and
corrected-cover cases.
@mutonby
mutonby requested a review from calesthio as a code owner September 29, 2026 20:54
Only definitive pre-acceptance rejections (400/401/402/403/404/413/422/429) and
a connect timeout may end as 'failed' and allow re-sending. A 5xx, a dropped
connection, a timeout or a 2xx without a JSON body may still have been
accepted: the tool polls the request id, keeps tracking it if it exists, and
otherwise records an explicit 'ambiguous' state in the ledger that blocks
re-sending — across runs and restarts — until the user confirms nothing went
out (confirm_not_published). This replaces the old 'not_received' state, which
treated an unconfirmed upload as safe to resend.

Tests: 503 and empty 2xx on the first submit -> re-run -> 0 extra uploads, also
from a fresh process reading the ledger; crash after 'submitting'; 503 after
server-side acceptance is tracked, not failed; 4xx and connect timeout still
allow a retry.
@mutonby

mutonby commented Sep 30, 2026

Copy link
Copy Markdown
Author

Pushed 4b62b6b: a hardening fix to how upload errors are classified.

Only definitive pre-acceptance rejections (400/401/402/403/404/413/422/429) and a connect timeout now end as failed and allow a retry. A 5xx, a dropped connection, a timeout or a 2xx without a JSON body may still have been accepted. In those cases the tool looks the request id up on the status endpoint and keeps tracking it if it exists. If Upload-Post has no record, it writes an explicit ambiguous state to the ledger instead of the old not_received. That state blocks re-sending across runs and restarts until the user confirms nothing went out (confirm_not_published: true). The server-side Idempotency-Key only covers 24 h, so this local gate is what actually prevents a second post.

New tests (no network):

  • 503 on the first submit, then a re-run → 0 extra uploads
  • the same with an empty 2xx
  • the same from a fresh process reading only the ledger
  • a crash after submitting
  • a 503 returned after server-side acceptance is tracked, not failed
  • 400/401/403/422 and a connect timeout still allow a retry

With the previous commit, the 503, empty-2xx, restart and dropped-connection tests fail (it re-sent). With this one, all 28 pass. The full suite shows 1857 passed and make lint is clean.

SanHsien added a commit to SanHsien/OpenMontage that referenced this pull request Sep 30, 2026
…issue OpenMontageInc#664, advance baseline

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

@calesthio calesthio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR and the work on this!

The new ambiguous-upload handling is a good improvement and all 28 supplied tests pass. A completed-request replay case still bypasses the duplicate guard.

[P1] Treat a locally completed submission as durable evidence even when the remote status endpoint later returns 404 (execute around the UNRESOLVED_STATES check). _conflicts skips the same request_id, while UNRESOLVED_STATES omits completed. I wrote a completed ledger entry, mocked _get_status to return None, and reran identical inputs: _submit was called once again, without either confirmation flag. This can create a duplicate after remote status/idempotency retention expires—the exact restart scenario the module promises to prevent. Return the recorded completion or require deliberate additional-post authorization; never infer non-publication from a missing remote record.

[P1] Fail closed on an unreadable/corrupt existing ledger. _read_ledger currently turns every OSError/ValueError into an empty history, which can discard the only evidence of an earlier accepted upload. Distinguish a genuinely absent ledger from a damaged/unreadable one and require recovery before publishing.

Please add restart tests for completed+404 and corrupt/unreadable ledger. All review reproductions used mocks; no content was published.

…corrupt ledgers never re-upload

- A publish recorded as completed is answered from the ledger when Upload-Post
  no longer has the request (status/idempotency retention expired, GET status
  404). The outcome (normalized results + publish_log) is stored with the entry
  on completion, so the replay returns the recorded completion instead of
  inferring 'never published' from the missing remote record. Only
  allow_additional_post=true sends the same publish again.
- _read_ledger distinguishes an absent ledger (empty history) from one that
  exists but cannot be read or parsed (OSError other than ENOENT, invalid JSON,
  unexpected structure): the latter raises LedgerError and execute/dry_run
  fail closed with ledger_unreadable=true, leaving the file untouched.
- Restart tests: completed + remote 404 (and with a platform failure),
  corrupt ledger, unreadable ledger, missing ledger.
@mutonby

mutonby commented Oct 5, 2026

Copy link
Copy Markdown
Author

Thanks for the two reproductions. Both are fixed in e8a2400, with restart tests for each.

[P1] Completed submission replayed after the remote record expires. A ledger entry in state completed is now durable evidence on its own. When GET /api/uploadposts/status answers 404 for that request id, execute returns the recorded completion (the normalized per-platform results and the publish_log are stored with the entry at completion time) with resumed: true, resumed_from: "ledger", and _submit is not called. Only allow_additional_post=true sends the same publish again. Your sequence (completed entry, _get_status → None, identical inputs) now yields one _submit call in total.

[P1] Unreadable or corrupt ledger. _read_ledger only treats FileNotFoundError as an empty history. Any other OSError, invalid JSON or an unexpected structure raises LedgerError; execute and dry_run fail closed with ledger_unreadable: true and the file is left untouched, so the only record of an earlier accepted upload is never discarded.

Tests added: test_completed_publish_is_not_resent_after_remote_record_expires (also covers allow_additional_post), test_completed_publish_with_a_platform_failure_replays_the_failure, test_corrupt_ledger_blocks_publishing (invalid JSON and wrong structure, execute + dry run), test_unreadable_ledger_blocks_publishing, test_missing_ledger_is_an_empty_history. 33 tests in the module pass, plus tests/test_network_guard.py and tests/contracts (1216 passed). All with the fake API; nothing was published.

@mutonby

mutonby commented Oct 30, 2026

Copy link
Copy Markdown
Author

@calesthio both reproductions from your last review are fixed in e8a2400, with restart tests for each (details in my comment above). Would you mind taking another look when you have a moment?

The CI run for e8a2400 is still waiting for maintainer approval, so it has not run yet. Thanks!

@calesthio calesthio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for addressing the earlier status and ledger feedback. I reviewed the updated implementation; the previous findings have been improved, but two current-head blockers remain.

[P1] Do not mark a partially pending publish as successful. In tools/publishers/upload_post_publisher.py::execute, success is bool(published) and not failed. A request with YouTube published and Instagram still processing therefore returns success=True. I reproduced this with a mocked processing response containing those two platform results. This lets the production advance while a requested publication is still incomplete. Require every requested platform to reach an accepted terminal state before reporting complete success, and preserve the request ID and explicit pending/partial status for polling/resume. Add a mixed completed/processing test. The provider documents asynchronous status polling.

[P1] Restore Windows test collection. tests/tools/test_upload_post_publisher.py:434 calls os.geteuid() in a skip condition without checking whether that function exists. Windows has Path.chmod but no os.geteuid, so the focused suite fails during collection with AttributeError and can prevent the full suite from collecting. Guard the platform/function before calling it and verify the test is skipped appropriately on Windows.

No real upload or external publication was performed.

…; tests collect on Windows

- success now requires every requested platform to be published (or delivered
  as a draft). One still processing returns success=False with an explicit
  status (partial_pending / pending / partial_failure), the request id and the
  pending platforms, and the ledger keeps the request in_flight so a re-run
  polls it instead of uploading again.
- the unreadable-ledger test looked up os.geteuid in its skip condition, which
  does not exist on Windows and broke collection. The condition now checks the
  platform and the function first.
@mutonby

mutonby commented Nov 12, 2026

Copy link
Copy Markdown
Author

@calesthio thanks for the second pass. Both blockers are fixed in 95e4d5a, and the branch is merged with current main (the only conflict was one line in docs/ARCHITECTURE.md).

[P1] Partially pending publish reported as successful. Reproduced as you described: YouTube published plus another platform still processing returned success=True.

  • execute now reports success only when every requested platform is published (or delivered as a TikTok inbox draft). Anything else is success=False.
  • The result carries an explicit status (complete, partial_pending, pending, partial_failure, failed) plus published_platforms, pending_platforms, failed_platforms and the request_id. The error names the platforms that are not done and says the request was not re-sent.
  • While any platform is pending the ledger keeps the request in_flight, even if the request-level status already says completed, so a re-run polls the same request ID and never uploads again. It is recorded as completed only once nothing is pending.
  • The replay from the ledger uses the same rule.

New tests, all failing on e8a2400:

  • test_one_platform_still_processing_is_not_a_successful_publish (mixed completed/processing, with the request-level status both in_progress and completed)
  • test_partially_pending_publish_resumes_by_polling_until_every_platform_is_done (three runs, one upload, success only on the last)
  • test_every_platform_still_processing_is_pending
  • test_a_failure_next_to_a_pending_platform_is_a_partial_failure

[P1] Windows test collection. The skip condition no longer calls os.geteuid() unguarded: it checks os.name == "posix" and hasattr(os, "geteuid") first. I do not have a Windows machine, so I checked it by deleting os.geteuid before collection on macOS: the old condition raises AttributeError, the new one collects and skips that test.

Results on Python 3.11: tests/tools/test_upload_post_publisher.py 38 passed. Full suite 2943 passed, 9 failed, all in files this PR does not touch (test_showcase_card_security.py, test_caption_workspace_isolation.py, test_audio_enhance_default_container.py, test_bmts_lite.py); the ones I looked at fail on my local FFmpeg build (drawtext with segoeui.ttf). CI has not run on this branch yet because it is waiting for workflow approval.

No real upload was made for this change; everything runs against the in-memory fake.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants