fix(fal): retry definite uploader rejections before direct CDN fallback

[dennybritz]

With FAL_USE_LOCAL_UPLOADER=1, File.from_path("out.mp4") currently fails when the node-local uploader is full or draining. Those requests were never accepted, so retrying or uploading directly to CDN is safe. A lost response or storage failure can already have accepted work and must stop instead.

This combines PLAT-5704 and PLAT-5748, stacked on #1182 with base cdn-uploader. The companion HTTP status change is isolate-cloud#10637.

Behavior

HTTP 429 raises LocalUploadRejected. save and save_file share one retry policy: try locally three times, waiting 100 ms and 200 ms, then call FalFileRepositoryV3 directly with the original arguments. This bypasses the local-uploader flag and preserves multipart and lifecycle settings.

File.from_bytes(...) / File.from_path("out.mp4")
  -> local upload
       429 -> wait 100 ms -> retry
       429 -> wait 200 ms -> retry
       429 -> direct CDN

Every attempt repeats the complete save operation. Bytes are reusable; large files reopen at the beginning. Small files retain their data for File.as_bytes(). The low-level HTTP client classifies responses without implementing retry policy.

Generic 503s, lost responses, and invalid acceptance receipts stop immediately, even after a 429. falls_back=False still prevents File's outer fallback list from creating a second URL. Missing sockets and other validation/authentication failures keep their existing behavior.

Scope

Unknown-size streams retain the parent PR's behavior: no retries, temporary-file buffering, or direct CDN fallback. This removes the extra stream workflow, replay factories, and rejection JSON protocol. Older uploaders still return 503 for capacity and remain conservative until the companion backend update is deployed.

Validation

• Local toolkit unit suite: 486 passed, 1 skipped (GCLOUD_SA_JSON unset).
• Four real-uploader process cases passed, including 80/101 MiB acceptance followed by rejection retries and direct fallback with no extra reservation.
• Unix-socket tests include a cloudpickled SDK in a separate process, complete body replay, and stopping after uncertain failures.
• pre-commit run --all-files passed, including mypy.

Run locally with FAL_CONFIG_PATH pointing to an empty config and a dummy FAL_KEY, plus PYTHONPATH=projects/fal/src:projects/isolate_proto/src:projects/fal_client/src:projects/fal/openapi-fal-rest, then python -m pytest projects/fal/tests/unit/toolkit -q. The real-uploader regression uses CDN_UPLOADER_TEST_BINARY=<isolate-cloud>/monorust/target/debug/cdn-uploader and a controlled local storage peer; it needs no cloud credentials.

---

Note

Medium Risk
Changes file upload routing when the local uploader rejects work, but retries are limited to explicit 429s and direct CDN fallback preserves prior safeguards against duplicate URLs on uncertain outcomes.

Overview
When the node-local uploader is full or draining, uploads no longer fail outright: HTTP 429 is treated as a definite admission rejection via new LocalUploadRejected, while ambiguous failures (503, lost connections, uncertain acceptance) still stop immediately.

LocalFileRepository.save / save_file now retry the full local upload up to three times (100 ms / 200 ms backoff), then bypass the local uploader and call FalFileRepositoryV3 with the same multipart and lifecycle options. The low-level LocalUploader client only classifies 429; it does not implement retries. Unknown-size save_stream behavior is unchanged.

Tests cover replay-then-CDN for bytes/small/large files, halting when acceptance becomes uncertain, integration with a real uploader binary, and serialized-runner socket retries.

^{Reviewed by Cursor Bugbot for commit 09b873e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[linear-code]

PLAT-5748

PLAT-5704

[claude]

I reviewed this PR and didn't find any bugs. Because it reworks the retry/fallback logic across three upload paths (save, save_file, save_stream) in a critical file-upload code path, a human look would still be worthwhile.

What was reviewed:

• New LocalUploadRejected marker parsing (HTTP 503 + rejection field) and its scoping to only queue_full/draining, leaving other 503s conservative (acceptance_uncertain)
• Replay semantics: byte bodies, file bodies (seek(0) + fresh iterator per attempt), and unknown-size stream sessions (retry only before body is sent)
• Fallback to FalFileRepositoryV3 in save/save_file/save_stream, including the stream-to-tempfile spooling path, which only happens after rejection and before the producer is touched
• Checked whether unknown-size streaming uploads could leak an unhandled LocalUploadRejected — save_stream itself catches it and falls back, so this is handled; checked whether a save()-internal direct-CDN failure bypasses the caller's own fallback_repository=['fal'] list — it does, but that list's 'fal' entry is the same repository already attempted, so no fallback capability is actually lost

Extended reasoning...

The diff adds an explicit-rejection exception type and retry/fallback logic (3 attempts with 0.1s/0.2s backoff) across LocalFileRepository.save, save_file, and save_stream, including a spool-to-tempfile path for unknown-size streams, in a core opt-in local-CDN upload path with no auth/crypto surface touched. Test coverage is substantial (unit, integration against a real Rust uploader binary, and a Unix-socket subprocess scenario) and the replay/retry semantics check out against the code; no CODEOWNERS restricts this path and no prior reviewer feedback is outstanding (no prior review events in the timeline). Despite finding nothing wrong after a thorough multi-round bug hunt, the change is non-trivial reliability-critical logic spanning several code paths, so a human sanity check is still warranted before merge.

[claude]

I reviewed this PR and didn't find any bugs in the retry/fallback logic. Because it changes how node-local upload rejections are retried and fall back to direct CDN (including auth and multipart-option handling), and the approach was substantially refactored since an earlier review of this PR, a human look would still be worthwhile.

What was reviewed:

• _retry_or_fallback in providers/local.py: 3 attempts with 0.1s/0.2s backoff before calling FalFileRepositoryV3 directly, matching the documented policy; LocalUploadRejected is scoped to explicit HTTP 429 only, other failures keep existing acceptance_uncertain handling.
• The CDN fallback path requires fetch_auth_credentials() to succeed, so a request authorized only via a per-request X-Fal-CDN-Token (no FAL_KEY) still fails on fallback — but this matches pre-PR behavior where no fallback existed at all, so the overall request outcome is unchanged, only the error differs.
• save_file's non-multipart branch now forwards multipart/threshold/chunk/concurrency kwargs to super().save_file so the fallback path keeps the caller's multipart settings.

Extended reasoning...

The diff touches fal/toolkit/file/_local_uploader.py (unix-socket HTTP client, classifies 429 as a new LocalUploadRejected) and fal/toolkit/file/providers/local.py (adds a retry/fallback decorator around save/save_file that replays the whole upload up to 3 times then falls back directly to the CDN repository), plus matching unit/integration tests; it touches auth-header construction indirectly (credential lookup on the fallback path) but no injection or permission surface. Three candidate concerns (auth-only-via-CDN-token on fallback, same-body replay on every 429, full-body replay for large files) were checked against the code and PR description and found to match intended, documented behavior rather than regressions. I'm deferring rather than approving because this is a second review of the PR with a substantial refactor since the last pass, and the prior review's content isn't available to me to confirm its concerns were addressed.