Repository navigation
fix(toolkit): keep every failure when upload initiation and fallback both fail (PLAT-5848) - #1214
Conversation
…both fail (PLAT-5848) Upload initiation errors dropped the server response body, and when the GCS fallback was denied its 400 hid the CDN failure that triggered it. - providers/fal.py: initiate errors include the requested storage type and up to 4 KiB of the server response, chained from the HTTPError. - file.py: when the primary and every fallback fail, raise FileUploadFallbackException (a FileUploadException) listing each attempt's error. Without a fallback the original exception is re-raised untouched.
The isolate agent pickles raised exceptions back to the client, and BaseException.__reduce__ re-invokes __init__(*args). With only the message in args, unpickling failed on the missing failures argument.
There was a problem hiding this comment.
I reviewed this PR and have nothing to add beyond the inline finding. Beyond that, I also checked the new fallback summary message (file.py:195-202) for the same kind of unbounded-size concern the PR addresses with its 4 KiB cap on CDN error bodies — but the summary only ever joins the small, fixed number of fallback attempts actually configured, so it isn't a practical issue.
Extended reasoning...
The PR adds FileUploadFallbackException (exceptions.py) to aggregate upload failures across fallback attempts in file.py's _try_with_fallback, and enriches HTTPError messages in providers/fal.py with up to 4 KiB of server response body across six upload-initiation paths; it's diagnostics-only and doesn't touch auth, crypto, or data-access logic. Beyond the confirmed inline finding, I checked the uncapped string concatenation of per-attempt failure messages at file.py:202 against the PR's own 4096-byte cap for CDN response bodies and ruled it out, since the fallback chain is short in practice.
Drop the unused `failures` attribute and custom constructor; the message already lists every attempt. Stop dumping the traceback on each fallback: one line per failed attempt is enough, and the final exception chains the last cause.
…pload-fallback-errors # Conflicts: # projects/fal/src/fal/toolkit/file/file.py
Why
An app saw
Error initiating upload. Status 400: Bad Requestand lost 32 requests(PLAT-5848). That message was useless twice over:
{"detail":"Invalid storage type"}never reached the app(
LEGACY_STORAGE_UPLOAD_ALLOWED_USERS); the CDN failure that triggered the fallbackwas only in runner stdout
What
providers/fal.py: initiate errors now include the requested storage type and upto 4 KiB of the server response, chained
fromthe originalHTTPError. Applied toall six initiation paths (single/multipart, GCS/V3/internal V3).
file.py_try_with_fallback: when the primary and every fallback fail, raiseFileUploadFallbackException(plain subclass ofFileUploadException) whose messagelists each attempt's error. With no fallback configured the original exception is
re-raised untouched. Exceptions with
falls_back = Falseare re-raised immediately, as on main.(
Failed to save to repository fal_v3: <Type>: <msg>, falling back to fal). The finalexception chains the last cause.
Before:
After:
No retry-policy change: 400 stays non-retryable.
Tests
Unit tests for all six initiation paths (JSON / plain / invalid UTF-8 / empty /
truncated bodies, exception chaining, attempt counts) and for the fallback wrapper
(both fail, mixed exception types, no-fallback keeps original exception, pickle
round-trip, one print line per fallback). Verified end to end with
fal runand aforced failure on both attempts.
Note
Medium Risk
Changes exception types/messages on the hot file-upload path when fallbacks exhaust, but behavior for
falls_back=Falseand single-repo failures is preserved; existingFileUploadExceptionhandlers still apply.Overview
Improves file upload error reporting when CDN initiation fails and optional repository fallbacks also fail, so apps see the primary failure and server details instead of only the last fallback error.
Initiate-upload errors (
providers/fal.py): a shared_initiate_upload_errorhelper now adds the storage type, up to 4 KiB of the HTTP error body (with truncation and safe UTF-8 handling), and chains the originalHTTPError. All GCS/V3 initiation paths use it; retry behavior for 4xx vs 5xx is unchanged.Repository fallback (
file.py): when every attempt fails, raises newFileUploadFallbackException(still aFileUploadException) with a message listing each repo’s error in order; single-repo / no-fallback still re-raises the original exception. Fallback logging drops full tracebacks in favor of one line per failed attempt (without leaking credentials from repo objects).Reviewed by Cursor Bugbot for commit 063905a. Bugbot is set up for automated code reviews on this repo. Configure here.