Context
For S3-backed SoF exports ($viewdefinition-export / $sqlquery-export), pre-signed GET URLs are generated once at shard-write time and baked into the job's CompletedFile.url (crates/rest/src/export/sink.rs, S3Sink::write_shard). Every subsequent status poll returns those same URLs in the manifest.
The spec (operations-common / #365) requires output.location download URLs to remain valid for at least 24 hours after export completion, and the completion poll advertises Expires: completed_at + 24h. The default presign TTL is now 24h (HFS_EXPORT_PRESIGN_TTL_SECS=86400), but the TTL clock starts at write time, not completion time — and a client that re-polls the manifest 23 hours in still receives URLs that are about to expire (or already have, if the operator configured a shorter TTL).
Proposed change
Store the S3 object key (not the pre-signed URL) in CompletedFile, and generate fresh pre-signed URLs at manifest-render time (build_completion_manifest in crates/rest/src/handlers/sof/export.rs). Each poll then hands out URLs with a full TTL window, and the Expires header can reflect the actual presign expiry.
Notes:
FilesystemSink URLs are server-routed and unaffected; the change can be scoped to how the manifest resolves a CompletedFile into a URL (e.g. an ExportSink::download_url(job_id, filename) method, with the FS/in-memory sinks returning their stable URLs).- Presigning requires an async call into the AWS SDK from the status handler; the existing
block_in_place pattern in S3Sink can be reused or the handler made to await it directly.
- Related: pre-signed URLs are also capped by the signing credentials' lifetime (e.g. STS session duration), which can silently undercut the requested TTL — worth a docs note for operators using temporary credentials.
Context
For S3-backed SoF exports (
$viewdefinition-export/$sqlquery-export), pre-signed GET URLs are generated once at shard-write time and baked into the job'sCompletedFile.url(crates/rest/src/export/sink.rs,S3Sink::write_shard). Every subsequent status poll returns those same URLs in the manifest.The spec (operations-common / #365) requires
output.locationdownload URLs to remain valid for at least 24 hours after export completion, and the completion poll advertisesExpires: completed_at + 24h. The default presign TTL is now 24h (HFS_EXPORT_PRESIGN_TTL_SECS=86400), but the TTL clock starts at write time, not completion time — and a client that re-polls the manifest 23 hours in still receives URLs that are about to expire (or already have, if the operator configured a shorter TTL).Proposed change
Store the S3 object key (not the pre-signed URL) in
CompletedFile, and generate fresh pre-signed URLs at manifest-render time (build_completion_manifestincrates/rest/src/handlers/sof/export.rs). Each poll then hands out URLs with a full TTL window, and theExpiresheader can reflect the actual presign expiry.Notes:
FilesystemSinkURLs are server-routed and unaffected; the change can be scoped to how the manifest resolves aCompletedFileinto a URL (e.g. anExportSink::download_url(job_id, filename)method, with the FS/in-memory sinks returning their stable URLs).block_in_placepattern inS3Sinkcan be reused or the handler made to await it directly.