Who is trusted, what the code defends against, and the two assumptions that fall out of the design and cannot be engineered away.
| Party | Trusted? | Notes |
|---|---|---|
| The operator's machine — CLI, config, storage keys, Salad API key | Yes | Holds every credential. |
| The SaladCloud control plane | Yes | Issues gateway URLs, schedules groups, is the billing authority. |
The rented GPU node, and the sf-agent running on it |
No | Consumer hardware owned by strangers. It executes your command and can read anything inside the container. |
End users of serve --proxy |
No | The gateway fronts it with auth=false; the app behind it enforces its own auth. |
The load-bearing consequence: no credential ever enters a container. The agent receives presigned URLs, plus the instance-metadata JWT for S4 — never the SaladCloud API key, never storage keys. Both assumptions below are the price of that choice.
So that the assumptions below are read in context, these are already handled in code:
- Every field of the untrusted
ResultEnvelopeis validated before use: artifact names must be plain relative paths and must match an output the run actually declared; the reported part count is capped at the run'smax_artifact_parts; a zero-part artifact is refused rather than silently satisfying the SHA-256 gate. - Archive extraction is confined by
unpack_in(no path traversal, no absolute or symlinked entries) and capped at 100× the compressed size, so a hostiletar | zstdcannot fill the collector's disk. - The
serve --proxyreverse proxy targets a fixed loopback address, strips hop-by-hop headers, and does not follow upstream redirects. - The session API compares its bearer token in constant time and fails closed when the token is unset.
- Local secrets are written owner-only at creation (0600 file, 0700 directory), and a hand-written, world-readable API key file is flagged on use.
ResultEnvelope is written by sf-agent on the rented node, through a presigned PUT
the node fully controls. saladfingers run maps its status / exit_code onto the CLI's
own process exit code, which is exactly what makes saladfingers run -- cargo test usable
in CI.
A compromised node can therefore report Succeeded / exit 0 for work it never ran, or
report failure for work that succeeded. The per-artifact sha256 does not close this: it
protects against a torn multi-part reassembly (a node that died mid-upload leaving a
mixed-generation stream), not against a malicious node, which can simply make its bytes
and its hash agree.
What is guaranteed is the envelope's shape, not its semantics: a hostile envelope cannot escape the output directory, name an artifact you never asked for, or exhaust your disk — but whether the job genuinely passed is the node's word.
This is not fixable in software here. The node runs the work, so any attestation it produces is produced by code the node controls. Real assurance would need hardware attestation (a TEE), which a consumer GPU fleet does not offer.
Guidance. Treat the exit code as a correctness signal, not an integrity guarantee. Do not gate a security-relevant decision — a release, a signing step, a deployment — solely on a rented node's self-report. If you need integrity, verify independently on trusted hardware: have the job emit an artifact you re-check locally, or compare against a digest computed somewhere you trust.
The JobSpec handed to each run contains every presigned URL that run needs: input GETs,
output PUTs, checkpoint PUT/GETs, the gate probe, and the attempts/result PUT+GET. The spec
itself is fetched through a presigned GET passed to the container as SF_JOB_URL.
Lifetime is deliberately generous, because the agent may still need to upload hours later after a reallocation, long after the CLI process is gone:
expiry = clamp(2 × max_duration, floor = 72 h, ceiling = 7 days)
Note the 72-hour floor: shortening a run's --max-duration does not shrink the
window below three days. The result-envelope GET is likewise pinned at 72 h.
Anyone who obtains a spec — or the URLs inside it — can, until expiry, read that run's inputs, overwrite its outputs and checkpoints, and forge its result envelope. And a presigned URL cannot be individually revoked: the only remedies are rotating the storage credentials that signed it (which invalidates every outstanding URL) or deleting the objects it points at.
The blast radius is bounded: URLs are scoped to specific object keys under
runs/<run_id>/, not to the bucket, so a leaked spec exposes that run's objects and
nothing else.
Guidance. Treat a JobSpec as a secret — it is a bundle of live capabilities, not
metadata. If one leaks, rotate the storage credentials named by [storage] access_key_env / secret_key_env; that invalidates outstanding URLs across all runs.
saladfingers gc removes a finished run's remote objects, which closes the window early
for that run.
The two assumptions are accepted residual risk, not defects. Assumption 1 (a node self-reports its own result) and Assumption 2 (a presigned URL is a live, unrevocable capability) both fall out of the load-bearing design choice that no credential ever enters a container. Neither can be engineered away at this layer — closing them would need hardware attestation and per-request URL revocation that a consumer GPU fleet and plain presigned storage do not offer. They are documented and bounded, not open findings.
How the properties are actually assured. Every protection in What is defended is backed by code-review reasoning plus unit and integration tests, and every fix in the hardening series landed with a regression test that asserts the guard trips on the crafted input that motivated it. That establishes the checks are present and fire on the inputs we thought to write.
What has not happened: adversarial testing. The system has never been run against a
deliberately hostile SaladCloud node, and there has been no penetration test or red-team
exercise. The unit tests assert the intended check fires on a crafted input; none of
them stand up the full stack with a genuinely malicious node in the loop and observe the
outcome end to end. The one exception is the reverse proxy: request-target host injection
is covered by an adversarial raw-TCP integration test
(crates/saladfingers-agent/tests/proxy.rs) that writes @host, //host, absolute-form,
and percent-encoded-@ request targets straight to the running proxy socket and asserts a
separate attacker listener is never contacted. That is the closest thing to adversarial
testing in the tree today — the exception, not the rule.
An adversarial-test checklist. A hostile-node or red-team exercise should drive the actual untrusted-node surface — a malicious node writing the result envelope and serving the agent/proxy — and confirm each attempt is refused, bounded, or logged, never a compromise. Concretely, what to test:
- Envelope artifact name, path traversal: a
ResultEnvelopeartifact whosenamecarries..or an absolute/rooted path, aiming to write outside the run's output tree (should hit theis_safe_relativepath-shape gate inadmit_output,runner.rs). - Envelope artifact name, undeclared output: a well-formed relative name the run
never declared, aiming to materialise an unexpected file (should hit the declared-output
allow-list in
admit_output). - Envelope part count, resource exhaustion: an artifact claiming more
partsthan the run'smax_artifact_parts, aiming to drive(0..parts)presigned-URL generation to OOM the host (should hit the part-count cap inadmit_output). - Envelope zero-part artifact: an artifact reported present with zero parts and the
empty-string SHA-256, aiming to satisfy the integrity gate vacuously (should hit the "no
parts to download" refusal in
transfer.rs::download_artifact). - Torn / mixed-generation reassembly: multi-part objects from different upload
generations stitched together, aiming to pass a partial or stale artifact off as whole
(should hit the per-artifact SHA-256 check in
download_artifact). - Decompression bomb: a tiny
tar|zstdoutput or checkpoint that expands enormously, aiming to fill the collector's disk (should hit the 100×MAX_DECOMPRESS_RATIOcap intransfer.rs::decompress). - Tar entry escaping the destination: an archive entry with a traversal, absolute, or
symlinked path, aiming to write outside the extraction root (should hit the
unpack_inguard indecompress). - Proxy request-target host injection (SSRF): crafted request targets steering the proxy's upstream fetch off its fixed loopback address (already covered by the raw-TCP test above — re-run it as the template for any new vector).
- Proxy open-redirect → SSRF amplification: the fronted app returns a 3xx pointing at
an attacker host, aiming to make the proxy chase it server-side (should be neutralised by
redirect(Policy::none()), which returns the redirect to the client instead of following it). - Agent bearer token, brute force / timing: many or near-miss tokens against the
session /
serve --proxyAPI, probing for a length- or content-dependent timing signal (should be flattened by the constant-timect_eqcompare; also confirm no route beyond the health check answers unauthenticated). - Agent API with
SF_AGENT_TOKENunset: start the agent with no token and confirm it fails closed — refusing to serve the exec/file API unless--allow-unauthenticatedis passed explicitly. - Spec substitution: overwrite the
job.jsonobject between the CLI's upload and the agent's fetch, aiming to swap in a different spec (should hit the now-armedSF_JOB_SHA256check, where the agent recomputes the digest and refuses to run on a mismatch). - Presigned-URL abuse inside the validity window: using a leaked input / output /
checkpoint / envelope URL from a
JobSpecto read that run's inputs or forge its outputs (the Assumption 2 surface — expected to succeed withinruns/<run_id>/until expiry; this test documents the blast radius, not a defended boundary).
Bottom line. The untrusted-node boundary has been reviewed repeatedly, and every concrete finding to date is fixed and regression-tested. But the honest next increase in assurance is adversarial testing against a genuinely hostile node — exercising the checklist above with a real node in the loop — not another pass of static review.