feat(fal): add SSH lease provider

[coygeek]

Closes #694

Summary

Adds a direct fal Compute SSH lease provider with Crabbox-managed SSH and sync support, local-claim-owned cleanup, and the fal-ai alias.

This also adds fal provider docs, provider matrix metadata, benchmark category generation, and an opt-in guarded live smoke script that defaults to no live provider mutation unless CRABBOX_LIVE=1 and fal is selected.

Lifecycle safety

• Uses the fal Compute Idempotency-Key header with the Crabbox lease ID for creates.
• Fsyncs the SSH keypair, exact create-request binding, and pre-mutation intent before any paid provider call; an untouched intent can be cancelled without creating a resource.
• Publishes a unique in-flight claim revision for every same-key POST and atomically records a known ID, retains an indeterminate outcome, or removes a first-attempt rejection.
• Bounds ambiguous-create replay to fal's documented idempotency window and treats rate limits as explicit rejections.
• Applies that bound as an actual request deadline in both immediate and stop-time replay, preserving a safety margin before provider idempotency expiry.
• Persists a cleanup-only provisional claim immediately after a provider ID is known, before readiness or SSH setup.
• Binds claims to the canonical API endpoint and a one-way credential fingerprint; lifecycle calls fail closed after endpoint or account changes.
• Serializes destructive actions under compare-and-swap claim locks, including mixed-credential cleanup.
• Serializes ambiguous-create recovery, winner adoption, and persistence-failure cleanup so concurrent stops cannot delete the winning instance or orphan a known billable instance.
• If a provisional claim disappears mid-acquire, proves account-wide absence or reclaims/deletes the exact in-memory instance under an expected-absent claim lock; ambiguous cleanup recreates the known-ID recovery claim.
• Reconciles ambiguous unclaimed creates against complete paginated provider inventory without deleting resources Crabbox cannot prove it owns.
• Requires the documented complete fal inventory envelope before accepting a zero-resource proof, and polls raw provider inventory back to the pre-create fingerprint after stop.
• Rolls back callback/readiness/bootstrap failures unless --keep explicitly owns a failed-acquire recovery claim.
• Cleanup removes provider-absent local claims before TTL filtering and only deletes local-claim-owned fal instances.
• Rejects cross-origin bearer redirects, method-rewriting mutation redirects, escaped API base paths, and unsafe SSH login names before any credential-bearing request or SSH invocation.
• Classifies fal Compute credit and automatic-top-up rejections as billing_blocked, with regression coverage, instead of reporting them as generic validation failures.

Verification

• Exact candidate: 7c54ce304c9379e78f6fb11a482552b384228ee8
• Rebased onto current main; generated provider metadata and docs checks pass.
• go vet ./...
• go test -race ./...
• go test -race ./internal/providers/fal ./internal/cli -run 'Fal|fal|RunStopCommandRedactsProviderURLUserinfo|LeaseOptionsFromConfigCanonicalizesProviderScope' -count=1
• go run golang.org/x/tools/cmd/deadcode@v0.45.0 -test ./... — no unreachable functions
• bash -n scripts/live-fal-smoke.sh
• node --test scripts/live-fal-smoke.test.js scripts/live-smoke.test.js (68 passing)
• ./scripts/check-docs.sh (54 command docs, 74-provider matrix, 214 Markdown files, docs-site build)
• git diff --check
• AutoReview: clean, no accepted/actionable findings
• independent transaction and durability audits: clean, no remaining actionable findings

The rebased candidate also fixes the default 1x H100 create path: it no longer sends the 8x-only sector field, even when stale configuration supplies one.

Authenticated live gate

• Authenticated fal Compute doctor and inventory listing passed with a temporary admin-scoped test key.
• The prior live candidate reached create authorization, then fal rejected the request before resource creation because the account did not satisfy its required $500 Compute balance and automatic-top-up threshold.
• The smoke classified that response as billing_blocked; exact post-attempt inventory was empty, so no funded resource or lease residue exists.
• All temporary test keys were revoked after the proof.
• The current exact head has not performed a funded create; the control plane still requires the same account balance/top-up threshold.

Keep this pull request unmerged until the exact candidate completes real create, SSH/use, destroy, and zero-residue proof, or that item-specific funded-provider gate is explicitly waived.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 9, 2026, 10:54 AM ET / 14:54 UTC.

Summary
Adds a built-in direct fal Compute SSH lease provider with env-only auth, claim-bound lifecycle cleanup, docs/provider metadata, and guarded live-smoke coverage.

Reproducibility: not applicable. this is a new provider PR rather than a broken existing behavior report. Source and release checks show current main and v0.36.0 do not already contain fal support.

Review metrics: 2 noteworthy metrics.

• Diff size: 39 files changed, +6860/-7. The PR is a broad new provider addition with docs, tests, scripts, core helpers, and provider code, so proof and owner review matter before merge.
• Provider surface: 1 built-in billable SSH lease provider added. A new direct GPU provider can create and delete paid external resources under user credentials.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #694
Summary: This PR is the implementation candidate for the canonical fal Compute provider issue.

Members:

• canonical: Add a fal.ai Compute provider #694 - The PR body says it closes the issue, and the issue describes the same built-in direct fal Compute SSH lease provider.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• Post redacted exact-head funded create, SSH/use, destroy, cleanup, and zero-residue proof.
• Remove the CHANGELOG.md release-note edit.

Proof guidance:

• [P1] Needs stronger real behavior proof before merge: Needs real behavior proof before merge: partial doctor/inventory and billing-blocked create evidence is useful, but the exact head still needs redacted terminal output, logs, or a transcript for funded create, SSH/use, destroy, and zero-residue cleanup; after updating the PR body, ClawSweeper should re-review automatically, or a maintainer can comment @clawsweeper re-review.

Risk before merge

• [P1] Merging before exact-head live proof could ship a billable provider path whose create, SSH/use, delete, and zero-residue cleanup behavior has not been demonstrated against fal's funded control plane.
• [P1] The PR adds a new built-in provider and several core claim/key durability helpers, so maintainer review should explicitly accept the lifecycle and cleanup safety bar before merge.

Maintainer options:

1. Prove the funded lifecycle (recommended)
   Require exact-head redacted proof for create, SSH/use, stop/delete, cleanup, and empty post-run inventory before merge.
2. Accept first-use provider risk
   Merge with a maintainer waiver that explicitly owns the missing funded-provider proof for the initial release.
3. Pause the provider
   Keep the branch unmerged or close it if the live proof cost or provider uncertainty is not worth carrying in core now.

Next step before merge

• [P1] Manual review is needed because the remaining blocker is proof sufficiency or an explicit waiver for a new billable provider lifecycle, not a safe ClawSweeper repair.

Maintainer decision needed

• Question: Should this fal provider PR require exact-head funded create, SSH/use, destroy, and zero-residue proof before merge, or may that item-specific live gate be waived?
• Rationale: This is a new billable provider lifecycle surface, and automation should not decide whether partial doctor/inventory plus billing-blocked proof is enough to accept create/delete risk.
• Likely owner: steipete — This person has the strongest recent history on provider lifecycle safety and also authored the latest fal hardening commits.
• Options:
  • Require funded lifecycle proof (recommended): Keep the PR open until redacted terminal output, logs, or a transcript shows exact-head create, SSH/use, destroy, and zero-residue cleanup.
  • Waive the live gate: A maintainer can explicitly accept first-use provider risk and merge after ordinary review and CI without funded lifecycle proof.
  • Pause built-in fal support: Hold or close the PR if built-in fal Compute support should wait for stronger API, billing, or ownership guarantees.

Security
Cleared: No concrete security or supply-chain defect was found in the diff; fal credentials are env-only and the client/live-smoke paths include redaction and redirect guards.

Review findings

• [P3] Remove the release-owned changelog entry — CHANGELOG.md:13

Review details

Best possible solution:

Land the provider only after removing the release-owned changelog edit and either posting redacted exact-head live lifecycle proof or getting an explicit maintainer waiver for that funded-provider gate.

Do we have a high-confidence way to reproduce the issue?

Not applicable; this is a new provider PR rather than a broken existing behavior report. Source and release checks show current main and v0.36.0 do not already contain fal support.

Is this the best way to solve the issue?

Mostly yes; the direct SSH-lease implementation matches the linked provider request and Crabbox's provider architecture. The safer merge path is to keep the implementation but require proof or a maintainer waiver before accepting the billable lifecycle.

Full review comments:

• [P3] Remove the release-owned changelog entry — CHANGELOG.md:13
  This repository treats CHANGELOG.md as release-owned, so normal feature PRs should not add release notes there. Please keep this context in the PR body or commit message and leave the changelog for release prep.
  Confidence: 0.98

Overall correctness: patch is correct
Overall confidence: 0.82

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against e5b1f2f54691.

Label changes

Label justifications:

• P2: This is a normal-priority provider feature with bounded but non-trivial lifecycle and billing impact.
• merge-risk: 🚨 other: The main unresolved merge risk is billable external-provider lifecycle proof, which is meaningful but outside the specific owned merge-risk categories.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Needs real behavior proof before merge: partial doctor/inventory and billing-blocked create evidence is useful, but the exact head still needs redacted terminal output, logs, or a transcript for funded create, SSH/use, destroy, and zero-residue cleanup; after updating the PR body, ClawSweeper should re-review automatically, or a maintainer can comment @clawsweeper re-review.

Evidence reviewed

What I checked:

• AGENTS.md policy applied: Repository policy was read fully; its provider-neutral core boundary, token handling, and release-process guidance are relevant to this provider PR. (AGENTS.md:13, e5b1f2f54691)
• Current main lacks fal provider: Current main has no internal/providers/fal package or fal provider registration, so the requested feature is not already implemented. (internal/providers/all/all.go, e5b1f2f54691)
• Latest release lacks fal provider: v0.36.0 also has no fal import in the built-in provider registry. (internal/providers/all/all.go, 4ea85a4f1b7e)
• PR provider contract: The PR registers fal as a Linux-only ProviderKindSSHLease with ssh, crabbox-sync, cleanup, and CoordinatorNever. (internal/providers/fal/provider.go:26, 7c54ce304c93)
• Security-sensitive auth path inspected: The new client requires env-loaded fal credentials, sends Authorization: Key, adds Idempotency-Key for creates, limits response bodies, and rejects unsafe redirects. (internal/providers/fal/client.go:60, 7c54ce304c93)
• Live proof gap is explicit: The PR body says authenticated doctor/inventory and billing-blocked create were tried, but the current exact head has not completed funded create, SSH/use, destroy, and zero-residue proof. (7c54ce304c93)

Likely related people:

• steipete: Current-main history shows recent work on exact direct-provider cleanup, runtime credential redaction, and provider lifecycle safety, and the PR branch also has many fal lifecycle hardening commits from this person. (role: recent provider lifecycle contributor and likely proof decision owner; confidence: high; commits: 65634c2cecaf, d2ad413b1011, c41c6e1cc740; files: internal/cli/claim.go, internal/cli/provider_backend.go, internal/providers/fal/backend.go)
• coygeek: Current-main history includes the Sealos Devbox SSH-lease provider addition, which is structurally adjacent to a new built-in direct SSH lease provider. (role: adjacent SSH-lease provider contributor; confidence: high; commits: b2d058605239, 6f8c55c565a5, 8e008af07eea; files: internal/providers/sealosdevbox, internal/providers/fal, docs/providers/provider-metadata.json)
• Yossi Eliaz: Recent commits touched provider registration and generated provider docs/metadata, which are part of this PR's review surface. (role: recent provider registry and docs contributor; confidence: medium; commits: 8c8a3c3e57ca, b469f051198a; files: internal/providers/all/all.go, docs/providers/README.md, docs/providers/provider-metadata.json)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (12 earlier review cycles; latest 8 shown)

• reviewed 2026-07-04T21:25:27.227Z sha 81d72c3 :: needs real behavior proof before merge. :: [P2] Default fal SSH to ubuntu | [P3] Remove the release-owned changelog entry
• reviewed 2026-07-04T21:29:42.980Z sha 81d72c3 :: needs real behavior proof before merge. :: [P2] Default fal SSH to ubuntu | [P3] Remove the release-owned changelog entry
• reviewed 2026-07-09T13:03:44.282Z sha d843438 :: needs real behavior proof before merge. :: [P3] Move release notes out of CHANGELOG.md
• reviewed 2026-07-09T13:09:55.674Z sha 6c015ab :: needs real behavior proof before merge. :: [P3] Move release notes out of CHANGELOG.md
• reviewed 2026-07-09T13:30:18.867Z sha 7c464ac :: needs real behavior proof before merge. :: [P3] Move release notes out of CHANGELOG.md
• reviewed 2026-07-09T13:39:28.951Z sha 7c464ac :: needs real behavior proof before merge. :: [P3] Move release notes out of CHANGELOG.md
• reviewed 2026-07-09T14:31:45.959Z sha c3e45cb :: needs real behavior proof before merge. :: [P3] Move release notes out of CHANGELOG.md
• reviewed 2026-07-09T14:37:40.090Z sha c3e45cb :: needs real behavior proof before merge. :: [P3] Move release notes out of CHANGELOG.md

[coygeek]

Context on the Go CI timeout update in 605caed0:

The first red Go run was a real static-analysis issue, not runtime: deadcode reported internal/providers/fal/core.go:35:6: unreachable func: inventoryDoctorResult. That helper was an unused wrapper around the shared provider doctor result helper, so it was removed in 039ebdf3.

After that fix, the next Go run passed the earlier gates: formatting, go vet ./..., go run golang.org/x/tools/cmd/deadcode@v0.45.0 -test ./..., go test -race ./..., and scripts/test-go-modules.sh. It then entered scripts/check-go-coverage.sh 90.0 and was canceled by the Go job's 15-minute workflow timeout while coverage was still running. The log showed GitHub cancellation during Coverage, not a coverage assertion failure.

To verify the diagnosis before changing CI, I ran the same coverage command locally. It completed successfully with Go core coverage 91.3% >= 90.0%; the long-running package work, including internal/providers/xcpng, finished cleanly. Based on that, the timeout update was scoped to the Go job only: timeout-minutes: 15 -> 30, giving the existing full Go gauntlet enough wall-clock budget without weakening any check.

Result: the replacement GitHub Actions run passed: https://github.com/openclaw/crabbox/actions/runs/28198853668. The Go job passed in 14m17s, including Deadcode, Test, Test all Go modules, Coverage, and Build: https://github.com/openclaw/crabbox/actions/runs/28198853668/job/83532759285.