Filecoin storage support: which provider, given Lighthouse retrieval is payment-gated?

[umeradl]

Background

DIN currently uses Filebase as its default IPFS pinning provider. All model artifacts (client updates, aggregated models, manifests, service files) flow through dincli/services/ipfs.py and are referenced on-chain by CID only — the contract layer is fully provider-agnostic. Adding a Filecoin-backed provider therefore requires only an adapter behind the existing custom provider path: no contract changes, no CID format changes.

We want Filecoin-backed storage because:

• Decentralisation: a trust-minimised ML coordination network shouldn't depend on a single centralised pinning vendor.
• Workload fit: model weight updates are write-once, content-addressed, and referenced across training rounds — permanency and auditability matter more than ultra-low read latency.
• Cost: at Filecoin rates, a typical round (100 validators × 10 MB updates ≈ 1 GB) costs a fraction of a cent — effectively a rounding error in any fee structure.

The internal design doc (Developer/discussion/add-filecoin-support.md) recommended Lighthouse as the first integration, and PR #16 implemented a Lighthouse provider adapter.

What we found: Lighthouse retrieval is payment-gated ⚠️

While testing PR #16 with a real Lighthouse account (not just mocks):

• ✅ Upload works exactly as documented and returns a valid CID.
• ❌ Retrieval does not. GET https://gateway.lighthouse.storage/ipfs/<cid> returns 402 Payment Required — including for the file's own uploader, with and without the uploader's own API key attached (identical response either way, so this is not an auth-header issue).
• ❌ The content is not discoverable on the public IPFS network at all. A generic public gateway (ipfs.io) fails for the same CID with "no providers found for the CID".

This breaks DIN's core assumption: an artifact's CID must be retrievable by other participants (aggregators fetching client updates, auditors fetching models) — not just uploadable. A provider where even the uploader can't read their own file back without a paid retrieval plan is not viable as-is.

The question

How should we add Filecoin-backed storage, and which provider should we choose?

Candidates we're aware of:

Option	Pros	Concerns
Storacha (Web3.Storage)	UCAN delegated-upload capabilities are a first-class primitive — exactly the scoped credential model we need for sponsored uploads in P3; multi-provider Filecoin deals	Need to verify retrieval is genuinely open (public gateway + IPFS DHT announcement), not gated like Lighthouse
Lighthouse + paid retrieval plan	Adapter already written in PR #16	Retrieval gated behind payment; unclear if a plan makes CIDs publicly resolvable or only via their gateway
Filebase Filecoin bucket	Zero migration friction — same API and credential model we use today	Filebase remains the centralised intermediary; least decentralisation gain
Direct Filecoin deals / FVM	Maximum trustlessness; smart-contract-native storage deals (escrow, SLA verification, auto-renewal)	Deal lifecycle management is significant operational complexity; retrieval latency
Something else?	—	—

Specific questions:

1. Is Storacha's retrieval path actually open? Before we swap adapters, has anyone verified that content uploaded to Storacha resolves via public IPFS gateways / is announced to the DHT?
2. Is there any Filecoin-backed provider where uploaded CIDs are reliably retrievable by third parties for free (or where retrieval cost is bounded and payable by the model owner, not each reader)?
3. Should we reconsider an FVM-native approach — storage deals managed by contract (escrow, provider SLA verification, automated renewal) — even if it's more work, given it aligns with the P3 sponsored-upload/storage-budget design?
4. Is a hybrid sensible? Filecoin for durable archival + a hot IPFS pin (even Filebase) for the active round's retrieval path?

Any hands-on experience with these providers' retrieval behaviour — especially failure modes like the one we hit with Lighthouse — would be extremely valuable.

cc @Abidoyesimze — you've built on FVM (storage-deal escrow with SLA verification and auto-renewal), which is directly relevant here; would love your take on the provider choice and whether an FVM-native path is worth the complexity.
cc @Santiagocetran (PR #16 author)

[Abidoyesimze]

Hey, dug into this a bit.

1. Yeah, Storacha retrieval is just standard IPFS gateway stuff — storacha.link or any public gateway, no payment step. Content's actually discoverable on the wider IPFS network too, not just their infra. Only catch is their own gateway rate-limits at 200 req/min per IP, shouldn't matter for us.
2. Storacha Forge (their FOC-backed tier) is the one that actually works for us here — storage cost is on the uploader ($5.99/TB/month, PDP-backed), retrieval through open IPFS gateway is free for anyone. That's what we want: model owner pays, readers don't.
   Lighthouse's a dead end btw — their docs now openly describe it as "x402 pay-per-use retrieval," every fetch returns 402, you pay in USDC on Base, no exception even for the uploader. So what we hit in PR feat(validator-readiness): keystore security hardening + Filecoin (Lighthouse) provider adapter #16 wasn't a bug, that's just how it works now.
3. Bigger thing — Filecoin launched "Filecoin Onchain Cloud" in Nov 2025, and it's basically already built most of what we'd have hand-rolled: PDP proofs check storage every ~24h with a real penalty system (10%/25%/50% on repeated failures, cuts off after 10), Filecoin Pay streams payment and pauses automatically if proofs fail, and there's no fixed-term deal to renew — rails just keep going as long as they're funded.
   The part that actually matters for P3: Filecoin Pay has an "operator" concept built in — a sponsor deposits funds and approves a contract to manage rails on their behalf, capped by a rate + lockup allowance. That's basically a native storage-budget primitive already.
   So I don't think we need custom FVM escrow logic. But I also wouldn't just default to the hosted Synapse SDK flow if we want sponsor-controlled budgets — we'd want a small DIN contract that gets approved as the Filecoin Pay operator, so we keep our own sponsorship rules but inherit their proof/payment mechanics. Still real contract work, just way less than building escrow + SLA + renewal from scratch.
4. Given answer 2 and 3, I don't think we need the hybrid anymore — Storacha Forge already is that hybrid (UCAN uploads + open retrieval + Filecoin-backed durability in one product).

[umeradl]

Hey @Abidoyesimze ! Please look into how we can integrate Storacha (this is the rebranded version of web3.storage).

Specifically, I need you to figure out and document:

Web UI: How to set up accounts and manage data.

CLI: The workflow for uploading and retrieving data.

Authentication: How they handle "API keys"—heads up, they use decentralized UCANs instead of traditional static keys, so pay close attention to how we handle delegations for our backend/CI environment.

Let's get a quick markdown guide or snippet ready on this. Thanks!

[Abidoyesimze]

Okay will do that and get back.