Phase 2: fula-ingest - verified byte ingress (client-side CID, quota-gated)

[ehsan6sha]

Phase 2 — fula-ingest: verified byte ingress + e2e suites

The decentralized-upload byte path: any approved operator can run a fula-ingest node that accepts encrypted chunk BYTES so uploads stop depending on a single gateway, with tamper-evidence enforced server-side.

What's in

• docker/fula-ingest/ (Go): PUT /v0/block?cid=<declared> verifies the blake3 raw-leaf CID over the body before anything touches the blockstore (one flipped bit ⇒ 422, nothing stored), then kubo block/put?cid-codec=raw&mhtype=blake3 with a cross-check of kubo's key. S1 quota gate mirrors the gateway exactly (GET {STORAGE_API_URL}/api/v1/storage with the client's Bearer; INGEST_QUOTA_MODE=open = gateway-parity fail-open, strict denies when unverifiable) with a 30s per-token cache (one quota call per window, not per chunk). Size cap, bounded concurrency, /health, graceful shutdown, Dockerfile.
• e2e suites (tests/e2e/phase-2/): 40-ingest-drills.sh (I1–I10) + 60-fidelity.sh (F1–F4).

Evidence (clean Ubuntu 24.04 test master, real daemons)

• Go unit: 10/10 (containerized) — incl. tampered-never-stored and quota-denied-never-stored.
• Live drills: 20/20 — valid block stored; tampered ⇒ 422 + absent from kubo; suspended user ⇒ 402 + absent; no token ⇒ 401; bytes still ingested with the gateway container STOPPED; gateway rebuilt from the fula-api Phase-2 branch flips /fula/capabilities false→true; live empty-body remote-cid mapping PUT ⇒ 200 with ETag=<declared cid> and a byte-exact GET round-trip; absent cid ⇒ bounded 409 (client-fallback contract).

Cross-repo: functionland/fula-api#31 (client routing + gateway mapping PUT; functionland/fula-api#30 lands first — the gateway image recipe).

In simple words

When your phone uploads a file today, the encrypted pieces must pass through one company server. With this, the pieces can go to any approved relay computer instead — and every piece carries a fingerprint, so a relay that corrupts even one byte gets caught and rejected instantly, and nobody can upload without an account in good standing.

Closes #74

🤖 Generated with Claude Code

[ehsan6sha]

Fidelity suite evidence (test master, real daemons)

Leg	Result
F1 — live client→ingest round-trip (1.5 MB, CID on)	✅ + server-side proof: kubo block count grew 36→75 (bytes flowed via the ingest node)
F2 — client-CID-off legacy parity	✅
F3 — FxFiles-faithful offline_e2e single-object flow	✅
F3 — FxFiles-faithful offline_e2e chunked flow (>768 KiB, the photo/video path)	✅
F4 — 1 GiB via ingest (~4096 chunks)	🔄 rerunning with a 12 h token — the first run uploaded 4019/4096 chunks over ~2 h on the contended 4-core box, then the 2 h fixed-expiry TEST token died (real clients refresh). Result will be posted here.

Scale measurement worth recording: chunked-upload throughput on this path is metadata-commit-bound (each chunk's mapping/bytes PUT serializes through the per-bucket write lock + an index-tree flush; observed ~3× object amplification per chunk). This is a pre-existing property of the gateway's chunked path — Phase 2 neither worsens nor improves it (byte transport moves off the master; commit costs are identical). The plan's Phase-7 client-computed recursive root / batched commits is the designed fix.

🤖 Generated with Claude Code

[ehsan6sha]

Large-file scale leg — honest disposition

The ≥large-file e2e (live_large_file_chunked_via_ingest, env-scalable via FULA_BIG_MB) cannot complete in reasonable wall-clock on the 7.8 GB / 4-core test box, at any size ≥256 MB — and the cause is infrastructure, not this change:

• Measured throughput on the box: ~0.4 IPFS objects/sec → a 512 MB file (~2048 chunks) projects to ~4 h; 1 GiB to 12 h+ (observed). The wall is CPU-bound per-chunk metadata-commit serialization (each chunk: encrypt → ingest PUT → mapping PUT → IPFS pin → prolly-tree flush, serialized on the per-bucket lock) on 4 cores — the documented Phase-7 throughput limit, amplified by a small box — plus the product's whole-file-in-RAM buffering (put_flat_from_path → fs::read → one Bytes).
• This is not a correctness gap. Every large-file attempt moved thousands of chunks through the exact ingest path under test with zero integrity errors before the box-resource wall hit; the path is architecturally identical to the multi-chunk cases that DO pass (below) — only the iteration count differs.

The test is committed and env-ready: FULA_BIG_MB=1024 cargo test … live_large_file_chunked_via_ingest runs the literal ≥1 GiB on a prod-class box. Recommend running it there as a release gate.

Scale invariant otherwise PROVEN by fast green tests

• Multi-chunk file via the ingest route, byte-exact round-trip + server-side kubo block-count proof — F1 ✅
• FxFiles-faithful chunked (>768 KiB) flow — F3 ✅
• Many files + nested folders — dir_index_v8_real_server_e2e (60+ files / 35 dirs / nested) ✅ (pre-existing)
• Live ingest drills 20/20 incl. mapping-PUT → byte-exact GET.

Product findings flagged for prod sizing (pre-existing, not introduced here)

1. Large-file upload buffers the whole file in RAM on both client and gateway (~1 GiB+ free needed for a 1 GiB file).
2. Per-chunk metadata commits serialize on the bucket lock (~3× object amplification; metadata-commit-bound throughput) — Phase 7's client-computed recursive root is the designed fix.

🤖 Generated with Claude Code