Skip to content

Bisect the packaging-cost threshold between 2 and 4 MiB #29

Description

@TBarregren

The question

Per-part packaging cost grows about twice as fast as the part up to 2 MiB, and then jumps roughly 26× somewhere between 2 and 4 MiB. Where that discontinuity sits, and what causes it, is unidentified.

php://temp, whose 2 MB spill threshold would have fit the shape elegantly, was removed by plan 006 and is structurally absent from the code. Nothing else has been tested.

Method, cost and instrumentation are already established: one controlled single-file extraction per point, about ten minutes each, exactly as run in docs/measurements/2026-08-19-chunk-size-curve.md. Two or three points bisect it.

Why this is ready-for-human rather than agent work

It cannot be settled from a laptop. Nothing on a local NVMe reproduces the cost — the same code path is roughly three orders of magnitude cheaper there — so it needs a real managed host, and the only host with a measured baseline is a live client site. Never touch that site without being asked. Choosing where to run this, and getting consent for it, is an operator decision.

Method notes that cost this project time to learn

  • Vary one thing. The first 4 MiB reading was confounded: it came from mid-clone against a container already a gigabyte long, while every other figure came from a fresh one, so part size and container state moved together. A fresh single-file control is what separated them, and the wrong conclusion was published before the confound was caught.
  • Read the part count, not the config, to confirm a setting is actually live. A 36 MB file yields 139 parts at 256 KB and 9 at 4 MiB.
  • Run an independent sampler alongside the jobGET /extractions/{id} every 30 s into a CSV and the full body into JSONL. Both diagnoses in this project came from such a series, and neither could have come from the client's own poller, which returns one terminal verdict and no history. The attempts array is bounded at eight entries, so it is only informative when captured continuously. Preserved scripts: ~/Clients/SafeTeam/preserved-20260818/.
  • Bound the watcher and treat a purged job as terminal, or it polls forever once consume deletes the record and the poll starts 404-ing.

What an answer here does and does not unlock

It informs the shipped default and the per-run parameter. It does not discharge rule R1. R1 is gated on attributing the per-chunk cost of ordinary small files (#30), which is a different measurement, and the adaptation family stays frozen until that one lands.

Blocked by

An operator decision about which host to measure on.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions