MILAB-6725: size exportClones from the .clns instead of a flat 12 GiB

[PaulNewling]

Ticket: MILAB-6725 — https://app.notion.com/3b23a83ff4af81d78906f6ff5c9dbd11
Reported in: https://platforma-bio.slack.com/archives/C0B0KCF0VDF/p1785763510195099

What broke

A customer running 10x single-cell BCR on EKS lost three block outputs at once —
clonotypes, clonotypeTables and qcReportTable. All three are one failure: the
IGK/IGL single-cell exportClones died with OutOfMemoryError (exit 2), and all three
depend on its tsvForSingleCell.

The failing command, verbatim from the report:

java -Xmx8788m -Xms6144m ... exportClones --dont-split-files --drop-default-fields \
  --reset-export-clone-table-splitting --chains IGK,IGL --export-productive-clones-only \
  --split-by-tags Cell -tags Cell -nFeature VDJRegion -vGene -jGene \
  -isProductive VDJRegionInFrame -uniqueTagCount Molecule clones.clns clones.tsv

The chain:

1. mixcr-export.tpl.tengo sized the export exec at a flat 12 GiB
   (max(12, perProcessMemGB / 4)).
2. The memory-from-limits entrypoint turns a container grant R into
   -Xmx = max(min(0.85·R, R − 3500 MiB), 0.5·R). At R = 12288 MiB that gives
   -Xmx8788m, matching the observed command line exactly. The flat 3500 MiB
   reserve dominates until R ≈ 23 GiB, so small grants lose a disproportionate share
   to non-heap.
3. exportClones loads the whole CloneSet into heap (CommandExportClones.kt:238).
   Single-cell exports then call divideClonesByTags (:308), which materialises a
   second, expanded list — one clone per (clonotype × cell), each with its own split
   TagCount — then sorts and re-ranks it (mixcr-algo CloneSet.kt:146-150).
4. --chains filters only after that division (:348), so exporting one chain group
   still pays the whole-file cost. This block loops chain groups and runs bulk plus
   single-cell per group: four of these per two-chain-group BCR sample.
5. Heap exhausted → Main.kt:107 catches OutOfMemoryError → exit 2.

MiXCR's message misleads. "This run used approximately 8792m of memory, this machine
has 12288 Mb in total" reads as "used 8.6 of 12, plenty spare", but 8792m is
Runtime.getRuntime().maxMemory() — the heap ceiling, not usage.

The dataset was small — 26,744 clones, ~10.7k cell barcodes. The 12 GiB was never
connected to anything. The same block sizes analyze at
clamp(192 GiB + 4·size(reads), 192 GiB, 256 GiB) for this preset — a 16× gap against
a step that holds an expanded copy of the clone set.

This is the second incident on this container. 83c4eb99 in the mixcr repo raised the
entrypoint's non-heap reserve 2500 → 3500 MiB because "Export commands on 12 GiB
containers were getting OOM-killed". That fixed a container kill by lowering -Xmx,
which set up today's heap OOM.

The change

Both export execs now derive RAM from the .clns:

ram = clamp(perByte × size(clns), floor, 128 GiB)

Export	floor	perByte	why
bulk	16 GiB	16	loads the CloneSet and writes it out
single-cell	24 GiB	32	additionally pays the divideClonesByTags expansion

size() sums the stored blob size via getBlobSize — a metadata read, no pre-exec
pass. The floor carries small inputs, where the in-heap tag-count graph rather than the
stored file dominates; the linear term carries large ones. A 24 GiB grant yields
20889 MiB of heap, 2.4× the ceiling that failed.

The floors are deliberately tight. -Xms is half the grant, so a request is a hard
pre-allocation rather than just a ceiling, and a sample runs one bulk plus one
single-cell export per chain group — the floors are paid several times over
concurrently. A separate incident this week had a mixcr analyze step granted 484 GiB,
pre-touch 242 GiB at JVM start, and get OOM-killed before finishing. Over-estimating a
request is not free.

between() supplies the floor rather than .plus(gib(floor)). The additive form
double-counts: the floors are total-memory values, so adding them to a size-proportional
term charges for the data twice.

Two changes follow from it:

• The Advanced Settings memory override now applies as-is. perProcessMemGB reaches
  the export exec unquartered, matching mixcr-analyze.tpl.tengo. Previously a user
  setting 48 GiB got 12 GiB here, so the documented escape hatch barely moved the failing
  step. Projects that set it will request 4× more for the export step than before —
  the one behaviour change worth pushing back on.
• The two PTabler steps now inherit workflow-tengo 6.8's input-volume formula
  (ram = clamp(2 GiB + 4·size, 2 GiB, 64 GiB)), which tracks the exported TSV better
  than a fixed share of a sibling step's budget. They previously took ⅔ of
  exportMemGB, which cannot survive that value becoming a formula. fix: size single-cell PTabler steps via workflow-tengo 6.8 formulas #204 gave the rest
  of the single-cell pipeline the same treatment; an explicit .mem()/.cpu() here was
  suppressing it.

The template's hash_override stays as-is. Resource requests ride as CID-transparent
meta inputs so a re-run with more memory reuses everything else, and the failed step
re-runs regardless — a failure is not a cached success.

Verified

Live, on a local backend, running the block's own simple sc project test — same
10x-sc-xcr-vdj preset and the same IGH + IGK,IGL chain groups as the customer:

• Test green. The per-block log shows the export execs receiving -Xmx12884m (bulk,
  16 GiB grant) and -Xmx27852m (single-cell, at the 32 GiB floor this ran under),
  both matching the entrypoint arithmetic exactly. Before the fix all eight were
  -Xmx8788m. This proves formula.size("clns") resolves at run time — an unmatched tag
  would have hard-failed the template — and that bulk and single-cell get different
  grants. The floor has since been lowered to 24 GiB; the mechanism is unchanged.
• analyze was observed requesting 192 GiB and receiving -Xmx33934m — the silent
  bucket clamp, live.
• The failure signature reproduces. Running the customer's exact exportClones
  argument list directly against a real single-cell .clns below ~64 MiB of heap gives
  Not enough memory for run command and exit 2 — the reported signature. The
  failure is heap-bound and this is the controlling knob.
• Resource changes are CID-transparent, confirmed the hard way: bumping the grant,
  changing the template hash, adding a distinguishing env var and rebuilding the block
  pack all failed to invalidate the cached exec. Worth knowing before anyone tries to
  bisect memory through the UI.
• pnpm run build:dev-local — 9/9 tasks green.
• size accepts .clns: _assertMetricCompatible in formula.lib.tengo restricts only
  lineCount to line-oriented text formats.
• The compute bucket clamps over-large requests (bucket.go:199-204 / :301-307) and
  feeds the clamped value to {system.ram.mib} rendering, so the JVM gets a consistent
  -Xmx. Raising the floor cannot strand a desktop user with an unschedulable request.

Not verified

• The slopes (16 / 32) are estimates — and now measurably so. Sweeping minimum
  passing heap across two real .clns files (86 KB and 1.24 MB, a 14× range) gave
  64 MiB for both bulk and single-cell on both inputs. At that scale heap is entirely
  JVM plus reference-library overhead; the data contributes nothing measurable. The
  largest .clns available locally holds 345 clone×cell pairs against the customer's
  tens of thousands, so no local dataset can calibrate the slope or discriminate bulk
  from single-cell. That is a measurement, not an assumption.
• The 24 GiB floor has not itself been run live; the live run above used 32 GiB. The
  mechanism is identical.
• The customer's own data has not been run.

The .clns from the failed run — or just its size — would turn the coefficient into a
measurement. It is derived data, not raw sequence. A calibration plan against the
studies library is drafted separately.

How to verify locally

The bug is "grant too small for the data", which is two independent claims. The mechanism
is testable today; the sufficiency of these specific numbers needs larger data.

1. Force the failure, then clear it. Scale the grant down rather than the data up.
   Pin the export to mem("1GiB") (→ -Xmx512m) and run the single-cell fixture in
   test/assets: expect the same Not enough memory for run command / exit 2 signature
   and the same error chain through tsvForSingleCell. Restore the formula and it
   passes. Proves the failure is heap-bound and that this is the controlling knob.
2. Read back the allocated grant. Start the backend with --log-blocks-enabled and
   grep the per-block log for the java -Xmx… line. Before: -Xmx8788m. After: the
   24 GiB request, clamped by the medium bucket to whatever the host allows. This is also
   the only check that proves formula.size("clns") resolves at run time.
3. Bisect -Xmx against a fixture .clns for a measured heap-per-clns-byte ratio.
   Treat it as a lower bound — tag-count overhead does not scale linearly from a 1.2 MB
   fixture to a real sample.

Follow-up

The same flat-12-GiB export sizing exists in mixcr-shm-trees (12 GiB on the light
queue, the worst of the set), miltenyi-tcr-bcr-clonotyping (also single-cell, same
multiplier) and mixcr-amplicon-alignment. Staged so this one can be confirmed against
the customer first.

Interim workaround

Set Advanced Settings → memory per process = 192 GB. On the current release that
gives the export 48 GiB (~41 GiB heap). Use 192 specifically: the override also replaces
analyze's formula, and 192 GiB is the floor analyze already computes for MiTool
presets, so nothing regresses.

Greptile Summary

The PR replaces fixed exportClones memory grants with bounded formulas based on .clns blob size, preserves direct Advanced Settings overrides, and delegates PTabler sizing to workflow-tengo 6.8.2 defaults.

• exportClones: MiXCR command that loads a CloneSet and emits clone tables; its bulk and single-cell executions now receive separate size-based RAM formulas.
• .clns: Stored MiXCR CloneSet artifact; it is tagged as clns so its blob size can drive runtime resource calculation.
• CloneSet: In-memory collection of clones loaded by exportClones; the new bulk formula uses a 16 GiB floor, a 16× stored-size multiplier, and a 128 GiB cap.
• divideClonesByTags: Single-cell expansion that creates clone-per-cell representations; the corresponding export uses a 24 GiB floor and a 32× multiplier.
• perProcessMemGB: Advanced Settings memory override; it now applies directly to export executions instead of being divided by four.
• PTabler: TSV-processing workflow used after export; explicit CPU and RAM settings were removed so workflow-tengo can derive resources from input volume.
• Adds a patch changeset documenting the incident, sizing model, override behavior, and PTabler resource change.

Confidence Score: 5/5

The PR appears safe to merge, with no blocking failure remaining in the eligible follow-up-review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
workflow/src/mixcr-export.tpl.tengo	Replaces fixed export and PTabler resource assignments with tagged-input formulas, distinct bulk and single-cell bounds, direct overrides, and workflow defaults.
.changeset/export-clones-memory.md	Documents the export heap failure, the new resource-sizing formulas, override semantics, and PTabler sizing behavior.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[.clns blob] --> B[Tagged input: clns]
    B --> C{Export mode}
    C -->|Bulk| D[RAM: clamp 16x size, 16 GiB, 128 GiB]
    C -->|Single-cell| E[RAM: clamp 32x size, 24 GiB, 128 GiB]
    F[perProcessMemGB override] -->|When configured| D
    F -->|When configured| E
    D --> G[exportClones]
    E --> H[exportClones plus tag expansion]
    G --> I[Exported TSV]
    H --> J[Single-cell TSV]
    I --> K[PTabler default input-volume sizing]
    J --> L[PTabler default input-volume sizing]

Loading

_{Reviews (3): Last reviewed commit: "fix: lower the single-cell export floor ..." | Re-trigger Greptile}

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[PaulNewling]

@greptileai

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[PaulNewling]

@greptileai