[Bug] MiniMax H3 text encoder needs ~97 GB resident to load a 63 GB bf16 checkpoint(FAS-494)

[KyleNeverGivesUp]

Describe the bug

Loading MiniMax H3's text encoder takes about 97 GB of resident memory for a
63 GB bf16 checkpoint. There is no dtype conversion to account for the
difference, and the extra ~34 GB does not appear when the weights are read. It
accumulates slowly over the five minutes after the read finishes.

On a DGX Spark (GB10, 121 GB unified memory) this is fatal: the process is
OOM-killed during component loading, before the DiT is ever touched.

The checkpoint is uniformly bf16. Header of the first text_encoder shard
and its config.json:

shards: 14
dtype histogram: {'BF16': 43}
config.json: "dtype": "bfloat16", hidden_size 5120, num_hidden_layers 64

That is roughly 31.5 B parameters, and 31.5e9 x 2 bytes matches the 63 GB on
disk. Nothing is being upcast.

Measured memory, sampled every 5 s with free, aligned against the loader's
own log timestamps. free's used column excludes buff/cache, so this is
process memory, not page cache. Baseline was 8 GB, the machine was otherwise
idle.

time	used	what the log says
00:45:39	8 GB	idle baseline
00:45:54	23 GB	text encoder read begins
00:46:08		Loading weights took 17.52 seconds
00:46:14	75 GB	~67 GB arrived in 20 s, consistent with the 63 GB on disk
00:47:09	72 GB	brief dip, then a steady climb
00:49:04	85 GB
00:51:04	101 GB
00:51:44	105 GB
00:51:57		Loaded module text_encoder (5 min 49 s after the read)
00:51:58	108 GB	Loading vae ...
00:52:14	114 GB	7 GB left
00:52:15		worker gone
00:52:19	51 GB	memory released

So the text encoder alone goes from 8 GB to about 105 GB. The weights explain
67 GB of that in the first 20 seconds. The remaining ~34 GB arrives afterwards,
gradually, while the loader is still working on the same component.

What is not explained

I measured this, I did not diagnose it. I do not know where the extra 34 GB
comes from or why it takes five minutes rather than being an allocation spike.
A dtype cast or a transient second copy would both be fast and would fall back
down; a slow monotonic climb looks more like per-parameter work that does not
release as it goes, but that is a guess.

I also have no control: I have not measured another large text encoder on the
same machine, so I cannot say whether this is specific to H3 or general to how
FastVideo loads large encoders. If it is general, anything with an encoder this
size will hit it, not just this device.

Reproduction

python examples/inference/basic/basic_minimax_h3_t2v.py \
  --prompt "A robotic arm pours liquid into a glass, cinematic lighting." \
  --num-gpus 1 --height 480 --width 832 --num-frames 25 --steps 8 \
  --output outputs/h3_smoke

Sample memory alongside it:

while true; do echo "$(date +%T) $(free -g | awk '/^Mem:/ {print $3}')"; sleep 5; done

Component sizes on disk, for reference:

text_encoder    63.0 GB      <- larger than the DiT
transformer     38.0 GB      (62 GB for MiniMaxAI/MiniMax-H3)
vae              9.8 GB
audio_vae        0.6 GB

Downstream effects

The failure is unattributable from the logs. The worker is SIGKILLed by
the OOM killer, so it never raises and never prints a traceback. The only
surface error is:

Exception: WorkerMultiprocProc initialization failed due to an exception in a
background process. See stack trace for root cause.

There is no stack trace to see, and nothing anywhere in the log mentions
memory. I spent two runs chasing the wrong cause before adding an external
memory sampler. A hint when a worker dies without an exception would save that.

The rank-reduced checkpoint from #1699 does not help here. Swapping
MiniMaxAI/MiniMax-H3 (DiT 62 GB) for noctuashap/MiniMax-H3-pruned-r16
(DiT 38 GB) moved peak memory from 113 GB to 114 GB, because the DiT is never
reached. The full run dies while loading the transformer; the reduced run dies
one component earlier, in the VAE. Worth noting for anyone who reads
"-23 GiB VRAM" and expects it to close a gap during loading.

Minor, separate: the VAE encoder is loaded for text-to-video and never
used. VAEConfig.load_encoder defaults to True and
MiniMaxH3PipelineConfig does not override it. _encode_visual_rows and
_encode_keyframe_latents in minimax_h3_latent_preparation.py only run for
reference-conditioned modes, so a plain t2v run pays for the encoder without
touching it. HunyuanVideo 1.5 sets load_encoder = False in its T2V config for
exactly this reason, but its t2v and i2v are separate config classes, whereas
MiniMaxH3PipelineConfig serves both, so the same fix is not a one-liner here.
Roughly 5 GB, plus 0.6 GB for audio_vae on a run with no audio conditioning.
Not the cause of this bug, just adjacent waste found while measuring.

Environment

DGX Spark, GB10, aarch64, sm_121a, CUDA 13.0, 121 GB unified memory usable,
single GPU. FastVideo at 8208536. Reproduced twice, with both the full and the
rank-reduced checkpoint, with the same curve shape both times.

Context

This came out of "set up MiniMax H3 on Spark". The practical conclusion for
that task is that H3 does not currently load on a single GB10 at all, and the
reason is loading overhead rather than the size of the model itself: the four
components total 111 GB on disk against 121 GB of memory, which is tight but
not impossible, while the measured requirement is well past it.

Happy to run further measurements on this hardware if it helps narrow the
34 GB down, including a control against another large text encoder.

[speedyshreya]

Hello,
Can I work on the VAE part at the bottom of this issue? Could you assign it to me if so?

Here's what I gathered: on a plain t2v run the VAE encoder gets built but never used, since _encode_visual_rows and _encode_keyframe_latents only run for reference-conditioned modes — so that's ~5 GB, plus 0.6 GB of audio_vae when there's no audio conditioning. HunyuanVideo 1.5 just sets load_encoder = False in its T2V config, but that won't work directly here since MiniMaxH3PipelineConfig serves both t2v and i2v. So either split the configs, or decide load_encoder at run time based on whether the run is reference-conditioned. I'd go with the runtime check unless you'd rather see it done differently, open to suggestions!

[linear]

from shh075:

Sure, go ahead.

[speedyshreya]

Small correction -
I went through the code and the ref2va gate won't work. _encode_keyframe_latents also gets called from _encode_fl2va_conditions, and FL2VA goes through MiniMaxH3Pipeline with ref2va=False. So FL2VA needs the encoder too. The check has to be whether the run has image conditioning at all, not just whether it's ref2va.

Two additional issues I found -

AutoencoderKLMiniMaxH3.init builds self.encoder no matter what — it never checks config.load_encoder, unlike wanvae.py and hunyuan15vae.py. So setting that flag would do nothing.

component_loader.py puts AutoencoderKLMiniMaxH3 in the strict=True set, so if the encoder isn't built, the encoder.* weights in the checkpoint become unexpected keys and the load fails. I'd rather drop those keys from the state dict when load_encoder=False than turn strict off, so the decoder still gets checked properly.

Also, the ~5 GB looks estimated from the VAE file size rather than measured. I'll count the encoder's actual parameters and put a real number in the PR.

tldr; make the VAE honor the flag, guard _encode_clip so it fails with a clear message instead of an AttributeError, filter the state dict, and gate it on T2VA.

Let me know if this doesn't sound right