Skip to content

chroma image CUDA kernel incompatibility with Blackwell GPUs (RTX PRO 6000) #127

Description

@lemon07r

Summary

The chroma chute (chute_id 1fc2c2e4-bb6e-5f73-8bd8-ed225a394dba, image comfyui:0.3.48-052aefbbac64) lists pro_6000 in supported_gpus but fails on RTX PRO 6000 Blackwell GPUs with:

CUDA error: no kernel image is available for execution on the device

The error occurs in the EmptyLatentImage ComfyUI node during warmup (torch.zeros on CUDA device). The image's PyTorch build does not include kernels for Blackwell compute capability 12.0 (sm_120).

Reproduction

Every chroma deployment on our 4× RTX PRO 6000 Blackwell passes PoVW verification, then crashes during initialize() warmup approximately 3 minutes later. Reproduced 4+ times with identical results.

Error trace from pod logs

Error occurred when executing EmptyLatentImage:
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

This triggers during initialize() → workflow warmup → EmptyLatentImage node → torch.zeros(..., device="cuda").

Environment

  • GPU: 4× NVIDIA RTX PRO 6000 Blackwell Server Edition, 96 GB VRAM each
  • Compute capability: 12.0 (sm_120)
  • CUDA driver: 590.48
  • CUDA runtime: 13.1
  • Hotkey: 5CCzhyifft3ei1Z5Z7xF8RnUZZXg2JoJTfxfrqpyRRCLGxov on subnet 64

Impact

chroma is the only scalable public chute that supports pro_6000. All other public chutes have reached their target capacity (validator rejects with "reached its target capacity of 2 instances").

Because chroma always fails on Blackwell, miners with RTX PRO 6000 GPUs:

  1. Cannot get more than 3 active public GPU instances (whisper=1, Affine=2)
  2. Cannot reach the 4 active public GPU threshold required for private chute deployments
  3. Are permanently locked out of private chutes — the highest-value workloads

This is a catch-22 specific to Blackwell miners on this subnet.

Request

Either:

  1. Rebuild the chroma image with a PyTorch version that includes Blackwell (sm_120) CUDA kernels
  2. Or remove pro_6000 from chroma's supported_gpus list so it stops being offered to Blackwell miners

Option 1 is strongly preferred as it would unblock Blackwell miners entirely.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions