Skip to content

Nvidia 50 series might crash on Triton, add option to skip specific attention modes #1

Description

@x5nder

Nvidia GeForce RTX 5060 Ti 16GB

sageattention 2.2.0+cu130torch2.9.0andhigher.post4
torch 2.9.1+cu130
triton 3.5.1.post24

Attention Optimizer crashes on Triton:

[Benchmark] Running... (head_dim=128, seq_len=8192)
[Benchmark] Available: ['basic', 'sub_quad', 'split', 'pytorch', 'sage_auto', 'sage_cuda', 'sage_triton', 'sage_fp8_cuda', 'sage_fp8_cuda_fast', 'flash']
__triton_launcher.c
   Creating library C:\Users\x5nde\AppData\Local\Temp\tmprie7271v\__triton_launcher.cp312-win_amd64.lib and object C:\Users\x5nde\AppData\Local\Temp\tmprie7271v\__triton_launcher.cp312-win_amd64.exp
Exception Code: 0x80000003
 #0 0x00007ffb0fb2dc25 mlir::detail::FallbackTypeIDResolver::registerImplicitTypeID(class llvm::StringRef) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\triton\_C\libtriton.pyd+0x3d0dc25)
 #1 0x00007ffc5b6d1989 (C:\WINDOWS\System32\ucrtbase.dll+0xc1989)
 #2 0x00007ffc5b6b4ab1 (C:\WINDOWS\System32\ucrtbase.dll+0xa4ab1)
 #3 0x00007ffc5b6d19de (C:\WINDOWS\System32\ucrtbase.dll+0xc19de)
 #4 0x00007ffbd4351911 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\VCRUNTIME140_1.dll+0x1911)
 #5 0x00007ffbd435218f (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\VCRUNTIME140_1.dll+0x218f)
 #6 0x00007ffbd43521e9 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\VCRUNTIME140_1.dll+0x21e9)
 #7 0x00007ffbd4354019 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\VCRUNTIME140_1.dll+0x4019)
 #8 0x00007ffc5ddc62ff (C:\WINDOWS\SYSTEM32\ntdll.dll+0x1662ff)
 #9 0x00007ffc5dc72327 (C:\WINDOWS\SYSTEM32\ntdll.dll+0x12327)
#10 0x00007ffc5dc6a961 (C:\WINDOWS\SYSTEM32\ntdll.dll+0xa961)
#11 0x00007ffc5ae7a80a (C:\WINDOWS\System32\KERNELBASE.dll+0xca80a)
#12 0x00007ffbd50b5267 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\VCRUNTIME140.dll+0x5267)
#13 0x00007ffc2fbbbcfb c10::cuda::c10_cuda_check_implementation(int, char const *, char const *, int, bool) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\c10_cuda.dll+0x3bcfb)
#14 0x00007ffc2fbc01c5 c10::cuda::warn_or_error_on_sync(void) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\c10_cuda.dll+0x401c5)
#15 0x00007ffc2fc5909e c10::StorageImpl::~StorageImpl(void) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\c10.dll+0x2909e)
#16 0x00007ffb1ddd77e5 at::DynamicLibrary::sym(char const *) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\torch_cpu.dll+0x61477e5)
#17 0x00007ffc2fc44882 c10::ConstantSymNodeImpl<__int64>::~ConstantSymNodeImpl<__int64>(void) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\c10.dll+0x14882)
#18 0x00007ffc2fc7f67c c10::TensorImpl::~TensorImpl(void) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\c10.dll+0x4f67c)
#19 0x00007ffb1ddd7825 at::DynamicLibrary::sym(char const *) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\torch_cpu.dll+0x6147825)
#20 0x00007ffb1dd75f58 at::TensorBase::reset(void) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\torch_cpu.dll+0x60e5f58)
#21 0x00007ffbf247e395 initModule (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\torch_python.dll+0x4be395)
#22 0x00007ffbf252ba9f THPVariable_Wrap(class at::TensorBase const &) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\torch_python.dll+0x56ba9f)
#23 0x00007ffbf252bda3 THPVariable_Wrap(class at::TensorBase const &) (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib\torch_python.dll+0x56bda3)
#24 0x00007ffbd4a824ea (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x824ea)
#25 0x00007ffbd4a8304b (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x8304b)
#26 0x00007ffbd4a8103c (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x8103c)
#27 0x00007ffbd4a7be18 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x7be18)
#28 0x00007ffbd4ab286b (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0xb286b)
#29 0x00007ffbd4ab275f (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0xb275f)
#30 0x00007ffbd4a866ee (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x866ee)
#31 0x00007ffbd4a4b201 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x4b201)
#32 0x00007ffbd4c44f58 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x244f58)
#33 0x00007ffbd4c3b57d (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x23b57d)
#34 0x00007ffc4d5e7334 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\DLLs\_asyncio.pyd+0x7334)
#35 0x00007ffc4d5e6c7e (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\DLLs\_asyncio.pyd+0x6c7e)
#36 0x00007ffbd4a6560e (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x6560e)
#37 0x00007ffbd4c696e0 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x2696e0)
#38 0x00007ffbd4c699c0 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x2699c0)
#39 0x00007ffbd4a68376 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x68376)
#40 0x00007ffbd4ab2815 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0xb2815)
#41 0x00007ffbd4ab275f (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0xb275f)
#42 0x00007ffbd4a866ee (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x866ee)
#43 0x00007ffbd4a8103c (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x8103c)
#44 0x00007ffbd4a7be6c (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x7be6c)
#45 0x00007ffbd4ab2815 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0xb2815)
#46 0x00007ffbd4ab275f (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0xb275f)
#47 0x00007ffbd4a2304c (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x2304c)
#48 0x00007ffbd4a22ff2 (C:\Users\x5nde\AppData\Local\Programs\Python\Python312\python312.dll+0x22ff2)
#49 0x00007ffc5b6137b0 (C:\WINDOWS\System32\ucrtbase.dll+0x37b0)
#50 0x00007ffc5cf6e8d7 (C:\WINDOWS\System32\KERNEL32.DLL+0x2e8d7)
#51 0x00007ffc5dcec53c (C:\WINDOWS\SYSTEM32\ntdll.dll+0x8c53c)

Apparently this is a known issue with Nvidia 50-series card. It would be nice to be able to skip a specific attention mode (sage_triton, in this case) so ComfyUI won't crash while testing the other modes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions