tutel version

[uuuzk]

hello，i can't install tutel. what is the version of the tutel library?

[yxbian23]

hello，i can't install tutel. what is the version of the tutel library?

Did you install by pip install --verbose --upgrade git+https://github.com/microsoft/tutel@main?

[uuuzk]

Thanks for your reply.
Yes – I did try. but I ran into this error log:
(motioncraft) (base) root@uzk_544:~/projects/MotionCraft# pip install --verbose --upgrade git+https://github.com/microsoft/tutel@main
Using pip 25.1.1 from /opt/conda/envs/motion/lib/python3.9/site-packages/pip (python 3.9)
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting git+https://github.com/microsoft/tutel@main
Cloning https://github.com/microsoft/tutel (to revision main) to /tmp/pip-req-build-zmk70hwn
Running command git version
git version 2.17.1
Running command git clone --filter=blob:none https://github.com/microsoft/tutel /tmp/pip-req-build-zmk70hwn
Cloning into '/tmp/pip-req-build-zmk70hwn'...
Checking out files: 1% (2/127)
Checking out files: 2% (3/127)
Checking out files: 3% (4/127)
Checking out files: 4% (6/127)
Checking out files: 5% (7/127)
Checking out files: 6% (8/127)
Checking out files: 7% (9/127)
Checking out files: 8% (11/127)
Checking out files: 9% (12/127)
Checking out files: 10% (13/127)
Checking out files: 11% (14/127)
Checking out files: 12% (16/127)
Checking out files: 13% (17/127)
Checking out files: 14% (18/127)
Checking out files: 15% (20/127)
Checking out files: 16% (21/127)
Checking out files: 17% (22/127)
Checking out files: 18% (23/127)
Checking out files: 19% (25/127)
Checking out files: 20% (26/127)
Checking out files: 21% (27/127)
Checking out files: 22% (28/127)
Checking out files: 23% (30/127)
Checking out files: 24% (31/127)
Checking out files: 25% (32/127)
Checking out files: 26% (34/127)
Checking out files: 27% (35/127)
Checking out files: 28% (36/127)
Checking out files: 29% (37/127)
Checking out files: 30% (39/127)
Checking out files: 31% (40/127)
Checking out files: 32% (41/127)
Checking out files: 33% (42/127)
Checking out files: 34% (44/127)
Checking out files: 35% (45/127)
Checking out files: 36% (46/127)
Checking out files: 37% (47/127)
Checking out files: 38% (49/127)
Checking out files: 39% (50/127)
Checking out files: 40% (51/127)
Checking out files: 41% (53/127)
Checking out files: 42% (54/127)
Checking out files: 43% (55/127)
Checking out files: 44% (56/127)
Checking out files: 45% (58/127)
Checking out files: 46% (59/127)
Checking out files: 47% (60/127)
Checking out files: 48% (61/127)
Checking out files: 49% (63/127)
Checking out files: 50% (64/127)
Checking out files: 51% (65/127)
Checking out files: 52% (67/127)
Checking out files: 53% (68/127)
Checking out files: 54% (69/127)
Checking out files: 55% (70/127)
Checking out files: 56% (72/127)
Checking out files: 57% (73/127)
Checking out files: 58% (74/127)
Checking out files: 59% (75/127)
Checking out files: 60% (77/127)
Checking out files: 61% (78/127)
Checking out files: 62% (79/127)
Checking out files: 63% (81/127)
Checking out files: 64% (82/127)
Checking out files: 65% (83/127)
Checking out files: 66% (84/127)
Checking out files: 67% (86/127)
Checking out files: 68% (87/127)
Checking out files: 69% (88/127)
Checking out files: 70% (89/127)
Checking out files: 71% (91/127)
Checking out files: 72% (92/127)
Checking out files: 73% (93/127)
Checking out files: 74% (94/127)
Checking out files: 75% (96/127)
Checking out files: 76% (97/127)
Checking out files: 77% (98/127)
Checking out files: 78% (100/127)
Checking out files: 79% (101/127)
Checking out files: 80% (102/127)
Checking out files: 81% (103/127)
Checking out files: 82% (105/127)
Checking out files: 83% (106/127)
Checking out files: 84% (107/127)
Checking out files: 85% (108/127)
Checking out files: 86% (110/127)
Checking out files: 87% (111/127)
Checking out files: 88% (112/127)
Checking out files: 89% (114/127)
Checking out files: 90% (115/127)
Checking out files: 91% (116/127)
Checking out files: 92% (117/127)
Checking out files: 93% (119/127)
Checking out files: 94% (120/127)
Checking out files: 95% (121/127)
Checking out files: 96% (122/127)
Checking out files: 97% (124/127)
Checking out files: 98% (125/127)
Checking out files: 99% (126/127)
Checking out files: 100% (127/127)
Checking out files: 100% (127/127), done.
Running command git show-ref main
ae44c2bc08fad09efa7660944b3dd3601661a42a refs/heads/main
ae44c2bc08fad09efa7660944b3dd3601661a42a refs/remotes/origin/main
Running command git symbolic-ref -q HEAD
refs/heads/main
Resolved https://github.com/microsoft/tutel to commit ae44c2bc08fad09efa7660944b3dd3601661a42a
Running command git rev-parse HEAD
ae44c2bc08fad09efa7660944b3dd3601661a42a
Running command python setup.py egg_info
/opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!

      ********************************************************************************
      Please consider removing the following classifiers in favor of a SPDX license expression:

      License :: OSI Approved :: MIT License

      See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
      ********************************************************************************

!!
self._finalize_license_expression()
Try installing with NCCL extension..
running egg_info
creating /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info
writing /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/dependency_links.txt
writing requirements to /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/requires.txt
writing top-level names to /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/SOURCES.txt'
/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/utils/cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
reading manifest file '/tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/SOURCES.txt'
Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in /opt/conda/envs/motioncraft/lib/python3.9/site-packages (from tutel==0.4) (1.23.2)
Building wheels for collected packages: tutel
Running command git rev-parse HEAD
ae44c2bc08fad09efa7660944b3dd3601661a42a
DEPRECATION: Building 'tutel' using the legacy setup.py bdist_wheel mechanism, which will be removed in a future version. pip 25.3 will enforce this behaviour change. A possible replacement is to use the standardized build interface by setting the --use-pep517 option, (possibly combined with --no-build-isolation), or adding a pyproject.toml file to the source tree of 'tutel'. Discussion can be found at pypa/pip#6334
Running command python setup.py bdist_wheel
Try installing with NCCL extension..
/opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!

      ********************************************************************************
      Please consider removing the following classifiers in favor of a SPDX license expression:

      License :: OSI Approved :: MIT License

      See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
      ********************************************************************************

!!
self._finalize_license_expression()
running bdist_wheel
/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/utils/cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend.
warnings.warn(msg.format('we could not find ninja.'))
running build
running build_py
creating build/lib.linux-x86_64-cpython-39/tutel
copying tutel/system.py -> build/lib.linux-x86_64-cpython-39/tutel
copying tutel/init.py -> build/lib.linux-x86_64-cpython-39/tutel
copying tutel/net.py -> build/lib.linux-x86_64-cpython-39/tutel
copying tutel/moe.py -> build/lib.linux-x86_64-cpython-39/tutel
copying tutel/jit.py -> build/lib.linux-x86_64-cpython-39/tutel
creating build/lib.linux-x86_64-cpython-39/tutel/launcher
copying tutel/launcher/run.py -> build/lib.linux-x86_64-cpython-39/tutel/launcher
copying tutel/launcher/execl.py -> build/lib.linux-x86_64-cpython-39/tutel/launcher
copying tutel/launcher/init.py -> build/lib.linux-x86_64-cpython-39/tutel/launcher
creating build/lib.linux-x86_64-cpython-39/tutel/gates
copying tutel/gates/top.py -> build/lib.linux-x86_64-cpython-39/tutel/gates
copying tutel/gates/init.py -> build/lib.linux-x86_64-cpython-39/tutel/gates
copying tutel/gates/cosine_top.py -> build/lib.linux-x86_64-cpython-39/tutel/gates
creating build/lib.linux-x86_64-cpython-39/tutel/custom
copying tutel/custom/init.py -> build/lib.linux-x86_64-cpython-39/tutel/custom
creating build/lib.linux-x86_64-cpython-39/tutel/jit_kernels
copying tutel/jit_kernels/sparse.py -> build/lib.linux-x86_64-cpython-39/tutel/jit_kernels
copying tutel/jit_kernels/init.py -> build/lib.linux-x86_64-cpython-39/tutel/jit_kernels
copying tutel/jit_kernels/gating.py -> build/lib.linux-x86_64-cpython-39/tutel/jit_kernels
creating build/lib.linux-x86_64-cpython-39/tutel/ops
copying tutel/ops/init.py -> build/lib.linux-x86_64-cpython-39/tutel/ops
creating build/lib.linux-x86_64-cpython-39/tutel/parted
copying tutel/parted/solver.py -> build/lib.linux-x86_64-cpython-39/tutel/parted
copying tutel/parted/init.py -> build/lib.linux-x86_64-cpython-39/tutel/parted
copying tutel/parted/patterns.py -> build/lib.linux-x86_64-cpython-39/tutel/parted
copying tutel/parted/spmdx.py -> build/lib.linux-x86_64-cpython-39/tutel/parted
creating build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/helloworld_amp.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/nccl_all_to_all_v.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/moe_cifar10.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/nccl_allreduce_perf.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/bandwidth_test.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/helloworld_ddp.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/moe_mnist.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/helloworld_ddp_tutel.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/nccl_all_gather_v.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/init.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/helloworld_custom_gate_expert.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/helloworld_custom_expert_sharded.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/helloworld_from_scratch.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/helloworld_switch.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
copying tutel/examples/helloworld.py -> build/lib.linux-x86_64-cpython-39/tutel/examples
creating build/lib.linux-x86_64-cpython-39/tutel/impls
copying tutel/impls/init.py -> build/lib.linux-x86_64-cpython-39/tutel/impls
copying tutel/impls/communicate.py -> build/lib.linux-x86_64-cpython-39/tutel/impls
copying tutel/impls/losses.py -> build/lib.linux-x86_64-cpython-39/tutel/impls
copying tutel/impls/fast_dispatch.py -> build/lib.linux-x86_64-cpython-39/tutel/impls
copying tutel/impls/moe_layer.py -> build/lib.linux-x86_64-cpython-39/tutel/impls
copying tutel/impls/jit_compiler.py -> build/lib.linux-x86_64-cpython-39/tutel/impls
copying tutel/impls/overlap.py -> build/lib.linux-x86_64-cpython-39/tutel/impls
creating build/lib.linux-x86_64-cpython-39/tutel/checkpoint
copying tutel/checkpoint/gather.py -> build/lib.linux-x86_64-cpython-39/tutel/checkpoint
copying tutel/checkpoint/scatter.py -> build/lib.linux-x86_64-cpython-39/tutel/checkpoint
copying tutel/checkpoint/init.py -> build/lib.linux-x86_64-cpython-39/tutel/checkpoint
creating build/lib.linux-x86_64-cpython-39/tutel/experts
copying tutel/experts/ffn.py -> build/lib.linux-x86_64-cpython-39/tutel/experts
copying tutel/experts/init.py -> build/lib.linux-x86_64-cpython-39/tutel/experts
copying tutel/experts/llama_ffn.py -> build/lib.linux-x86_64-cpython-39/tutel/experts
creating build/lib.linux-x86_64-cpython-39/tutel/custom/extensions
copying tutel/custom/extensions/init.py -> build/lib.linux-x86_64-cpython-39/tutel/custom/extensions
creating build/lib.linux-x86_64-cpython-39/tutel/parted/backend
copying tutel/parted/backend/init.py -> build/lib.linux-x86_64-cpython-39/tutel/parted/backend
creating build/lib.linux-x86_64-cpython-39/tutel/parted/backend/torch
copying tutel/parted/backend/torch/executor.py -> build/lib.linux-x86_64-cpython-39/tutel/parted/backend/torch
copying tutel/parted/backend/torch/config.py -> build/lib.linux-x86_64-cpython-39/tutel/parted/backend/torch
copying tutel/parted/backend/torch/init.py -> build/lib.linux-x86_64-cpython-39/tutel/parted/backend/torch
running egg_info
creating tutel.egg-info
writing tutel.egg-info/PKG-INFO
writing dependency_links to tutel.egg-info/dependency_links.txt
writing requirements to tutel.egg-info/requires.txt
writing top-level names to tutel.egg-info/top_level.txt
writing manifest file 'tutel.egg-info/SOURCES.txt'
reading manifest file 'tutel.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'tutel.egg-info/SOURCES.txt'
/opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'tutel.ops.cuda' is absent from the packages configuration.
!!

      ********************************************************************************
      ############################
      # Package would be ignored #
      ############################
      Python recognizes 'tutel.ops.cuda' as an importable package[^1],
      but it is absent from setuptools' `packages` configuration.

      This leads to an ambiguous overall configuration. If you want to distribute this
      package, please make sure that 'tutel.ops.cuda' is explicitly added
      to the `packages` configuration field.

      Alternatively, you can also rely on setuptools' discovery methods
      (for example by using `find_namespace_packages(...)`/`find_namespace:`
      instead of `find_packages(...)`/`find:`).

      You can read more about "package discovery" on setuptools documentation page:

      - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

      If you don't want 'tutel.ops.cuda' to be distributed and are
      already explicitly excluding 'tutel.ops.cuda' via
      `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
      you can try to use `exclude_package_data`, or `include-package-data=False` in
      combination with a more fine grained `package-data` configuration.

      You can read more about "package data files" on setuptools documentation page:

      - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


      [^1]: For Python, any directory (with suitable naming) can be imported,
            even if it does not contain any `.py` files.
            On the other hand, currently there is no concept of package data
            directory, all directories are treated like packages.
      ********************************************************************************

!!
check.warn(importable)
/opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'tutel.ops.rocm' is absent from the packages configuration.
!!

      ********************************************************************************
      ############################
      # Package would be ignored #
      ############################
      Python recognizes 'tutel.ops.rocm' as an importable package[^1],
      but it is absent from setuptools' `packages` configuration.

      This leads to an ambiguous overall configuration. If you want to distribute this
      package, please make sure that 'tutel.ops.rocm' is explicitly added
      to the `packages` configuration field.

      Alternatively, you can also rely on setuptools' discovery methods
      (for example by using `find_namespace_packages(...)`/`find_namespace:`
      instead of `find_packages(...)`/`find:`).

      You can read more about "package discovery" on setuptools documentation page:

      - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

      If you don't want 'tutel.ops.rocm' to be distributed and are
      already explicitly excluding 'tutel.ops.rocm' via
      `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
      you can try to use `exclude_package_data`, or `include-package-data=False` in
      combination with a more fine grained `package-data` configuration.

      You can read more about "package data files" on setuptools documentation page:

      - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


      [^1]: For Python, any directory (with suitable naming) can be imported,
            even if it does not contain any `.py` files.
            On the other hand, currently there is no concept of package data
            directory, all directories are treated like packages.
      ********************************************************************************

!!
check.warn(importable)
creating build/lib.linux-x86_64-cpython-39/tutel/ops/cuda
copying tutel/ops/cuda/deepseek_r1_sigmoid_top_k_routed_scaled_f32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda
copying tutel/ops/cuda/qwen3_moe_top_k_routed_scaled_f32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda
copying tutel/ops/cuda/scatter_sample_ids_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda
copying tutel/ops/cuda/scatter_top_ids_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda
copying tutel/ops/cuda/to_bfloat16_3d.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda
copying tutel/ops/cuda/token_sort_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda
creating build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/deepseek_r1_sigmoid_top_k_routed_scaled_f32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_blkvect_phase_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_blkvect_phase_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_blockscal_stage_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_blockscal_stage_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_blockscal_stage_3.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_blockscal_stage_4.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_blockscal_vector_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_blockscal_vector_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_f16xf4_stage_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_f16xf4_stage_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_w8a16_stage_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_w8a16_stage_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_w8a16_stage_3.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_w8a16_vector_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fmoe_w8a16_vector_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/fused_silu_mul_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/gemm_down_weight_sum_bf16xf8_s_16x16_fnuz.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/gemm_down_weight_sum_bf16xf8_s_16x16_fnuz_bs4_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/gemm_down_weight_sum_bf16xf8_s_16x16_fnuz_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/gemm_gate_up_silu_bf16xf8_s_16x16_fnuz.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/gemm_gate_up_silu_bf16xf8_s_16x16_fnuz_bs4_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/gemm_gate_up_silu_bf16xf8_s_16x16_fnuz_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/gemv_nt_bf16xfp8_block.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/gemv_nt_bf16xfp8_block_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/gemv_nt_bf16xfp8_row.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/qwen3_moe_top_k_routed_scaled_f32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/qwen3_norm_rotary_kvcache_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/rmsnorm2_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/rmsnorm_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/rope_gmv_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/rope_kt_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/rope_mla_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/rope_q_out_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/rope_qt_bf16_put.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/scatter_sample_ids_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/scatter_top_ids_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/to_bfloat16_3d.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/to_float32_3d.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/to_float8_block.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/to_float8_per_token.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
copying tutel/ops/rocm/token_sort_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm
running build_ext
building 'tutel_custom_kernel' extension
creating build/temp.linux-x86_64-cpython-39/tutel/custom
g++ -pthread -B /opt/conda/envs/motioncraft/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /opt/conda/envs/motioncraft/include -I/opt/conda/envs/motioncraft/include -fPIC -O2 -isystem /opt/conda/envs/motioncraft/include -fPIC -I/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/include -I/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/include/TH -I/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda-12.9/include -I/opt/conda/envs/motioncraft/include/python3.9 -c ./tutel/custom/custom_kernel.cpp -o build/temp.linux-x86_64-cpython-39/tutel/custom/custom_kernel.o -w -DUSE_GPU -DUSE_NCCL -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=tutel_custom_kernel -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
./tutel/custom/custom_kernel.cpp:5:10: fatal error: torch/csrc/distributed/c10d/ProcessGroupGloo.hpp: No such file or directory
#include <torch/csrc/distributed/c10d/ProcessGroupGloo.hpp>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/g++' failed with exit code 1
error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /opt/conda/envs/motioncraft/bin/python3.9 -u -c '
exec(compile('"'"''"'"''"'"'

This is -- a caller that pip uses to run setup.py

- It imports setuptools before invoking setup.py, to enable projects that directly

import from distutils.core to work with newer packaging standards.

- It provides a clear error message when setuptools is not installed.

- It sets sys.argv[0] to the underlying setup.py, when invoking setup.py so

setuptools doesn'"'"'t think the script is -c. This avoids the following warning:

manifest_maker: standard file '"'"'-c'"'"' not found".

- It generates a shim setup.py, for handling setup.cfg-only projects.

import os, sys, tokenize, traceback

try:
import setuptools
except ImportError:
print(
"ERROR: Can not execute setup.py since setuptools failed to import in "
"the build environment with exception:",
file=sys.stderr,
)
traceback.print_exc()
sys.exit(1)

file = %r
sys.argv[0] = file

if os.path.exists(file):
filename = file
with tokenize.open(file) as f:
setup_py_code = f.read()
else:
filename = ""
setup_py_code = "from setuptools import setup; setup()"

exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/tmp/pip-req-build-zmk70hwn/setup.py'"'"',), "", "exec"))' bdist_wheel -d /tmp/pip-wheel-tgclcj3h
cwd: /tmp/pip-req-build-zmk70hwn/
Building wheel for tutel (setup.py) ... error
ERROR: Failed building wheel for tutel
Running setup.py clean for tutel
Running command python setup.py clean
Try installing with NCCL extension..
/opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!

      ********************************************************************************
      Please consider removing the following classifiers in favor of a SPDX license expression:

      License :: OSI Approved :: MIT License

      See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
      ********************************************************************************

!!
self._finalize_license_expression()
running clean
removing 'build/temp.linux-x86_64-cpython-39' (and everything under it)
removing 'build/lib.linux-x86_64-cpython-39' (and everything under it)
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.9' does not exist -- can't clean it
removing 'build'
Failed to build tutel
ERROR: Failed to build installable wheels for some pyproject.toml based projects (tutel)

Because of that I tried checking out older tags:
v0.1.x: the code is too old and doesn’t match the current repository layout
v0.2.x: installs, but then fails with runtime errors like
v0.4.0: seems to conflict with PyTorch 1.12.1
not find v0.3.x

[yxbian23]

Thanks for your reply. Yes – I did try. but I ran into this error log: (motioncraft) (base) root@uzk_544:~/projects/MotionCraft# pip install --verbose --upgrade git+https://github.com/microsoft/tutel@main Using pip 25.1.1 from /opt/conda/envs/motion/lib/python3.9/site-packages/pip (python 3.9) Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting git+https://github.com/microsoft/tutel@main Cloning https://github.com/microsoft/tutel (to revision main) to /tmp/pip-req-build-zmk70hwn Running command git version git version 2.17.1 Running command git clone --filter=blob:none https://github.com/microsoft/tutel /tmp/pip-req-build-zmk70hwn Cloning into '/tmp/pip-req-build-zmk70hwn'... Checking out files: 1% (2/127) Checking out files: 2% (3/127) Checking out files: 3% (4/127) Checking out files: 4% (6/127) Checking out files: 5% (7/127) Checking out files: 6% (8/127) Checking out files: 7% (9/127) Checking out files: 8% (11/127) Checking out files: 9% (12/127) Checking out files: 10% (13/127) Checking out files: 11% (14/127) Checking out files: 12% (16/127) Checking out files: 13% (17/127) Checking out files: 14% (18/127) Checking out files: 15% (20/127) Checking out files: 16% (21/127) Checking out files: 17% (22/127) Checking out files: 18% (23/127) Checking out files: 19% (25/127) Checking out files: 20% (26/127) Checking out files: 21% (27/127) Checking out files: 22% (28/127) Checking out files: 23% (30/127) Checking out files: 24% (31/127) Checking out files: 25% (32/127) Checking out files: 26% (34/127) Checking out files: 27% (35/127) Checking out files: 28% (36/127) Checking out files: 29% (37/127) Checking out files: 30% (39/127) Checking out files: 31% (40/127) Checking out files: 32% (41/127) Checking out files: 33% (42/127) Checking out files: 34% (44/127) Checking out files: 35% (45/127) Checking out files: 36% (46/127) Checking out files: 37% (47/127) Checking out files: 38% (49/127) Checking out files: 39% (50/127) Checking out files: 40% (51/127) Checking out files: 41% (53/127) Checking out files: 42% (54/127) Checking out files: 43% (55/127) Checking out files: 44% (56/127) Checking out files: 45% (58/127) Checking out files: 46% (59/127) Checking out files: 47% (60/127) Checking out files: 48% (61/127) Checking out files: 49% (63/127) Checking out files: 50% (64/127) Checking out files: 51% (65/127) Checking out files: 52% (67/127) Checking out files: 53% (68/127) Checking out files: 54% (69/127) Checking out files: 55% (70/127) Checking out files: 56% (72/127) Checking out files: 57% (73/127) Checking out files: 58% (74/127) Checking out files: 59% (75/127) Checking out files: 60% (77/127) Checking out files: 61% (78/127) Checking out files: 62% (79/127) Checking out files: 63% (81/127) Checking out files: 64% (82/127) Checking out files: 65% (83/127) Checking out files: 66% (84/127) Checking out files: 67% (86/127) Checking out files: 68% (87/127) Checking out files: 69% (88/127) Checking out files: 70% (89/127) Checking out files: 71% (91/127) Checking out files: 72% (92/127) Checking out files: 73% (93/127) Checking out files: 74% (94/127) Checking out files: 75% (96/127) Checking out files: 76% (97/127) Checking out files: 77% (98/127) Checking out files: 78% (100/127) Checking out files: 79% (101/127) Checking out files: 80% (102/127) Checking out files: 81% (103/127) Checking out files: 82% (105/127) Checking out files: 83% (106/127) Checking out files: 84% (107/127) Checking out files: 85% (108/127) Checking out files: 86% (110/127) Checking out files: 87% (111/127) Checking out files: 88% (112/127) Checking out files: 89% (114/127) Checking out files: 90% (115/127) Checking out files: 91% (116/127) Checking out files: 92% (117/127) Checking out files: 93% (119/127) Checking out files: 94% (120/127) Checking out files: 95% (121/127) Checking out files: 96% (122/127) Checking out files: 97% (124/127) Checking out files: 98% (125/127) Checking out files: 99% (126/127) Checking out files: 100% (127/127) Checking out files: 100% (127/127), done. Running command git show-ref main ae44c2bc08fad09efa7660944b3dd3601661a42a refs/heads/main ae44c2bc08fad09efa7660944b3dd3601661a42a refs/remotes/origin/main Running command git symbolic-ref -q HEAD refs/heads/main Resolved https://github.com/microsoft/tutel to commit ae44c2bc08fad09efa7660944b3dd3601661a42a Running command git rev-parse HEAD ae44c2bc08fad09efa7660944b3dd3601661a42a Running command python setup.py egg_info /opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated. !!

      ********************************************************************************
      Please consider removing the following classifiers in favor of a SPDX license expression:

      License :: OSI Approved :: MIT License

      See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
      ********************************************************************************

!! self._finalize_license_expression() Try installing with NCCL extension.. running egg_info creating /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info writing /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/PKG-INFO writing dependency_links to /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/dependency_links.txt writing requirements to /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/requires.txt writing top-level names to /tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/top_level.txt writing manifest file '/tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/SOURCES.txt' /opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/utils/cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend. warnings.warn(msg.format('we could not find ninja.')) reading manifest file '/tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file '/tmp/pip-pip-egg-info-kam0rm5p/tutel.egg-info/SOURCES.txt' Preparing metadata (setup.py) ... done Requirement already satisfied: numpy in /opt/conda/envs/motioncraft/lib/python3.9/site-packages (from tutel==0.4) (1.23.2) Building wheels for collected packages: tutel Running command git rev-parse HEAD ae44c2bc08fad09efa7660944b3dd3601661a42a DEPRECATION: Building 'tutel' using the legacy setup.py bdist_wheel mechanism, which will be removed in a future version. pip 25.3 will enforce this behaviour change. A possible replacement is to use the standardized build interface by setting the --use-pep517 option, (possibly combined with --no-build-isolation), or adding a pyproject.toml file to the source tree of 'tutel'. Discussion can be found at pypa/pip#6334 Running command python setup.py bdist_wheel Try installing with NCCL extension.. /opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated. !!

      ********************************************************************************
      Please consider removing the following classifiers in favor of a SPDX license expression:

      License :: OSI Approved :: MIT License

      See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
      ********************************************************************************

!! self._finalize_license_expression() running bdist_wheel /opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/utils/cpp_extension.py:411: UserWarning: Attempted to use ninja as the BuildExtension backend but we could not find ninja.. Falling back to using the slow distutils backend. warnings.warn(msg.format('we could not find ninja.')) running build running build_py creating build/lib.linux-x86_64-cpython-39/tutel copying tutel/system.py -> build/lib.linux-x86_64-cpython-39/tutel copying tutel/init.py -> build/lib.linux-x86_64-cpython-39/tutel copying tutel/net.py -> build/lib.linux-x86_64-cpython-39/tutel copying tutel/moe.py -> build/lib.linux-x86_64-cpython-39/tutel copying tutel/jit.py -> build/lib.linux-x86_64-cpython-39/tutel creating build/lib.linux-x86_64-cpython-39/tutel/launcher copying tutel/launcher/run.py -> build/lib.linux-x86_64-cpython-39/tutel/launcher copying tutel/launcher/execl.py -> build/lib.linux-x86_64-cpython-39/tutel/launcher copying tutel/launcher/init.py -> build/lib.linux-x86_64-cpython-39/tutel/launcher creating build/lib.linux-x86_64-cpython-39/tutel/gates copying tutel/gates/top.py -> build/lib.linux-x86_64-cpython-39/tutel/gates copying tutel/gates/init.py -> build/lib.linux-x86_64-cpython-39/tutel/gates copying tutel/gates/cosine_top.py -> build/lib.linux-x86_64-cpython-39/tutel/gates creating build/lib.linux-x86_64-cpython-39/tutel/custom copying tutel/custom/init.py -> build/lib.linux-x86_64-cpython-39/tutel/custom creating build/lib.linux-x86_64-cpython-39/tutel/jit_kernels copying tutel/jit_kernels/sparse.py -> build/lib.linux-x86_64-cpython-39/tutel/jit_kernels copying tutel/jit_kernels/init.py -> build/lib.linux-x86_64-cpython-39/tutel/jit_kernels copying tutel/jit_kernels/gating.py -> build/lib.linux-x86_64-cpython-39/tutel/jit_kernels creating build/lib.linux-x86_64-cpython-39/tutel/ops copying tutel/ops/init.py -> build/lib.linux-x86_64-cpython-39/tutel/ops creating build/lib.linux-x86_64-cpython-39/tutel/parted copying tutel/parted/solver.py -> build/lib.linux-x86_64-cpython-39/tutel/parted copying tutel/parted/init.py -> build/lib.linux-x86_64-cpython-39/tutel/parted copying tutel/parted/patterns.py -> build/lib.linux-x86_64-cpython-39/tutel/parted copying tutel/parted/spmdx.py -> build/lib.linux-x86_64-cpython-39/tutel/parted creating build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/helloworld_amp.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/nccl_all_to_all_v.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/moe_cifar10.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/nccl_allreduce_perf.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/bandwidth_test.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/helloworld_ddp.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/moe_mnist.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/helloworld_ddp_tutel.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/nccl_all_gather_v.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/init.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/helloworld_custom_gate_expert.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/helloworld_custom_expert_sharded.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/helloworld_from_scratch.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/helloworld_switch.py -> build/lib.linux-x86_64-cpython-39/tutel/examples copying tutel/examples/helloworld.py -> build/lib.linux-x86_64-cpython-39/tutel/examples creating build/lib.linux-x86_64-cpython-39/tutel/impls copying tutel/impls/init.py -> build/lib.linux-x86_64-cpython-39/tutel/impls copying tutel/impls/communicate.py -> build/lib.linux-x86_64-cpython-39/tutel/impls copying tutel/impls/losses.py -> build/lib.linux-x86_64-cpython-39/tutel/impls copying tutel/impls/fast_dispatch.py -> build/lib.linux-x86_64-cpython-39/tutel/impls copying tutel/impls/moe_layer.py -> build/lib.linux-x86_64-cpython-39/tutel/impls copying tutel/impls/jit_compiler.py -> build/lib.linux-x86_64-cpython-39/tutel/impls copying tutel/impls/overlap.py -> build/lib.linux-x86_64-cpython-39/tutel/impls creating build/lib.linux-x86_64-cpython-39/tutel/checkpoint copying tutel/checkpoint/gather.py -> build/lib.linux-x86_64-cpython-39/tutel/checkpoint copying tutel/checkpoint/scatter.py -> build/lib.linux-x86_64-cpython-39/tutel/checkpoint copying tutel/checkpoint/init.py -> build/lib.linux-x86_64-cpython-39/tutel/checkpoint creating build/lib.linux-x86_64-cpython-39/tutel/experts copying tutel/experts/ffn.py -> build/lib.linux-x86_64-cpython-39/tutel/experts copying tutel/experts/init.py -> build/lib.linux-x86_64-cpython-39/tutel/experts copying tutel/experts/llama_ffn.py -> build/lib.linux-x86_64-cpython-39/tutel/experts creating build/lib.linux-x86_64-cpython-39/tutel/custom/extensions copying tutel/custom/extensions/init.py -> build/lib.linux-x86_64-cpython-39/tutel/custom/extensions creating build/lib.linux-x86_64-cpython-39/tutel/parted/backend copying tutel/parted/backend/init.py -> build/lib.linux-x86_64-cpython-39/tutel/parted/backend creating build/lib.linux-x86_64-cpython-39/tutel/parted/backend/torch copying tutel/parted/backend/torch/executor.py -> build/lib.linux-x86_64-cpython-39/tutel/parted/backend/torch copying tutel/parted/backend/torch/config.py -> build/lib.linux-x86_64-cpython-39/tutel/parted/backend/torch copying tutel/parted/backend/torch/init.py -> build/lib.linux-x86_64-cpython-39/tutel/parted/backend/torch running egg_info creating tutel.egg-info writing tutel.egg-info/PKG-INFO writing dependency_links to tutel.egg-info/dependency_links.txt writing requirements to tutel.egg-info/requires.txt writing top-level names to tutel.egg-info/top_level.txt writing manifest file 'tutel.egg-info/SOURCES.txt' reading manifest file 'tutel.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file 'tutel.egg-info/SOURCES.txt' /opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'tutel.ops.cuda' is absent from the packages configuration. !!

      ********************************************************************************
      ############################
      # Package would be ignored #
      ############################
      Python recognizes 'tutel.ops.cuda' as an importable package[^1],
      but it is absent from setuptools' `packages` configuration.

      This leads to an ambiguous overall configuration. If you want to distribute this
      package, please make sure that 'tutel.ops.cuda' is explicitly added
      to the `packages` configuration field.

      Alternatively, you can also rely on setuptools' discovery methods
      (for example by using `find_namespace_packages(...)`/`find_namespace:`
      instead of `find_packages(...)`/`find:`).

      You can read more about "package discovery" on setuptools documentation page:

      - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

      If you don't want 'tutel.ops.cuda' to be distributed and are
      already explicitly excluding 'tutel.ops.cuda' via
      `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
      you can try to use `exclude_package_data`, or `include-package-data=False` in
      combination with a more fine grained `package-data` configuration.

      You can read more about "package data files" on setuptools documentation page:

      - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


      [^1]: For Python, any directory (with suitable naming) can be imported,
            even if it does not contain any `.py` files.
            On the other hand, currently there is no concept of package data
            directory, all directories are treated like packages.
      ********************************************************************************

!! check.warn(importable) /opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/command/build_py.py:212: _Warning: Package 'tutel.ops.rocm' is absent from the packages configuration. !!

      ********************************************************************************
      ############################
      # Package would be ignored #
      ############################
      Python recognizes 'tutel.ops.rocm' as an importable package[^1],
      but it is absent from setuptools' `packages` configuration.

      This leads to an ambiguous overall configuration. If you want to distribute this
      package, please make sure that 'tutel.ops.rocm' is explicitly added
      to the `packages` configuration field.

      Alternatively, you can also rely on setuptools' discovery methods
      (for example by using `find_namespace_packages(...)`/`find_namespace:`
      instead of `find_packages(...)`/`find:`).

      You can read more about "package discovery" on setuptools documentation page:

      - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html

      If you don't want 'tutel.ops.rocm' to be distributed and are
      already explicitly excluding 'tutel.ops.rocm' via
      `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
      you can try to use `exclude_package_data`, or `include-package-data=False` in
      combination with a more fine grained `package-data` configuration.

      You can read more about "package data files" on setuptools documentation page:

      - https://setuptools.pypa.io/en/latest/userguide/datafiles.html


      [^1]: For Python, any directory (with suitable naming) can be imported,
            even if it does not contain any `.py` files.
            On the other hand, currently there is no concept of package data
            directory, all directories are treated like packages.
      ********************************************************************************

!! check.warn(importable) creating build/lib.linux-x86_64-cpython-39/tutel/ops/cuda copying tutel/ops/cuda/deepseek_r1_sigmoid_top_k_routed_scaled_f32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda copying tutel/ops/cuda/qwen3_moe_top_k_routed_scaled_f32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda copying tutel/ops/cuda/scatter_sample_ids_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda copying tutel/ops/cuda/scatter_top_ids_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda copying tutel/ops/cuda/to_bfloat16_3d.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda copying tutel/ops/cuda/token_sort_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/cuda creating build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/deepseek_r1_sigmoid_top_k_routed_scaled_f32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_blkvect_phase_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_blkvect_phase_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_blockscal_stage_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_blockscal_stage_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_blockscal_stage_3.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_blockscal_stage_4.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_blockscal_vector_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_blockscal_vector_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_f16xf4_stage_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_f16xf4_stage_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_w8a16_stage_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_w8a16_stage_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_w8a16_stage_3.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_w8a16_vector_1.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fmoe_w8a16_vector_2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/fused_silu_mul_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/gemm_down_weight_sum_bf16xf8_s_16x16_fnuz.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/gemm_down_weight_sum_bf16xf8_s_16x16_fnuz_bs4_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/gemm_down_weight_sum_bf16xf8_s_16x16_fnuz_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/gemm_gate_up_silu_bf16xf8_s_16x16_fnuz.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/gemm_gate_up_silu_bf16xf8_s_16x16_fnuz_bs4_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/gemm_gate_up_silu_bf16xf8_s_16x16_fnuz_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/gemv_nt_bf16xfp8_block.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/gemv_nt_bf16xfp8_block_v2.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/gemv_nt_bf16xfp8_row.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/qwen3_moe_top_k_routed_scaled_f32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/qwen3_norm_rotary_kvcache_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/rmsnorm2_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/rmsnorm_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/rope_gmv_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/rope_kt_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/rope_mla_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/rope_q_out_bf16.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/rope_qt_bf16_put.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/scatter_sample_ids_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/scatter_top_ids_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/to_bfloat16_3d.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/to_float32_3d.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/to_float8_block.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/to_float8_per_token.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm copying tutel/ops/rocm/token_sort_i32.mod -> build/lib.linux-x86_64-cpython-39/tutel/ops/rocm running build_ext building 'tutel_custom_kernel' extension creating build/temp.linux-x86_64-cpython-39/tutel/custom g++ -pthread -B /opt/conda/envs/motioncraft/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -Wall -fPIC -O2 -isystem /opt/conda/envs/motioncraft/include -I/opt/conda/envs/motioncraft/include -fPIC -O2 -isystem /opt/conda/envs/motioncraft/include -fPIC -I/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/include -I/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/include/TH -I/opt/conda/envs/motioncraft/lib/python3.9/site-packages/torch/include/THC -I/usr/local/cuda-12.9/include -I/opt/conda/envs/motioncraft/include/python3.9 -c ./tutel/custom/custom_kernel.cpp -o build/temp.linux-x86_64-cpython-39/tutel/custom/custom_kernel.o -w -DUSE_GPU -DUSE_NCCL -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -DTORCH_EXTENSION_NAME=tutel_custom_kernel -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14 ./tutel/custom/custom_kernel.cpp:5:10: fatal error: torch/csrc/distributed/c10d/ProcessGroupGloo.hpp: No such file or directory #include <torch/csrc/distributed/c10d/ProcessGroupGloo.hpp> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated. error: command '/usr/bin/g++' failed with exit code 1 error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip. full command: /opt/conda/envs/motioncraft/bin/python3.9 -u -c ' exec(compile('"'"''"'"''"'"'

This is -- a caller that pip uses to run setup.py

- It imports setuptools before invoking setup.py, to enable projects that directly

import from distutils.core to work with newer packaging standards.

- It provides a clear error message when setuptools is not installed.

- It sets sys.argv[0] to the underlying setup.py, when invoking setup.py so

setuptools doesn'"'"'t think the script is -c. This avoids the following warning:

manifest_maker: standard file '"'"'-c'"'"' not found".

- It generates a shim setup.py, for handling setup.cfg-only projects.

import os, sys, tokenize, traceback

try: import setuptools except ImportError: print( "ERROR: Can not execute setup.py since setuptools failed to import in " "the build environment with exception:", file=sys.stderr, ) traceback.print_exc() sys.exit(1)

file = %r sys.argv[0] = file

if os.path.exists(file): filename = file with tokenize.open(file) as f: setup_py_code = f.read() else: filename = "" setup_py_code = "from setuptools import setup; setup()"

exec(compile(setup_py_code, filename, "exec")) '"'"''"'"''"'"' % ('"'"'/tmp/pip-req-build-zmk70hwn/setup.py'"'"',), "", "exec"))' bdist_wheel -d /tmp/pip-wheel-tgclcj3h cwd: /tmp/pip-req-build-zmk70hwn/ Building wheel for tutel (setup.py) ... error ERROR: Failed building wheel for tutel Running setup.py clean for tutel Running command python setup.py clean Try installing with NCCL extension.. /opt/conda/envs/motioncraft/lib/python3.9/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated. !!

      ********************************************************************************
      Please consider removing the following classifiers in favor of a SPDX license expression:

      License :: OSI Approved :: MIT License

      See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
      ********************************************************************************

!! self._finalize_license_expression() running clean removing 'build/temp.linux-x86_64-cpython-39' (and everything under it) removing 'build/lib.linux-x86_64-cpython-39' (and everything under it) 'build/bdist.linux-x86_64' does not exist -- can't clean it 'build/scripts-3.9' does not exist -- can't clean it removing 'build' Failed to build tutel ERROR: Failed to build installable wheels for some pyproject.toml based projects (tutel)

Because of that I tried checking out older tags: v0.1.x: the code is too old and doesn’t match the current repository layout v0.2.x: installs, but then fails with runtime errors like v0.4.0: seems to conflict with PyTorch 1.12.1 not find v0.3.x

This seems very weird... I haven't encountered such error before. But I guess is your cuda or torch environment setting misalignment? Because I saw the key error said:

"./tutel/custom/custom_kernel.cpp:5:10: fatal error: torch/csrc/distributed/c10d/ProcessGroupGloo.hpp: No such file or directory
#include <torch/csrc/distributed/c10d/ProcessGroupGloo.hpp>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/bin/g++' failed with exit code 1
error: subprocess-exited-with-error"

[uuuzk]

Thanks, so, it work with environment: "pytorch==1.12.1 torchvision==0.13.1 cudatoolkit=11.3" and tutel@main(the latest v0.4.1),
, maybe my cuda or torch environment setting was wrong?