Environment
Hardware Environment(Ascend/GPU/CPU):
/device gpu
Software Environment:
- MindSpore version (source or binary): 2.4.0rc1
- Python version (e.g., Python 3.7.5): 3.9.11
- OS platform and distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 24.04
- GCC/Compiler version (if compiled from source): 9.4.0
Describe the current behavior
[WARNING] DEVICE(1234498,7f778ee25740,python):2026-01-04-16:35:46.341.353 [mindspore/ccsrc/runtime/device/gsm/io_handle.cc:48] LoadAio] Loading libaio_plugin.so failed. Error message: libaio_plugin.so: cannot open shared object file: No such file or directory
[ERROR] DEVICE(1234498,7f7600b336c0,python):2026-01-04-16:35:59.584.915 [mindspore/ccsrc/plugin/device/gpu/hal/device/gpu_device_address.cc:226] SyncDeviceToDevice] Format or type is different, src(format:NHWC, type_id:Float16), dst(format:DefaultFormat), type_id:Float16
Describe the expected behavior
无报错,能够正常运行 GPU 训练及 memory offload 功能
Steps to reproduce the issue
- /home/test/MindSpore/models/benchmark/ascend/resnet/train.py 代码set_parameter()函数中添加如下内容:
memory_offload
offload_config = {"offload_path": "./offload/",
"auto_offload": True,
"offload_param": "cpu",
"offload_cpu_size": "512GB",
"offload_disk_size": "1024GB",
"host_mem_block_size":"1GB",
"enable_aio": False,
"enable_pinned_mem": True}
ms.set_context(mode=ms.GRAPH_MODE, memory_offload='ON', max_device_memory='10GB')
ms.set_offload_context(offload_config=offload_config)
- 运行 resnet GPU训练脚本:bash run_standalone_train_gpu.sh /home/test/MindSpore/dataset/imagenet/ /home/test/MindSpore/models/benchmark/ascend/resnet/config/resnet152_imagenet2012_config_gpu.yaml
- 运行报错
[ERROR] DEVICE(1234498,7f7600b336c0,python):2026-01-04-16:35:59.584.915 [mindspore/ccsrc/plugin/device/gpu/hal/device/gpu_device_address.cc:226] SyncDeviceToDevice] Format or type is different, src(format:NHWC, type_id:Float16), dst(format:DefaultFormat), type_id:Float16
Related log / screenshot
RuntimeError: Copy device tensor failed: 18_Default/network-TrainOneStepCell/network-WithLossCell/_backbone-ResNet/layer1-SequentialCell/0-ResidualBlock/conv1-Conv2d/Conv2D-op0
terminate called after throwing an instance of 'std::runtime_error'
what(): Create device context failed, please make sure target device:GPU is available, error message of loading plugins:
Special notes for this issue
Environment
Hardware Environment(
Ascend/GPU/CPU):Software Environment:
Describe the current behavior
[WARNING] DEVICE(1234498,7f778ee25740,python):2026-01-04-16:35:46.341.353 [mindspore/ccsrc/runtime/device/gsm/io_handle.cc:48] LoadAio] Loading libaio_plugin.so failed. Error message: libaio_plugin.so: cannot open shared object file: No such file or directory
[ERROR] DEVICE(1234498,7f7600b336c0,python):2026-01-04-16:35:59.584.915 [mindspore/ccsrc/plugin/device/gpu/hal/device/gpu_device_address.cc:226] SyncDeviceToDevice] Format or type is different, src(format:NHWC, type_id:Float16), dst(format:DefaultFormat), type_id:Float16
Describe the expected behavior
无报错,能够正常运行 GPU 训练及 memory offload 功能
Steps to reproduce the issue
memory_offload
offload_config = {"offload_path": "./offload/","auto_offload": True,
"offload_param": "cpu",
"offload_cpu_size": "512GB",
"offload_disk_size": "1024GB",
"host_mem_block_size":"1GB",
"enable_aio": False,
"enable_pinned_mem": True}
ms.set_context(mode=ms.GRAPH_MODE, memory_offload='ON', max_device_memory='10GB')
ms.set_offload_context(offload_config=offload_config)
[ERROR] DEVICE(1234498,7f7600b336c0,python):2026-01-04-16:35:59.584.915 [mindspore/ccsrc/plugin/device/gpu/hal/device/gpu_device_address.cc:226] SyncDeviceToDevice] Format or type is different, src(format:NHWC, type_id:Float16), dst(format:DefaultFormat), type_id:Float16
Related log / screenshot
RuntimeError: Copy device tensor failed: 18_Default/network-TrainOneStepCell/network-WithLossCell/_backbone-ResNet/layer1-SequentialCell/0-ResidualBlock/conv1-Conv2d/Conv2D-op0
terminate called after throwing an instance of 'std::runtime_error'
what(): Create device context failed, please make sure target device:GPU is available, error message of loading plugins:
Special notes for this issue