Skip to content

运行 memory offload 模式报错 #17

Description

@double-corner

Environment

Hardware Environment(Ascend/GPU/CPU):

/device gpu

Software Environment:

  • MindSpore version (source or binary): 2.4.0rc1
  • Python version (e.g., Python 3.7.5): 3.9.11
  • OS platform and distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 24.04
  • GCC/Compiler version (if compiled from source): 9.4.0

Describe the current behavior

[WARNING] DEVICE(1234498,7f778ee25740,python):2026-01-04-16:35:46.341.353 [mindspore/ccsrc/runtime/device/gsm/io_handle.cc:48] LoadAio] Loading libaio_plugin.so failed. Error message: libaio_plugin.so: cannot open shared object file: No such file or directory
[ERROR] DEVICE(1234498,7f7600b336c0,python):2026-01-04-16:35:59.584.915 [mindspore/ccsrc/plugin/device/gpu/hal/device/gpu_device_address.cc:226] SyncDeviceToDevice] Format or type is different, src(format:NHWC, type_id:Float16), dst(format:DefaultFormat), type_id:Float16

Describe the expected behavior

无报错,能够正常运行 GPU 训练及 memory offload 功能

Steps to reproduce the issue

  1. /home/test/MindSpore/models/benchmark/ascend/resnet/train.py 代码set_parameter()函数中添加如下内容:

    memory_offload

    offload_config = {"offload_path": "./offload/",
    "auto_offload": True,
    "offload_param": "cpu",
    "offload_cpu_size": "512GB",
    "offload_disk_size": "1024GB",
    "host_mem_block_size":"1GB",
    "enable_aio": False,
    "enable_pinned_mem": True}
    ms.set_context(mode=ms.GRAPH_MODE, memory_offload='ON', max_device_memory='10GB')
    ms.set_offload_context(offload_config=offload_config)
  2. 运行 resnet GPU训练脚本:bash run_standalone_train_gpu.sh /home/test/MindSpore/dataset/imagenet/ /home/test/MindSpore/models/benchmark/ascend/resnet/config/resnet152_imagenet2012_config_gpu.yaml
  3. 运行报错
    [ERROR] DEVICE(1234498,7f7600b336c0,python):2026-01-04-16:35:59.584.915 [mindspore/ccsrc/plugin/device/gpu/hal/device/gpu_device_address.cc:226] SyncDeviceToDevice] Format or type is different, src(format:NHWC, type_id:Float16), dst(format:DefaultFormat), type_id:Float16

Related log / screenshot

RuntimeError: Copy device tensor failed: 18_Default/network-TrainOneStepCell/network-WithLossCell/_backbone-ResNet/layer1-SequentialCell/0-ResidualBlock/conv1-Conv2d/Conv2D-op0

terminate called after throwing an instance of 'std::runtime_error'
what(): Create device context failed, please make sure target device:GPU is available, error message of loading plugins:

Special notes for this issue

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions