MI250X cannot initialize the driver

[KegangWangCCNU]

I have a MI250X, using ubuntu 22.04, and installed amdgpu-dkms according to the AMD website, but it cannot be initialized. After loading the driver, it freezes and automatically restarts after a few minutes. I can't provide a detailed log because I can't find the path to it.

Linux system log when loading driver:

[ 7659.094781] amdkcl: loading out-of-tree module taints kernel.
[ 7659.094804] amdkcl: module verification failed: signature and/or required key missing - tainting kernel
[ 7659.138731] AMD-Vi: AMD IOMMUv2 loaded and initialized
[ 7660.215273] [drm] amdgpu kernel modesetting enabled.
[ 7660.215275] [drm] amdgpu version: 6.2.4
[ 7660.215276] [drm] OS DRM version: 6.2.0
[ 7660.215292] amdgpu: vga_switcheroo: detected switching method \_SB_.PCI0.GPP8.SWUS.SWDS.VGA_.ATPX handle
[ 7660.215407] ATPX version 1, functions 0x00000000
[ 7660.227388] amdgpu: Ignoring ACPI CRAT on non-APU system
[ 7660.227391] amdgpu: Virtual CRAT table created for CPU
[ 7660.227404] amdgpu: Topology: Add CPU node
[ 7660.247176] amdgpu: PeerDirect support was initialized successfully
[ 7660.247303] amdgpu 0000:0c:00.0: enabling device (0000 -> 0003)
[ 7660.247355] [drm] initializing kernel modesetting (ALDEBARAN 0x1002:0x740C 0x1002:0x0B0C 0x01).
[ 7660.247368] [drm] register mmio base: 0xFCA00000
[ 7660.247370] [drm] register mmio size: 524288
[ 7660.247408] [drm] add ip block number 0 <soc15_common>
[ 7660.247409] [drm] add ip block number 1 <gmc_v9_0>
[ 7660.247411] [drm] add ip block number 2 <vega20_ih>
[ 7660.247412] [drm] add ip block number 3 <psp>
[ 7660.247413] [drm] add ip block number 4 <smu>
[ 7660.247414] [drm] add ip block number 5 <gfx_v9_0>
[ 7660.247416] [drm] add ip block number 6 <sdma_v4_0>
[ 7660.247417] [drm] add ip block number 7 <vcn_v2_6>
[ 7660.247418] [drm] add ip block number 8 <jpeg_v2_6>
[ 7684.811780] watchdog: BUG: soft lockup - CPU#24 stuck for 27s! [modprobe:41476]
[ 7684.811814] Modules linked in: ib_uverbs ib_core amdgpu(OE+) amddrm_ttm_helper(OE) amdttm(OE) amdxcp(OE) iommu_v2 amddrm_buddy(OE) amd_sched(OE) amdkcl(OE) drm_display_helper cec rc_core drm_kms_helper i2c_algo_bit syscopyarea sysfillrect sysimgblt binfmt_misc nls_iso8859_1 snd_hda_codec_realtek snd_hda_codec_generic intel_rapl_msr intel_rapl_common snd_hda_codec_hdmi edac_mce_amd mt7921e uvcvideo mt7921_common snd_hda_intel snd_intel_dspcfg kvm_amd mt76_connac_lib videobuf2_vmalloc snd_intel_sdw_acpi mt76 snd_hda_codec snd_usb_audio kvm snd_usbmidi_lib videobuf2_memops mac80211 irqbypass snd_hda_core snd_rawmidi videobuf2_v4l2 snd_seq_device snd_hwdep videodev rapl snd_pcm videobuf2_common input_leds eeepc_wmi joydev cfg80211 asus_nb_wmi snd_timer wmi_bmof mc k10temp snd ccp libarc4 soundcore mac_hid sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efi_pstore drm ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq
[ 7684.811883]  async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic usbhid uas hid usb_storage crct10dif_pclmul mfd_aaeon crc32_pclmul polyval_clmulni asus_wmi polyval_generic ghash_clmulni_intel video sha512_ssse3 ledtrig_audio aesni_intel crypto_simd r8169 nvme sparse_keymap platform_profile cryptd i2c_piix4 realtek ahci nvme_core xhci_pci libahci xhci_pci_renesas nvme_common wmi gpio_amdpt
[ 7684.811915] CPU: 24 PID: 41476 Comm: modprobe Tainted: G           OE      6.2.0-34-generic #34~22.04.1-Ubuntu
[ 7684.811918] Hardware name: ASUS System Product Name/TUF GAMING B550M-PLUS WIFI II, BIOS 3002 02/23/2023
[ 7684.811919] RIP: 0010:memcpy_fromio+0x7d/0xd0
[ 7684.811923] Code: ff e9 a7 18 83 00 48 85 db 74 e6 41 f6 c4 01 75 51 48 83 fb 01 76 06 41 f6 c4 02 75 31 48 89 d9 4c 89 ef 4c 89 e6 48 c1 e9 02 <f3> a5 f6 c3 02 74 02 66 a5 f6 c3 01 74 01 a4 5b 41 5c 41 5d 5d 31
[ 7684.811925] RSP: 0018:ffffaaa915a9f7e0 EFLAGS: 00000216
[ 7684.811927] RAX: 0000000000000000 RBX: 0000000000040000 RCX: 000000000000ff86
[ 7684.811928] RDX: 0000000000000000 RSI: ffffaaa903b001e8 RDI: ffff9355cd4001e8
[ 7684.811929] RBP: ffffaaa915a9f7f8 R08: 0000000000000000 R09: 0000000000000000
[ 7684.811931] R10: 0000000000000000 R11: 0000000000000000 R12: ffffaaa903b00000
[ 7684.811932] R13: ffff9355cd400000 R14: ffffaaa915a9f848 R15: ffff9355d1000000
[ 7684.811933] FS:  00007f47b5b5ac40(0000) GS:ffff93746f000000(0000) knlGS:0000000000000000
[ 7684.811934] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7684.811935] CR2: 00007fbed48e7240 CR3: 000000011645c000 CR4: 0000000000750ee0
[ 7684.811937] PKRU: 55555554
[ 7684.811938] Call Trace:
[ 7684.811939]  <IRQ>
[ 7684.811942]  ? show_regs+0x72/0x90
[ 7684.811945]  ? watchdog_timer_fn+0x1e7/0x250
[ 7684.811948]  ? __pfx_watchdog_timer_fn+0x10/0x10
[ 7684.811950]  ? __hrtimer_run_queues+0x107/0x280
[ 7684.811952]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.811957]  ? hrtimer_interrupt+0x101/0x250
[ 7684.811960]  ? __sysvec_apic_timer_interrupt+0x64/0x140
[ 7684.811964]  ? sysvec_apic_timer_interrupt+0x91/0xb0
[ 7684.811966]  </IRQ>
[ 7684.811967]  <TASK>
[ 7684.811968]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[ 7684.811973]  ? memcpy_fromio+0x7d/0xd0
[ 7684.811977]  igp_read_bios_from_vram+0x9e/0x100 [amdgpu]
[ 7684.812179]  amdgpu_get_bios+0x398/0x6e0 [amdgpu]
[ 7684.812315]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.812318]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.812320]  ? amdgpu_device_rreg+0x3a/0x50 [amdgpu]
[ 7684.812447]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.812449]  ? nbio_v7_4_get_rev_id+0xc2/0x130 [amdgpu]
[ 7684.812594]  amdgpu_device_init+0x12fb/0x2d40 [amdgpu]
[ 7684.812721]  ? pci_bus_read_config_word+0x4a/0x90
[ 7684.812725]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.812727]  ? do_pci_enable_device+0xe1/0x110
[ 7684.812731]  amdgpu_driver_load_kms+0x1a/0x200 [amdgpu]
[ 7684.812858]  amdgpu_pci_probe+0x1c0/0x4f0 [amdgpu]
[ 7684.812985]  local_pci_probe+0x4b/0xb0
[ 7684.812988]  pci_call_probe+0x55/0x190
[ 7684.812991]  pci_device_probe+0x84/0x120
[ 7684.812993]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.812996]  really_probe+0x1ed/0x450
[ 7684.812999]  __driver_probe_device+0x8a/0x190
[ 7684.813001]  driver_probe_device+0x23/0xd0
[ 7684.813003]  __driver_attach+0x10f/0x220
[ 7684.813005]  ? __pfx___driver_attach+0x10/0x10
[ 7684.813007]  bus_for_each_dev+0x83/0xe0
[ 7684.813010]  driver_attach+0x1e/0x30
[ 7684.813012]  bus_add_driver+0x152/0x250
[ 7684.813014]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.813017]  driver_register+0x83/0x160
[ 7684.813019]  ? __pfx_init_module+0x10/0x10 [amdgpu]
[ 7684.813144]  __pci_register_driver+0x68/0x80
[ 7684.813147]  amdgpu_init+0x9a/0xff0 [amdgpu]
[ 7684.813272]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.813274]  do_one_initcall+0x49/0x240
[ 7684.813279]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.813280]  ? kmalloc_trace+0x2a/0xb0
[ 7684.813284]  do_init_module+0x52/0x240
[ 7684.813287]  load_module+0xb96/0xd60
[ 7684.813289]  ? security_kernel_post_read_file+0x5c/0x80
[ 7684.813295]  __do_sys_finit_module+0xcc/0x150
[ 7684.813297]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.813299]  ? __do_sys_finit_module+0xcc/0x150
[ 7684.813304]  __x64_sys_finit_module+0x18/0x30
[ 7684.813306]  do_syscall_64+0x5c/0x90
[ 7684.813309]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.813311]  ? do_syscall_64+0x69/0x90
[ 7684.813312]  ? syscall_exit_to_user_mode+0x38/0x60
[ 7684.813315]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.813316]  ? do_syscall_64+0x69/0x90
[ 7684.813318]  ? exit_to_user_mode_prepare+0x3b/0xd0
[ 7684.813320]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.813322]  ? syscall_exit_to_user_mode+0x38/0x60
[ 7684.813324]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.813326]  ? do_syscall_64+0x69/0x90
[ 7684.813327]  ? srso_alias_return_thunk+0x5/0x7f
[ 7684.813329]  ? do_syscall_64+0x69/0x90
[ 7684.813331]  entry_SYSCALL_64_after_hwframe+0x73/0xdd
[ 7684.813333] RIP: 0033:0x7f47b531ea7d
[ 7684.813336] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 83 a3 0f 00 f7 d8 64 89 01 48
[ 7684.813337] RSP: 002b:00007ffe3803f008 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 7684.813339] RAX: ffffffffffffffda RBX: 000055b59d553cc0 RCX: 00007f47b531ea7d
[ 7684.813340] RDX: 0000000000000000 RSI: 000055b59c8a0cd2 RDI: 0000000000000012
[ 7684.813341] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000002
[ 7684.813343] R10: 0000000000000012 R11: 0000000000000246 R12: 000055b59c8a0cd2
[ 7684.813344] R13: 000055b59d54c580 R14: 000055b59d552500 R15: 000055b59d555bc0
[ 7684.813347]  </TASK>

After entering modprobe -v amdgpu to load the driver, it will crash at amdgpu.ko. If you need any information please let me know how to do it, it is very important for me to get this MI250X running, any help is greatly appreciated!

[srinivamd]

The firmware version 113-D65201-X06 seems not right. Could you please update the firmware/IFWI on the GPU? Please see: https://www.amd.com/en/support/server-accelerators/amd-instinct/amd-instinct-mi-series/amd-instinct-mi210

Could you also please collect the rocm_techsupport logs and post it? The rocm_techsupport instructions can be found here: https://github.com/amddcgpuce/rocmtechsupport

The rocm_techsupport script collects a bunch of info including dkms status. Could you post the output of dkms status here?

Recommended steps: Update firmware to the latest version using amdfwflash tool. Reboot. Retest. May need to reinstall the amdgpu driver packages but not sure. Will need rocm_techsupport logs to analyze.

[KegangWangCCNU]

Thank you for your suggestion. Regarding reinstalling the driver, I have tried many times on different versions of the system and Linux kernel, but the same error occurred. I tried updating vbios without success, I will provide you the logs.

When trying to update vbios, it fails.

linux log:

[ 1980.068232] x86/PAT: amdfwflash:28127 map pfn expected mapping type uncached-minus for [mem 0xca6c7000-0xca6c7fff], got write-back

Run rocm_techsupport, the following information about the MI250X is output:

0c:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran (rev 01)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 255
	IOMMU group: 18
	Region 0: Memory at 5000000000 (64-bit, prefetchable) [disabled] [size=64G]
	Region 2: Memory at 6000000000 (64-bit, prefetchable) [disabled] [size=256M]
	Region 4: I/O ports at f000 [disabled] [size=256]
	Region 5: Memory at fca00000 (32-bit, non-prefetchable) [disabled] [size=512K]
	Expansion ROM at fca80000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 32GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 32GT/s (ok), Width x16 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
			 10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Form Factor Dev Specific, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
			 AtomicOpsCtl: ReqEn-
		LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 32GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [200 v1] Physical Resizable BAR
		BAR 0: current size: 64GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB 64GB
		BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
	Capabilities: [240 v1] Power Budgeting <?>
	Capabilities: [250 v1] Dynamic Power Allocation <?>
	Capabilities: [270 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable+, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000100, Page Request Allocation: 00000000
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [410 v1] Physical Layer 16.0 GT/s <?>
	Capabilities: [450 v1] Lane Margining at the Receiver <?>
	Capabilities: [500 v1] Extended Capability ID 0x2a
	Kernel modules: amdgpu

Full log:
ww-server.23-10-13-15-19-45.rocm_techsupport.log

dkms status output:

Thanks for any help!

[arikamidotnet]

If you figured this out please let me know i have the same issue

[srinivamd]

@KegangWangCCNU Could you please check/update the system settings as follows and try updating the firmware again with amdfwflash? I think, it would be good to blacklist the amdgpu driver when updating the firmware -- please check amdfwflash user guide.
It should address the error amdfwflash:1897 map pfn expected mapping type uncached-minus for [mem 0xca6d6000-0xca6d6fff], got write-back from the amdfwflash tool, failing to update firmware.

SBIOS settings:

SRIOV - enabled
CSM –  enabled
Video - UEFI
ACPI PCI AER – Enabled
IOMMU– enabled
Above 4G Decoding - enabled

OS boot params:

Update `/etc/default/grub`

Add “amd_iommu=on iommu=pt” to GRUB_CMDLINE line in grub
GRUB_CMDLINE_LINUX="amd_iommu=pt"

run update-grub
reboot

[KegangWangCCNU]

Hi, I tried your suggestions and still get the same error. The message thrown by amdfwflash is not an error, it will throw this message when working normally.

I continued to search for relevant information and found that the mi250x on the ex235a has the same vbios version, so this is the vbios of the liquid-cooled mi250x. During the driver initialization, I found by tracing the stack that the driver could not read the bios, but I could extract the bios normally using amdfwflash, which showed that it was not a hardware failure.

The label in the lower right corner of this picture, D65201 corresponds to the vbios version. This label is exactly the same as my GPU.

This GPU comes from the ex235a node and is consistent with the version on frontier. Please refer to the following article.
https://www.servethehome.com/behold-the-amd-instinct-mi250x-oam-at-sc21/

I must admit, I am a master's student and lack a high-performance GPU, I purchased a second-hand GPU to replace the overly expensive A100. But I didn't expect that driver installation would be so difficult. The only possibility I speculate is that, unlike MI250, MI250X requires a customized version of the driver. If you know of any way to get it working, or anyone who might be able to help me out, I'd be grateful.

thanks again!

[srinivamd]

@KegangWangCCNU I just realized I made a mistake, misread the part number D65201 that you have. The information I provided was for D65210. Let me analyze the information and get back to you.

[ShuraWW]

Hi @KegangWangCCNU,

Did you ever try using PSPTool to analyze the firmware dump?

I'm wondering if the write protection checks the signature or just
compares against the existing firmware. If we could patch the dumped
firmware (e.g., modify device/subsystem IDs or disable the OEM lock)
and write it back, it might work?

Have you made any progress since then?

[Gracana]

@ShuraWW I might pick up one of these modules. FWIW, you can get the latest RMFW and IFWI for them from here, but I don't know if that helps to make them work on non-HP systems.

I tried unsuccessfully to inspect them with psptool.

[arikamidotnet]

I have the same issues and im using a supermicro OAM board with the same hpe module. this is the cheapest practical setup. if you find anymore out or have anything for me to test id love to try

[mgjaggers]

I have the same issues and im using a supermicro OAM board with the same hpe module. this is the cheapest practical setup. if you find anymore out or have anything for me to test id love to try

Do you have a discord I can message you about this?

[arikamidotnet]

little.cross1291@fastmail.com

you can email me

[LucienMP]

I would be interested to work on this with you guys together, sent email - i bring experience in hw design, and driver engineering/reverse engineering.

[LucienMP]

What systems and carrier base boards are you testing against?