Log entries report ConnectX4 is not rdma capable

[dcontiveros]

This appears to be a regression or an issue with multiple RDMA capable NICs. Here is my setup:

# lspci output
03:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
04:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function];

# ksmbd entries after starting

Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(enp3s0) rdma capable : true
Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(enp3s0) rdma capable : true
Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(ibp4s0) rdma capable : false
Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(ibp4s0) rdma capable : false
Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(vlan10) rdma capable : false
Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(vlan10) rdma capable : false   

The ConnectX4 NIC is running in IB only mode. I have compiled 6.16 manually. Here are all IB options:

CONFIG_INFINIBAND=y
CONFIG_INFINIBAND_USER_MAD=y
CONFIG_INFINIBAND_USER_ACCESS=y
CONFIG_INFINIBAND_USER_MEM=y
CONFIG_INFINIBAND_ON_DEMAND_PAGING=y
CONFIG_INFINIBAND_ADDR_TRANS=y
CONFIG_INFINIBAND_ADDR_TRANS_CONFIGFS=y
CONFIG_INFINIBAND_VIRT_DMA=y
# CONFIG_INFINIBAND_BNXT_RE is not set
# CONFIG_INFINIBAND_CXGB4 is not set
# CONFIG_INFINIBAND_EFA is not set
# CONFIG_INFINIBAND_ERDMA is not set
# CONFIG_INFINIBAND_IRDMA is not set
# CONFIG_MANA_INFINIBAND is not set
# CONFIG_MLX4_INFINIBAND is not set
CONFIG_MLX5_INFINIBAND=m
# CONFIG_INFINIBAND_MTHCA is not set
# CONFIG_INFINIBAND_OCRDMA is not set
# CONFIG_INFINIBAND_QEDR is not set
# CONFIG_INFINIBAND_USNIC is not set
# CONFIG_INFINIBAND_VMWARE_PVRDMA is not set
# CONFIG_INFINIBAND_RDMAVT is not set
CONFIG_INFINIBAND_IPOIB=y
CONFIG_INFINIBAND_IPOIB_CM=y
CONFIG_INFINIBAND_IPOIB_DEBUG=y
# CONFIG_INFINIBAND_IPOIB_DEBUG_DATA is not set
CONFIG_INFINIBAND_SRP=y
# CONFIG_INFINIBAND_SRPT is not set
CONFIG_INFINIBAND_ISER=y
# CONFIG_INFINIBAND_ISERT is not set
# CONFIG_INFINIBAND_RTRS_CLIENT is not set
# CONFIG_INFINIBAND_RTRS_SERVER is not set
# CONFIG_INFINIBAND_OPA_VNIC is not set
CONFIG_SECURITY_INFINIBAND=y`

Fortunately I had kernel 6.13 lying around (been a while since I touched this), and ibp4s0 works. Windows output is as follows:

# With 6.13
> Get-SmbMultichannelConnection

Server Name Selected Client IP   Server IP   Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
----------- -------- ---------   ---------   ---------------------- ---------------------- ------------------ -------------------
192.168.0.1 True     192.168.0.2 192.168.0.1 5                      3                      False              True

# With 6.16
Server Name Selected Client IP   Server IP   Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
----------- -------- ---------   ---------   ---------------------- ---------------------- ------------------ -------------------
192.168.0.1 True     192.168.0.2 192.168.0.1 5                      3                      True               False

Any help is appreciated.

[namjaejeon]

Can you check the following patch cause this issue in linux-6.16 ?

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/fs/smb/server?h=v6.16&id=1a81ea738c030dfa83929a954e8c2cd15a91a883

Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(enp3s0) rdma capable : true
Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(enp3s0) rdma capable : true
Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(ibp4s0) rdma capable : false
Aug 07 00:58:44 HOSTNAME_REDACTED kernel: ksmbd: smb_direct: netdev(ibp4s0) rdma capable : false

enp3s0 is a RDMA NIC ? Why is "rdma capable" set to true on it?

[dcontiveros]

Will test this patch today. enp3s0 is a virtual function of a ConnectX4-LX. I am interested in RDMA of the ConenctX4 VF though.

[dcontiveros]

Interesting I did:

$ cd GIT_REPO

$ git branch
* (HEAD detached at v6.16)
  master

$` git revert 1a81ea738c030dfa83929a954e8c2cd15a91a883

*fixed conflicts*
*compiled kernel*

This did not resolve the issue. I am still seeing ibp4s0 as rdma capable false.

[dcontiveros]

Apologies for the mistake. Reverting that commit did actually work:

Aug 08 00:44:58 REDACTED_HOSTNAME kernel: ksmbd: credits: requested[1] granted[1] total_granted[33]
Aug 08 00:44:58 REDACTED_HOSTNAME kernel: ksmbd: smb_direct: netdev(enp3s0) rdma capable : true
Aug 08 00:44:58 REDACTED_HOSTNAME kernel: ksmbd: smb_direct: netdev(enp3s0) rdma capable : true
Aug 08 00:44:58 REDACTED_HOSTNAME kernel: ksmbd: smb_direct: netdev(ibp4s0) rdma capable : true
Aug 08 00:44:58 REDACTED_HOSTNAME kernel: ksmbd: smb_direct: netdev(ibp4s0) rdma capable : true
Aug 08 00:44:58 REDACTED_HOSTNAME kernel: ksmbd: smb_direct: netdev(vlan10) rdma capable : false
Aug 08 00:44:58 REDACTED_HOSTNAME kernel: ksmbd: smb_direct: netdev(vlan10) rdma capable : false  

# kernel output
$ uname -r
6.16.0-dirty

# Windows verification
PS C:\Users\dontiveros> Get-SmbMultichannelConnection

Server Name Selected Client IP   Server IP   Client Interface Index Server Interface Index Client RSS Capable Client RDMA Capable
----------- -------- ---------   ---------   ---------------------- ---------------------- ------------------ -------------------
192.168.0.1 True     192.168.0.2 192.168.0.1 5                      3                      False              True

[namjaejeon]

Sorry, I don't have LX/VF ConnectX device. I may have limitations in solving this problem.

If you use the following function, Can ibp4s0 be judged as an rdma capable device?

bool ksmbd_rdma_capable_netdev(struct net_device *netdev)
{
    struct smb_direct_device *smb_dev;
    int i;
    bool rdma_capable = false;

    read_lock(&smb_direct_device_lock);
    list_for_each_entry(smb_dev, &smb_direct_device_list, list) {
        for (i = 0; i < smb_dev->ib_dev->phys_port_cnt; i++) {
            struct net_device *ndev;

            ndev = ib_device_get_netdev(smb_dev->ib_dev, i + 1);
            if (!ndev)
                continue;

            if (ndev == netdev ||
                netdev->ifindex == ndev->ifindex ||
                strcmp(netdev->name, ndev->name) == 0) {
                dev_put(ndev);
                rdma_capable = true;
                goto out;
            }
            dev_put(ndev);
        }
    }
out:
    read_unlock(&smb_direct_device_lock);

    if (rdma_capable == false) {
        struct ib_device *ibdev;

        ibdev = ib_device_get_by_netdev(netdev, RDMA_DRIVER_UNKNOWN);
        if (ibdev) {
            rdma_capable = rdma_frwr_is_supported(&ibdev->attrs);
            ib_device_put(ibdev);
        }
    }

    ksmbd_debug(RDMA, "netdev(%s) rdma capable : %s\n",
                netdev->name, str_true_false(rdma_capable));

    return rdma_capable;
}

[namjaejeon]

@dcontiveros Can you check if the following branch fix your issue ?

git clone https://github.com/namjaejeon/ksmbd --branch=next

[dcontiveros]

Apologies for the delay in responding to this matter. I was in the process of setting this up again on the following:

# lspci | grep -i mel
03:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function]
04:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4 Virtual Function]

# ibstat
CA 'mlx5_0'
        CA type: MT4118
        Number of ports: 1
        Firmware version: 14.32.1900
        Hardware version: 0
        Node GUID: 0xdeadfafffede0000
        System image GUID: 0x0c42a10300e47ba4
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x00010000
                Port GUID: 0xdcadfafffede0000
                Link layer: Ethernet
CA 'mlx5_1'
        CA type: MT4116
        Number of ports: 1
        Firmware version: 12.28.2302
        Hardware version: 0
        Node GUID: 0xbabefacade000000
        System image GUID: 0x248a070300b6940a
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 100
                Base lid: 6
                LMC: 0
                SM lid: 1
                Capability mask: 0x2659ec48
                Port GUID: 0xcafefacade000000
                Link layer: InfiniBand

# journalctl -f | grep -i rdma
Mar 09 19:26:32 HOSTNAME_REDACTED  kernel: ksmbd: smb_direct: init RDMA listener. cm_id=0000000088f9ad24
Mar 09 19:26:39 HOSTNAME_REDACTED  kernel: ksmbd: smb_direct: netdev(ibp4s0) rdma capable : false
Mar 09 19:26:39 HOSTNAME_REDACTED  kernel: ksmbd: smb_direct: netdev(ibp4s0) rdma capable : false

# uname -a
Linux HOSTNAME_REDACTED  6.19.6 #1 SMP PREEMPT_DYNAMIC Mon Mar  9 17:35:15 EDT 2026 x86_64 GNU/Linux

# cat /etc/redhat-release
AlmaLinux release 10.1 (Heliotrope Lion)

This is a custom compiled kernel with config taken from a clean install on AlmaLinux10. So I am still experiencing the same regression. Is the above function still relevant? Can test this week for sure.

[dcontiveros]

This is not an issue where one NIC takes precedence over another. I disabled RoCE on my ConnectX4-LX VF. ibstat only shows my ConnectX4, yet when I start the ksmbd server, I get the rdma capable: false message.