ICX intermittent failures: command timeout triggered, timeout value is 45 secs

[yamjoepobuda]

SUMMARY

Intermittently, across multiple switches (using multiple firmware versions, all > 8090/UFI), we run into the following error:

failed: [switch] (item=192.168.1.10) => {"ansible_loop_var": "item", "changed": false, "item": "192.168.1.10", "msg": "command timeout triggered, timeout value is 45 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."}

In recent testing with firmware versions 8095j, we see this issue a lot less. I can run this play 10 times in a row, and it might succeed 80% of the time some days, 20% of the time other days.

ISSUE TYPE

• Bug Report

COMPONENT NAME

icx_config
icx_ping

ANSIBLE VERSION

15:27:52 + ansible --version
15:27:52 ansible [core 2.15.1]
15:27:52   config file = /etc/ansible/ansible-nocache.cfg
15:27:52   configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
15:27:52   ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
15:27:52   ansible collection location = /usr/share/ansible/collections
15:27:52   executable location = /usr/local/bin/ansible
15:27:52   python version = 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] (/usr/bin/python3)
15:27:52   jinja version = 3.1.2
15:27:52   libyaml = False

COLLECTION VERSION

15:27:52 + ansible-galaxy collection list
15:27:53 
15:27:53 # /usr/local/lib/python3.10/dist-packages/ansible_collections
15:27:53 Collection                    Version
15:27:53 ----------------------------- -------
15:27:53 amazon.aws                    6.1.0  
15:27:53 ansible.netcommon             5.1.1  
15:27:53 ansible.posix                 1.5.4  
15:27:53 ansible.utils                 2.10.3 
15:27:53 ansible.windows               1.14.0 
15:27:53 arista.eos                    6.0.1  
15:27:53 awx.awx                       22.3.0 
15:27:53 azure.azcollection            1.16.0 
15:27:53 check_point.mgmt              5.1.1  
15:27:53 chocolatey.chocolatey         1.4.0  
15:27:53 cisco.aci                     2.6.0  
15:27:53 cisco.asa                     4.0.1  
15:27:53 cisco.dnac                    6.7.2  
15:27:53 cisco.intersight              1.0.27 
15:27:53 cisco.ios                     4.6.1  
15:27:53 cisco.iosxr                   5.0.3  
15:27:53 cisco.ise                     2.5.12 
15:27:53 cisco.meraki                  2.15.1 
15:27:53 cisco.mso                     2.4.0  
15:27:53 cisco.nso                     1.0.3  
15:27:53 cisco.nxos                    4.4.0  
15:27:53 cisco.ucs                     1.8.0  
15:27:53 cloud.common                  2.1.3  
15:27:53 cloudscale_ch.cloud           2.3.1  
15:27:53 community.aws                 6.0.0  
15:27:53 community.azure               2.0.0  
15:27:53 community.ciscosmb            1.0.6  
15:27:53 community.crypto              2.14.0 
15:27:53 community.digitalocean        1.23.0 
15:27:53 community.dns                 2.5.6  
15:27:53 community.docker              3.4.8  
15:27:53 community.fortios             1.0.0  
15:27:53 community.general             7.1.0  
15:27:53 community.google              1.0.0  
15:27:53 community.grafana             1.5.4  
15:27:53 community.hashi_vault         5.0.0  
15:27:53 community.hrobot              1.8.0  
15:27:53 community.libvirt             1.2.0  
15:27:53 community.mongodb             1.6.0  
15:27:53 community.mysql               3.7.2  
15:27:53 community.network             5.0.0  
15:27:53 community.okd                 2.3.0  
15:27:53 community.postgresql          2.4.2  
15:27:53 community.proxysql            1.5.1  
15:27:53 community.rabbitmq            1.2.3  
15:27:53 community.routeros            2.8.2  
15:27:53 community.sap                 1.0.0  
15:27:53 community.sap_libs            1.4.1  
15:27:53 community.skydive             1.0.0  
15:27:53 community.sops                1.6.2  
15:27:53 community.vmware              3.7.0  
15:27:53 community.windows             1.13.0 
15:27:53 community.zabbix              2.0.1  
15:27:53 containers.podman             1.10.2 
15:27:53 cyberark.conjur               1.2.0  
15:27:53 cyberark.pas                  1.0.19 
15:27:53 dellemc.enterprise_sonic      2.2.0  
15:27:53 dellemc.openmanage            7.6.1  
15:27:53 dellemc.powerflex             1.6.0  
15:27:53 dellemc.unity                 1.6.0  
15:27:53 f5networks.f5_modules         1.25.0 
15:27:53 fortinet.fortimanager         2.2.0  
15:27:53 fortinet.fortios              2.3.0  
15:27:53 frr.frr                       2.0.2  
15:27:53 gluster.gluster               1.0.2  
15:27:53 google.cloud                  1.1.3  
15:27:53 grafana.grafana               2.0.0  
15:27:53 hetzner.hcloud                1.11.0 
15:27:53 hpe.nimble                    1.1.4  
15:27:53 ibm.qradar                    2.1.0  
15:27:53 ibm.spectrum_virtualize       1.12.0 
15:27:53 infinidat.infinibox           1.3.12 
15:27:53 infoblox.nios_modules         1.5.0  
15:27:53 inspur.ispim                  1.3.0  
15:27:53 inspur.sm                     2.3.0  
15:27:53 junipernetworks.junos         5.1.0  
15:27:53 kubernetes.core               2.4.0  
15:27:53 lowlydba.sqlserver            2.0.0  
15:27:53 microsoft.ad                  1.2.0  
15:27:53 netapp.aws                    21.7.0 
15:27:53 netapp.azure                  21.10.0
15:27:53 netapp.cloudmanager           21.22.0
15:27:53 netapp.elementsw              21.7.0 
15:27:53 netapp.ontap                  22.7.0 
15:27:53 netapp.storagegrid            21.11.1
15:27:53 netapp.um_info                21.8.0 
15:27:53 netapp_eseries.santricity     1.4.0  
15:27:53 netbox.netbox                 3.13.0 
15:27:53 ngine_io.cloudstack           2.3.0  
15:27:53 ngine_io.exoscale             1.0.0  
15:27:53 ngine_io.vultr                1.1.3  
15:27:53 openstack.cloud               2.1.0  
15:27:53 openvswitch.openvswitch       2.1.1  
15:27:53 ovirt.ovirt                   3.1.2  
15:27:53 purestorage.flasharray        1.19.1 
15:27:53 purestorage.flashblade        1.11.0 
15:27:53 purestorage.fusion            1.5.0  
15:27:53 sensu.sensu_go                1.13.2 
15:27:53 servicenow.servicenow         1.0.6  
15:27:53 splunk.es                     2.1.0  
15:27:53 t_systems_mms.icinga_director 1.33.1 
15:27:53 theforeman.foreman            3.11.0 
15:27:53 vmware.vmware_rest            2.3.1  
15:27:53 vultr.cloud                   1.8.0  
15:27:53 vyos.vyos                     4.1.0  
15:27:53 wti.remote                    1.0.5  
15:27:53 
15:27:53 # /usr/share/ansible/collections/ansible_collections
15:27:53 Collection                    Version
15:27:53 ----------------------------- -------
15:27:53 amazon.aws                    6.1.0  
15:27:53 ansible.netcommon             5.1.1  
15:27:53 ansible.posix                 1.5.4  
15:27:53 ansible.utils                 2.10.3 
15:27:53 ansible.windows               2.0.0  
15:27:53 chocolatey.chocolatey         1.5.0  
15:27:53 cisco.ios                     4.6.1  
15:27:53 cisco.meraki                  2.15.1 
15:27:53 commscope.icx                 1.0.5  
15:27:53 community.aws                 6.0.0  
15:27:53 community.crypto              2.14.1 
15:27:53 community.docker              3.4.8  
15:27:53 community.general             7.1.0  
15:27:53 community.network             5.0.0  
15:27:53 community.windows             2.0.0  
15:27:53 fortinet.fortios              2.3.0 

CONFIGURATION

15:27:52 + ansible-config dump --only-changed
15:27:52 ANSIBLE_NOCOWS(/etc/ansible/ansible-nocache.cfg) = True
15:27:52 ANSIBLE_PIPELINING(/etc/ansible/ansible-nocache.cfg) = True
15:27:52 COLLECTIONS_PATHS(/etc/ansible/ansible-nocache.cfg) = ['/usr/share/ansible/collections']
15:27:52 CONFIG_FILE() = /etc/ansible/ansible-nocache.cfg
15:27:52 DEFAULT_FORKS(/etc/ansible/ansible-nocache.cfg) = 50
15:27:52 DEFAULT_POLL_INTERVAL(/etc/ansible/ansible-nocache.cfg) = 5
15:27:52 DEFAULT_REMOTE_USER(/etc/ansible/ansible-nocache.cfg) = ansible
15:27:52 DEFAULT_ROLES_PATH(/etc/ansible/ansible-nocache.cfg) = ['/etc/ansible/roles']
15:27:52 DEFAULT_TIMEOUT(/etc/ansible/ansible-nocache.cfg) = 20
15:27:52 DISPLAY_SKIPPED_HOSTS(/etc/ansible/ansible-nocache.cfg) = False
15:27:52 HOST_KEY_CHECKING(/etc/ansible/ansible-nocache.cfg) = False
15:27:52 PERSISTENT_COMMAND_TIMEOUT(/etc/ansible/ansible-nocache.cfg) = 30
15:27:52 PERSISTENT_CONNECT_TIMEOUT(/etc/ansible/ansible-nocache.cfg) = 40
15:27:52 RETRY_FILES_ENABLED(/etc/ansible/ansible-nocache.cfg) = False

OS / ENVIRONMENT

Switch

SSH@switch#sh ver
  Copyright (c) Ruckus Networks, Inc. All rights reserved.
    UNIT 1: compiled on Apr 11 2023 at 00:06:40 labeled as SPR08095j
      (33554432 bytes) from Primary SPR08095j.bin (UFI)
        SW: Version 08.0.95jT213
      Compressed Primary Boot Code size = 786944, Version:10.1.26T215 (spz10126)
       Compiled on Tue Nov 29 05:43:15 2022

  HW: Stackable ICX7450-48F
  Internal USB: Serial #: 99013190710xxxxx
      Vendor: ATP Electronics, Total size = 1919 MB
==========================================================================
UNIT 1: SL 1: ICX7450-48F 48-port Management Module
      Serial  #:CYS330xxxxx
      Software Package: ICX7450_L3_SOFT_PACKAGE   (LID: eauIIFHtFGH)
      Current License: l3-prem
      P-ASIC  0: type B548, rev 01  Chip BCM56548_A0
==========================================================================
UNIT 1: SL 2: ICX7400-4X10GF 4-port 40G Module
      Serial  #:CYV333xxxxx
==========================================================================
 1000 MHz ARM processor ARMv7 88 MHz bus
    8 MB boot flash memory
    2 GB code flash memory
    2 GB DRAM
STACKID 1  system uptime is 4 day(s) 16 hour(s) 2 minute(s) 52 second(s)
The system started at 21:01:09 Arizona Fri Aug 04 2023

Kernel

15:27:53 + uname -a
15:27:53 Linux 070822a60e46 5.19.0-1028-aws #29~22.04.1-Ubuntu SMP Tue Jun 20 19:12:11 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

OS

15:27:53 + cat /etc/os-release
15:27:53 PRETTY_NAME="Ubuntu 22.04.2 LTS"
15:27:53 NAME="Ubuntu"
15:27:53 VERSION_ID="22.04"
15:27:53 VERSION="22.04.2 LTS (Jammy Jellyfish)"
15:27:53 VERSION_CODENAME=jammy
15:27:53 ID=ubuntu
15:27:53 ID_LIKE=debian
15:27:53 HOME_URL="https://www.ubuntu.com/"
15:27:53 SUPPORT_URL="https://help.ubuntu.com/"
15:27:53 BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
15:27:53 PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
15:27:53 UBUNTU_CODENAME=jammy

apt

15:27:53 + apt list --installed
15:27:53 
15:27:53 WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
15:27:53 
15:27:53 Listing...
15:27:53 adduser/jammy,now 3.118ubuntu5 all [installed]
15:27:53 apt/jammy-updates,now 2.4.9 amd64 [installed]
15:27:53 base-files/jammy-updates,now 12ubuntu4.3 amd64 [installed]
15:27:53 base-passwd/jammy,now 3.5.52build1 amd64 [installed]
15:27:53 bash/jammy,now 5.1-6ubuntu1 amd64 [installed]
15:27:53 binutils-common/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 binutils-x86-64-linux-gnu/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 binutils/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 bsdutils/jammy,now 1:2.37.2-4ubuntu3 amd64 [installed]
15:27:53 ca-certificates/jammy-updates,jammy-security,now 20230311ubuntu0.22.04.1 all [installed,automatic]
15:27:53 coreutils/jammy,now 8.32-4.1ubuntu1 amd64 [installed]
15:27:53 cpp-11/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 cpp/jammy,now 4:11.2.0-1ubuntu1 amd64 [installed,automatic]
15:27:53 curl/jammy-updates,jammy-security,now 7.81.0-1ubuntu1.10 amd64 [installed]
15:27:53 dash/jammy,now 0.5.11+git20210903+057cd650a4ed-3build1 amd64 [installed]
15:27:53 debconf/jammy,now 1.5.79ubuntu1 all [installed]
15:27:53 debianutils/jammy,now 5.5-1ubuntu2 amd64 [installed]
15:27:53 diffutils/jammy,now 1:3.8-0ubuntu2 amd64 [installed]
15:27:53 dpkg/jammy-updates,now 1.21.1ubuntu2.2 amd64 [installed]
15:27:53 e2fsprogs/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 findutils/jammy,now 4.8.0-1ubuntu3 amd64 [installed]
15:27:53 gcc-11-base/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 gcc-11/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 gcc-12-base/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed]
15:27:53 gcc/jammy,now 4:11.2.0-1ubuntu1 amd64 [installed]
15:27:53 git-man/jammy-updates,jammy-security,now 1:2.34.1-1ubuntu1.9 all [installed,automatic]
15:27:53 git/jammy-updates,jammy-security,now 1:2.34.1-1ubuntu1.9 amd64 [installed]
15:27:53 gpgv/jammy-updates,jammy-security,now 2.2.27-3ubuntu2.1 amd64 [installed]
15:27:53 grep/jammy,now 3.7-1build1 amd64 [installed]
15:27:53 gzip/jammy-updates,now 1.10-4ubuntu4.1 amd64 [installed]
15:27:53 hostname/jammy,now 3.23ubuntu2 amd64 [installed]
15:27:53 init-system-helpers/jammy,now 1.62 all [installed]
15:27:53 iputils-ping/jammy,now 3:20211215-1 amd64 [installed]
15:27:53 libacl1/jammy,now 2.3.1-1 amd64 [installed]
15:27:53 libapt-pkg6.0/jammy-updates,now 2.4.9 amd64 [installed]
15:27:53 libasan6/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 libatomic1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libattr1/jammy,now 1:2.5.1-1build1 amd64 [installed]
15:27:53 libaudit-common/jammy,now 1:3.0.7-1build1 all [installed]
15:27:53 libaudit1/jammy,now 1:3.0.7-1build1 amd64 [installed]
15:27:53 libbinutils/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 libblkid1/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 libbrotli1/jammy,now 1.0.9-2build6 amd64 [installed,automatic]
15:27:53 libbsd0/jammy,now 0.11.5-1 amd64 [installed,automatic]
15:27:53 libbz2-1.0/jammy,now 1.0.8-5build1 amd64 [installed]
15:27:53 libc-bin/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed]
15:27:53 libc-dev-bin/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed,automatic]
15:27:53 libc6-dev/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed,automatic]
15:27:53 libc6/jammy-updates,now 2.35-0ubuntu3.1 amd64 [installed]
15:27:53 libcap-ng0/jammy,now 0.7.9-2.2build3 amd64 [installed]
15:27:53 libcap2-bin/jammy-updates,jammy-security,now 1:2.44-1ubuntu0.22.04.1 amd64 [installed,automatic]
15:27:53 libcap2/jammy-updates,jammy-security,now 1:2.44-1ubuntu0.22.04.1 amd64 [installed]
15:27:53 libcbor0.8/jammy,now 0.8.0-2ubuntu1 amd64 [installed,automatic]
15:27:53 libcc1-0/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libcom-err2/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 libcrypt-dev/jammy,now 1:4.4.27-1 amd64 [installed,automatic]
15:27:53 libcrypt1/jammy,now 1:4.4.27-1 amd64 [installed]
15:27:53 libctf-nobfd0/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 libctf0/jammy-updates,jammy-security,now 2.38-4ubuntu2.2 amd64 [installed,automatic]
15:27:53 libcurl3-gnutls/jammy-updates,jammy-security,now 7.81.0-1ubuntu1.10 amd64 [installed,automatic]
15:27:53 libcurl4/jammy-updates,jammy-security,now 7.81.0-1ubuntu1.10 amd64 [installed,automatic]
15:27:53 libdb5.3/jammy,now 5.3.28+dfsg1-0.8ubuntu3 amd64 [installed]
15:27:53 libdebconfclient0/jammy,now 0.261ubuntu1 amd64 [installed]
15:27:53 libedit2/jammy,now 3.1-20210910-1build1 amd64 [installed,automatic]
15:27:53 liberror-perl/jammy,now 0.17029-1 all [installed,automatic]
15:27:53 libexpat1-dev/jammy-updates,jammy-security,now 2.4.7-1ubuntu0.2 amd64 [installed,automatic]
15:27:53 libexpat1/jammy-updates,jammy-security,now 2.4.7-1ubuntu0.2 amd64 [installed,automatic]
15:27:53 libext2fs2/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 libffi8/jammy,now 3.4.2-4 amd64 [installed]
15:27:53 libfido2-1/jammy,now 1.10.0-1 amd64 [installed,automatic]
15:27:53 libgcc-11-dev/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 libgcc-s1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed]
15:27:53 libgcrypt20/jammy,now 1.9.4-3ubuntu3 amd64 [installed]
15:27:53 libgdbm-compat4/jammy,now 1.23-1 amd64 [installed,automatic]
15:27:53 libgdbm6/jammy,now 1.23-1 amd64 [installed,automatic]
15:27:53 libgmp10/jammy,now 2:6.2.1+dfsg-3ubuntu1 amd64 [installed]
15:27:53 libgnutls30/jammy-updates,jammy-security,now 3.7.3-4ubuntu1.2 amd64 [installed]
15:27:53 libgomp1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libgpg-error0/jammy,now 1.43-3 amd64 [installed]
15:27:53 libgpm2/jammy,now 1.20.7-10build1 amd64 [installed,automatic]
15:27:53 libgssapi-krb5-2/jammy-updates,now 1.19.2-2ubuntu0.2 amd64 [installed]
15:27:53 libhogweed6/jammy,now 3.7.3-1build2 amd64 [installed]
15:27:53 libidn2-0/jammy,now 2.3.2-2build1 amd64 [installed]
15:27:53 libisl23/jammy,now 0.24-2build1 amd64 [installed,automatic]
15:27:53 libitm1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libjs-jquery/jammy,now 3.6.0+dfsg+~3.5.13-1 all [installed,automatic]
15:27:53 libjs-sphinxdoc/jammy,now 4.3.2-1 all [installed,automatic]
15:27:53 libjs-underscore/jammy,now 1.13.2~dfsg-2 all [installed,automatic]
15:27:53 libk5crypto3/jammy-updates,now 1.19.2-2ubuntu0.2 amd64 [installed]
15:27:53 libkeyutils1/jammy,now 1.6.1-2ubuntu3 amd64 [installed]
15:27:53 libkrb5-3/jammy-updates,now 1.19.2-2ubuntu0.2 amd64 [installed]
15:27:53 libkrb5support0/jammy-updates,now 1.19.2-2ubuntu0.2 amd64 [installed]
15:27:53 libldap-2.5-0/jammy-updates,now 2.5.14+dfsg-0ubuntu0.22.04.2 amd64 [installed,automatic]
15:27:53 liblsan0/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 liblz4-1/jammy,now 1.9.3-2build2 amd64 [installed]
15:27:53 liblzma5/jammy,now 5.2.5-2ubuntu1 amd64 [installed]
15:27:53 libmd0/jammy,now 1.0.4-1build1 amd64 [installed,automatic]
15:27:53 libmount1/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 libmpc3/jammy,now 1.2.1-2build1 amd64 [installed,automatic]
15:27:53 libmpdec3/jammy,now 2.5.1-2build2 amd64 [installed,automatic]
15:27:53 libmpfr6/jammy,now 4.1.0-3build3 amd64 [installed,automatic]
15:27:53 libncurses6/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 amd64 [installed]
15:27:53 libncursesw6/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 amd64 [installed]
15:27:53 libnettle8/jammy,now 3.7.3-1build2 amd64 [installed]
15:27:53 libnghttp2-14/jammy,now 1.43.0-1build3 amd64 [installed,automatic]
15:27:53 libnsl-dev/jammy,now 1.3.0-2build2 amd64 [installed,automatic]
15:27:53 libnsl2/jammy,now 1.3.0-2build2 amd64 [installed]
15:27:53 libp11-kit0/jammy,now 0.24.0-6build1 amd64 [installed]
15:27:53 libpam-modules-bin/jammy-updates,jammy-security,now 1.4.0-11ubuntu2.3 amd64 [installed]
15:27:53 libpam-modules/jammy-updates,jammy-security,now 1.4.0-11ubuntu2.3 amd64 [installed]
15:27:53 libpam-runtime/jammy-updates,jammy-security,now 1.4.0-11ubuntu2.3 all [installed]
15:27:53 libpam0g/jammy-updates,jammy-security,now 1.4.0-11ubuntu2.3 amd64 [installed]
15:27:53 libpcre2-8-0/jammy-updates,jammy-security,now 10.39-3ubuntu0.1 amd64 [installed]
15:27:53 libpcre3/jammy-updates,jammy-security,now 2:8.39-13ubuntu0.22.04.1 amd64 [installed]
15:27:53 libperl5.34/jammy-updates,jammy-security,now 5.34.0-3ubuntu1.2 amd64 [installed,automatic]
15:27:53 libprocps8/jammy,now 2:3.3.17-6ubuntu2 amd64 [installed]
15:27:53 libpsl5/jammy,now 0.21.0-1.2build2 amd64 [installed,automatic]
15:27:53 libpython3-dev/jammy-updates,now 3.10.6-1~22.04 amd64 [installed,automatic]
15:27:53 libpython3-stdlib/jammy-updates,now 3.10.6-1~22.04 amd64 [installed,automatic]
15:27:53 libpython3.10-dev/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 libpython3.10-minimal/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 libpython3.10-stdlib/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 libpython3.10/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 libquadmath0/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libreadline8/jammy,now 8.1.2-1 amd64 [installed,automatic]
15:27:53 librtmp1/jammy,now 2.4+20151223.gitfa8646d.1-2build4 amd64 [installed,automatic]
15:27:53 libsasl2-2/jammy-updates,now 2.1.27+dfsg2-3ubuntu1.2 amd64 [installed,automatic]
15:27:53 libsasl2-modules-db/jammy-updates,now 2.1.27+dfsg2-3ubuntu1.2 amd64 [installed,automatic]
15:27:53 libseccomp2/jammy,now 2.5.3-2ubuntu2 amd64 [installed]
15:27:53 libselinux1/jammy,now 3.3-1build2 amd64 [installed]
15:27:53 libsemanage-common/jammy,now 3.3-1build2 all [installed]
15:27:53 libsemanage2/jammy,now 3.3-1build2 amd64 [installed]
15:27:53 libsepol2/jammy,now 3.3-1build1 amd64 [installed]
15:27:53 libsmartcols1/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 libsodium23/jammy,now 1.0.18-1build2 amd64 [installed,automatic]
15:27:53 libsqlite3-0/jammy-updates,jammy-security,now 3.37.2-2ubuntu0.1 amd64 [installed,automatic]
15:27:53 libss2/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 libssh-4/jammy-updates,jammy-security,now 0.9.6-2ubuntu0.22.04.1 amd64 [installed,automatic]
15:27:53 libssl-dev/jammy-updates,jammy-security,now 3.0.2-0ubuntu1.10 amd64 [installed]
15:27:53 libssl3/jammy-updates,jammy-security,now 3.0.2-0ubuntu1.10 amd64 [installed]
15:27:53 libstdc++6/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed]
15:27:53 libsystemd0/jammy-updates,now 249.11-0ubuntu3.9 amd64 [installed]
15:27:53 libtasn1-6/jammy,now 4.18.0-4build1 amd64 [installed]
15:27:53 libtinfo6/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 amd64 [installed]
15:27:53 libtirpc-common/jammy-updates,jammy-security,now 1.3.2-2ubuntu0.1 all [installed]
15:27:53 libtirpc-dev/jammy-updates,jammy-security,now 1.3.2-2ubuntu0.1 amd64 [installed,automatic]
15:27:53 libtirpc3/jammy-updates,jammy-security,now 1.3.2-2ubuntu0.1 amd64 [installed]
15:27:53 libtsan0/jammy-updates,jammy-security,now 11.3.0-1ubuntu1~22.04.1 amd64 [installed,automatic]
15:27:53 libubsan1/jammy-updates,jammy-security,now 12.1.0-2ubuntu1~22.04 amd64 [installed,automatic]
15:27:53 libudev1/jammy-updates,now 249.11-0ubuntu3.9 amd64 [installed]
15:27:53 libunistring2/jammy,now 1.0-1 amd64 [installed]
15:27:53 libuuid1/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 libxxhash0/jammy,now 0.8.1-1 amd64 [installed]
15:27:53 libzstd1/jammy,now 1.4.8+dfsg-3build1 amd64 [installed]
15:27:53 linux-libc-dev/jammy-updates,jammy-security,now 5.15.0-76.83 amd64 [installed,automatic]
15:27:53 login/jammy-updates,jammy-security,now 1:4.8.1-2ubuntu2.1 amd64 [installed]
15:27:53 logsave/jammy-updates,jammy-security,now 1.46.5-2ubuntu1.1 amd64 [installed]
15:27:53 lsb-base/jammy,now 11.1.0ubuntu4 all [installed]
15:27:53 mawk/jammy,now 1.3.4.20200120-3 amd64 [installed]
15:27:53 media-types/jammy,now 7.0.0 all [installed,automatic]
15:27:53 mount/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 ncurses-base/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 all [installed]
15:27:53 ncurses-bin/jammy-updates,jammy-security,now 6.3-2ubuntu0.1 amd64 [installed]
15:27:53 openssh-client/jammy-updates,now 1:8.9p1-3ubuntu0.1 amd64 [installed]
15:27:53 openssl/jammy-updates,jammy-security,now 3.0.2-0ubuntu1.10 amd64 [installed,automatic]
15:27:53 passwd/jammy-updates,jammy-security,now 1:4.8.1-2ubuntu2.1 amd64 [installed]
15:27:53 perl-base/jammy-updates,jammy-security,now 5.34.0-3ubuntu1.2 amd64 [installed]
15:27:53 perl-modules-5.34/jammy-updates,jammy-security,now 5.34.0-3ubuntu1.2 all [installed,automatic]
15:27:53 perl/jammy-updates,jammy-security,now 5.34.0-3ubuntu1.2 amd64 [installed,automatic]
15:27:53 procps/jammy,now 2:3.3.17-6ubuntu2 amd64 [installed]
15:27:53 python3-dev/jammy-updates,now 3.10.6-1~22.04 amd64 [installed]
15:27:53 python3-distutils/jammy-updates,now 3.10.6-1~22.04 all [installed,automatic]
15:27:53 python3-lib2to3/jammy-updates,now 3.10.6-1~22.04 all [installed,automatic]
15:27:53 python3-minimal/jammy-updates,now 3.10.6-1~22.04 amd64 [installed,automatic]
15:27:53 python3-pip/jammy-updates,jammy-security,now 22.0.2+dfsg-1ubuntu0.3 all [installed]
15:27:53 python3-pkg-resources/jammy-updates,jammy-security,now 59.6.0-1.2ubuntu0.22.04.1 all [installed,automatic]
15:27:53 python3-setuptools/jammy-updates,jammy-security,now 59.6.0-1.2ubuntu0.22.04.1 all [installed,automatic]
15:27:53 python3-wheel/jammy-updates,jammy-security,now 0.37.1-2ubuntu0.22.04.1 all [installed,automatic]
15:27:53 python3.10-dev/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 python3.10-minimal/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 python3.10/jammy-updates,jammy-security,now 3.10.6-1~22.04.2ubuntu1.1 amd64 [installed,automatic]
15:27:53 python3/jammy-updates,now 3.10.6-1~22.04 amd64 [installed]
15:27:53 readline-common/jammy,now 8.1.2-1 all [installed,automatic]
15:27:53 rpcsvc-proto/jammy,now 1.4.2-0ubuntu6 amd64 [installed,automatic]
15:27:53 sed/jammy,now 4.8-1ubuntu2 amd64 [installed]
15:27:53 sensible-utils/jammy,now 0.0.17 all [installed]
15:27:53 sshpass/jammy,now 1.09-1 amd64 [installed]
15:27:53 sysvinit-utils/jammy,now 3.01-1ubuntu1 amd64 [installed]
15:27:53 tar/jammy-updates,jammy-security,now 1.34+dfsg-1ubuntu0.1.22.04.1 amd64 [installed]
15:27:53 ubuntu-keyring/jammy,now 2021.03.26 all [installed]
15:27:53 unzip/jammy-updates,jammy-security,now 6.0-26ubuntu3.1 amd64 [installed]
15:27:53 usrmerge/jammy,now 25ubuntu2 all [installed]
15:27:53 util-linux/jammy,now 2.37.2-4ubuntu3 amd64 [installed]
15:27:53 vim-common/jammy-updates,jammy-security,now 2:8.2.3995-1ubuntu2.8 all [installed,automatic]
15:27:53 vim-runtime/jammy-updates,jammy-security,now 2:8.2.3995-1ubuntu2.8 all [installed,automatic]
15:27:53 vim/jammy-updates,jammy-security,now 2:8.2.3995-1ubuntu2.8 amd64 [installed]
15:27:53 wget/jammy,now 1.21.2-2ubuntu1 amd64 [installed]
15:27:53 xxd/jammy-updates,jammy-security,now 2:8.2.3995-1ubuntu2.8 amd64 [installed,automatic]
15:27:53 zlib1g-dev/jammy-updates,jammy-security,now 1:1.2.11.dfsg-2ubuntu9.2 amd64 [installed,automatic]
15:27:53 zlib1g/jammy-updates,jammy-security,now 1:1.2.11.dfsg-2ubuntu9.2 amd64 [installed]

pip

15:27:53 + pip3 list
15:27:54 Package            Version
15:27:54 ------------------ ---------
15:27:54 ansible            8.1.0
15:27:54 ansible-compat     4.1.2
15:27:54 ansible-core       2.15.1
15:27:54 ansible-lint       6.17.2
15:27:54 ansible-pylibssh   1.1.0
15:27:54 async-timeout      4.0.2
15:27:54 attrs              23.1.0
15:27:54 awscli             1.27.163
15:27:54 bcrypt             4.0.1
15:27:54 black              23.3.0
15:27:54 boto               2.49.0
15:27:54 boto3              1.26.163
15:27:54 botocore           1.29.163
15:27:54 bracex             2.3.post1
15:27:54 certifi            2023.5.7
15:27:54 cffi               1.15.1
15:27:54 charset-normalizer 3.1.0
15:27:54 click              8.1.3
15:27:54 colorama           0.4.4
15:27:54 cryptography       41.0.1
15:27:54 datadog            0.45.0
15:27:54 datadog-api-client 2.14.0
15:27:54 dnspython          2.3.0
15:27:54 docker             6.1.3
15:27:54 docutils           0.16
15:27:54 filelock           3.12.2
15:27:54 fortiosapi         1.0.5
15:27:54 httplib2           0.22.0
15:27:54 idna               3.4
15:27:54 Jinja2             3.1.2
15:27:54 jmespath           1.0.1
15:27:54 jsonschema         4.17.3
15:27:54 markdown-it-py     3.0.0
15:27:54 MarkupSafe         2.1.3
15:27:54 mdurl              0.1.2
15:27:54 mitogen            0.3.3
15:27:54 mypy-extensions    1.0.0
15:27:54 netaddr            0.8.0
15:27:54 oyaml              1.0
15:27:54 packaging          23.1
15:27:54 paramiko           3.2.0
15:27:54 passlib            1.7.4
15:27:54 pathspec           0.11.1
15:27:54 pip                23.1.2
15:27:54 platformdirs       3.8.0
15:27:54 pyasn1             0.5.0
15:27:54 pycparser          2.21
15:27:54 Pygments           2.15.1
15:27:54 PyNaCl             1.5.0
15:27:54 pyOpenSSL          23.2.0
15:27:54 pyparsing          3.1.0
15:27:54 pyrsistent         0.19.3
15:27:54 pyspnego           0.9.1
15:27:54 python-dateutil    2.8.2
15:27:54 pywinrm            0.4.3
15:27:54 PyYAML             5.4.1
15:27:54 redis              4.6.0
15:27:54 requests           2.31.0
15:27:54 requests-credssp   2.0.0
15:27:54 requests-ntlm      1.2.0
15:27:54 resolvelib         1.0.1
15:27:54 rich               13.4.2
15:27:54 rsa                4.7.2
15:27:54 ruamel.yaml        0.17.32
15:27:54 ruamel.yaml.clib   0.2.7
15:27:54 s3transfer         0.6.1
15:27:54 setuptools         59.6.0
15:27:54 six                1.16.0
15:27:54 subprocess-tee     0.4.1
15:27:54 tomli              2.0.1
15:27:54 typing_extensions  4.7.0
15:27:54 urllib3            1.26.16
15:27:54 wcmatch            8.4.1
15:27:54 websocket-client   1.6.1
15:27:54 wheel              0.37.1
15:27:54 xmltodict          0.13.0
15:27:54 yamllint           1.32.0
15:27:54 
15:27:54 [notice] A new release of pip is available: 23.1.2 -> 23.2.1
15:27:54 [notice] To update, run: python3 -m pip install --upgrade pip

STEPS TO REPRODUCE

The issue is intermittent and not reproducible on-demand. We see the error multiple times per job and use ignore_errors: true to work around it.

Example playbook

- name: ICX AAA role
  hosts: icx_*
  gather_facts: no
  ignore_unreachable: true
  become: yes
  serial: 10
  roles:
    - icx-aaa
  vars:
    icx_username: "{{ hostvars['localhost']['network_user'] }}"
    icx_password: "{{ hostvars['localhost']['network_password'] }}"
    icx_enable_password: "{{ icx_password }}"
    icx_radius_psk: "{{ hostvars['localhost']['radius_psk'] }}"
    ansible_command_timeout: 45

Example tasks that generate the error

- name: Configure RADIUS attributes
  community.network.icx_config:
    lines:
      - radius-server retransmit {{ aaa_radius_retransmit }}
      - radius-server timeout {{ aaa_radius_timeout }}
  register: _aaa_radius_attr
  ignore_errors: "{{ icx_bypass_errors }}"

- name: Test reachability to RADIUS servers
  community.network.icx_ping:
    dest: "{{ item }}"
  loop: "{{ radius_servers }}"
  register: _validation

EXPECTED RESULTS

I expect that it completes successfully every run, but intermittently the task fails as indicated above.

15:06:52 TASK [icx-aaa : Configure RADIUS attributes] ***********************************
15:06:52 ok: [switch]

ACTUAL RESULTS

15:30:29 Loading collection ansible.netcommon from /usr/share/ansible/collections/ansible_collections/ansible/netcommon
15:30:29 Loading collection ansible.utils from /usr/share/ansible/collections/ansible_collections/ansible/utils
15:30:29 redirecting (type: become) ansible.builtin.enable to ansible.netcommon.enable
15:30:29 <switch> attempting to start connection
15:30:29 <switch> using connection plugin ansible.netcommon.network_cli
15:30:29 Found ansible-connection at path /usr/local/bin/ansible-connection
15:30:29 <switch> found existing local domain socket, using it!
15:30:29 <switch> updating play_context for connection
15:30:29 <switch> Loading collection ansible.builtin from 
15:30:29 <switch> local domain socket path is /root/.ansible/pc/b146f0a1d8
15:30:29 Using module file /usr/share/ansible/collections/ansible_collections/community/network/plugins/modules/icx_config.py
15:30:29 Pipelining is enabled.
15:30:29 <switch> ESTABLISH LOCAL CONNECTION FOR USER: root
15:30:29 <switch> EXEC /bin/sh -c '/usr/bin/python3 && sleep 0'
15:30:29 
15:30:29 TASK [icx-aaa : Configure RADIUS attributes] ***********************************
15:30:29 task path: /etc/ansible/roles/icx-aaa/tasks/icx_aaa.yml:26
15:30:29 The full traceback is:
15:30:29   File "/tmp/ansible_community.network.icx_config_payload_546o4akj/ansible_community.network.icx_config_payload.zip/ansible_collections/community/network/plugins/module_utils/network/icx/icx.py", line 34, in run_commands
15:30:29     return connection.run_commands(commands=commands, check_rc=check_rc)
15:30:29   File "/tmp/ansible_community.network.icx_config_payload_546o4akj/ansible_community.network.icx_config_payload.zip/ansible/module_utils/connection.py", line 200, in __rpc__
15:30:29     raise ConnectionError(to_text(msg, errors='surrogate_then_replace'), code=code)
15:30:29 fatal: [switch]: FAILED! => {
15:30:29     "changed": false,
15:30:29     "invocation": {
15:30:29         "module_args": {
15:30:29             "after": null,
15:30:29             "backup": false,
15:30:29             "before": null,
15:30:29             "defaults": false,
15:30:29             "diff_against": null,
15:30:29             "diff_ignore_lines": null,
15:30:29             "intended_config": null,
15:30:29             "lines": [
15:30:29                 "radius-server retransmit 5",
15:30:29                 "radius-server timeout 10"
15:30:29             ],
15:30:29             "match": "line",
15:30:29             "multiline_delimiter": "@",
15:30:29             "parents": null,
15:30:29             "replace": "line",
15:30:29             "running_config": null,
15:30:29             "save_when": "never",
15:30:29             "src": null
15:30:29         }
15:30:29     },
15:30:29     "msg": "command timeout triggered, timeout value is 45 secs.\nSee the timeout setting options in the Network Debug and Troubleshooting Guide."
15:30:29 }
15:30:29 ...ignoring
15:30:30 Loading collection ansible.netcommon from /usr/share/ansible/collections/ansible_collections/ansible/netcommon
15:30:30 Loading collection ansible.utils from /usr/share/ansible/collections/ansible_collections/ansible/utils
15:30:30 redirecting (type: become) ansible.builtin.enable to ansible.netcommon.enable
15:30:30 <switch> attempting to start connection
15:30:30 <switch> using connection plugin ansible.netcommon.network_cli
15:30:30 Found ansible-connection at path /usr/local/bin/ansible-connection
15:30:30 <switch> found existing local domain socket, using it!
15:30:30 <switch> invoked shell using ssh_type: libssh
15:30:30 <switch> updating play_context for connection
15:30:30 <switch> Loading collection ansible.builtin from 
15:30:30 <switch> local domain socket path is /root/.ansible/pc/b146f0a1d8