ci: add labgrid-coordinator-driven hardware tests - #744
Open
tfcollins wants to merge 19 commits into
Open
Conversation
New self-hosted-runner workflow runs marker-selected pytest legs against real hardware via the labgrid coordinator (10.0.0.41:20408), one matrix job per available place. Each leg acquires the place, fresh-boots via the labgrid Strategy, runs tests, and releases. - test/conftest.py override (gated by LG_ENV) provides iio_uri from labgrid NetworkService.address; emu/unit path is untouched when LG_ENV is unset. - Per-place env yamls under test/hw/env/ describe targets/drivers/boot. - .github/hw-nodes.json maps places to markers (one entry per place). - Helper scripts: bootstrap-uv.sh, install-pyadi-iio-venv.sh, register-hw-runners.sh. - Workflow gated by external-pr environment for fork PRs (mirrors iio-context-coverage.yml's pattern). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
test/conftest.py imports test.rf.spec which pulls in scipy via scipy.signal.windows.kaiser; pyadi-iio[jesd] alone is not enough. Mirror the existing emu-CI install_pydeps.sh pattern by installing requirements_dev.txt into the labgrid runner's venv. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Without adi-labgrid-plugins, the coordinator-forwarded resources (KuiperRelease, VesyncPowerDriver, KuiperDLDriver, etc.) fail to deserialize on the runner side with InvalidConfigError. The plugin package transitively pulls in the tfcollins/plugin-support labgrid fork, replacing the explicit LABGRID_PIP entry. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hardware Test Results 2 files 2 suites 52s ⏱️ Results for commit 34eee7b. ♻️ This comment has been updated with latest results. |
|
Generated documentation for this PR is available at Link |
The package source repo is tfcollins/labgrid-plugins (public), not analogdevicesinc/adi-labgrid-plugins (does not exist). Previous URL was invented from the package name. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Test Results1 812 tests ±0 507 ✅ - 1 4m 18s ⏱️ -5s Results for commit 34eee7b. ± Comparison against base commit ee07c9b. This pull request removes 1 and adds 1 tests. Note that renamed tests count towards both.This pull request skips 1 test.♻️ This comment has been updated with latest results. |
bq + mini2 fail at strategy construction because BootFPGASoC[TFTP] calls KuiperDLDriver.get_boot_files_from_release() during __attrs_post_init__, which transitively imports pytsk3. pytsk3 is in the [kuiper] extra of adi-labgrid-plugins; manylinux wheels exist so no libtsk-dev install is required. nuc needs the runner host (nemo) to SSH-key-auth into nuc itself for XilinxJTAGDriver._run_xsdb's scp call. That key isn't yet provisioned; drop nuc from the manifest until it is. preflight will simply not emit a matrix leg for nuc. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bq.yaml's TFTPServerDriver runs on the runner host (nemo) and serves
boot files to the bq board via auto-detected runner IP. Default root
/var/lib/tftpboot isn't writable by the runner user; switch to
/tmp/pyadi-iio-tftpboot which the driver auto-creates.
Restoring nuc to the manifest now that SSH key auth from nemo to each
lab host (bq, mini2, nuc) will be provisioned out-of-band via
ssh-copy-id. The labgrid drivers (USBSDMuxDriver on mini2,
XilinxJTAGDriver on nuc, etc.) ssh/scp from the runner into the host
that owns each physical resource, so key auth nemo->{bq,mini2,nuc} is
load-bearing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
mini2 regressed on the prior run because shell.run() captured stderr into stdout — when the kernel hadn't brought up networking yet, 'RTNETLINK answers: Network is unreachable' became the 'IP', and the fixture happily built ip:RTNETLINK... and then failed iio.Context. Wrap the IP read in a 60s poll loop and validate the line is dotted- quad before accepting it. Drop bq from the place manifest. The Kuiper release's adrv9371 uImage/devicetree.dtb (5c9e342a... / 30KB dtb) don't match what bq's known-working bootflow uses (2545d6f9... / 82KB dtb) — different kernel build, different DTS. pyadi-dt builds these locally per-run on the bq host; pyadi-iio's runner-on-nemo path can't easily mirror that without either an rsync hop from bq or a build-from-source flow. Re-add when one of those is in place. The bq exporter config changes (kernel_path, device_tree_path) stay applied so when this is revisited, the resource side is ready. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pytest_collection_modifyitems treated a single-string iio_hardware
arg as a sequence and used `key in hardware_list` for matching,
which silently became a string-substring check:
@pytest.mark.iio_hardware("ad9081_full_bw")
-> "ad9081" in "ad9081_full_bw" # True (substring!)
That made test_full_bw_rx pick up every marker from
test_map["ad9081"] (incl. zynqmp_zcu102_rev10_ad9081_m8_l4_vcxo122p88)
and be selected by `-m <board-marker>` runs of unrelated configs.
Wrap a single string in a list so the `in` check is element
equality. Tests with a list-form @iio_hardware were already correct.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wrap the iio_uri yield in try/finally and transition the strategy to 'powered_off' on teardown. All three strategies in current use (BootFPGASoC, BootFPGASoCTFTP, BootFabric) define Status.powered_off so a single transition string works across legs. Cleanup errors are caught + warned, never re-raised, so a flaky power-off can't mask a real test failure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the manifest-pinned `-m <board-marker>` selection model with
pytest-libiio's built-in discovery: pytest_configure boots the lab
board, polls for the DHCP-assigned IP and IIO daemon readiness, and
sets config.option.uri. pytest-libiio's _contexts fixture then scans
that URI, identifies the hardware against
test/emu/hardware_map.yml's existing entries, and skips any test
whose @iio_hardware(...) decorator doesn't match the discovered
hardware identifier.
Adding a new place is now {place, env_remote} in hw-nodes.json plus a
new env yaml — no marker math, no test_map.py cross-reference, and
bitstream/dtb variant drift no longer requires manifest edits.
- test/conftest.py: drop the labgrid_iio_uri/iio_uri override
fixtures; introduce pytest_configure (boot + set --uri) and
pytest_sessionfinish (powered_off teardown). The dotted-quad poll
and iio.Context retry are lifted into _wait_for_ipv4 /
_wait_for_iio helpers.
- .github/workflows/hardware-test.yml: drop -m and --skip-scan from
the pytest invocation; pytest-libiio now handles selection.
- .github/hw-nodes.json: drop the marker field.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two pre-existing test bugs that were silently filtered out by the old pinned-marker selection model and now surface under discovery-driven CI on AD9081 hardware: 1. test_ad9081_attr[attr=tx_main_ffh_index] used start=1, stop=31 but the AD9081 driver documents the valid range as [0, 30] (adi/ad9081.py:527). Random pick of val=31 is rejected, readback differs, and tol=0 trips the assert. Change parametrize to (0, 30, 1, 0, 10) to match the driver's documented range. 2. test_ad9081_loopback (and test_dac_zeros) call sdr.loopback = 1, which on AD9081 silently creates a no-op instance attribute (the class has loopback_mode for RX->TX, not loopback for TX->RX). Internal loopback is never enabled, RX captures ambient/random data, and the bit-exact comparison fails. Skip the test when the device class doesn't define a `loopback` property; hasattr() before any assignment is the reliable pre-write check (Python's default __setattr__ would otherwise mask the absence). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The lab-side path has multiple sources of transient flakes:
- mDNS resolution of mini2/nuc for rfc2217 serial URLs
(socket.gaierror during SerialDriver.open)
- SSH ControlMaster bring-up to exporter hosts
- Stale serial console state from a prior aborted run
Each manifests as a hard error during strategy.transition('shell')
or Environment construction, but a second attempt 10s later
typically succeeds.
Wrap pytest_configure's labgrid boot in a 3-attempt retry; only
fail the session if every attempt errors. Successful boots return
on attempt 1 with no observable change in cost.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Preflight now parses `labgrid-client -v places` to read each place's carrier= / daughter-board= tags, then includes the place in the matrix iff a corresponding test/hw/env/<place>.yaml exists in the repo. Adding a new lab board becomes: 1. tag the place via labgrid-client -p <name> set-tags carrier=... 2. drop test/hw/env/<name>.yaml in this repo No manifest edit needed. hw-nodes.json is kept as an optional override layer (currently empty) for places that need a non-default yaml or that lack the carrier/daughter-board tags. bq is currently auto-discoverable (tags exist) but its boot artifacts in the Kuiper release don't match the working bq bootflow. Remove test/hw/env/bq.yaml so auto-discovery skips it until the artifact path is sorted; restore from git history when ready. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Microblaze IIO daemon on nuc occasionally drops connections under load, surfacing as BrokenPipeError on iio.Context() construction or attribute reads. Pattern is intermittent — sometimes one test, last run took out 17 — but always with the exact same BrokenPipeError signature. Add pytest-rerunfailures to the runner venv and pass: --reruns 2 --reruns-delay 5 --only-rerun "BrokenPipeError" The --only-rerun filter is a regex against the failure message, so real assertion failures (peak_min, attribute mismatch, etc.) stay visible — only the daemon-drop class of failure gets retried. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
pytest-rerunfailures 16.0 introduced a subprocess-based "rerun server" that segfaults during pytest_runtest_protocol when the session has loaded the libiio C extension (and probably labgrid's pytest plugin too): Fatal Python error: Segmentation fault File ".../pytest_rerunfailures.py", line 466 in run_server File ".../pytest_rerunfailures.py", line 593 in pytest_runtest_protocol v15 uses the simpler in-process rerun loop which doesn't fork and doesn't cross any C-extension boundaries. Pin to <16 until upstream fixes the server-mode crash. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New self-hosted-runner workflow that, per labgrid place exposed by the coordinator (
10.0.0.41:20408), acquires the place, fresh-boots via the labgrid Strategy, runs marker-selected pytest, and releases. Each leg posts a per-place check run; a final job aggregates JUnit results into a single PR comment.test/conftest.pyoverride (gated byLG_ENV) providesiio_urifromlabgrid.NetworkService.address. WhenLG_ENVis unset the existing emu/unit-test path is untouched.test/hw/env/{bq,mini2,nuc}.yamldescribe targets/drivers/boot strategy..github/hw-nodes.jsonmaps places to pytest markers (one entry per place).bootstrap-uv.sh,install-pyadi-iio-venv.sh,register-hw-runners.sh(single local runner, mirrored from pyadi-dt's pattern but simplified — pyadi-iio's tests don't need per-board Vivado-equipped runners).external-prGitHub environment (matchesiio-context-coverage.yml).Test plan
available_nodeslist (or empty if coordinator unreachable; either is graceful).hw-coordmatrix leg acquires, runspytest -m <marker>, and releases.dorny/test-reporterposts a per-place check on the PR.EnricoMi/publish-unit-test-result-actionposts an aggregated comment._LABGRID_MODE=Falsepath: existingTest-Emulated-HWworkflow continues to pass unchanged on this PR.🤖 Generated with Claude Code