Add prism_report pytest plugin (annotated spectra + DUT context) - #740
Open
tfcollins wants to merge 19 commits into
Open
Add prism_report pytest plugin (annotated spectra + DUT context)#740tfcollins wants to merge 19 commits into
tfcollins wants to merge 19 commits into
Conversation
Pure function `analyze(payload) -> AnalysisResult` that uses genalyzer.simplified_beta when available and falls back to a plain numpy FFT (without component annotations) otherwise. No I/O. Adds Component / AnalysisResult dataclasses (consumed unchanged by the upcoming render.py) and synthetic IQ fixtures used by tests.
Replace the scaffolded plugin.py with the fully wired version that assembles Config, OutputDir, capture_session_pre/post, analyze, render_spectrum, and upload into pytest hooks. The plugin now: - pytest_configure: builds Config, creates OutputDir, points JUnit XML into the run dir. - pytest_sessionstart: runs capture_session_pre and writes iio_info, boot.log, and dmesg_pre artifacts. - pytest_runtest_makereport: drains pytest.data_log per case, runs analyze + render_spectrum, writes spectrum.html + iq.npz + metrics.json under cases/<safe_id>/. Falls through gracefully on payload, analyze, or render errors. - pytest_sessionfinish: runs capture_session_post, writes dmesg_post + dmesg_diff, finalizes the manifest, and (if --prism-url is set) invokes upload(). Side fixes: - Config.from_pytest now uses parse_known_args so pytest's own flags (-q, -v, ...) on invocation_params.args don't blow up the prism parser. - tests/conftest.py inner-pytester fixture now also injects -p test.plugins.prism_report.plugin so pytester runs see the plugin's addoptions. End-to-end test populates pytest.data_log with synthetic IQ and asserts manifest.json + spectrum.html (containing plotly) appear in the out-dir. All 46 prism_report tests pass.
Round out the plugin README with the complete flag/env/default table, a troubleshooting matrix, and the artifact-ownership table. Also relocate `pytest_plugins = ["pytester"]` from the prism_report self-tests' conftest to the top-level test/conftest.py: pytest 9.x rejects pytest_plugins in non-rootdir conftests as a hard error during collection, which broke `pytest --collect-only test/`. The pytester fixture override stays in the inner conftest; only the registration moved.
Test Results1 861 tests +49 553 ✅ +45 4m 54s ⏱️ +31s For more details on these failures, see this check. Results for commit 78bed86. ± Comparison against base commit ee07c9b. ♻️ This comment has been updated with latest results. |
Real-prism upload was failing with HTTP 422 "metadata field required": upload.py was building a hand-rolled multipart with field names "project"/"run_name"/"tag.<k>"/"artifacts" that don't match what the prism API expects (a single "metadata" JSON form field plus "junit" and optional "archive" file parts). The prior fake-prism test only checked body length, so the field-name drift slipped through. - Replace the hand-rolled _multipart() construction with a delegated call to the vendored _prism_client.upload_run() which builds the correct payload shape. - Tighten the fake handler in test_upload_fake_prism.py to actually parse the multipart body and require "metadata" + "junit" form fields (returns 422 otherwise, matching real prism). - Round-trip test now asserts metadata.project_slug, metadata.name, metadata.tags, and that "archive" is sent. Caught when running the bench loop against a live prism docker stack.
Pytest parametrize ids that include large dicts (e.g. test_ad9364_sfdr's full param_set) sanitize to 400+ chars, exceeding ext4's 255-byte limit on path components and crashing pytest with INTERNALERROR: OSError [Errno 36] inside makereport. Truncate to 200 bytes and append a 4-byte blake2b digest of the original nodeid to keep collisions impossible. Test added: two huge nodeids differing only in the last char must yield distinct dirs. Caught when running test_ad9364_sfdr against a real Pluto.
…nvention
Prism's worker parses each archive basename as
`{suite}__{case}__{label}.{ext}` to attach artifacts to the right case.
Our local-export layout is `cases/<safe_id>/<filename>`, which prism's
parser saw as run-scoped, so per-case spectra were uploaded but never
attached to their cases.
In `_build_artifacts_zip`:
- Read the testsuite name from junit.xml.
- Read manifest.json to map safe_id -> case_nodeid -> JUnit case name
(the part of the nodeid after `::`).
- Rewrite per-case archive entries to `{suite}__{case}__{filename}`.
- Run-level artifacts stay bare (run-scoped, unchanged).
Test now opens the uploaded archive and asserts the rewrite produces
`pytest__c__spectrum.html` for a per-case entry and a bare
`iio_info.txt` for a run-level entry.
Caught when running test_ad9364_sfdr against a live prism.
Previously each case got `spectrum.html` only. When prism's UI tried to display it via <iframe>, MinIO returned the bytes with `Content-Type: binary/octet-stream` + `X-Content-Type-Options: nosniff`, so browsers refused to render the page (blank frame). Setting the content-type at upload would fix that one case but the iframe approach also fights with cross-origin cookies, mixed content, and host isolation in remote-browser scenarios. Better fit: ship the Plotly figure as JSON alongside the HTML. The host UI (prism) fetches the JSON via a proxying API endpoint and renders it with its own react-plotly.js stack — no iframe, no MinIO direct fetch, no content-type sniffing, fully theme-integrated. - `render.py`: extract `_build_spectrum_figure()` from `render_spectrum` so HTML and JSON share identical figure construction. New `render_spectrum_figure_json()` returns `fig.to_json()`. - `plugin.py`: in the makereport hook, call both renderers and write `spectrum.html` + `spectrum.json` per case. The HTML stays for offline viewing (open the file in a browser); the JSON is what the host UI consumes inline. - `test_render.py`: new test asserts the JSON is a valid Plotly figure spec containing the expected shapes (carrier + harmonic vlines) and annotations (HD2/HD3 labels, SFDR metrics card).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a new opt-in pytest plugin under
test/plugins/prism_report/that turns existing controlled-tone loopback tests into a richer test report:boot.log(passive labgrid console capture),dmesg_pre.log/dmesg_post.log/dmesg_diff.log(SSH preferred, labgrid console fallback),iio_info.txt(binary or pylibiio walk fallback),run_meta.jsonprism/scripts/_prism_client.py--prism-report(orPRISM_REPORT=1)5 existing loopback fixtures in
dma_tests.pyupgraded to emit the new richer payload through the existingpytest.data_log/do_html_logconvention. Behavior when--prism-reportis off is unchanged — legacy{"html": "..."}payloads remain tolerated.Architecture
8 small modules, one job each:
config.py,manifest.py,analyze.py,render.py,capture.py,upload.py,plugin.py(the only file that importspytest), and a vendored_prism_client.py. Plugin registered viapytest_pluginsintest/conftest.py(NOT a setuptools entry-point — the plugin lives intest/and pyadi-iio's wheel correctly excludes that tree).Spec:
docs/superpowers/specs/2026-04-30-pyadi-prism-report-design.mdPlan:
docs/superpowers/plans/2026-04-30-pyadi-prism-report-plan.mdNotable
include = ["adi"]; exclude = ["test*"]preserved)BENCH_PLUTO_URI)prism_reportextras (plotly, pyyaml only)Test plan
pytest test/plugins/prism_report/ -v→ 46 passed locallypytest --prism-report --prism-out=/tmp/run-001 test/test_ad9364_p.py::test_ad9364_sfdragainst simulated emu — spot-check that artifacts land in/tmp/run-001/BENCH_PLUTO_URI=ip:192.168.2.1 pytest test/plugins/prism_report/tests/bench/ -m bench_pluto -vagainst a wired-up Pluto — verify SFDR thresholds intests/bench/thresholds.yamlare reasonable--prism-url=..., verify run + per-case spectrum HTML appear in the Prism UIspectrum.htmlfiles in a browser — confirm carriers/harmonics annotations render correctly