Skip to content

test/ci: repair compliance suite; demote weekly-compliance to placeholder until 0.2.0 on CRAN#143

Merged
MaartenMarsman merged 2 commits into
mainfrom
chore/fix-weekly-compliance-remotes
Jun 4, 2026
Merged

test/ci: repair compliance suite; demote weekly-compliance to placeholder until 0.2.0 on CRAN#143
MaartenMarsman merged 2 commits into
mainfrom
chore/fix-weekly-compliance-remotes

Conversation

@MaartenMarsman

@MaartenMarsman MaartenMarsman commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator

The weekly-compliance job had been red since April. Investigating it surfaced a chain of issues — fixed the real ones, and demoted the CI job to a placeholder because bitwise-in-CI is fundamentally unworkable until there's a matching CRAN release.

Genuine fixes (kept)

  • CI infra: the job died before running anything — remotes missing, a 404ing/unused mixedGM install, and library(bgms) with the package never installed. (Moot now that the job is a placeholder, but the diagnosis drove the rest.)
  • Stale output-property names: the harness read fit$posterior_coclustering_matrix (renamed → posterior_mean_coclustering_matrix in 0.2.0) and fit$posterior_mean_indicator for bgmCompare (gone; bgmCompare has only posterior_summary_indicator). Updated to the current API. (Confirmed these were renames, not removals, and not from this session's refactors.)
  • Config drift → single source of truth: generate_fixtures.R and test_compliance.R duplicated the config matrix in different formats and had diverged (test side pinned the CRAN-era target_accept=0.6/0.65; generator used the default). Extracted a shared tests/compliance/config_defs.R both source, with target_accept pinned explicitly per config (NUTS 0.80, adaptive-metropolis 0.44).
  • Re-baselined fixtures to the current 0.2.0 build (bitwise-vs-CRAN-0.1.6.3 is no longer meaningful).

Why the CI job is now a placeholder

Bitwise compliance against committed fixtures can't run in CI: NUTS leapfrog trajectories are chaotically FP-sensitive, so fixtures reproduce bitwise only on the machine that generated them — verified: ubuntu CI 0/32, macOS CI 8/32, vs 32/32 on the generating machine. So the bitwise check is inherently local (test_compliance.R, documented LOCAL-ONLY).

weekly-compliance.yaml is reduced to a green placeholder that documents the planned replacement: once 0.2.0 is on CRAN, install the CRAN release + dev build on one runner and diff them bitwise on the fly — same FP environment, so differences are real code changes; platform-independent; no committed fixtures.

Not in scope

  • Continuous GGM / mixed-MRF configs (suite is discrete-only; the CRAN baseline predated them) — add when the CRAN-vs-dev job lands.
  • The nightly-validation failures (2 Rhat-threshold tests + ~964 deprecation warnings) are a separate S9 thread.

The workflow called remotes::install_github("MaartenMarsman/mixedGM") but
setup-r-dependencies did not install remotes, so the step failed immediately
with 'there is no package called remotes' (weekly job red since April). Add
remotes to extra-packages, matching the nightly job's devtools pattern.
…tself

Dispatching the job revealed two more problems past the remotes fix:
- The mixedGM install (remotes::install_github MaartenMarsman/mixedGM) 404s: the
  repo is inaccessible to the repo-scoped CI token, and tests/compliance/ does not
  use mixedGM at all (it is only referenced by skip-guarded validation helpers in
  the regular suite). Remove the step (and the remotes dep added for it).
- test_compliance.R does library(bgms), but setup-r-dependencies installs only
  dependencies, not the package. Add extra-packages: local::. so bgms is installed.

This leaves: checkout -> setup-r -> deps+bgms -> (gen fixtures if missing) ->
run bitwise compliance.
@MaartenMarsman MaartenMarsman changed the title ci(weekly-compliance): install remotes before remotes::install_github ci(weekly-compliance): fix the job (drop unused mixedGM, install bgms) Jun 4, 2026
@codecov

codecov Bot commented Jun 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.74%. Comparing base (bab7ca2) to head (22f6546).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #143      +/-   ##
==========================================
- Coverage   87.75%   87.74%   -0.01%     
==========================================
  Files          87       87              
  Lines       12867    12862       -5     
==========================================
- Hits        11291    11286       -5     
  Misses       1576     1576              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@MaartenMarsman MaartenMarsman merged commit 8e9206e into main Jun 4, 2026
10 of 11 checks passed
@MaartenMarsman MaartenMarsman deleted the chore/fix-weekly-compliance-remotes branch June 4, 2026 22:03
@MaartenMarsman MaartenMarsman changed the title ci(weekly-compliance): fix the job (drop unused mixedGM, install bgms) ci/test: repair weekly-compliance (infra + re-baseline fixtures to 0.2.0) Jun 5, 2026
@MaartenMarsman MaartenMarsman changed the title ci/test: repair weekly-compliance (infra + re-baseline fixtures to 0.2.0) test/ci: repair compliance suite; demote weekly-compliance to placeholder until 0.2.0 on CRAN Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant