Skip to content

feat(analysis): exploratory ranks + multi-session claim tooling - #74

Merged
leo-gan merged 2 commits into
masterfrom
feat/a1-a6-claims-multiplicity
Jul 25, 2026
Merged

feat(analysis): exploratory ranks + multi-session claim tooling#74
leo-gan merged 2 commits into
masterfrom
feat/a1-a6-claims-multiplicity

Conversation

@leo-gan

@leo-gan leo-gan commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Summary

  • Multi-way effect-vs-fastest now uses median reference, Mann–Whitney tests, and within-group Holm adjustment; Results pages label ranks as exploratory (prefer pairwise A/B for confirmatory checks).
  • Sidecars gain machine_id / L1 claim hint; --multi-session aggregates rank stability and L1/L2/L3 claim levels; soft check-research-env.sh for research hosts.
  • Student docs: claims ladder (CLAIMS_AND_REPLICATION.md), methodology sections, Wikipedia links for statistical terms; strip internal ticket tags from public docs.

Validation

  • Tests: analysis 83 pass; python 29 pass; javascript 9 pass; rust 13 pass; cpp roundtrips OK. Java/mvn unavailable (warn only).
  • Benchmarks: skipped (no benchmark-runner language changes; analysis/docs only).
  • Error CSVs: N/A (no rebench).
  • Analysis: no forced full-matrix Results regen (banners apply on next analyze-benchmarks).
  • Dashboard: sync-data.py verified latest stems present; no commit of timestamp-only gzip churn.

Notes

  • Default Results remain L1 single-session / single-host; multi-session is opt-in CLI.
  • Older CSVs without machine_id in sidecars report L2_multi_session_host_unknown when aggregating ≥3 sessions.
  • Example: analyze-benchmarks --multi-session python:STEM1 --multi-session python:STEM2 --multi-session python:STEM3

Attach Mann–Whitney + within-group Holm vs the median-fastest codec,
label multi-way Results as exploratory, and add L1/L2/L3 claim tooling
(machine_id sidecars, --multi-session aggregates, claims docs, soft
research-host checks). Student methodology links Wikipedia for stats terms.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@leo-gan
leo-gan merged commit 6470d67 into master Jul 25, 2026
11 checks passed
@leo-gan
leo-gan deleted the feat/a1-a6-claims-multiplicity branch July 25, 2026 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant