Skip to content

Update MAP expected values to match map_at_k in notebooks - #2372

Open
miguelgfierro wants to merge 6 commits into
stagingfrom
miguelgfierro/fix-map-expected-values
Open

Update MAP expected values to match map_at_k in notebooks#2372
miguelgfierro wants to merge 6 commits into
stagingfrom
miguelgfierro/fix-map-expected-values

Conversation

@miguelgfierro

@miguelgfierro miguelgfierro commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Description

The notebooks were migrated from map to map_at_k and re-executed, but the tests were not updated. So the tests compare a map_at_k result against an old map expectation, and fail.

map_at_k divides by min(k, n_relevant) instead of n_relevant, so it is always greater than or equal to map. That is why every value below goes up.

This PR only moves the test expectations to the values the notebooks already produce.

Test Notebook Size Old New MAP@K in the notebook
test_sar_single_node_smoke sar_movielens.ipynb 100k 0.110591 0.244722 0.244722
test_sar_single_node_functional sar_movielens.ipynb 1m 0.060579 0.185099 notebook is committed at 100k
test_sar_single_node_functional sar_movielens.ipynb 10m 0.098745 0.203722 notebook is committed at 100k
test_baseline_deep_dive_smoke baseline_deep_dive.ipynb 100k 0.052850 0.138672 0.138664
test_baseline_deep_dive_functional baseline_deep_dive.ipynb 1m 0.033914 0.139016 notebook is committed at 100k
test_cornac_bivae_smoke cornac_bivae_deep_dive.ipynb 100k 0.146552 0.330255 0.330255

Each value comes from running the notebook with the parameters the test uses. The 1m and 10m rows have no notebook value to cite because those notebooks are committed at 100k.

Only MAP moved. Every other metric in these tests still holds and is left untouched.

BPR is fixed separately in #2371, because it changed for a different reason.

Benefit

Unblocks group_cpu_002 and group_cpu_003 of the CPU nightly, which has had no green run since 2026-05-04. A real ranking regression will now surface instead of hiding behind an assertion that already fails.

Risk

  • test_vw_deep_dive_smoke and test_vw_deep_dive_functional are skipped (VW pip package has installation incompatibilities), so they do not affect the nightly. Their MAP expectations are stale for the same reason as the rest of this PR, and so is the committed MAP of 0.012535 in vowpal_wabbit_deep_dive.ipynb, which predates the redefinition. Neither can be corrected until VW installs again: vowpalwabbit>=8.9.0,<9 has no cp310 or cp311 wheels, so the notebook cannot be re-run on a supported Python.
  • The GPU nightly has the same stale MAP expectations, but most GPU tests override EPOCHS, so their notebook outputs are not comparable and correct values need GPU runs. test_cornac_bivae_smoke is the only one whose parameters match, and it is included here.

Checklist:

  • I have followed the contribution guidelines and code style for this project.
  • I have added tests covering my contributions.
  • I have updated the documentation accordingly.
  • This PR is being made to staging branch AND NOT TO main BRANCH.

Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Signed-off-by: miguelgfierro <miguelgfierro@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant