Skip to content

test: simplify test setup and strengthen assertions - #2537

Open
joaovv-Vitor wants to merge 14 commits into
NVIDIA-NeMo:developfrom
letdnts:test/remaining-test-smells
Open

joaovv-Vitor wants to merge 14 commits into
NVIDIA-NeMo:developfrom
letdnts:test/remaining-test-smells

Conversation

@joaovv-Vitor

Copy link
Copy Markdown
Contributor

Description

Follow up on the test-maintenance proposal after the LFU timing changes were
merged in #2530. This PR contains the remaining changes from the team's
experimental branch in 14 self-contained commits, with each commit addressing
one issue or a category of related issues.

  • Extract focused setup for LFU workloads, benchmark latency checks,
    evaluation CLI arguments, Clavata payloads, Patronus chat configuration,
    and manifest action modules.
  • Make LFU invariant assertions and embedding import-error cleanup explicit.
  • Parameterize predefined embedding models and separate generation
    serialization surfaces and LLMRails registration methods into focused cases.
  • Replace the HF classifier loading sleep with explicit synchronization.
    The test observes the second caller entering the loading sentinel's wait
    path before releasing the first loader, checks that only one pipeline is
    constructed, and propagates worker failures through futures.

Changes are limited to 11 test files. The LFU clock and concurrent-clear tests
accepted in #2530 are preserved, along with newer upstream tests. Production
code, dependencies, lockfiles, and generated changelogs are unchanged.

Related Issue(s)

Related to #2518 and follows #2530.

Commit organization follows @Pouyanpi's
request for a single PR with self-contained commits.

Issue assignee: @joaovv-Vitor.

Verification

  • Before these changes: 438 passed, 3 skipped across the 11 affected test files.
  • After these changes: 452 passed, 3 skipped, executed together.
  • Every commit passed the full affected test file and all applicable
    repository pre-commit checks before proceeding to the next commit.
  • The final combined pre-commit run passed, including Ruff, formatting,
    whitespace, license checks, and ty. Hooks without applicable files were skipped.
  • All 14 commits have SSH signatures verified locally and by GitHub
    (Verified), with DCO sign-offs.
  • Scope review confirmed preservation of the four LFU timing/concurrency
    methods merged in test: make LFU cache tests deterministic #2530 and the newer generation and LLMRails tests.
  • Unit-test execution excluded live-provider credentials and used mocked services.

AI Assistance

The changes were adapted from João Vitor and Maria Letícia Dantas's team
branch, with source-commit references and human coauthor credit preserved
where applicable. Codex assisted with separating the changes, adapting them
to current develop, strengthening HF loading synchronization, validation,
and drafting this description.

  • No AI tools were used.
  • AI tools were used; a human reviewed and can explain every change (tool: Codex).

Checklist

  • I've read the CONTRIBUTING guidelines.
  • This PR links to a triaged issue assigned to me.
  • My PR title follows the project commit convention.
  • I've updated the documentation if applicable (not applicable: tests only).
  • I've added tests if applicable (existing scenarios were reorganized and strengthened).
  • I've noted any verification beyond CI and any checks I couldn't run.
  • I did not update generated changelog files manually.
  • I addressed all CodeRabbit, Greptile, and other review comments, or replied with why no change is needed.
  • @mentions of the person or team responsible for reviewing proposed changes.

joaovv-Vitor and others added 14 commits October 29, 2026 16:18
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 991f934.
Codex assisted with adaptation, implementation, and validation.

Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 9cf2e4a, 81263cd, 96013d2.
Codex assisted with adaptation, implementation, and validation.

Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 330097f, 89214de, 8ee7a6a, 6d4741d, 088d217, a2121b7, a90a557, 7dfbb38.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 2b59c67.
Codex assisted with adaptation, implementation, and validation.

Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: df14b83.
Codex assisted with adaptation, implementation, and validation.

Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: c2c4065, 0e9b833, 1707914, 44fbf3a.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 102596e, 6dfe48f, c8b6ce1, 3fa9c6a.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 926b286, 5063181.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 2dabd22.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 0e6c88e, 1f60a9a.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: b2b5fa7.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 95fd210.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 2c217fb.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
A focused follow-up to NVIDIA-NeMo#2518 after NVIDIA-NeMo#2530, adapted from the team branch.

Source commits: 2dc3908.
Codex assisted with adaptation, implementation, and validation.

Co-authored-by: Maria Letícia Dantas <mleticiadantas@alu.ufc.br>
Signed-off-by: joaovv_vitor <jrodriguescontatato@gmail.com>
@github-actions github-actions Bot added status: needs triage New issues that have not yet been reviewed or categorized. size: L labels Oct 29, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size: L status: needs triage New issues that have not yet been reviewed or categorized.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant