Skip to content

Fix gradient accumulation#7

Open
rockerBOO wants to merge 7 commits into
team-approx-bayes:mainfrom
rockerBOO:gradient-accumulation
Open

Fix gradient accumulation#7
rockerBOO wants to merge 7 commits into
team-approx-bayes:mainfrom
rockerBOO:gradient-accumulation

Conversation

@rockerBOO

@rockerBOO rockerBOO commented Jun 18, 2025

Copy link
Copy Markdown

Fixes #6 #2

  • Tests added to showcase that it is working.
  • Added robustness tests to infer that it is working as before.
  • Test that tensors are on the same device (when we move tensors around or offload before starting the optimizer)
  • Use autocast to make sure we handle mixed-precision appropriately and cast to float32 when appropriate

Use requirements-test.txt for testing requirements.

I think these changes do fix the issues but a performance overview will be important to consider. The tests should give a good grounding to make sure things continue to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gradient accumulation causes error when stepping

1 participant