Repository navigation
Conversation
|
Codecov Report✅ All modified and coverable lines are covered by tests.
Additional details and impacted files@@ Coverage Diff @@
## master #22005 +/- ##
=========================================
- Coverage 87% 79% -8%
=========================================
Files 270 267 -3
Lines 24015 23959 -56
=========================================
- Hits 20855 18827 -2028
- Misses 3160 5132 +1972 |
|
The previous Test PyTorch run had two matrix jobs cancelled externally mid-run (no test failures — both were past 74% and passing), which failed |
|
Gentle nudge on this one. All six workflows on the current head (
Could someone approve the runs? For context on the earlier attempt: the previous Test PyTorch failure was two matrix jobs cancelled externally mid-run (both past 74%, no test failures), which took down Thanks! |
…edProfiler (Lightning-AI#9136) `AdvancedProfiler.teardown()` cleared `profiled_actions` unconditionally, so a profiling context that was still active at teardown had its action dropped from the dict. When that context exited, its `finally` block called `stop()` on an action that no longer existed and raised: ValueError: Attempting to stop recording an action (run_test_evaluation) which was never started. Changes: - `teardown()` now disables any still-running profiles before clearing the dict, so no profile is left enabled after the profiler goes away. - `stop()` logs at debug level and returns instead of raising when the action was never started. This path is reachable from `teardown()` by construction, so raising there was never actionable for the user. - Removes the now-inaccurate `Raises:` section from the `AdvancedProfiler` docstring. - Adds a regression test for stopping an action that was never started, and for teardown while an action is still active. Fixes Lightning-AI#9136
ac08704 to
50324fc
Compare
What does this PR do?
Fixes #9136.
AdvancedProfiler.teardown()clearedprofiled_actionsunconditionally. If a profiling context was still active when teardown ran, its action had already been dropped from the dict, so the context'sfinallyblock then calledstop()on an action that no longer existed:That is reachable in normal use: the profiler is torn down at the end of a stage while a
profile(...)block around the test evaluation can still be unwinding.Changes
teardown()disables any still-running profiles before clearing the dict, so nothing is left enabled once the profiler goes away.stop()logs at debug level and returns instead of raising when the action was never started. This path is reachable fromteardown()by construction, so the exception was never actionable for the caller.Raises:entry from theAdvancedProfilerdocstring.Tests
Two regression tests in
tests/tests_pytorch/profilers/test_profiler.py:test_advanced_profiler_stop_unstarted_action-stop()on an action that was never started returns without raising.test_advanced_profiler_teardown_disables_active_action- an action still active atteardown()is disabled and cleared.What I ran and what I did not: I did not run the suite locally, because this workspace has no torch environment installed. Those two tests plus the existing profiler tests are what CI needs to run; if either fails I will fix it on this branch.
No breaking changes: the only behaviour removed is an exception that could not be recovered from and always followed a teardown.