fix(core): only defer hashing on dependencies that produce outputs - #36923
Draft
FrozenPandaz wants to merge 3 commits into
Draft
fix(core): only defer hashing on dependencies that produce outputs#36923FrozenPandaz wants to merge 3 commits into
FrozenPandaz wants to merge 3 commits into
Conversation
A task with a dependentTasksOutputFiles input was deferred from up-front hashing whenever it had any dependency, and the batch path re-ran it whenever any direct dependency missed the cache. The native hasher already ignores dependencies with no outputs, so those deferrals bought nothing: the task's hash could not change. Add getDependenciesWithOutputsToHash, mirroring the native BFS, and gate all three sites on it. Tasks whose only relevant dependencies are output-less now get a hash up front, which lets Nx Cloud consider them for a remote cache hit at DTE start and lets batch executors replay them instead of re-running. (cherry picked from commit d986b06498ed5a7439ad3f2097fed8944888abf2)
✅ Deploy Preview for nx-dev ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for nx-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
|
View your CI Pipeline Execution ↗ for commit 54a65b4
☁️ Nx Cloud last updated this comment at |
…okup check-task-files already walks a task's dependencies with the same direct-or-transitive switch. Lift it into task-graph-utils and use it for getDependenciesWithOutputsToHash instead of a second traversal.
![nx-cloud[bot]](https://avatars.githubusercontent.com/in/80458?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Nx Cloud has identified a flaky task in your failed CI:
🔂 Since the failure was identified as flaky, we triggered a CI rerun by adding an empty commit to this branch.
🔔 Heads up, your workspace has pending recommendations ↗ to auto-apply fixes for similar failures.
🎓 Learn more about Self-Healing CI on nx.dev
…elf-Healing CI Rerun]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Current Behavior
A task with a
dependentTasksOutputFilesinput is deferred from up-front hashing whenever it has any dependency, and the batch path re-runs it whenever any direct dependency misses the cache. The native hasher already ignores dependencies that declare nooutputs, so those deferrals buy nothing: the task's hash cannot change.Consequences:
Example in this repo:
e2e-maven:e2e-ci--*depends onnx-maven-plugin:install, which has no outputs and cannot affect the e2e task's hash.Expected Behavior
Hashing is deferred only when a dependency that actually produces outputs feeds the task's hash.
getDependenciesWithOutputsToHashmirrors the nativecollect_task_dependenciesBFS and returns the ids of dependency tasks with non-emptyoutputs, walking transitively when anydependentTasksOutputFilesentry is transitive.hashTasksThatDoNotDependOnOutputsOfOtherTaskshashes a task up front when that set is empty.applyBatchCachedResultsdefers a task only when an output-bearing dependency was a cache miss, bringing the batch and non-batch paths onto one rule.Tasks whose only relevant dependencies are output-less now get a hash up front, which lets Nx Cloud consider them for a remote cache hit and lets batch executors replay them instead of re-running.
Related Issue(s)
N/A
View Polygraph session ↗