Remove duplicated file queries in pipeline staging - #116
Merged
Conversation
The pipeline counted the same files in more than one place: `waiting` re-scanned the input directory for files middleware already receives as `candidates`, and the finished-directory scan was repeated in the staging context, completion logging, and the inactivity check. - Drop `waiting` from `StagingContext`; the remaining counts describe files that are staged, completed, or failed - Add `_count_finished` and `_all_tracked_files_settled` as the single source for those counts
yoonspark
added
bug
The previous commit dropped `waiting` because middleware already receives those files as `candidates`. That holds only for the first step in the chain: once a step filters the list, later steps can no longer see the full input backlog. Recovering the count would mean a second scan of the input directory, so merge the candidate scan into the context builder instead: one scan feeds both, and `waiting` comes from the list the chain receives. The builder is renamed `_prepare_staging_inputs` to match.
Collaborator
Author
|
Separate from this PR, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_prepare_staging_inputsreplaces_build_staging_contextand returns both the candidate files and theStagingContext, withwaitingderived from that same list._count_finishedand_all_tracked_files_settledso the finished-file count and the "everything settled" check live in one place instead of being repeated in_handle_processed_filesand_check_inactivity.