validator: make bg-download pending-eval cap configurable - #120
Open
isabella618033 wants to merge 1 commit into
Open
validator: make bg-download pending-eval cap configurable#120isabella618033 wants to merge 1 commit into
isabella618033 wants to merge 1 commit into
Conversation
Move DOWNLOAD_PENDING_EVAL_CAP from a module-level constant in background_download_worker.py to ValidatorCheckpointCfg.download_pending_eval_cap so operators can tune the on-disk staging backlog (each pending submission is ~3.6 GB) without forking the image. Default unchanged (10).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DOWNLOAD_PENDING_EVAL_CAPfrom a module-level constant inconnito/validator/background_download_worker.pytoValidatorCheckpointCfg.download_pending_eval_capinconnito/shared/config.py.10);PositiveIntvalidation rejects0/negative values.self.config.ckpt.download_pending_eval_caponce per loop iteration, so an operator can override the cap fromvalidator.yaml(underckpt:) without rebuilding the image.Why
Each pending submission in
Round.downloaded_poolis ~3.6 GB on disk. With the cap hardcoded at 10, the staging area can grow to ≈ 40 GB before backpressure kicks in. Operators on smaller/datavolumes had no way to tune this without forking. Moving it to config matches howminer_submission_max_age_cycles,miner_submission_archive_max_files, anddownload_concurrencyare already exposed on the sameCheckpointCfg.Test plan
from connito.shared.config import ValidatorCheckpointCfg— default is10.ValidatorCheckpointCfg(download_pending_eval_cap=4).download_pending_eval_cap == 4.ValidatorCheckpointCfg(download_pending_eval_cap=0)raisesValidationError(PositiveInt).ast.parseclean on the worker module.ckpt.download_pending_eval_cap: 4invalidator.yamland confirm thebg-download: pausing — eval backlog above cap cap=4 pending_eval=5log appears at the new threshold.