fix(core): stop nist-nvd2 extracts from filling the node - #3795
Merged
Conversation
Oban timeouts kill the worker with :kill, so after-cleanup never runs and each retry downloaded a new zip. Keep at most one nist-nvd2 run dir, prune leftovers on every scheduler tick, and refuse a download when staging is over budget. Demo turns the CPE catalog back on.
The 60s feed scheduler only reaped as a side effect of enqueue, and Oban :kill still skips after-cleanup. Give leftover extracts a real self-scheduling job, drop a run dir when acquire fails, and keep at most one nist-nvd2 dir while a load is executing.
Flipping current on ~360k advisory rows hit demo CNPG's default statement_timeout. Oban then retried the whole download. Raise the timeout for the generation swap so a finished load can become current.
current_by_device ORDER BY kev/exploit/cvss was using the global priority index and filtering device_uid after scanning ~74k active rows. Partial index on (device_uid, kev, exploit, cvss, last_seen) WHERE status=active lets LIMIT 50 stop after the device's top rows.
Elixir Quality --check-formatted failed on the nist-nvd2 disk-safety changes.
mfreeman451
force-pushed
the
fix/nist-nvd2-disk-budget
branch
from
August 19, 2026 02:17
06ec9ed to
32e160c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Demo's Software tab showed CPE catalog is not current because
nist-nvd2was disabled after leftover extracts filledk8s-cp3-worker3(~255 GiB). The PVC islocal-path(no quota). Each Oban attempt uses a newrun_id, and a timeout kills the worker with:kill, soaftercleanup never runs. The next retry downloaded another zip.What
nist-nvd2staging dir; prune leftovers on every scheduler tickSERVICERADAR_ADVISORY_NIST_NVD2_ENABLEDfrom Helm (demo overlay turns the feed back on)Demo is being rolled to a prune/budget core image separately. This PR is the product-side fix so a future release cannot refill a node the same way.
Test plan
Staging.prune_feed/2keep 1 / keep 0ensure_budget/1fails over the capreap_orphans/1drops extra nist-nvd2 dirs even when they are freshsha-db61aeec8bcore roll: one staging dir, generationcurrent=true, Software banner gone