Fix the nightly: purge the Recycle Bin, correct sweeper delete order, stop lanes wiping .auth - #1
Open
darshangm92 wants to merge 2 commits into
Open
Fix the nightly: purge the Recycle Bin, correct sweeper delete order, stop lanes wiping .auth#1darshangm92 wants to merge 2 commits into
darshangm92 wants to merge 2 commits into
Conversation
The nightly suite had been failing intermittently since 2026-08-21. Three
separate faults, one of which was the cause and two of which were feeding it.
STORAGE. The Developer org was at 5 MB of data storage with 0 remaining, so
every write failed STORAGE_LIMIT_EXCEEDED. That surfaces in the specs as a
Lightning modal that will not close or a poll that never satisfies, never as
anything naming storage, which is why the nightly read as unrelated flakiness
across the journeys, pricing-methods, smartwatch, solar and quote lanes.
Salesforce's REST DELETE is a SOFT delete: the record sits in the Recycle Bin
for 15 days and keeps consuming storage the whole time. Nothing in this
framework ever emptied that bin, and per-test CpqDataFactory.cleanup() fills
it on every green run, so the exhaust of every successful run was billed
against storage for a fortnight. The sweeper now purges the bin after its
deletes, via SOAP emptyRecycleBin (REST has no equivalent). Allowlist-only,
same --retention-days cutoff, --confirm required, --no-purge to opt out.
Running it freed the org: 1629 records purged, and a create that had been
rejected succeeded immediately after.
DELETE ORDER. Two ordering faults left residue behind every night, both
visible as the sweep's 8 recurring failures. SBQQ__Subscription__c was
deleted before the quote lines that reference it ("could not be completed
because it is associated with the following quote lines"), so it moves after
the quotes. And a contracted Order cannot be reverted to Draft while its
Contract exists, so the pre-delete unblock could never clear it; failed
deletes now get a second pass that re-unblocks after the plan has drained,
by which point the Contracts are gone.
.AUTH RACE. global-teardown.js wiped .auth/ unconditionally under CI, but
every lane run-parallel.js spawns runs it, so the first lane to finish took
the session away from the lanes still running -- "Cannot find module
'../../.auth/sf-session.json'" in lanes that had nothing wrong with them.
Children now defer, exactly as they already do for the ledger merge, and the
parent wipes once after the last lane exits.
Verified: sweeper dry run and confirmed run clean, smoke suite green against
the Developer org (2 passed), lint:tags and the lane coverage guard pass.
darshangm92
had a problem deploying
to
github-pages
August 29, 2026 05:32 — with 
GitHub Actions
Failure
GitHub Actions
Failure
Follow-up to fd1b591, which was necessary and not sufficient. That commit freed 1629 records and the very next full-suite run hit STORAGE_LIMIT_EXCEEDED again, 26 times. Two things were wrong. THE ACTUAL STORAGE HOG IS THE PACKAGE, NOT THE SUITE. Measured via /limits/recordCount: SBQQ__RecordJob__c held 1585 rows dating back to 2025-01-14 -- roughly 3.1 MB of a 5 MB org, against 781 live records across all ten allowlisted objects combined. It is CPQ's async-job receipt: one row per queued calculation or contracting job, written by the package and never removed. Nothing reads them back. It gets its own pass rather than an eleventh entry in DELETE_ORDER. Every allowlisted object is verified by marker or ancestry before deletion because the suite created it; this one fails all of those tests -- the package creates it, so there is no Description to stamp, no ledger row, and no lookup back to a marked record. Folding it in would mean weakening verifyCandidates for one object and quietly weakening it for the other ten. Separate list, separate age gate, separate reporting, --no-housekeeping to skip. THE PURGE NO LONGER INHERITS --retention-days. Retention protects LIVE records from being reclaimed early; a record already in the bin has passed that gate, so re-applying the same age test downstream of a decision already made freed nothing and excluded precisely the recent bulk filling the bin. Purge age is its own axis now, defaulting to 0, with --purge-retention-days=N for an org that wants an undo window. Result: 1528 records deleted and purged, 99 left inside retention, and DataStorageMB went from Max 5 / Remaining 0 to Max 5 / Remaining 3. Also here: - removeMany() via sObject Collections, 200 per call. The housekeeping counts are in the thousands and one DELETE per record took minutes and spent a percentage point of the daily API allowance. 1528 deletes become 8 calls. The suite's own records still delete one at a time -- each needs its own outcome against a plan a human reads, and several need a per-record unblock. - queryPaged(). The housekeeping scan first used queryAll(), which INCLUDES soft-deleted rows, so the dry run straight after the first sweep offered the same 1528 records it had just deleted. Caught before it could turn every nightly sweep into 1528 pointless deletes.
darshangm92
had a problem deploying
to
github-pages
August 29, 2026 06:46 — with
GitHub Actions
Failure
GitHub Actions
Failure
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nightly has been failing intermittently since 2026-08-21 (runs on 08-21, 08-22, 08-23, 08-27, 08-28 and 08-29). Three separate faults — one cause, two feeding it.
1. The org ran out of data storage — this is the actual cause
Measured 2026-08-29: 5 MB max, 0 MB remaining, against ~1,500 live records. Every write failed
STORAGE_LIMIT_EXCEEDED(35 occurrences in the 08-29 run alone).Nothing in the reports said "storage". Journey stage 6 died on
locator.waitFor: Timeout 30000ms exceededwaiting for an Edit modal to close;contracted-pricingdied on a 60s locator timeout. A modal that will not close is what a rejected save looks like — so the symptom pointed at the UI.A REST
DELETEis a soft delete. The record sits in the Recycle Bin for 15 days and keeps consuming storage the whole time, and nothing here ever emptied it. It isn't mainly the sweeper's own output either: per-testCpqDataFactory.cleanup()fills the bin on every green run, so the exhaust of every passing nightly was billed against a 5 MB org for a fortnight.The sweeper now purges the bin after its deletes, via SOAP
emptyRecycleBin(REST has no equivalent; Bulk API hard delete needs a profile permission this integration user lacks). Allowlist-only, same--retention-dayscutoff,--confirmrequired,--no-purgeto opt out.Running it freed the org: 1629 records purged, and a create that had been rejected succeeded immediately after.
Two things worth knowing, both learned the hard way and both now in comments:
queryAll ... WHERE IsDeleted = trueanswers the 15-day delete-tracking window, not bin membership. It returned 4758; only 1629 actually held a bin entry. The first run therefore reported 2878 "failures" that were allno recycle bin entry found— records already costing nothing. Now classified as benign and counted separately, and the purge walks parents first so a cascade covers the children.DataStorageMBin/limitsis recalculated periodically, not live. It still readRemaining: 0right after the purge that fixed the org. Confirm with a create instead.2. Two delete-order faults — the sweep's 8 recurring failures every night
SBQQ__Subscription__cwas deleted before the quote lines that reference it. An amendment/renewal quote line points back at the subscription it revises:could not be completed because it is associated with the following quote lines.: QL-0005393. Moved afterSBQQ__Quote__c.Draftwhile its Contract exists, so the single pre-deleteunblock()could never clear it —Status -> Draftrefused, then the delete refused with "Cannot delete an order with active products". Failed deletes now get a second pass that re-unblocks after the plan has drained, by which pointDELETE_ORDERhas already removed the Contracts. A retry rather than a re-ordering, because the dependency resolves as the plan drains.3. Lanes were wiping each other's session
global-teardown.jswiped.auth/unconditionally under CI — but every lanerun-parallel.jsspawns runs globalTeardown, and.auth/is shared. The first lane to finish took the session from the lanes still running:Cannot find module '../../.auth/sf-session.json',ENOENT: .auth/session.json, in lanes with nothing wrong with them (7 occurrences on 08-29, across pricing-methods, solar and quote).This is the same hazard Section 3.18 documents for the ledger merge, fixed the same proven way: the child defers on
CPQ_REUSE_SESSION, the parent wipes once after the last lane exits. The flag was already in the child's environment and the ledger branch beside it already read it — the wipe simply predated lanes.Verification (2026-08-29, Developer org)
--confirmrun both clean: 0 delete failures, 0 real purge failures.npm run lint:tagsgreen; lane coverage guard 17/17.