Repository navigation
fix(telematics): retention pruner no longer scans the whole device_events table every run - #381
Merged
roncodes merged 1 commit intoOct 11, 2026
Conversation
…e_events table every run fleetops:prune-telematics-data ran every 15 minutes and, on a 2.3M-row device_events table, held the database for minutes per batch even though every telematics connection was inactive and nothing was left to prune: - batches were ordered by id while the scope ranged over created_at, so MySQL walked the primary key instead of the (company_uuid, created_at) retention index; batches now order by the ranged column - the orphan scope was 'company_uuid IS NULL OR NOT IN (select uuid from companies)', which cannot use an index; the gone company uuids are now resolved once per table from a grouped index scan and the scope becomes IS NULL / IN (...) - the compaction sweep re-read every already-compacted row on every run; a per-scope watermark (cache) now limits it to rows that aged into the window since the last complete sweep - the soft-deleted purge had no index on deleted_at; two migrations add (company_uuid, deleted_at) on device_events and positions Polling itself was already correct: fleetops:sync-telematics only queues connections in active/connected states.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
fleetops:prune-telematics-dataruns every 15 minutes. On a dev database with a 2.3M-rowdevice_eventstable it held MySQL for minutes per batch (single statements running 300–500 s, database at 200–320 % CPU, API requests timing out) even though every telematics connection wasinactiveand there was nothing left to prune.Polling itself was already correct:
fleetops:sync-telematicsonly queues connections inactive/connectedstates, so inactive connections generate no device-event traffic. The load was the retention sweep.Why it scanned everything
idwhile the scope ranged overcreated_at, so MySQL walked the primary key instead of the(company_uuid, created_at)retention index. Batches now order by the ranged column.company_uuid IS NULL OR company_uuid NOT IN (select uuid from companies), which cannot use an index. The gone company uuids are now resolved once per table from a grouped index scan, and the scope becomesIS NULL/IN (...).deleted_at, so it walked the table even when nothing was trashed. Two migrations add(company_uuid, deleted_at)ondevice_eventsandpositions, guarded like the existing retention index migrations.Validation
Validation is deferred to CI and the maintainer's run (per the agreed workflow):
php -lpasses on the three files; behaviour of the existingPruneTelematicsDataCommandTestis unchanged (same rows deleted/compacted, only ordering and scope shape differ). The watermark uses theCachefacade in try/catch so test bootstraps without a cache store are unaffected.Operator notes
php artisan migrateto add the two indexes (an online index build on a large table takes a while).Docs / API
No user-facing or API change; no docs or postman updates needed.
Stacked on
refactor/identity-stub-from-fleetops-data(#380) because that is the checked-out dev branch; rebase onto the release branch if preferred.