Skip to content

Batch triggerer unused trigger cleanup deletes#68269

Draft
BeauDevCode wants to merge 2 commits into
apache:mainfrom
BeauDevCode:batch-trigger-clean-unused-deletes
Draft

Batch triggerer unused trigger cleanup deletes#68269
BeauDevCode wants to merge 2 commits into
apache:mainfrom
BeauDevCode:batch-trigger-clean-unused-deletes

Conversation

@BeauDevCode

@BeauDevCode BeauDevCode commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

closes: #68243

Why

Trigger.clean_unused() currently removes all unreferenced trigger rows in one unbounded delete. On busy deployments this can hold locks on the trigger table for the duration of a large cleanup and stall the triggerer loop.

What changed

  • Select unused trigger IDs before deleting, preserving the existing unused-trigger criteria.
  • Delete matching trigger rows in configurable batches, committing between batches.
  • Add [triggerer] unreferenced_triggers_cleanup_batch_size, defaulting to 500; setting it to 0 keeps single-transaction cleanup behavior.
  • Add a unit test that uses a batch size of 2 and verifies five unused triggers are removed across three committed delete batches.

Related context: PR #68244 changes the anti-join shape in the same method, but does not add bounded delete batches. This PR intentionally focuses on the batching behavior requested in #68243.

Tests

  • git diff --check
  • python3 -m compileall -q airflow-core/src/airflow/models/trigger.py airflow-core/tests/unit/models/test_trigger.py
  • Parsed airflow-core/src/airflow/config_templates/config.yml with PyYAML and verified the new option shape

Not run locally: focused pytest/config validation could not complete because this environment lacks parts of Airflow's development dependency set (pathspec during test collection; requests for dev/validate_version_added_fields_in_config.py).


Important

🛠️ Maintainer triage note for @BeauDevCode · by @potiuk · 2026-06-17 14:51 UTC

Helpful heads-up from the maintainers — please address before this PR can be reviewed:

  • Static / docs checks failing (CI image checks / Build documentation (--spellcheck-only)). Run them locally with prek run --all-files (or pre-commit run --all-files) and push the fixes.
  • See the Pull Request quality criteria.

The ball is in your court — you've been assigned to this PR. Fix the above, then mark it Ready for review.

Automated triage — may be imperfect; a maintainer takes the next look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Batch the triggerer's Trigger.clean_unused() cleanup deletes

1 participant