Skip to content

HA: Reset stalled processing queue entries & use updatable last_update timestamp#466

Merged
oxzi merged 2 commits into
mainfrom
retention-ha-reset-processing
Jul 14, 2026
Merged

HA: Reset stalled processing queue entries & use updatable last_update timestamp#466
oxzi merged 2 commits into
mainfrom
retention-ha-reset-processing

Conversation

@oxzi

@oxzi oxzi commented Jul 13, 2026

Copy link
Copy Markdown
Member
retention: Reset stalled processing queue entries

Instead of deleting event queue entries in a stalled processing state,
reset them to the pending state.

Initially, there was only one kind of retention, which deleted affected
rows. With the last refactoring[0], the logic became easier to adjust.
This enabled this change, introducing the ResetPruner, which performs a
UPDATE query instead of a DELETE.

As old event queue entries in the processing state imply a crashed
daemon or another fatal error, they should be reset and retried.
Otherwise, we would lose events, which we accept prior to this commit.

As a drive by change, the queue state variables from the event package
were exported. This allowed getting rid of magic numbers in the static
dbPruner configuration.

[0]: fb63975c2178fa64e7d492869bff6db62f3f8ebc

HA: Dedicated last_update timestamp

Use a dedicated last_update timestamp instead of the event time for the
event queue. While this timestamp is used where the prior time - now
renamed to event_time - is used, it can be updated for each event queue
object modification.

This ensures that the retention uses the defined durations instead of
the duration minus an unknown delta, depending on when the event was
processed or how long it waited in the queue.

Refs #394.

@oxzi oxzi added this to the 1.0 milestone Jul 13, 2026
@oxzi oxzi requested a review from yhabteab July 13, 2026 12:54
@oxzi oxzi self-assigned this Jul 13, 2026
@cla-bot cla-bot Bot added the cla/signed CLA is signed by all contributors of a PR label Jul 13, 2026
Instead of deleting event queue entries in a stalled processing state,
reset them to the pending state.

Initially, there was only one kind of retention, which deleted affected
rows. With the last refactoring[0], the logic became easier to adjust.
This enabled this change, introducing the ResetPruner, which performs a
UPDATE query instead of a DELETE.

As old event queue entries in the processing state imply a crashed
daemon or another fatal error, they should be reset and retried.
Otherwise, we would lose events, which we accept prior to this commit.

As a drive by change, the queue state variables from the event package
were exported. This allowed getting rid of magic numbers in the static
dbPruner configuration.

[0]: fb63975
@oxzi oxzi force-pushed the retention-ha-reset-processing branch from 18d6093 to 62b8763 Compare July 13, 2026 14:08
@nilmerg nilmerg moved this from Todo to In progress in Icinga Notifications 1.0 Jul 14, 2026
Use a dedicated last_update timestamp instead of the event time for the
event queue. While this timestamp is used where the prior time - now
renamed to event_time - is used, it can be updated for each event queue
object modification.

This ensures that the retention uses the defined durations instead of
the duration minus an unknown delta, depending on when the event was
processed or how long it waited in the queue.
@oxzi oxzi changed the title retention: Reset stalled processing queue entries HA: Reset stalled processing queue entries and use update-able last_update timestamp Jul 14, 2026
@oxzi

oxzi commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

I have pushed a second commit, introducing a second update-able timestamp used for retention - as discussed yesterday. More details are in the commit message and in the updated top post.

@yhabteab yhabteab left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LFTM! Ty.

@yhabteab yhabteab changed the title HA: Reset stalled processing queue entries and use update-able last_update timestamp HA: Reset stalled processing queue entries & use updatable last_update timestamp Jul 14, 2026
@oxzi

oxzi commented Jul 14, 2026

Copy link
Copy Markdown
Member Author

Just verified the schema upgrade: no diff.

@oxzi oxzi merged commit c90f8b1 into main Jul 14, 2026
23 checks passed
@oxzi oxzi deleted the retention-ha-reset-processing branch July 14, 2026 08:17
@github-project-automation github-project-automation Bot moved this from In progress to Done in Icinga Notifications 1.0 Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla/signed CLA is signed by all contributors of a PR

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants