“Failed to load …” silently discards schedule entries

[niksite]

Problem description

Whenever RedBeat encounters a ZSET member that has lost its HASH (e.g. evicted by Redis), the scheduler does:

except KeyError:
    logger.warning('beat: Failed to load %s, removing', key)
    client.zrem(self.app.redbeat_conf.schedule_key, key)

Impact:

• The task silently disappears from the schedule.
• Only a WARNING is logged, which is easy to miss in production.
• If the entry was originally defined in app.conf.beat_schedule, it is not restored until the next Beat restart.

In systems with aggressive eviction policies (allkeys-lfu, etc.) this leads to unpredictable loss of periodic
tasks.

Expected behaviour

1. Emit a logger.error (or even critical) so that operators notice.
2. Attempt self-healing:
   • If the missing key matches a static entry in app.conf.beat_schedule, recreate it automatically instead of simply removing it.
   • For dynamic entries, keep the current behaviour (remove + error) because Beat doesn’t know how to recreate them.

Steps to reproduce

# 1. define static task "hello" in app.conf.beat_schedule
# 2. start celery beat -S redbeat
# 3. IN ANOTHER SHELL:
redis-cli DEL   redbeat:hello            # delete the hash
# don't remove it from the ZSET, simulate eviction only
# 4. wait until the next tick

Observe log:
beat: Failed to load redbeat:hello, removing (level=WARNING).
Task never runs again until Beat restarts.

Suggested fix (pseudo-patch)

diff --git a/redbeat/schedulers.py b/redbeat/schedulers.py
@@
-         except KeyError:
-             logger.warning('beat: Failed to load %s, removing', key)
-             client.zrem(self.app.redbeat_conf.schedule_key, key)
+        except KeyError:
+            static_entry = self.app.redbeat_conf.schedule.get(entry_name_from(key))
+            if static_entry:
+                logger.error('beat: Corrupted entry %s detected, restoring from '
+                             'app.conf.beat_schedule', key)
+                self._maybe_entry(entry_name_from(key), static_entry).save()
+            else:
+                logger.error('beat: Lost dynamic entry %s, removing from schedule', key)
+                client.zrem(self.app.redbeat_conf.schedule_key, key)

Where entry_name_from(key) strips the key_prefix.

Why this matters

Periodic tasks often execute business-critical workflows (billing, data sync, alerts).
An unnoticed eviction effectively turns them off and violates reliability expectations.

Work-arounds

• Switch to volatile-* or noeviction policy in Redis.
• Pin Beat to a dedicated Redis instance with maxmemory 0.
• Monitor logs for the warning and trigger Beat restart.

But a built-in auto-recover mechanism would eliminate the need for manual mitigation.

---

Thanks for maintaining RedBeat! Let me know if more details are required.

[sibson]

Upgrading the message to an error in this case seems reasonable. Redbeat isn't designed to be tolerant of evictions, this is something that should be monitored. Having some evictions auto-heal and other not creates inconsistency. Additionally, if the job definition was changed the operator might not be ready to the new entry to be auto-reloaded. tldr; +1 on error, -1 on autoheal.

[niksite]

Could it be an opt-in auto-heal?

[sibson]

Is it possible to create a custom RedBeatScheduler with custom RedBeatSchedulerEntry with from_key() overridden with the behaviour you desire? If you are using static entries the default in-memory Celery Beat Scheduler might be a better fit.

I'd be more inclined to have RedBeat error out if it discovers a incompatible eviction policy or missing entry due to suspected eviction. An alternative would be some thing like a dead letter queue that could be monitored by the application.

[niksite]

A full crash when a key is missing is totally fine – anything that doesn't just silently ignore it is good. A supervisor will automatically restart it in case of a crash.