overload: add shutdown overload action - #46932
Open
salman-frs wants to merge 1 commit into
Open
Conversation
An overload condition is meant to be transient, but some never clear. A memory leak or heap fragmentation can hold the fixed heap monitor above the stop_accepting_requests threshold indefinitely, so Envoy stays up rejecting everything, and no OOM killer fires because the process is not actually at a hard limit. Adds envoy.overload_actions.shutdown: when the action stays saturated without interruption for saturation_duration, Envoy optionally drains for drain_time and then exits, leaving the restart to the supervisor. Requiring continuous saturation is what bounds the restart rate, since a freshly started Envoy has to be saturated for saturation_duration all over again before it can shut down. Signed-off-by: salman-frs <sallman.alffarisi@gmail.com>
salman-frs
requested a deployment
to
external-contributors
August 25, 2026 04:18 — with
GitHub Actions
Waiting
|
Hi @salman-frs, welcome and thank you for your contribution. We will try to review your Pull Request as quickly as possible. In the meantime, please take a look at the contribution guidelines if you have not done so already. |
|
CC @envoyproxy/api-shepherds: Your approval is needed for changes made to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #41492.
An overload condition is meant to be transient, but some never clear. A memory leak or heap
fragmentation can hold the fixed heap monitor above the
stop_accepting_requeststhresholdindefinitely, so Envoy stays up rejecting everything, and no OOM killer fires because the process is
not actually at a hard limit. The only way out today is for something outside Envoy to notice and
restart it.
This adds a well known overload action,
envoy.overload_actions.shutdown, with a requiredShutdownConfig. When the action stays saturated without interruption forsaturation_duration,Envoy drains and exits. Envoy does not restart itself, that is the
supervisor's job, hence
shutdownrather thanrestart.Requiring continuous saturation is what bounds the restart rate: any dip cancels the countdown, so a
freshly started Envoy has to be saturated for
saturation_durationall over again before it canshut down.
max_jitterkeeps a fleet that shares one overload condition from exiting in lockstep.Shutdown reuses the existing graceful path rather than adding a second notion of drain time: fail
the health check, stop accepting, drain over
--drain-time-s, exit on drain completion.The countdown lives in
source/server/overload_shutdown.ccrather than inOverloadManagerImplbecause it needs
Server::Instanceto drain and exit. It followsMemory::HeapShrinker.Risk Level: low -- opt-in, nothing changes unless the action is configured.
Testing: unit tests in overload_shutdown_test.cc, config parsing tests in overload_manager_impl_test.cc, integration test in overload_integration_test.cc.
Docs Changes: new Shutdown section in docs/root/configuration/operations/overload_manager/overload_manager.rst, a row in the overload actions table, and an example config.
Release Notes: changelogs/current/new_features/overload_manager__shutdown-action.rst
Platform Specific Features: n/a.