Skip to content

overload: add shutdown overload action - #46932

Open
salman-frs wants to merge 1 commit into
envoyproxy:mainfrom
salman-frs:feat/overload-shutdown-action
Open

overload: add shutdown overload action#46932
salman-frs wants to merge 1 commit into
envoyproxy:mainfrom
salman-frs:feat/overload-shutdown-action

Conversation

@salman-frs

@salman-frs salman-frs commented Aug 25, 2026

Copy link
Copy Markdown

Fixes #41492.

An overload condition is meant to be transient, but some never clear. A memory leak or heap
fragmentation can hold the fixed heap monitor above the stop_accepting_requests threshold
indefinitely, so Envoy stays up rejecting everything, and no OOM killer fires because the process is
not actually at a hard limit. The only way out today is for something outside Envoy to notice and
restart it.

This adds a well known overload action, envoy.overload_actions.shutdown, with a required
ShutdownConfig. When the action stays saturated without interruption for saturation_duration,
Envoy drains and exits. Envoy does not restart itself, that is the
supervisor's job, hence shutdown rather than restart.

Requiring continuous saturation is what bounds the restart rate: any dip cancels the countdown, so a
freshly started Envoy has to be saturated for saturation_duration all over again before it can
shut down. max_jitter keeps a fleet that shares one overload condition from exiting in lockstep.

Shutdown reuses the existing graceful path rather than adding a second notion of drain time: fail
the health check, stop accepting, drain over --drain-time-s, exit on drain completion.

The countdown lives in source/server/overload_shutdown.cc rather than in OverloadManagerImpl
because it needs Server::Instance to drain and exit. It follows Memory::HeapShrinker.

Risk Level: low -- opt-in, nothing changes unless the action is configured.
Testing: unit tests in overload_shutdown_test.cc, config parsing tests in overload_manager_impl_test.cc, integration test in overload_integration_test.cc.
Docs Changes: new Shutdown section in docs/root/configuration/operations/overload_manager/overload_manager.rst, a row in the overload actions table, and an example config.
Release Notes: changelogs/current/new_features/overload_manager__shutdown-action.rst
Platform Specific Features: n/a.

An overload condition is meant to be transient, but some never clear. A
memory leak or heap fragmentation can hold the fixed heap monitor above
the stop_accepting_requests threshold indefinitely, so Envoy stays up
rejecting everything, and no OOM killer fires because the process is not
actually at a hard limit.

Adds envoy.overload_actions.shutdown: when the action stays saturated
without interruption for saturation_duration, Envoy optionally drains for
drain_time and then exits, leaving the restart to the supervisor.

Requiring continuous saturation is what bounds the restart rate, since a
freshly started Envoy has to be saturated for saturation_duration all
over again before it can shut down.

Signed-off-by: salman-frs <sallman.alffarisi@gmail.com>
@salman-frs
salman-frs requested a deployment to external-contributors August 25, 2026 04:18 — with GitHub Actions Waiting
@repokitteh-read-only

Copy link
Copy Markdown

Hi @salman-frs, welcome and thank you for your contribution.

We will try to review your Pull Request as quickly as possible.

In the meantime, please take a look at the contribution guidelines if you have not done so already.

🐱

Caused by: #46932 was opened by salman-frs.

see: more, trace.

@repokitteh-read-only

Copy link
Copy Markdown

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @adisuissa
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #46932 was opened by salman-frs.

see: more, trace.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: Add overload action to restart Envoy when it gets stuck

2 participants