Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,35 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [0.5.0] - 2026-06-26

### Changed
- **grpc 1.0 support.** Bumped `grpc` to `~> 1.0` (was `0.11.5`). grpc 1.0 is
client-only (`GRPC.Server` was removed) and makes `:gun` optional, so `gun ~> 2.2`
is now a direct dependency (the pool uses the default Gun adapter). `:public_key`
and `:ssl` are declared in `extra_applications` (used by `Config`).
- **BREAKING (telemetry): the `[:grpc_connection_pool, :channel, :gun_down]` and
`[:grpc_connection_pool, :channel, :gun_error]` events were removed.** Under
grpc 1.0 the Gun adapter owns the socket in its own process, so those gun
messages never reach the pool. A single adapter-agnostic
`[:grpc_connection_pool, :channel, :connection_down]` event (metadata
`pool_name`, `reason`) is emitted instead. `:disconnected` and
`:reconnect_scheduled` are unchanged. Consumers handling `:gun_down`/`:gun_error`
should switch to `:connection_down` (or `:disconnected`).
- **Disconnect detection reworked.** grpc 1.0's Gun adapter keeps the live
connection inside its own `ConnectionProcess`, which lingers as a zombie after a
drop, so the old `gun_down`/`gun_error` message handlers were dead code. The
worker now monitors the inner gun process and pairs it with
`adapter_opts: [retry: 0]` so gun fails fast on a drop and the pool's own
`Backoff` governs reconnection. This also restores fast-fail connects (a dead
endpoint now errors in ~0ms instead of stalling ~5s per attempt).

### Fixed
- **License declaration** now correctly reports **Apache-2.0** (matching the committed
`LICENSE` file) instead of MIT in `mix.exs` and README — resolves the Hex.pm
mismatch (#6).

## [0.4.0] - 2026-06-01

### Changed
- **Security:** production configs now default to verifying TLS. `Config.production/1`
Expand Down
29 changes: 21 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,12 +68,18 @@ Add `grpc_connection_pool` to your dependencies in `mix.exs`:
```elixir
def deps do
[
{:grpc_connection_pool, "~> 0.3.0"},
{:grpc, "~> 0.11.5"} # Required peer dependency
{:grpc_connection_pool, "~> 0.5.0"},
{:grpc, "~> 1.0"}, # Required peer dependency
{:gun, "~> 2.2"} # grpc >= 1.0 makes gun optional; the default Gun adapter needs it
]
end
```

> **grpc 1.0:** this library targets grpc `~> 1.0`, which is client-only and makes
> `:gun` an optional dependency. Since the pool uses the default Gun adapter, add
> `:gun` explicitly. See the [Changelog](CHANGELOG.md) for the 0.5.0 migration notes
> (telemetry event change, disconnect-detection rework).

## Quick Start

### 1. Basic Usage
Expand Down Expand Up @@ -146,7 +152,7 @@ The library supports flexible configuration through `GrpcConnectionPool.Config`:
keepalive: 30_000, # HTTP/2 keepalive interval
ping_interval: 25_000, # Ping interval to keep connections warm
health_check: true, # Enable connection health monitoring
suppress_connection_errors: false # Suppress gun_down/gun_error logs (useful for GCP endpoints)
suppress_connection_errors: false # Suppress connection-down logs (useful for GCP endpoints)
]
])
```
Expand Down Expand Up @@ -723,10 +729,17 @@ The library emits telemetry events for observability. Attach handlers to monitor
| `[:grpc_connection_pool, :channel, :connection_failed]` | `duration` | `pool_name`, `error` |
| `[:grpc_connection_pool, :channel, :disconnected]` | `duration` | `pool_name`, `reason` |
| `[:grpc_connection_pool, :channel, :ping]` | `duration` | `pool_name`, `result` (`:ok` or `:error`) |
| `[:grpc_connection_pool, :channel, :gun_down]` | - | `pool_name`, `reason`, `protocol` |
| `[:grpc_connection_pool, :channel, :gun_error]` | - | `pool_name`, `reason` |
| `[:grpc_connection_pool, :channel, :connection_down]` | - | `pool_name`, `reason` |
| `[:grpc_connection_pool, :channel, :reconnect_scheduled]` | `delay_ms`, `attempt` | `pool_name`, `reason` |

> **Changed in 0.5.0 (grpc 1.0):** the `:gun_down` and `:gun_error` events were
> removed. Under grpc 1.0 the Gun adapter owns the socket in its own process and
> those gun messages never reach the pool. A single adapter-agnostic
> `:channel, :connection_down` event (metadata `pool_name`, `reason`) is now emitted
> when the monitored connection process dies. The `:disconnected` and
> `:reconnect_scheduled` events are unchanged. If you handled `:gun_down`/`:gun_error`,
> switch to `:connection_down` (or `:disconnected`).

#### Example: Attaching Telemetry Handlers

```elixir
Expand Down Expand Up @@ -971,7 +984,7 @@ If you see frequent reconnections:

### Connection Error Messages with GCP Services

GCP gRPC endpoints have hardcoded timeouts that periodically close connections, causing `gun_down` error messages. This is normal behavior and the connection pool will automatically replace the workers. To suppress these expected error messages:
GCP gRPC endpoints have hardcoded timeouts that periodically close connections, causing connection-down error messages. This is normal behavior and the connection pool will automatically replace the workers. To suppress these expected error messages:

```elixir
config :my_app, GrpcConnectionPool,
Expand All @@ -981,7 +994,7 @@ config :my_app, GrpcConnectionPool,
port: 443
],
connection: [
suppress_connection_errors: true # Suppress expected gun_down errors from GCP
suppress_connection_errors: true # Suppress expected connection-down errors from GCP
]
```

Expand Down Expand Up @@ -1009,7 +1022,7 @@ mix test

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.

## Acknowledgments

Expand Down
10 changes: 10 additions & 0 deletions lib/grpc_connection_pool/config.ex
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ defmodule GrpcConnectionPool.Config do
- `:health_check` - Whether to enable connection health checks (default: true)
- `:ping_interval` - Interval to send ping to keep connection warm (default: 25_000)

> Note: gun's internal reconnect is disabled (`adapter_opts: [retry: 0]`) so the
> pool's own backoff governs reconnection and connects to a dead endpoint fail
> fast instead of stalling. This is not user-configurable.

## Examples

### Production Configuration
Expand Down Expand Up @@ -377,6 +381,12 @@ defmodule GrpcConnectionPool.Config do
defp build_connection_opts(endpoint, connection) do
base_opts = [
adapter_opts: [
# Disable gun's internal reconnect (default 100 retries). Under grpc 1.0 the
# adapter's internal retry keeps a dead connection's owner process alive and
# makes connects to a dead endpoint block ~5s per attempt. With retry: 0 gun
# fails fast and dies on a drop, so this pool's own Backoff governs reconnects
# and the connection monitor in Worker can detect drops. See SPIKE-1.
retry: 0,
http2_opts: %{keepalive: connection.keepalive}
]
]
Expand Down
149 changes: 98 additions & 51 deletions lib/grpc_connection_pool/worker.ex
Original file line number Diff line number Diff line change
Expand Up @@ -9,10 +9,26 @@ defmodule GrpcConnectionPool.Worker do
Features:
- Channels stored in ETS for O(1) pool access (no GenServer.call in hot path)
- Automatic reconnection with exponential backoff and jitter
- Active disconnect detection (gun_down/gun_error messages)
- Active disconnect detection by monitoring the gRPC connection process
- Self-registration in Registry for health tracking
- Optional periodic ping to keep connections warm
- Configurable max reconnect attempts before crash

## Disconnect detection (grpc 1.0)

grpc 1.0's Gun adapter owns the socket inside its own supervised
`GRPC.Client.Adapters.Gun.ConnectionProcess`, so gun's `:gun_down`/`:gun_error`
messages no longer reach this worker. Worse, when the connection drops the inner
gun process dies but the `ConnectionProcess` (the `conn_pid` in
`channel.adapter_payload`) lingers as a zombie — so monitoring `conn_pid` cannot
detect a drop.

Instead we monitor the **inner gun process**, read out of the adapter's
`ConnectionProcess` state, and pair it with `adapter_opts: [retry: 0]` (see
`GrpcConnectionPool.Config`) so gun gives up immediately on a drop and this
pool's own `Backoff` governs reconnection. Reaching into the adapter state
couples us to a grpc internal; it is guarded and falls back to monitoring
`conn_pid` if the state shape changes.
"""
use GenServer
require Logger
Expand All @@ -32,7 +48,8 @@ defmodule GrpcConnectionPool.Worker do
:ets_table,
:connection_start,
:reconnect_attempt,
:slot_index
:slot_index,
:conn_monitor_ref
]
end

Expand Down Expand Up @@ -87,7 +104,8 @@ defmodule GrpcConnectionPool.Worker do
ets_table: ets_table,
connection_start: nil,
reconnect_attempt: 0,
slot_index: nil
slot_index: nil,
conn_monitor_ref: nil
}

{:ok, state}
Expand All @@ -112,6 +130,10 @@ defmodule GrpcConnectionPool.Worker do

Logger.debug("gRPC connection established for pool #{inspect(state.pool_name)}")

# Monitor the connection so a drop triggers reconnection. We monitor the
# inner gun process (which dies on drop), not the lingering conn_pid.
conn_monitor_ref = monitor_connection(channel)

# Register in Registry for health tracking
Registry.register(state.registry_name, :channels, nil)

Expand Down Expand Up @@ -148,7 +170,8 @@ defmodule GrpcConnectionPool.Worker do
backoff_state: new_backoff,
connection_start: now,
reconnect_attempt: 0,
slot_index: slot
slot_index: slot,
conn_monitor_ref: conn_monitor_ref
}}

{:error, reason} ->
Expand Down Expand Up @@ -217,50 +240,31 @@ defmodule GrpcConnectionPool.Worker do
end
end

# Active disconnect detection — Gun adapter
def handle_info({:gun_down, _conn_pid, protocol, reason, _killed_streams}, state) do
# Active disconnect detection — the monitored gRPC connection process died.
def handle_info(
{:DOWN, ref, :process, _pid, reason},
%State{conn_monitor_ref: ref} = state
) do
:telemetry.execute(
[:grpc_connection_pool, :channel, :gun_down],
%{},
%{pool_name: state.pool_name, reason: reason, protocol: protocol}
)

unless state.config.connection.suppress_connection_errors do
Logger.debug(
"gRPC connection closed by server for pool #{inspect(state.pool_name)}: #{inspect(reason)}"
)
end

handle_disconnect(state, {:gun_down, reason})
end

def handle_info({:gun_error, _conn_pid, reason}, state) do
:telemetry.execute(
[:grpc_connection_pool, :channel, :gun_error],
[:grpc_connection_pool, :channel, :connection_down],
%{},
%{pool_name: state.pool_name, reason: reason}
)

unless state.config.connection.suppress_connection_errors do
Logger.error(
"Gun connection error for pool #{inspect(state.pool_name)}: #{inspect(reason)}"
Logger.debug(
"gRPC connection down for pool #{inspect(state.pool_name)}: #{inspect(reason)}"
)
end

handle_disconnect(state, {:gun_error, reason})
handle_disconnect(state, {:connection_down, reason})
end

def handle_info({:gun_up, _conn_pid, _protocol}, state) do
Logger.debug("Gun connection is up for pool #{inspect(state.pool_name)}")
# A DOWN we no longer care about (already torn down / demonitored late). Ignore.
def handle_info({:DOWN, _ref, :process, _pid, _reason}, state) do
{:noreply, state}
end

# Active disconnect detection — Mint adapter
def handle_info({:elixir_grpc, :connection_down, _pid}, state) do
Logger.debug("Mint gRPC connection down for pool #{inspect(state.pool_name)}")
handle_disconnect(state, :mint_connection_down)
end

def handle_info({:EXIT, _pid, _reason}, state) do
{:noreply, state}
end
Expand All @@ -279,13 +283,22 @@ defmodule GrpcConnectionPool.Worker do
GRPC.Stub.connect("#{host}:#{port}", opts)
end

defp send_ping(channel) do
case channel do
%GRPC.Channel{adapter_payload: %{conn_pid: pid}} when is_pid(pid) ->
if Process.alive?(pid), do: :ok, else: :error
# The connection monitor is the primary drop detector; this ping is a warm-keep
# plus backstop. It checks the inner gun process (the thing that actually dies on
# a drop) rather than conn_pid, which lingers as a zombie.
defp send_ping(%GRPC.Channel{adapter_payload: %{conn_pid: conn_pid}}) when is_pid(conn_pid) do
if Process.alive?(conn_pid), do: gun_alive?(conn_pid), else: :error
end

_ ->
:error
defp send_ping(_channel), do: :error

# Liveness of the inner gun process. If it can't be introspected we return :ok to
# avoid false reconnects — the connection monitor (on the gun pid, set at connect)
# is the primary drop detector; this ping is only a warm-keep + backstop.
defp gun_alive?(conn_pid) do
case gun_pid(conn_pid) do
pid when is_pid(pid) -> if Process.alive?(pid), do: :ok, else: :error
_ -> :ok
end
end

Expand All @@ -310,18 +323,13 @@ defmodule GrpcConnectionPool.Worker do
defp cleanup_connection(%State{channel: channel, ping_timer: timer}) do
cancel_ping_timer(timer)

# GRPC.Stub.disconnect/1 is the correct teardown under grpc 1.0 — it stops the
# adapter's ConnectionProcess (reaping the zombie left behind after a drop) and
# is safe/idempotent on an already-dropped channel. Guarded because this is a
# best-effort teardown path that must never crash terminate/2.
try do
case channel do
%GRPC.Channel{adapter_payload: %{conn_pid: pid}} when is_pid(pid) ->
if Process.alive?(pid) do
:gun.close(pid)
end

_ ->
GRPC.Stub.disconnect(channel)
end
GRPC.Stub.disconnect(channel)
rescue
FunctionClauseError -> :ok
_ -> :ok
catch
:exit, _ -> :ok
Expand Down Expand Up @@ -353,6 +361,10 @@ defmodule GrpcConnectionPool.Worker do
defp handle_disconnect(%State{} = state, reason) do
now = System.monotonic_time()

# Drop the connection monitor and flush any DOWN already in the mailbox so a
# stale DOWN can't trigger a second disconnect after we've torn down.
demonitor_connection(state)

# Remove channel from ETS and Registry
remove_channel_from_ets(state)

Expand Down Expand Up @@ -392,7 +404,42 @@ defmodule GrpcConnectionPool.Worker do
backoff_state: new_backoff,
connection_start: nil,
reconnect_attempt: new_attempt,
slot_index: nil
slot_index: nil,
conn_monitor_ref: nil
}}
end

# Monitors the connection so a drop triggers reconnection. grpc 1.0 keeps the live
# gun process *inside* the Gun adapter's ConnectionProcess and never exposes it;
# on a drop the gun process dies but ConnectionProcess lingers as a zombie, so we
# monitor gun directly. gun also dies when its owner (conn_pid) dies, so this
# covers conn_pid crashes too. Falls back to monitoring conn_pid if the inner gun
# pid can't be read (grpc internal state shape changed).
defp monitor_connection(%GRPC.Channel{adapter_payload: %{conn_pid: conn_pid}})
when is_pid(conn_pid) do
Process.monitor(gun_pid(conn_pid) || conn_pid)
end

defp monitor_connection(_channel), do: nil

defp demonitor_connection(%State{conn_monitor_ref: nil}), do: :ok

defp demonitor_connection(%State{conn_monitor_ref: ref}) do
Process.demonitor(ref, [:flush])
:ok
end

# Reads the inner gun pid out of the Gun adapter's ConnectionProcess state. This
# couples us to a grpc internal (state shape %{gun_pid: pid, ...}); guarded and
# short-timed so a busy/dead ConnectionProcess can't block or crash the worker.
defp gun_pid(conn_pid) do
case :sys.get_state(conn_pid, 1_000) do
%{gun_pid: gun_pid} when is_pid(gun_pid) -> gun_pid
_ -> nil
end
rescue
_ -> nil
catch
:exit, _ -> nil
end
end
Loading