HotReloadingDesign

[ric-pro]

Shotover Hot Reloading Interface: Design

The proposed design to implement hot reloading mechanism will be based on handover of the listener socket from already running shotover to a new shotover instance. The new shotover will start accepting new connections while the old shotover will handle all the existing connections until all the connections are handled. Since there are multiple shotover instances running in parallel, the serialisation/ de-serialisation of any states to disks are not required.

As part of the design, those needed to be finalised at this stage will be the following:

1. The Communication Channel

• A unix domain socket can be used as the primary communication channel between the old and the new shotovers. The two main approaches that can be considered for managing the socket paths are:

  • Explicit Path - As in the user will pass the path to the unix socket as a CLI argument to both the old and new shotover instance.

  • Default path - There will be a default path for unix sockets. Whenever a new shotover starts, it checks if the socket is at the default path. If yes, the new instance will assume that it is entering in a hot reload mode.

Both approaches have its own benefits, a combination of both can result in flexibility. A new starting shotover should check for any CLI arguments in relation to hot reloading, if not found - it should also check for the unix socket at the default path.

2. How Hot Reloading is triggered

• a new shotover can be started with a particular CLI flag. This will make the shotover to connect to the unix sockets path the specified or default path. Once the connection is established, the new shotover will send a request to the old shotover for transferring the listening sockets.

• Once the request is received, the old shotover will check if there are any requests made for hot reloading, if yes, the listener sockets will be transferred to the new shotover and the old shotover will enter a 'draining state' where it will stop receiving the new connections and will continue with processing the existing connections.

• Once all the existing connections are processed, the old shotover will be shutdown.

3. The processes happening while hot reloading happens

• The processes that will happen in the shotover hot reload will be the following:

  • Old Shotover - The first shotover will be running. It has a unix socket at a specified path or default path. It is listening to the incoming messages at this socket.

  • New Shotover - A new shotover is started with a CLI flag which trigger the shotover to start in hot reload mode. This shotover will have the same configuration as the old shotover.

  • Handover - The new shotover will send a 'handover' request to the old shotover through the unix sockets. Once the old shotover receives the handover request, the file descriptors of the listening socket is transferred to the new shotover.

  • The New Shotover Up and running - Once the new shotover receives the listener sockets, all the new client connections will be processed through this shotover.

  • The Previous shotover in 'Drain' Mode - Since the listening socket is transferred to the new shotover, the old shotover will no longer receive any new connections. It will enter a 'drain' mode where it will handle all the existing connections. Once all the existing connections are processed, the old instance will shut down.

[Question: if we integrate a timer at which the old shooter gets forcefully shuts while still having connections to process]

4. The Integration Test

• The test should initially focus on verifying whether the handover and shutdown was successful.

  • This can be done by:

    • Start a shotover

    • Start another shotover with CLI Argument for hot reloading

    • Check if the handover is successful, the first shotover is shutdown and the second shotover is running successfully.

    • Check if the first shotover is no longer taking new connections and is being able to successfully process the existing connections

    • Check if the new shotover is able to receive new connections and is being able to successfully process those new connections.

[rukai]

Here is my thoughts on the unix socket protocol.

Unix Socket protocol

The protocol follows a request/response model.
The replacement shotover instance sends 1 request and the original shotover instance responds with 1 respond.
The original shotover instance cannot send requests, and it cannot send responses unless replying to a request.
Over the course of the hot reloading process the replacement shotover instance will drive the hot reloading process.
Every hot-reloading action taken by the original instance is done in response to a request from the replacement instance.

#[derive(Serialize, Deserialize)]
enum Request {
    // implement this request type for your next PR
    SendListeningSockets,

    // below are other request types, dont implement these yet, just demonstrating other kinds of actions we might want.

    BeginDrainingConnections,

    ShutdownOriginalNode,
}

/// Each response type is sent in response to the request type of the same name.
/// With the exception of the Error response which can be sent in response to any request.
#[derive(Serialize, Deserialize)]
enum Response {
    // implement this response type for your next PR
    SendListeningSockets {
        /// For every source configured in the original shotover node an entry is provided mapping the port to its corresponding file descriptor.
        port_to_fd: HashMap<u32, FileDescriptor>
    },

    // below are other response types, dont implement these yet, just demonstrating other kinds of actions we might want.

    BeginDrainingConnections,

    ShutdownOriginalNode,

    /// This can be returned in response to any request if the request cannot be fulfilled.
    Error(String)
}

Steps of hot reloading to be implemented in your next PR

1. Original shotover creates unix socket and is running and serving connections from clients.
2. Replacement shotover:
3. partially starts up without creating its own listening sockets.
4. connects to the original shotovers unix socket.
5. sends Request::SendListeningSockets serialized into JSON over the unix socket.
6. Original shotover:
7. receives the request and deserializes the JSON bytes into a Request::SendListeningSockets
8. stops listening for new connections
9. extracts the listening socket FDs into the port_to_fd map
10. sends Response::SendListeningSOckets serialized into JSON
11. Replacement shotover:
12. receives the response and deserializes the JSON bytes into a Response::SendListeningSockets
13. Takes the file descriptors from port_to_fd and starts listening for new incoming connections, using the source definitions as defined in the topology.yaml
14. Shotover startup is complete and it handles traffic normally.

[ric-pro]

Socket Handoff Investigations

1. Listener Socket Creation and Usage

• Listener socket created as - create_listener() function is at line 282 in server.rs.
• The listener sockets are stored in TcpCodecListener.listener: Option<TcpListener>.
• The listener sockets are used in the accept method (line 256-280) in order to receive incoming connections.

2. File Descriptors Extraction Methods

There are two different methods (identified so far) in which FD's can be accessed.

• Direct Raw FD Access - By making use of the AsRawFd trait to extract the file descriptor directly from the Tokio 'TcpListener'

• Standard Library Conversion - An approach where the tokio listener will be converted to a standard library listener and then extract the file descriptor.

Initial plan is to try and implement FD extraction is AsRawFd trait.

Unix Socket Communication Protocol

Protocol Design

The hot reload process will follow a request/response model where the replacement shotover instance will be driving the entire process. The original shotover instance only responds to request and cannot initiate any communication.

Communication Flow

1. Replacement Shotover Instance will connect to original instance's Unix socket.
2. Replace instance will send 'Request::SendListeningSockets' as JSON.
3. Original Shotover Instance receives and deserialises the request.
4. Original Shotover Instance extracts the FD and responds with 'Response::SendListeningSockets'.
5. Replacement Shotover will receive FD and completes the startup

Implementation Scope

Phase 1: Original Shotover Unix Socket Setup

• Original shotover is listening on unix sockets and continues serving client connects as normal.
• Implement capability to receive and process unix socket requests

Phase 2: Replacement Shotover Startup

• Starts without having a listener socket
• Connects to original shotover's unix socket
• Sends 'Request::SendListeningSockets' as serialised JSON

Phase 3: Socket Transfer Process

• Original shotover deserialises incoming requests
• Original shotover stop accepting new connections and socket extraction begins
• Creates 'ports_to_fd' mapping and sends 'Response::SendListeningSockets'

Phase 4: Replacement Shotover Completion

• Replacement shotover deserialises response and extracts FD
• Extracted FD's are used to create listening sockets
• completes start up and starts handling the traffic normally(expected).

[ric-pro]

Task division for Socket Handoff Implementation

Task 1 - Defining Protocol Messages and verifying serialisation works successfully within the system

Task 2 - Development of Unix Socket Server at the Original Shotover Instance

• to setup the communication channel between shotover instances

Task 3 - Development of Unix Socket Client at the Replacement Shotover Instance

• to complete establishment of communication channel between the shotovers.

Task 4 - Request/Response Protocol Implementation

Task 5 - FD Extraction

Task 6 - Socket Reconstruction at the replacement shotover

[ric-pro]

Where are we now - (As of 11/08/2025)

• Communication Infrastructure between shotover instances for hot reloading is established. Now need to implement real file descriptor transfer.
• Next task - Replaces the hard coded FD with real FD from TcpListener instances.

What Has Been Accomplished So Far

• Built unix socket communication between shotover instances.
• Original shotover creates a unix socket server
• Replacement shotover connects as a client
• JSON protocol communication(Request/Response)

Current File Structure

shotover/src/hot_reload/
├── protocol.rs - Request/Response type definitions
├── server.rs - Unix socket server (original shotover)
├── client.rs - Unix socket client (replacement shotover)
├── json_parsing.rs - JSON serialization utilities
└── mod.rs - Module exports and shared test helpers

Task 4 Implementation Design Details

• Current: Hot reload server returns a fake file descriptor (10).
• Goal: Hot reload server returns real FD from shotover's actual TcpListener.
• Result: Replacement shotover can use real FD to take over the listening socket.

[ric-pro]

Current Status (As of September 1, 2025)

COMPLETED - Real File Descriptor Transfer (PR #1921)

Replace hardcoded FD with real FD from TcpListener instances" has actually been fully implemented in PR #1921.

What Has Been Accomplished

• Communication Infrastructure (Previously completed)
• Unix socket communication between shotover instances
• Original shotover creates unix socket server
• Replacement shotover connects as client
• JSON protocol communication (Request/Response)
• Real File Descriptor Transfer (Completed in PR Listening Socket Extraction for Hot Reload #1921)
• Hot reload server now returns real FDs from actual TcpListener instances
• Channel-based communication between hot reload server and TcpCodecListener
• tokio::select! implementation for non-blocking socket extraction
• Integration with all source types (Cassandra, Kafka, Valkey, OpenSearch)

What's Next?

File Descriptor Utilization

• Current: Replacement shotover receives real FDs but cannot use them
• Goal: Convert received FDs back to TcpListener instances
• Result: New shotover instance can initialize with existing sockets

Hot Reload Orchestration

• End-to-end coordination of the complete hot reload process
• Configuration parsing and source initialization with existing listeners

The hardcoded FD issue has been resolved. The next phase is implementing the utilization of those real FDs.

[ric-pro]

FD Recreation

File Descriptor Utilities

First up creating functions to turn raw file descriptors from the old instance back into std::net::TcpListener objects.

Hot Reload Orchestrator

This will handle the full workflow: connecting to the existing instance over a Unix socket, requesting the file descriptors, converting them back to listeners using the utilities above, and then updating the Shotover config to use these listeners instead of binding new ones.

Source and Server Modifications

Enhancing TcpCodecListener with a new from_existing_listener() constructor, so it can accept pre-existing listeners. Update all the source types so they can optionally start up from existing listeners, rather than always binding to new addresses.

Runner Integration

Wirng the orchestrator into the main startup flow. That way, if we launch with the --hotreload-from-socket CLI flag, it’ll try the hot reload path first, and if that doesn’t work, it’ll just fall back to the normal startup.

[rukai]

In the Hot Reload Orchestrator section

converting them back to listeners using the utilities above,

I think I would have the orchestrator just deal with the FD and have the TcpCodecListener be responsible for constructing the TcpListener from the FD.
TcpCodecListener would then have a from_existing_fd method.
My reasoning is that the TcpStream is more a low level concern of TcpCodecListener whereas the hot reload orchestrator is running at a higher level.

And a heads up:
You might also need to call into the linux https://man7.org/linux/man-pages/man2/pidfd_getfd.2.html function to make it work.
This might require access to a linux VM.
But get everything else working first and if it doesnt work, we can look into this kind of thing.

Otherwise, sounds good, go ahead and get started.