Keep sessions alive across etserver restarts

When the router goes away, etterminal keeps its shell and re-registers
with the same credentials, retrying indefinitely with backoff capped at
30s, and the new server resumes the session with a reset. For 60s after
startup the server answers unknown ids with RETRY_LATER so clients wait
for re-registration. -T sessions end instead, since their packet-framed
stream cannot be resumed. Stopping etserver leaves shells running until
they exit or --disconnect-timeout closes them.

Author: epsalmond

.github/workflows/linux_ci.yml

@@ -185,6 +185,9 @@ jobs:
       - name: Named sessions survive client death
         run: ./test/system_tests/named_sessions.sh
 
+      - name: Sessions survive etserver restart
+        run: ./test/system_tests/router_restart.sh
+
       - name: Stop sshd for act
         if: ${{ always() }}
         run: |

CMakeLists.txt

@@ -833,7 +833,8 @@ else(WIN32)
         TEST_PREFIX "et-test."
         WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
         ADD_TAGS_AS_LABELS
-        PROPERTIES TIMEOUT 600
+        # RouterRestartTest runs full client/server/terminal stacks.
+        PROPERTIES TIMEOUT 900
       )
     else()
       add_test(NAME et-test COMMAND et-test "~[.ghostty]")

docs/protocol.md

@@ -54,9 +54,9 @@ Once etterminal launches:
 - It [locates the server fifo](https://github.com/MisterTea/EternalTerminal/blob/113fb23133eabce3d11681392d75ba4772814b44/src/terminal/ServerFifoPath.cpp) to connect to the etserver process:
   - If `/var/run/etserver.idpasskey.fifo` exists, when etserver is running as root, this path is used.
   - Otherwise, `$XDG_RUNTIME_DIR/etserver/etserver.ifpasskey.fifo` is used, resolving `$XDG_RUNTIME_DIR` to `$HOME/.local/share` if the environment variable is not set.
-- Once it connects to the server, it sends a `TERMINAL_USER_INFO` packet with [TerminalUserInfo](../proto/ETerminal.proto#L96-L102) containing the **client-id** and **passkey** to register the terminal with the server.  These are registered into the ServerConnection [`clientKeys` map](https://github.com/MisterTea/EternalTerminal/blob/113fb23133eabce3d11681392d75ba4772814b44/src/base/ServerConnection.hpp#L37-L40) awaiting a user connection.
+- Once it connects to the server, it sends a `TERMINAL_USER_INFO` packet with [TerminalUserInfo](../proto/ETerminal.proto#L97-L110) containing the **client-id** and **passkey** to register the terminal with the server.  These are registered into the ServerConnection [`clientKeys` map](https://github.com/MisterTea/EternalTerminal/blob/113fb23133eabce3d11681392d75ba4772814b44/src/base/ServerConnection.hpp#L37-L40) awaiting a user connection.
 - After etterminal connects to etserver, it outputs the **client-id** and **passkey**, to inform the client in cases where it regenerated them.
-- etterminal then waits for a client connect, waiting for a `TERMINAL_INIT` ([TermInit](../proto/ETerminal.proto#L89-L94)) packet.
+- etterminal then waits for a client connect, waiting for a `TERMINAL_INIT` ([TermInit](../proto/ETerminal.proto#L89-L95)) packet.
 - After receiving this packet UserTerminalHandler enters the `runUserTerminal` run loop, and proxies input/output until the terminal exits. See the [Terminal Run Loop](#terminal-run-loop).
 
 ## Client Connection
@@ -92,13 +92,13 @@ sequenceDiagram
 
 After the terminal launches, **et** connects to the **etserver** over the EternalTerminal port (defaults to 2022), and sends a [ConnectRequest](../proto/ET.proto#L12-L19) message containing the **client-id** and protocol version.  Since encryption is client-specific, this client-id is sent unencrypted.
 
-The server answers a version mismatch with `MISMATCHED_PROTOCOL` and closes the socket, and an unknown **client-id** with `INVALID_KEY`. For a registered client that sets `supportsChallenge`, the server first sends a [ConnectResponse](../proto/ET.proto#L28-L40) carrying a fresh `authChallenge`. The client answers with [ConnectAuth](../proto/ET.proto#L42-L45), a keyed proof over the client id, protocol version, and challenge. If the proof checks out, the server sends the final `NEW_CLIENT` or `RETURNING_CLIENT` response and creates or resumes the ServerClientConnection, which holds the BackedReader and BackedWriter used for EternalTCP buffering. A bad proof gets `INVALID_KEY`.
+The server answers a version mismatch with `MISMATCHED_PROTOCOL` and closes the socket, and an unknown **client-id** with `INVALID_KEY` (or `RETRY_LATER` during the recovery grace period after an etserver restart). For a registered client that sets `supportsChallenge`, the server first sends a [ConnectResponse](../proto/ET.proto#L29-L41) carrying a fresh `authChallenge`. The client answers with [ConnectAuth](../proto/ET.proto#L43-L46), a keyed proof over the client id, protocol version, and challenge. If the proof checks out, the server sends the final `NEW_CLIENT` or `RETURNING_CLIENT` response and creates or resumes the ServerClientConnection, which holds the BackedReader and BackedWriter used for EternalTCP buffering. A bad proof gets `INVALID_KEY`.
 
 The final response carries `resetProof`, a keyed proof over the challenge, `status`, `resetRequired`, and `resetSalt`, so it cannot be replayed into another handshake.
 
 ### Legacy handshake
 
-The challenge is a capability within protocol 6, not a version bump. A request without `supportsChallenge` gets the original single `ConnectResponse`, with no challenge or reset fields. A client that gets no `authChallenge` treats the first response as final; if it asked to reattach (`resetIntent`) and gets `RETURNING_CLIENT`, it fails with "Server does not support session reattach; upgrade etserver". The legacy path goes away at the next `PROTOCOL_VERSION` bump.
+The challenge is a capability within protocol 6, not a version bump. A request without `supportsChallenge` gets the original single `ConnectResponse`, with no challenge, reset fields, or `RETRY_LATER`. A client that gets no `authChallenge` treats the first response as final; if it asked to reattach (`resetIntent`) and gets `RETURNING_CLIENT`, it fails with "Server does not support session reattach; upgrade etserver". The legacy path goes away at the next `PROTOCOL_VERSION` bump.
 
 The client then sends an `INITIAL_PAYLOAD` (with an [InitialPayload](../proto/ETerminal.proto#L71-L78)), which contains port forwarding information or the jumphost flag, to which the server responds with an `INITIAL_RESPONSE` ([InitialResponse](../proto/ETerminal.proto#L80-L82)).  If there's an error during connect, the InitialResponse will contain an error string.
 
@@ -134,7 +134,9 @@ Based on this, a CatchupBuffer protobufs are swapped, containing the missing enc
 
 ### Reset recovery
 
-A fresh client process (`--attach`, or any initial connect) sets `ConnectRequest.resetIntent`, since it has no sequence history. When one side has lost its history, the final `ConnectResponse` sets `resetRequired` with a fresh `resetSalt`. Both peers echo the salt in their [SequenceHeader](../proto/ET.proto#L47-L55), exchange empty catchup buffers, and start over at sequence zero under a key derived from the salt. A `reset` bit that the authenticated handshake did not select is rejected.
+A fresh client process (`--attach`, or any initial connect) sets `ConnectRequest.resetIntent`, since it has no sequence history. When one side has lost its history, the final `ConnectResponse` sets `resetRequired` with a fresh `resetSalt`. Both peers echo the salt in their [SequenceHeader](../proto/ET.proto#L48-L56), exchange empty catchup buffers, and start over at sequence zero under a key derived from the salt. A `reset` bit that the authenticated handshake did not select is rejected.
+
+After an etserver restart, the surviving etterminal re-registers with `TerminalUserInfo.ptyactive` set, and the server resumes its shell instead of bootstrapping a new one. Until then clients get `RETRY_LATER`.
 
 ### Ending a session
 

proto/ET.proto

@@ -23,6 +23,7 @@ enum ConnectStatus {
   RETURNING_CLIENT = 2;
   INVALID_KEY = 3;
   MISMATCHED_PROTOCOL = 4;
+  RETRY_LATER = 5;
 }
 
 message ConnectResponse {

proto/ETerminal.proto

@@ -91,6 +91,7 @@ message TermInit {
   repeated string environmentvalues = 2;
   optional bool no_pty = 3 [default = false];
   optional string command = 4;
+  optional bool hadreversetunnels = 5 [default = false];
 }
 
 message TerminalUserInfo {
@@ -99,4 +100,11 @@ message TerminalUserInfo {
   optional int64 uid = 3;
   optional int64 gid = 4;
   optional int64 fd = 5;
+  // Set when a terminal re-registers after its router went away: the pty and
+  // its shell are already running, so the server must resume the session
+  // instead of bootstrapping a fresh one.
+  optional bool ptyactive = 6;
+  // Preserved by etterminal so a replacement server can warn that the
+  // server-side listeners from the original bootstrap no longer exist.
+  optional bool hadreversetunnels = 7 [default = false];
 }

src/base/ClientConnection.cpp

@@ -33,6 +33,9 @@ bool ClientConnection::connect() {
     et::ConnectResponse response;
     connectHandshake(socketFd, &response, resetOnConnect);
     lastStatus_.store(response.status());
+    if (response.status() == RETRY_LATER) {
+      throw std::runtime_error("Server is recovering; retry later");
+    }
     if (response.status() != NEW_CLIENT &&
         response.status() != RETURNING_CLIENT) {
       // Note: the response can be returning client if the client died while
@@ -205,7 +208,11 @@ void ClientConnection::pollReconnect() {
             socketHandler->close(newSocketFd);
             return;
           }
-          if (response.status() != RETURNING_CLIENT) {
+          if (response.status() == RETRY_LATER) {
+            LOG(INFO) << "Server is still recovering; retrying reconnect "
+                         "shortly.";
+            socketHandler->close(newSocketFd);
+          } else if (response.status() != RETURNING_CLIENT) {
             STERROR << "Error reconnecting to server: " << response.status()
                     << ": " << response.error();
             CLOG(INFO, "stdout")

src/base/ServerClientConnection.cpp

@@ -70,7 +70,8 @@ bool ServerClientConnection::recoverClient(int newSocketFd, bool forceReset,
 
   bool success = recover(newSocketFd, forceReset, resetSalt);
   if (success) {
-    if (oldSocketFd != -1) {
+    // A resumed connection was constructed with newSocketFd.
+    if (oldSocketFd != -1 && oldSocketFd != newSocketFd) {
       socketHandler->close(oldSocketFd);
     }
     return true;

src/base/ServerConnection.cpp

@@ -5,6 +5,7 @@ ServerConnection::ServerConnection(
     const SocketEndpoint& _serverEndpoint)
     : socketHandler(_socketHandler),
       serverEndpoint(_serverEndpoint),
+      startTime_(time(NULL)),
       clientHandlerThreadPool(new ThreadPool(8)) {
   socketHandler->listen(serverEndpoint);
 }
@@ -26,9 +27,13 @@ bool ServerConnection::acceptNewConnection(int fd) {
 }
 
 void ServerConnection::shutdown() {
-  lock_guard<std::recursive_mutex> guard(classMutex);
-  socketHandler->stopListening(serverEndpoint);
+  {
+    lock_guard<std::recursive_mutex> guard(classMutex);
+    socketHandler->stopListening(serverEndpoint);
+  }
+  // In-flight clientHandlers take classMutex, so join the pool without it.
   clientHandlerThreadPool.reset();
+  lock_guard<std::recursive_mutex> guard(classMutex);
   for (const auto& it : clientConnections) {
     it.second->shutdown();
   }
@@ -67,6 +72,7 @@ void ServerConnection::clientHandler(int clientSocketFd) {
     clientId = request.clientid();
     shared_ptr<ServerClientConnection> serverClientState = NULL;
     bool clientKeyExistsNow;
+    bool clientWasRemoved;
     string clientKey;
 
     {
@@ -80,6 +86,9 @@ void ServerConnection::clientHandler(int clientSocketFd) {
       if (clientKeyExistsNow) {
         clientKey = clientKeys.at(clientId);
       }
+      pruneRemovedClientIds(time(NULL));
+      clientWasRemoved =
+          removedClientIds.find(clientId) != removedClientIds.end();
     }
 
     // Legacy handshake for peers without the challenge capability. Remove at
@@ -119,7 +128,15 @@ void ServerConnection::clientHandler(int clientSocketFd) {
       std::ostringstream errorStream;
       errorStream << "Client is not registered";
       response.set_error(errorStream.str());
-      response.set_status(INVALID_KEY);
+      // Right after a restart, terminals may not have re-registered yet.
+      if (!legacyPeer && !clientWasRemoved &&
+          time(NULL) - startTime_ < recoveryGraceSeconds) {
+        LOG(INFO) << "Within the recovery grace window; asking client "
+                  << clientId << " to retry.";
+        response.set_status(RETRY_LATER);
+      } else {
+        response.set_status(INVALID_KEY);
+      }
       socketHandler->writeProto(clientSocketFd, response, true);
 
       socketHandler->close(clientSocketFd);
@@ -131,28 +148,59 @@ void ServerConnection::clientHandler(int clientSocketFd) {
       socketHandler->writeProto(clientSocketFd, response, true);
       socketHandler->close(clientSocketFd);
     } else if (createdClientConnection) {
+      // A known key with no connection is either a new session or a terminal
+      // that re-registered after an etserver restart.
+      const bool resume = shouldResumeAsReturning(clientId);
+      if (legacyPeer && resume) {
+        // Legacy peers can't reset; answer as a pre-restart-recovery server.
+        LOG(INFO) << "Legacy client " << clientId
+                  << " cannot resume after restart";
+        et::ConnectResponse response;
+        response.set_status(INVALID_KEY);
+        response.set_error("Client is not registered");
+        socketHandler->writeProto(clientSocketFd, response, true);
+        // The partial connection owns clientSocketFd and closes it.
+        destroyPartialConnection(clientId);
+        return;
+      }
+      const string resetSalt =
+          resume ? CryptoHandler::randomBytes(CryptoHandler::EPOCH_SALT_BYTES)
+                 : string();
+      const ConnectStatus status = resume ? RETURNING_CLIENT : NEW_CLIENT;
       et::ConnectResponse response;
-      response.set_status(NEW_CLIENT);
+      response.set_status(status);
       if (!legacyPeer) {
-        response.set_resetrequired(false);
-        response.set_resetsalt(string());
+        response.set_resetrequired(resume);
+        response.set_resetsalt(resetSalt);
         response.set_resetproof(CryptoHandler::resetDecisionProof(
-            clientKey, clientId, PROTOCOL_VERSION, authChallenge, NEW_CLIENT,
-            false, string()));
+            clientKey, clientId, PROTOCOL_VERSION, authChallenge, status,
+            resume, resetSalt));
       }
       socketHandler->writeProto(clientSocketFd, response, true);
 
-      LOG(INFO) << "New client.  Setting up connection";
-      VLOG(1) << "Created client with id " << clientId;
+      if (resume) {
+        LOG(INFO) << "Resuming existing session for " << clientId;
+        if (!serverClientState->recoverClient(clientSocketFd,
+                                              /*forceReset=*/true, resetSalt)) {
+          LOG(WARNING) << "Resume handshake failed for " << clientId;
+          // Keep the key: the terminal is still live.
+          destroyPartialConnection(clientId);
+        } else {
+          resumeClient(serverClientState);
+        }
+      } else {
+        LOG(INFO) << "New client.  Setting up connection";
+        VLOG(1) << "Created client with id " << clientId;
 
-      {
-        lock_guard<std::recursive_mutex> guard(classMutex);
+        {
+          lock_guard<std::recursive_mutex> guard(classMutex);
 
-        if (!newClient(serverClientState)) {
-          VLOG(1) << "newClient failed";
-          // Client creation failed, Destroy the new client
-          removeClient(clientId);
-          socketHandler->close(clientSocketFd);
+          if (!newClient(serverClientState)) {
+            VLOG(1) << "newClient failed";
+            // Client creation failed, Destroy the new client
+            removeClient(clientId);
+            socketHandler->close(clientSocketFd);
+          }
         }
       }
     } else {
@@ -225,6 +273,9 @@ bool ServerConnection::removeClient(const string& id) {
     if (clientKeys.find(id) == clientKeys.end()) {
       return false;
     }
+    const time_t now = time(NULL);
+    pruneRemovedClientIds(now);
+    removedClientIds[id] = now;
     clientKeys.erase(id);
     const auto it = clientConnections.find(id);
     if (it == clientConnections.end()) {
@@ -239,6 +290,17 @@ bool ServerConnection::removeClient(const string& id) {
   return true;
 }
 
+void ServerConnection::pruneRemovedClientIds(time_t now) {
+  for (auto it = removedClientIds.begin(); it != removedClientIds.end();) {
+    if (now >= it->second &&
+        now - it->second > static_cast<time_t>(recoveryGraceSeconds)) {
+      it = removedClientIds.erase(it);
+    } else {
+      ++it;
+    }
+  }
+}
+
 void ServerConnection::destroyPartialConnection(const string& clientId) {
   shared_ptr<ServerClientConnection> connection;
   {

src/base/ServerConnection.hpp

@@ -52,6 +52,7 @@ class ServerConnection {
   inline void addClientKey(const string& id, const string& passkey) {
     lock_guard<std::recursive_mutex> guard(classMutex);
     clientKeys[id] = passkey;
+    removedClientIds.erase(id);
   }
 
   /**
@@ -64,14 +65,20 @@ class ServerConnection {
    */
   bool removeClient(const string& id);
 
-  shared_ptr<ServerClientConnection> getClientConnection(
+  shared_ptr<ServerClientConnection> tryGetClientConnection(
       const string& clientId) {
     lock_guard<std::recursive_mutex> guard(classMutex);
     auto it = clientConnections.find(clientId);
-    if (it == clientConnections.end()) {
+    return it == clientConnections.end() ? nullptr : it->second;
+  }
+
+  shared_ptr<ServerClientConnection> getClientConnection(
+      const string& clientId) {
+    auto connection = tryGetClientConnection(clientId);
+    if (!connection) {
       STFATAL << "Error: Tried to get a client connection that doesn't exist";
     }
-    return it->second;
+    return connection;
   }
 
   /**
@@ -81,6 +88,11 @@ class ServerConnection {
   virtual bool newClient(
       shared_ptr<ServerClientConnection> serverClientState) = 0;
 
+  // A known id whose terminal re-registered with a live pty is resumed
+  // instead of bootstrapped.
+  virtual bool shouldResumeAsReturning(const string& clientId) { return false; }
+  virtual void resumeClient(shared_ptr<ServerClientConnection> state) {}
+
  protected:
   bool authenticateClient(int clientSocketFd, const string& clientId,
                           const string& clientKey, int protocolVersion,
@@ -97,6 +109,8 @@ class ServerConnection {
   SocketEndpoint serverEndpoint;
   /** @brief Map of client IDs to their registered passkeys. */
   std::unordered_map<string, string> clientKeys;
+  // Client ids whose sessions ended in this process, with removal time.
+  std::unordered_map<string, time_t> removedClientIds;
   /** @brief Active client connections indexed by ID. */
   std::unordered_map<string, shared_ptr<ServerClientConnection>>
       clientConnections;
@@ -106,6 +120,12 @@ class ServerConnection {
   recursive_mutex classMutex;
   /** @brief Serializes connect/disconnect events. */
   mutex connectMutex;
+  // After a restart, unknown ids get RETRY_LATER instead of INVALID_KEY for
+  // this long so terminals can re-register.
+  int recoveryGraceSeconds = 60;
+  time_t startTime_;
+
+  void pruneRemovedClientIds(time_t now);
 };
 }  // namespace et