Make the deploy crash-recovery restart respect COMPOSE_FILE (#441) (#452)

deploy/hetzner/remote-deploy.sh's crash-recovery trap hard-coded
docker-compose -f docker-compose.yml, dropping whatever overlay the
deployment actually runs with. The demo host runs docker-compose.yml +
docker-compose.demo.yml, and the overlay is what enables the public
read-only lock and the worker's CPU cap - a crash-recovery restart could
silently bring the stack back up without either.

scripts/lib/compose_args.sh extracts the COMPOSE_FILE-resolution logic
scripts/upgrade.sh already had into a shared, sourceable function, so the
trap and the demo-reseed exec (which had its own separate, cruder version)
both use the real overlay. Resolved once before the crash-recovery trap is
installed (from .env, which is untouched by the tag checkout) and again
after checkout, so the trap has a correct value even if the fetch/checkout
step itself is what failed, or the deploy is a rollback to a tag predating
this file.

Writing a test that actually executes the shared function (rather than
re-describing the bash in Python) caught a real bug in the first draft:
its last statement was a bare `[ cond ] && action`, which is a safe no-op
under set -e as a top-level script statement but becomes the function's
own failing return status once wrapped in a function - aborting the
entire deploy under set -e any time COMPOSE_FILE resolved to at least one
file, which is every real deployment including the demo server's own
config.


Claude-Session: https://claude.ai/code/session_01Q44qCG1TEkZxDo9cb9voYT

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Author: haksungjang

apps/backend/tests/unit/test_compose_args_shared_helper.py

@@ -0,0 +1,91 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 TRUSCA contributors
+"""
+scripts/lib/compose_args.sh's compose_args_from_env() (#441).
+
+Neither this helper nor the inline logic it replaced in scripts/upgrade.sh
+had ever been executed by anything CI runs - both were exercised only by an
+operator actually deploying. deploy/hetzner/remote-deploy.sh's own
+crash-recovery restart hard-coded `-f docker-compose.yml` with no COMPOSE_FILE
+handling at all before this change (#441), which this repo's shellcheck gate
+cannot catch (it checks syntax, not which env var wins). This drives the real
+function through subprocess with a real temp directory and .env file rather
+than re-describing the bash in Python, which would test the description
+instead of the shell.
+"""
+
+from __future__ import annotations
+
+import subprocess
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parents[4]
+COMPOSE_ARGS_SH = REPO_ROOT / "scripts" / "lib" / "compose_args.sh"
+
+
+def _compose_args(tmp_path: Path, *, env: dict[str, str] | None = None) -> list[str]:
+    """Source the helper and call compose_args_from_env() in tmp_path, cwd'd
+    there the way every real caller (upgrade.sh, remote-deploy.sh) already
+    does, and return the resulting COMPOSE_ARGS elements."""
+    script = (
+        f'cd "{tmp_path}" && source "{COMPOSE_ARGS_SH}" && compose_args_from_env '
+        '&& printf "%s\\n" "${COMPOSE_ARGS[@]}"'
+    )
+    result = subprocess.run(
+        ["bash", "-c", script],
+        capture_output=True,
+        text=True,
+        env=env,
+        timeout=10,
+        check=True,
+    )
+    return result.stdout.splitlines()
+
+
+def test_no_env_file_and_no_compose_file_var_defaults_to_the_base_file(
+    tmp_path: Path,
+) -> None:
+    assert _compose_args(tmp_path, env={"PATH": "/usr/bin:/bin"}) == ["-f", "docker-compose.yml"]
+
+
+def test_compose_file_already_in_the_environment_wins_over_env_file(
+    tmp_path: Path,
+) -> None:
+    (tmp_path / ".env").write_text("COMPOSE_FILE=ignored.yml\n")
+    args = _compose_args(
+        tmp_path,
+        env={"PATH": "/usr/bin:/bin", "COMPOSE_FILE": "docker-compose.yml:docker-compose.demo.yml"},
+    )
+    assert args == ["-f", "docker-compose.yml", "-f", "docker-compose.demo.yml"]
+
+
+def test_compose_file_declared_only_in_dot_env_is_picked_up(tmp_path: Path) -> None:
+    """The actual bug this PR fixes: a deployment declares its overlay in
+    .env (the documented way), not as a shell-exported variable."""
+    (tmp_path / ".env").write_text("COMPOSE_FILE=docker-compose.yml:docker-compose.demo.yml\n")
+    args = _compose_args(tmp_path, env={"PATH": "/usr/bin:/bin"})
+    assert args == ["-f", "docker-compose.yml", "-f", "docker-compose.demo.yml"]
+
+
+def test_dot_env_with_no_compose_file_line_defaults_to_the_base_file(
+    tmp_path: Path,
+) -> None:
+    (tmp_path / ".env").write_text("SOME_OTHER_KEY=value\n")
+    assert _compose_args(tmp_path, env={"PATH": "/usr/bin:/bin"}) == ["-f", "docker-compose.yml"]
+
+
+def test_a_trailing_colon_does_not_produce_an_empty_dash_f(tmp_path: Path) -> None:
+    (tmp_path / ".env").write_text("COMPOSE_FILE=docker-compose.yml:\n")
+    args = _compose_args(tmp_path, env={"PATH": "/usr/bin:/bin"})
+    assert args == ["-f", "docker-compose.yml"]
+
+
+def test_a_compose_file_that_resolves_to_nothing_falls_back_to_the_base_file(
+    tmp_path: Path,
+) -> None:
+    """COMPOSE_FILE set but entirely colons (":::") is the pathological
+    input that must not leave COMPOSE_ARGS empty - an empty array means every
+    caller's `docker-compose "${COMPOSE_ARGS[@]}" up -d` runs with no -f at
+    all, silently falling through to compose's own default file discovery."""
+    args = _compose_args(tmp_path, env={"PATH": "/usr/bin:/bin", "COMPOSE_FILE": ":::"})
+    assert args == ["-f", "docker-compose.yml"]

deploy/hetzner/remote-deploy.sh

@@ -37,6 +37,29 @@ IMG="${TAG#v}"   # image tags are published without the leading 'v' (see release
 
 cd "$REMOTE_PATH"
 
+# Compose file selection - follow the deploy's own overlay (#441). Shared
+# with scripts/upgrade.sh (scripts/lib/compose_args.sh's own docstring has
+# the full "why": a docker-compose call that drops the demo overlay silently
+# turns off the public read-only lock and the worker's CPU cap).
+#
+# Resolved TWICE, deliberately. .env (gitignored, untouched by the checkout
+# below) already holds the real COMPOSE_FILE before anything else in this
+# script runs, so waiting until after the checkout to resolve it at all would
+# leave restore_stack_on_failure's trap using the wrong overlay for the
+# entire fetch/checkout window - not just a first-deploy edge case, but every
+# ROLLBACK to a tag predating this file, where checkout would otherwise be
+# the only thing standing between the trap and a correct COMPOSE_ARGS
+# (security review on #441). Falling back to a bare default here only when
+# the shared helper genuinely does not exist yet on disk keeps that window
+# closed without hand-duplicating compose_args_from_env's own logic.
+if [ -f "$REMOTE_PATH/scripts/lib/compose_args.sh" ]; then
+  # shellcheck source=../../scripts/lib/compose_args.sh
+  source "$REMOTE_PATH/scripts/lib/compose_args.sh"
+  compose_args_from_env
+else
+  COMPOSE_ARGS=(-f docker-compose.yml)
+fi
+
 # Bring the stack back if any step below fails. upgrade.sh recreates services
 # one at a time, so a failure partway through leaves the earlier ones stopped —
 # a failed deploy became an outage that the next deploy could not clear on its
@@ -46,7 +69,7 @@ restore_stack_on_failure() {
   rc=$?
   [ "$rc" -eq 0 ] && exit 0
   echo "==> deploy failed (exit $rc) — attempting to restart the stack" >&2
-  if docker-compose -f docker-compose.yml up -d >&2; then
+  if docker-compose "${COMPOSE_ARGS[@]}" up -d >&2; then
     echo "==> stack is back up; the deploy itself still FAILED (exit $rc)" >&2
   else
     echo "==> could not restart the stack — it needs manual attention" >&2
@@ -67,6 +90,20 @@ git checkout -f "tags/$TAG"
 # logs (the fetch uses --force, so a moved upstream tag is accepted silently).
 echo "    $TAG resolves to commit $(git rev-parse --short HEAD)"
 
+# Re-resolve now that TAG's own copy of scripts/lib/compose_args.sh (or its
+# absence, on a rollback older than this file) is what is actually on disk.
+# If TAG predates this file, this deliberately lets `source` fail the script
+# via `set -e` rather than silently keeping a possibly-stale COMPOSE_ARGS:
+# restore_stack_on_failure still has the correct value from the resolution
+# above when that happens, since COMPOSE_ARGS is only overwritten once this
+# source succeeds.
+if [ -f "$REMOTE_PATH/scripts/lib/compose_args.sh" ]; then
+  # shellcheck source=../../scripts/lib/compose_args.sh
+  source "$REMOTE_PATH/scripts/lib/compose_args.sh"
+  compose_args_from_env
+fi
+echo "==> compose files: ${COMPOSE_ARGS[*]}"
+
 echo "==> pinning IMAGE_TAG=$IMG in .env"
 if [ ! -f .env ]; then
   echo "remote-deploy: .env not found in $REMOTE_PATH — run scripts/install.sh first" >&2
@@ -117,10 +154,6 @@ NO_PROMPT=1 UPGRADE_SKIP_DRAIN=1 bash scripts/upgrade.sh
 # transaction. Accounts outside it are untouched.
 if [ "${RESEED:-0}" = "1" ]; then
   echo "==> reseeding the demo dataset (RESEED=1)"
-  compose_args="-f docker-compose.yml"
-  if [ -f .env ] && grep -qE '^COMPOSE_FILE=' .env; then
-    compose_args=""   # docker-compose reads COMPOSE_FILE from .env itself
-  fi
   # < /dev/null: this whole script runs as `bash -s < remote-deploy.sh` over SSH,
   # so its own stdin is the SSH channel still streaming the rest of THIS file.
   # `docker-compose exec` keeps stdin open by default (-T only disables the
@@ -130,8 +163,7 @@ if [ "${RESEED:-0}" = "1" ]; then
   # error and no visible sign the tail never ran. Confirmed 2026-09-08: a
   # RESEED=1 deploy reported success with zero reset_demo output in the
   # backend logs and stale (pre-reseed) row IDs still live.
-  # shellcheck disable=SC2086  # compose_args is our own literal, word-splitting intended.
-  docker-compose $compose_args exec -T -e APP_ENV=demo backend python -m scripts.reset_demo < /dev/null
+  docker-compose "${COMPOSE_ARGS[@]}" exec -T -e APP_ENV=demo backend python -m scripts.reset_demo < /dev/null
   echo "==> reseed complete"
 else
   echo "==> skipping demo reseed (set RESEED=1 to rebuild the demo dataset)"

scripts/lib/compose_args.sh

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# SPDX-License-Identifier: Apache-2.0
+# Copyright 2026 TRUSCA contributors
+# TrustedOSS Portal - shared docker-compose file-selection helper.
+#
+# Exposes ``compose_args_from_env`` which sets the caller's ``COMPOSE_ARGS``
+# array to the ``-f`` flags a deployment's overlay actually needs.
+#
+# Usage:
+#   source "$ROOT_DIR/scripts/lib/compose_args.sh"
+#   compose_args_from_env
+#   docker-compose "${COMPOSE_ARGS[@]}" up -d
+#
+# Why this exists:
+#   scripts/upgrade.sh used to hard-code `-f docker-compose.yml` on every
+#   docker-compose call, which silently DROPS whatever overlay a deployment
+#   actually runs with. An explicit `-f` also overrides the standard
+#   COMPOSE_FILE variable, so declaring the overlay the documented way (in
+#   .env) had no effect there either. That was fixed in upgrade.sh directly;
+#   this extracts the same logic so deploy/hetzner/remote-deploy.sh's own
+#   docker-compose calls (the crash-recovery restart, and the demo reseed)
+#   can share it instead of re-deriving it a third time by hand (#441 - the
+#   reseed branch had its own ad-hoc, plain-string version, and the
+#   crash-recovery restart had no COMPOSE_FILE handling at all).
+#
+#   Concretely: the demo host runs `docker-compose.yml` +
+#   `docker-compose.demo.yml`, and the overlay is what passes
+#   DEMO_READ_ONLY into the backend and caps the worker at the box's 2 CPUs.
+#   A docker-compose call that drops the overlay rebuilds the stack WITHOUT
+#   the public read-only lock and with the 4.0 CPU default - a deploy or a
+#   crash-recovery restart quietly turning off a safety boundary.
+#
+# COMPOSE_FILE is read from .env when the environment does not already carry
+# it, because that is where an operator declares it and where docker-compose
+# itself looks. Unset (the single-file default) keeps the previous
+# behaviour. Must be called from the directory holding docker-compose.yml
+# and .env (every caller already `cd`s there first).
+
+compose_args_from_env() {
+  if [ -z "${COMPOSE_FILE:-}" ] && [ -f .env ]; then
+    COMPOSE_FILE="$(grep -E '^COMPOSE_FILE=' .env | tail -1 | cut -d= -f2- || true)"
+    if [ -n "$COMPOSE_FILE" ]; then
+      export COMPOSE_FILE
+    fi
+  fi
+  COMPOSE_ARGS=(-f docker-compose.yml)
+  if [ -n "${COMPOSE_FILE:-}" ]; then
+    COMPOSE_ARGS=()
+    local _f _compose_files
+    IFS=':' read -ra _compose_files <<< "$COMPOSE_FILE"
+    for _f in "${_compose_files[@]}"; do
+      [ -n "$_f" ] && COMPOSE_ARGS+=(-f "$_f")
+    done
+    if [ ${#COMPOSE_ARGS[@]} -eq 0 ]; then
+      COMPOSE_ARGS=(-f docker-compose.yml)
+    fi
+  fi
+  # Explicit, unconditional success: the two guards above are `if` blocks
+  # whose CONDITION can be false with no `else`, and under `set -e` (every
+  # caller sets it) a function's return status is its LAST executed
+  # command's - a false `if` condition with no `else` still exits 0 for the
+  # `if` construct itself, but this makes it impossible for a future edit to
+  # accidentally leave a bare `[ cond ] && action` as the final statement
+  # here, which WOULD make this function's return status (and therefore
+  # `compose_args_from_env && next_thing` at every call site) fail whenever
+  # cond happens to be false. That exact bug shipped once in this file's
+  # first version and was only caught by the test suite, not by shellcheck.
+  return 0
+}

scripts/upgrade.sh

@@ -53,22 +53,12 @@ command -v docker-compose >/dev/null 2>&1 || fail "docker-compose (V1) is requir
 # the base file alone rebuilt the stack WITHOUT the public read-only lock and
 # with the 4.0 CPU default — a deploy quietly turning off a safety boundary.
 #
-# COMPOSE_FILE is read from .env when the environment does not already carry
-# it, because that is where an operator declares it and where docker-compose
-# itself looks. Unset (the single-file default) keeps the previous behaviour.
-if [ -z "${COMPOSE_FILE:-}" ] && [ -f .env ]; then
-  COMPOSE_FILE="$(grep -E '^COMPOSE_FILE=' .env | tail -1 | cut -d= -f2- || true)"
-  [ -n "$COMPOSE_FILE" ] && export COMPOSE_FILE
-fi
-COMPOSE_ARGS=(-f docker-compose.yml)
-if [ -n "${COMPOSE_FILE:-}" ]; then
-  COMPOSE_ARGS=()
-  IFS=':' read -ra _compose_files <<< "$COMPOSE_FILE"
-  for _f in "${_compose_files[@]}"; do
-    [ -n "$_f" ] && COMPOSE_ARGS+=(-f "$_f")
-  done
-  [ ${#COMPOSE_ARGS[@]} -eq 0 ] && COMPOSE_ARGS=(-f docker-compose.yml)
-fi
+# The selection logic itself lives in scripts/lib/compose_args.sh (#441) so
+# deploy/hetzner/remote-deploy.sh's own docker-compose calls share it instead
+# of re-deriving it by hand.
+# shellcheck source=scripts/lib/compose_args.sh
+source "$ROOT_DIR/scripts/lib/compose_args.sh"
+compose_args_from_env
 note "compose files: ${COMPOSE_ARGS[*]}"
 
 # ---------------------------------------------------------------------------