fix: recover cleanly from OAuth authorization failures

Author: sumitake

CHANGELOG.md

@@ -2,6 +2,16 @@
 
 All notable project changes are recorded here. The README shows only the most recent entry.
 
+## 2026-07-14 - OAuth-Blocked Recovery
+
+- Classified refresh-token `invalid_grant` responses and unusable token files as explicit OAuth authorization blocks instead of ambiguous retry failures.
+- Added a mode-`0600` cross-process refresh lock and a single exact request replay when YouTube rejects a cached access token with HTTP 401.
+- Prevented a second post-refresh 401 from creating a supervisor busy loop by waiting on the post-refresh token fingerprint.
+- Added bounded local credential fingerprinting and signal-interruptible waiting that makes no API calls until token content changes, then resumes automatically.
+- Preserved already-public FFmpeg ingest under its local watchdog during an OAuth block and revalidated the exact cached lifecycle state after reauthorization.
+- Limited custom child-stopping signal handlers to visible test-pattern runs so device authorization terminates normally.
+- Added regression coverage and operator documentation for revoked, missing, malformed, unreadable, and oversized token states.
+
 ## 2026-07-11 - Description-Neutral Lifecycle Identity
 
 - Stopped setting `liveBroadcast` and `liveStream` descriptions; new insert payloads omit the description field entirely and privacy updates remain status-only.

README.md

@@ -61,6 +61,7 @@ Important files:
 - `youtube-autoencoder.env`: private service configuration.
 - `google-oauth-client.json`: Google OAuth client configuration.
 - `youtube-token.json`: OAuth access and refresh token cache.
+- `youtube-token.json.lock`: mode-`0600` advisory lock serializing access-token refreshes across the encoder, API helper, authorization flow, and optional telemetry.
 - `youtube-live-state.json`: versioned, non-secret lifecycle cache containing exact resource IDs, local create intent, privacy, lifecycle, and retry metadata.
 - `youtube-live-state.lock`: serialized mutation lock for create, bind, transition, privacy, and explicit completion operations.
 - `supervisor.lock`: process-lifetime lock preventing two encoder supervisors from running concurrently.
@@ -131,7 +132,8 @@ See the [recovery state machine](docs/architecture-and-flows.md#recovery-state-m
 | Insert response is lost or ambiguous | The durable `verify_create` intent polls under ambiguous backoff and never issues another insert automatically. |
 | Lifecycle state is missing or corrupt | Legacy markers may be read once for migration; otherwise conflicting same-stream or same-title resources block creation until an operator reconciles ownership. |
 | Ambiguous or unknown remote state | Reconciliation fails closed and creates nothing until the ambiguity is resolved. |
-| OAuth access token expires | API helper refreshes from the stored refresh token. |
+| OAuth access token expires locally or YouTube rejects the cached token with HTTP 401 | The API helper performs one serialized refresh and replays the exact buffered JSON request once. Concurrent helpers reuse the first successful refresh. |
+| OAuth refresh token is missing, invalid, expired, revoked, unreadable, or malformed | API mutation pauses without entering ambiguous backoff. The service waits for the token file to change and resumes automatically after `youtube-autoencoder-api authorize` saves a replacement. If a verified public stream is already running, FFmpeg remains under its local progress watchdog while the control plane is blocked. |
 | Previous broadcast is `complete`, `revoked`, or confirmed missing | One new unlisted generation may be staged after the source probe passes. |
 
 Recovery deadlines survive service and host restarts. Exponential backoff uses these class floors and caps:
@@ -143,6 +145,10 @@ Recovery deadlines survive service and host restarts. Exponential backoff uses t
 | Quota | 15 minutes | 6 hours | `userRequestsExceedRateLimit`, quota errors, HTTP 429. |
 | Ambiguous | 5 minutes | 1 hour | Multiple managed candidates, unknown lifecycle, conflicting state. |
 
+OAuth authorization failures are blocked conditions, not retry classes. They do not consume API quota while the token file is unchanged.
+
+Token refresh serialization uses an advisory `fcntl` lock and therefore requires a local filesystem. Every process that can refresh or replace the token must use the same lock path; override `YTA_YOUTUBE_TOKEN_REFRESH_LOCK_FILE` only when the encoder, helper, authorization command, and telemetry share that exact local path.
+
 The project never sets or updates YouTube `liveBroadcast` or `liveStream` descriptions. Normal recovery validates the exact broadcast ID in the private schema-v3 state file. Before a new insert, the helper persists the required title, scheduled start, privacy, creation window, and stream relationship; if the insert outcome is ambiguous, those fields may identify exactly one remote candidate, but they can never authorize another automatic insert. Legacy description markers are read only during one-way migration from schema v2.
 
 ### Test Pattern Flow
@@ -399,6 +405,7 @@ systemctl --user enable --now youtube-autoencoder.service
 | `invalid_client` | OAuth client type does not support the device-code flow. | Create a client for TVs and Limited Input devices, then replace `google-oauth-client.json`. |
 | `authorization_pending` | The browser approval has not completed yet. | Finish the device-code flow; the CLI will keep polling until the code expires. |
 | `slow_down` | Polling is too frequent. | The helper backs off automatically. |
+| `invalid_grant` | The refresh token expired, was revoked, or no longer belongs to the OAuth client. Testing-mode authorizations commonly expire after seven days. | Move the OAuth app to In production for unattended use, rerun `youtube-autoencoder-api authorize`, and select the account that owns or manages the intended channel. The running service detects the saved token and resumes automatically. |
 | Token works briefly then expires | App is still in Testing mode. | Add the correct test user for setup, then move the app to In production for unattended use and complete required verification. |
 | API calls fail despite valid OAuth | The account does not own/manage the YouTube channel, live streaming is not enabled, or quota/policy blocks the operation. | Reauthorize with the right channel account, enable live streaming, and check project quota and YouTube Studio restrictions. |
 
@@ -521,10 +528,12 @@ Each eligible collection performs one [`videos.list`](https://developers.google.
 
 Only the most recent changelog entry is shown here. See `CHANGELOG.md` for full history.
 
-### 2026-07-11 - Description-Neutral Lifecycle Identity
+### 2026-07-14 - OAuth-Blocked Recovery
 
-- New YouTube stream and broadcast inserts omit the description field, and normal lifecycle recovery never updates descriptions.
-- Schema-v3 exact-ID state and a durable create fingerprint retain duplicate-safe recovery; legacy markers are read only for one-way migration.
+- Refresh-token rejection and unusable token files now enter an explicit OAuth-blocked state instead of ambiguous retry backoff.
+- A cached token rejected with HTTP 401 now receives one cross-process serialized refresh and one exact request replay; a second rejection blocks on the post-refresh credential state without looping.
+- The supervisor waits without API calls until token content changes, preserves an already-public FFmpeg stream under its watchdog, and revalidates exact lifecycle state after reauthorization.
+- Device authorization keeps normal termination semantics; child-specific signal handling is limited to the visible test-pattern command.
 
 ## Repository Layout
 

bin/youtube-autoencoder

@@ -8,6 +8,7 @@ import contextlib
 import dataclasses
 import datetime as dt
 import fcntl
+import hashlib
 import json
 import os
 import pathlib
@@ -33,6 +34,11 @@ OBS_SERVICE_FILE = pathlib.Path(os.environ.get("YTA_OBS_SERVICE_FILE", OBS_PROFI
 OBS_SCENE_FILE = pathlib.Path(
     os.environ.get("YTA_OBS_SCENE_FILE", HOME / ".config/obs-studio/basic/scenes/Untitled.json")
 ).expanduser()
+CONFIG_DIR = pathlib.Path(os.environ.get("YTA_CONFIG_DIR", HOME / ".config/youtube-autoencoder")).expanduser()
+OAUTH_TOKEN_FILE = pathlib.Path(
+    os.environ.get("YTA_YOUTUBE_TOKEN_FILE", CONFIG_DIR / "youtube-token.json")
+).expanduser()
+OAUTH_TOKEN_MAX_BYTES = 1024 * 1024
 
 child: subprocess.Popen[Any] | None = None
 stopping = False
@@ -83,6 +89,7 @@ class ApiCommandError(RuntimeError):
         self.http_status = self.payload.get("http_status")
         self.reasons = tuple(str(reason) for reason in self.payload.get("reasons") or [])
         self.retry_class = str(self.payload.get("retry_class") or "fatal")
+        self.token_updated = self.payload.get("token_updated") is True
         self.public_fallback_allowed = self.payload.get("public_fallback_allowed") is not False
         retry_after = self.payload.get("retry_after")
         try:
@@ -199,9 +206,65 @@ def lifecycle_action(lifecycle: str) -> str:
 
 
 def classify_api_error(error: ApiCommandError) -> str:
+    if error.retry_class == "oauth":
+        return "oauth"
     return error.retry_class if error.retry_class in RETRY_LIMITS else "ambiguous"
 
 
+def oauth_token_fingerprint() -> tuple[Any, ...]:
+    try:
+        token_stat = OAUTH_TOKEN_FILE.stat()
+    except FileNotFoundError:
+        return ("missing",)
+    except OSError as exc:
+        return ("unreadable", exc.errno)
+
+    metadata = (
+        token_stat.st_dev,
+        token_stat.st_ino,
+        token_stat.st_mode & 0o7777,
+        token_stat.st_uid,
+        token_stat.st_gid,
+        token_stat.st_size,
+        token_stat.st_mtime_ns,
+    )
+    if token_stat.st_size > OAUTH_TOKEN_MAX_BYTES:
+        return ("oversized", *metadata)
+    try:
+        with OAUTH_TOKEN_FILE.open("rb") as handle:
+            raw = handle.read(OAUTH_TOKEN_MAX_BYTES + 1)
+    except OSError as exc:
+        return ("unreadable", *metadata, exc.errno)
+    if len(raw) > OAUTH_TOKEN_MAX_BYTES:
+        return ("oversized", *metadata)
+    return ("readable", *metadata, hashlib.sha256(raw).hexdigest())
+
+
+def oauth_blocked_poll_interval() -> float:
+    return max(0.1, float(env("YTA_OAUTH_BLOCKED_POLL_SEC", "5")))
+
+
+def oauth_wait_baseline(error: ApiCommandError, pre_attempt: tuple[Any, ...]) -> tuple[Any, ...]:
+    return oauth_token_fingerprint() if error.token_updated else pre_attempt
+
+
+def wait_for_oauth_token_change(
+    baseline: tuple[Any, ...],
+    *,
+    runtime: StreamRuntime | None = None,
+) -> bool:
+    while not stopping:
+        if oauth_token_fingerprint() != baseline:
+            log("OAuth credential change detected; resuming YouTube API recovery")
+            return True
+        duration = oauth_blocked_poll_interval()
+        if runtime is None:
+            sleep_interruptibly(duration)
+        else:
+            wait_with_runtime(runtime, duration)
+    return False
+
+
 def retry_delay(
     retry_class: str,
     *,
@@ -906,32 +969,44 @@ def revalidate_public_state(runtime: StreamRuntime, state: dict[str, Any]) -> di
 
 
 def manage_with_public_fallback(runtime: StreamRuntime, cached_state: dict[str, Any]) -> dict[str, Any]:
-    try:
-        return manage_youtube_lifecycle(runtime, prepared_state=cached_state)
-    except ApiCommandError as error:
-        if not verified_public_state(cached_state) or not error.public_fallback_allowed:
-            raise
-        ensure_encoder_running(runtime)
-        retry_class = classify_api_error(error)
-        attempt = retry_attempt(cached_state, retry_class)
-        delay = retry_delay(retry_class, attempt=attempt, retry_after=error.retry_after)
-        not_before = retry_not_before(delay)
+    while True:
+        token_fingerprint = oauth_token_fingerprint()
         try:
-            api_command_while_streaming(
-                ["set-retry", retry_class, str(attempt), not_before],
-                runtime,
-                timeout=30,
+            return manage_youtube_lifecycle(runtime, prepared_state=cached_state)
+        except ApiCommandError as error:
+            if not verified_public_state(cached_state) or not error.public_fallback_allowed:
+                raise
+            ensure_encoder_running(runtime)
+            retry_class = classify_api_error(error)
+            if retry_class == "oauth":
+                log(
+                    "YouTube OAuth authorization required; preserving verified public stream and waiting for "
+                    "credential change. Run: youtube-autoencoder-api authorize. "
+                    f"error={redact_text(str(error))}"
+                )
+                wait_baseline = oauth_wait_baseline(error, token_fingerprint)
+                if not wait_for_oauth_token_change(wait_baseline, runtime=runtime):
+                    return dict(cached_state)
+                continue
+            attempt = retry_attempt(cached_state, retry_class)
+            delay = retry_delay(retry_class, attempt=attempt, retry_after=error.retry_after)
+            not_before = retry_not_before(delay)
+            try:
+                api_command_while_streaming(
+                    ["set-retry", retry_class, str(attempt), not_before],
+                    runtime,
+                    timeout=30,
+                )
+            except EncoderStopped:
+                raise
+            except Exception as persist_error:  # noqa: BLE001
+                log(f"could not persist public-stream API cooldown: {redact_text(str(persist_error))}")
+            ensure_encoder_running(runtime)
+            log(
+                "YouTube API unavailable; preserving verified public stream without mutation "
+                f"class={retry_class} attempt={attempt} retry_not_before={not_before}"
             )
-        except EncoderStopped:
-            raise
-        except Exception as persist_error:  # noqa: BLE001
-            log(f"could not persist public-stream API cooldown: {redact_text(str(persist_error))}")
-        ensure_encoder_running(runtime)
-        log(
-            "YouTube API unavailable; preserving verified public stream without mutation "
-            f"class={retry_class} attempt={attempt} retry_not_before={not_before}"
-        )
-        return dict(cached_state)
+            return dict(cached_state)
 
 
 @contextlib.contextmanager
@@ -1082,13 +1157,22 @@ def supervisor_loop() -> int:
             sleep_interruptibly(cooldown)
             continue
         try:
+            token_fingerprint = oauth_token_fingerprint()
             rc = run_once(cached_state=state)
             if stopping:
                 break
             raise EncoderStopped(f"ffmpeg stream attempt exited rc={rc}")
         except Exception as exc:  # noqa: BLE001
             if stopping:
                 break
+            if isinstance(exc, ApiCommandError) and classify_api_error(exc) == "oauth":
+                log(
+                    "YouTube OAuth authorization required; pausing API recovery until the credential changes. "
+                    "Run: youtube-autoencoder-api authorize. "
+                    f"error={redact_text(str(exc))}"
+                )
+                wait_for_oauth_token_change(oauth_wait_baseline(exc, token_fingerprint))
+                continue
             retry_class = failure_retry_class(exc)
             try:
                 state = read_recovery_state()