fix(pro): enforce signed license heartbeat lifecycle (#312)

* fix(pro): enforce signed license heartbeat lifecycle

* fix(pro): fail heartbeat timeout promptly on Python 3.10

Author: rylinjames

infra/license-worker/README.md

@@ -13,9 +13,10 @@ Pairs with the Phase 1 telemetry worker at `infra/telemetry-worker/` (different
 | `/admin/issue` | POST | bearer | Sign + store new license, return activation code |
 | `/admin/revoke` | POST | bearer | Revoke a license_id |
 | `/admin/list` | GET | bearer | List all licenses with status + heartbeat info |
+| `/admin/signer` | GET | bearer | Cryptographically verify `SIGNING_KEY_ID` matches `PRIVATE_KEY` |
 | `/v1/pubkey` | GET | none | Return current Ed25519 public key (for offline verify) |
-| `/v1/activation/:code` | GET | none | Fetch signed license by one-time code (24h TTL, single-use) |
-| `/v1/heartbeat` | POST | none | Record heartbeat + check revocation/expiry |
+| `/v1/activation/:code` | POST | one-time code | Bind hardware and fetch signed v2 license (24h TTL, single-use) |
+| `/v1/heartbeat` | POST | nonce-bound | Record heartbeat + return signed status attestation |
 | `/v1/revocation/:license_id` | GET | none | Check if a license is revoked |
 
 ## Deploy (one-time setup, ~25 min)
@@ -53,9 +54,10 @@ curl -X POST https://reflex-licenses.<subdomain>.workers.dev/admin/init \
 # IMMEDIATELY:
 #   a. Copy the private_key_b64 and set it as a Worker Secret:
 #        echo '<private_key_b64>' | wrangler secret put PRIVATE_KEY
-#   b. Copy the public_key_b64 into src/tether/pro/_public_key.py
-#      (replace the BUNDLED_PUBLIC_KEY_B64 constant)
-#   c. Discard the private_key_b64 from your terminal scrollback (it never
+#   b. Bind the private key to the exact D1 public-key row:
+#        echo '<key_id>' | wrangler secret put SIGNING_KEY_ID
+#   c. Add key_id → public_key_b64 to TRUSTED_PUBLIC_KEYS_B64
+#   d. Discard the private_key_b64 from your terminal scrollback (it never
 #      needs to leave wrangler again)
 #
 # 9. Update src/tether/pro/license.py:DEFAULT_LICENSE_ENDPOINT to your worker URL
@@ -66,6 +68,30 @@ curl https://reflex-licenses.<subdomain>.workers.dev/healthz
 curl https://reflex-licenses.<subdomain>.workers.dev/v1/pubkey
 ```
 
+### Existing-install migration
+
+Older installs may already have one active D1 public-key row and a matching
+`PRIVATE_KEY`, but no `SIGNING_KEY_ID`. Migrate that install before deploying
+the stricter Worker:
+
+1. Query the active rows directly with `wrangler d1 execute reflex-licenses
+   --remote --json --command "SELECT key_id, public_key_b64 FROM master_keys
+   WHERE retired_at IS NULL"`.
+2. Compare the row's public key with the key already shipped in
+   `TRUSTED_PUBLIC_KEYS_B64`. If more than one row is active, do not guess: set
+   `TETHER_SIGNING_KEY_ID` to the row known to match the existing private key.
+3. Set that id with `wrangler secret put SIGNING_KEY_ID`, then deploy.
+4. Call authenticated `GET /admin/signer`. A 200 response with
+   `{ "verified": true, "key_id": "..." }` proves the configured private key
+   matches the selected non-retired D1 row. Any mismatch fails closed.
+
+`deploy.sh` performs these steps automatically for the normal one-active-key
+case. It refuses an ambiguous multi-key migration or an active public row with
+no `PRIVATE_KEY`; it never creates a replacement private key for an existing
+public row. Set `REFLEX_ADMIN_TOKEN` before running it so the script can verify
+the coupled signer immediately after deployment; it refuses to migrate an
+existing signer without that verification credential.
+
 ## Issue your first license
 
 ```bash
@@ -106,7 +132,18 @@ python -m tether.admin.revoke_license \
     --reason "Refund processed"
 ```
 
-Customer's running deployment will fail its next heartbeat (within 24h) and refuse to serve.
+Customer's running deployment will fail its next heartbeat and refuse to serve.
+
+## Signed heartbeat contract
+
+The client sends a fresh unpadded-base64url 16-byte `request_nonce`. The Worker
+returns an Ed25519-signed `tether.license.heartbeat` v1 attestation containing
+the echoed nonce, `license_id`, active signing `key_id`, Unix `issued_at`,
+`valid_until`, and status (`active`, `expired`, or `revoked`; clients also
+understand `suspended`). Active validity is capped at 24 hours and at the
+license expiry. Released clients accept five minutes of clock skew, persist
+only verified attestations, and fail paid requests closed when the signed
+deadline plus skew elapses.
 
 ## Privacy posture
 
@@ -142,14 +179,22 @@ WHERE l.revoked_at IS NULL
   );
 ```
 
-## Key rotation (Phase 2 — not implemented yet)
+## Key rotation
 
 The schema supports key rotation via the `master_keys.retired_at` column, but the rotation endpoint (`POST /admin/rotate`) isn't built yet. When you need it:
 
-1. Generate a new Ed25519 keypair (new POST /admin/init variant)
-2. New licenses get signed with the new key
-3. Old key stays valid for verification (grace period)
-4. Customer-side bundled key gets a list of N trusted keys instead of one
-5. Eventually retire the old key when no licenses signed with it remain
+1. Generate the new Ed25519 pair through an audited rotation procedure and
+   insert its public half as a new non-retired `master_keys` row. Keep the old
+   row active.
+2. Ship both old and new public keys in `TRUSTED_PUBLIC_KEYS_B64` and release
+   that client trust overlap before changing the signer.
+3. Set `PRIVATE_KEY` and `SIGNING_KEY_ID` to the new coupled pair in one
+   controlled deployment window, then require authenticated `/admin/signer`
+   to return the new id with `verified: true`. Roll back both secrets together
+   if verification fails.
+4. Keep the old public key trusted throughout the maximum license/attestation
+   overlap.
+5. Retire the old D1 row and remove its client trust only after that overlap
+   has elapsed.
 
 Plan to revisit when you have ~50 active licenses or a security incident requires rotation.

infra/license-worker/deploy.sh

@@ -15,10 +15,11 @@
 #   5. Applies the schema
 #   6. Generates a strong ADMIN_TOKEN, prints it ONCE, then sets it as a Worker Secret
 #   7. Optionally sets SLACK_WEBHOOK_URL
-#   8. Deploys the worker
-#   9. Calls /admin/init to generate the Ed25519 keypair
-#   10. Sets PRIVATE_KEY as a Worker Secret
-#   11. Prints the public_key_b64 + key_id you need to paste into
+#   8. Migrates an existing active signer id before deploying, when present
+#   9. Deploys the worker
+#   10. Calls /admin/init for a fresh install, or verifies the existing signer
+#   11. Sets PRIVATE_KEY + SIGNING_KEY_ID for a fresh install
+#   12. Prints the public_key_b64 + key_id you need to paste into
 #       src/reflex/pro/_public_key.py + the worker URL for src/reflex/pro/activate.py
 #
 # After this script: edit those two Python constants, commit + push, and your
@@ -36,6 +37,34 @@ warn() { printf "${YELLOW}⚠${NC} %s\n" "$*"; }
 err()  { printf "${RED}✗${NC} %s\n" "$*" >&2; }
 info() { printf "${CYAN}→${NC} %s\n" "$*"; }
 
+select_active_signing_key() {
+    python3 -c '
+import json, os, sys
+value = json.load(sys.stdin)
+batches = value if isinstance(value, list) else [value]
+rows = [row for batch in batches for row in batch.get("results", [])]
+requested = os.environ.get("TETHER_SIGNING_KEY_ID", "")
+if requested:
+    rows = [row for row in rows if row.get("key_id") == requested]
+    if len(rows) != 1:
+        print(f"TETHER_SIGNING_KEY_ID={requested!r} is not an active D1 key", file=sys.stderr)
+        raise SystemExit(4)
+elif not rows:
+    raise SystemExit(3)
+elif len(rows) != 1:
+    print("multiple active D1 keys; set TETHER_SIGNING_KEY_ID to the key matching PRIVATE_KEY", file=sys.stderr)
+    raise SystemExit(4)
+row = rows[0]
+print(f"{row['"'"'key_id'"'"']}\t{row['"'"'public_key_b64'"'"']}")
+'
+}
+
+# Pure selector mode exercises the exact branch logic without Cloudflare access.
+if [ "${1:-}" = "--select-active-key" ]; then
+    select_active_signing_key
+    exit $?
+fi
+
 cd "$(dirname "$0")"
 
 # ─── 1. wrangler install ─────────────────────────────────────────────────────
@@ -129,7 +158,38 @@ else
     fi
 fi
 
-# ─── 8. Deploy the worker ────────────────────────────────────────────────────
+# ─── 8. Pre-bind an existing signer before deploying the stricter Worker ────
+ACTIVE_KEYS_JSON=$(wrangler d1 execute "$DB_NAME" --remote --json --command \
+    "SELECT key_id, public_key_b64 FROM master_keys WHERE retired_at IS NULL ORDER BY generated_at DESC")
+set +e
+ACTIVE_KEY_ROW=$(printf '%s' "$ACTIVE_KEYS_JSON" | select_active_signing_key)
+KEY_SELECT_STATUS=$?
+set -e
+if [ "$KEY_SELECT_STATUS" -eq 0 ]; then
+    IFS=$'\t' read -r KEY_ID PUBKEY_B64 <<< "$ACTIVE_KEY_ROW"
+    if [ -z "${ADMIN_TOKEN:-}" ]; then
+        if [ -n "${REFLEX_ADMIN_TOKEN:-}" ]; then
+            ADMIN_TOKEN="$REFLEX_ADMIN_TOKEN"
+        else
+            err "Existing-signer migration requires REFLEX_ADMIN_TOKEN so /admin/signer can verify the binding immediately after deploy. SIGNING_KEY_ID and Worker code were not changed."
+            exit 1
+        fi
+    fi
+    if ! wrangler secret list 2>/dev/null | grep -q '"PRIVATE_KEY"'; then
+        err "D1 has active key ${KEY_ID}, but PRIVATE_KEY is missing. Restore the matching private key; do not generate a replacement for the existing public row."
+        exit 1
+    fi
+    info "Binding existing active D1 key ${KEY_ID} before deploying signer enforcement..."
+    echo -n "$KEY_ID" | wrangler secret put SIGNING_KEY_ID
+    ok "Existing SIGNING_KEY_ID set before deploy"
+elif [ "$KEY_SELECT_STATUS" -eq 3 ]; then
+    info "No active D1 signing key; fresh install will initialize one after deploy"
+else
+    err "Could not select the existing signing key safely. If rotation overlap leaves multiple active rows, set TETHER_SIGNING_KEY_ID to the row that matches PRIVATE_KEY."
+    exit 1
+fi
+
+# ─── 9. Deploy the worker ────────────────────────────────────────────────────
 info "Deploying worker..."
 DEPLOY_OUT=$(wrangler deploy 2>&1)
 echo "$DEPLOY_OUT" | tail -10
@@ -140,7 +200,7 @@ if [ -z "$WORKER_URL" ]; then
 fi
 ok "Worker deployed at: ${WORKER_URL}"
 
-# ─── 9. /admin/init ──────────────────────────────────────────────────────────
+# ─── 10. /admin/init ─────────────────────────────────────────────────────────
 # Skip init if we don't have the ADMIN_TOKEN locally (it was already set as a
 # Secret on a prior run and we don't have the value anymore).
 if [ -z "${ADMIN_TOKEN:-}" ]; then
@@ -173,20 +233,36 @@ elif echo "$INIT_OUT" | grep -q "keypair_generated"; then
     KEY_ID=$(echo "$INIT_OUT" | python3 -c "import sys,json; print(json.load(sys.stdin)['key_id'])")
     ok "Keypair generated (id=${KEY_ID})"
 
-    # ─── 10. Set PRIVATE_KEY ──────────────────────────────────────────────────
+    # ─── 11. Set PRIVATE_KEY + SIGNING_KEY_ID ─────────────────────────────────
     info "Setting PRIVATE_KEY as Worker Secret (this is the only place it goes — never persisted elsewhere)..."
     echo -n "$PRIVKEY_B64" | wrangler secret put PRIVATE_KEY
     ok "PRIVATE_KEY set as Worker Secret"
+    echo -n "$KEY_ID" | wrangler secret put SIGNING_KEY_ID
+    ok "SIGNING_KEY_ID set to ${KEY_ID}"
     unset PRIVKEY_B64
 else
     err "Unexpected response from /admin/init:"
     echo "$INIT_OUT"
     exit 1
 fi
 
-# ─── 11. Print next-steps with values to paste ───────────────────────────────
-PUBKEY_FILE="../../src/reflex/pro/_public_key.py"
-ACTIVATE_FILE="../../src/reflex/pro/activate.py"
+# This authenticated check invokes loadSigner(), which signs and verifies a
+# fixed challenge. Matching ids alone do not prove PRIVATE_KEY is the private
+# half of the selected D1 public key.
+info "Verifying configured signer/private-key binding..."
+SIGNER_OUT=$(curl -sS "${WORKER_URL}/admin/signer" \
+    -H "Authorization: Bearer ${ADMIN_TOKEN}")
+VERIFIED_KEY_ID=$(printf '%s' "$SIGNER_OUT" | python3 -c \
+    "import sys,json; value=json.load(sys.stdin); print(value.get('key_id', '')) if value.get('verified') is True else raise SystemExit(1)")
+if [ "$VERIFIED_KEY_ID" != "$KEY_ID" ]; then
+    err "Signer verification failed for ${KEY_ID}: ${SIGNER_OUT}"
+    exit 1
+fi
+ok "Signer binding cryptographically verified (id=${VERIFIED_KEY_ID})"
+
+# ─── 12. Print next-steps with values to paste ───────────────────────────────
+PUBKEY_FILE="../../src/tether/pro/_public_key.py"
+ACTIVATE_FILE="../../src/tether/pro/activate.py"
 
 printf "\n\n${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}\n"
 printf "${GREEN}DEPLOY COMPLETE${NC}\n"
@@ -199,8 +275,7 @@ printf "${YELLOW}Public key:${NC} ${PUBKEY_B64}\n\n"
 printf "${CYAN}Next steps (3 manual edits, ~2 min):${NC}\n\n"
 
 printf "1. Update ${PUBKEY_FILE}:\n"
-printf "   BUNDLED_PUBLIC_KEY_B64 = \"${PUBKEY_B64}\"\n"
-printf "   BUNDLED_KEY_ID         = \"${KEY_ID}\"\n\n"
+printf "   Add \"${KEY_ID}\": \"${PUBKEY_B64}\" to TRUSTED_PUBLIC_KEYS_B64\n\n"
 
 printf "2. Update ${ACTIVATE_FILE}:\n"
 printf "   DEFAULT_LICENSE_ENDPOINT = \"${WORKER_URL}\"\n\n"

infra/license-worker/package.json

@@ -0,0 +1,7 @@
+{
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "test": "node --test test/*.test.js"
+  }
+}

infra/license-worker/test/deploy-helper.test.js

@@ -0,0 +1,48 @@
+import assert from "node:assert/strict";
+import { spawnSync } from "node:child_process";
+import { readFileSync } from "node:fs";
+import test from "node:test";
+
+function select(results, env = {}) {
+  return spawnSync("bash", ["deploy.sh", "--select-active-key"], {
+    cwd: new URL("..", import.meta.url),
+    env: { ...process.env, TETHER_SIGNING_KEY_ID: "", ...env },
+    input: JSON.stringify([{ results, success: true }]),
+    encoding: "utf8",
+  });
+}
+
+test("deploy helper distinguishes fresh install from one existing active key", () => {
+  const fresh = select([]);
+  assert.equal(fresh.status, 3);
+  assert.equal(fresh.stdout, "");
+
+  const existing = select([{ key_id: "key_existing", public_key_b64: "cHVi" }]);
+  assert.equal(existing.status, 0, existing.stderr);
+  assert.equal(existing.stdout.trim(), "key_existing\tcHVi");
+});
+
+test("deploy helper refuses ambiguous rotation rows unless id is explicit", () => {
+  const rows = [
+    { key_id: "key_old", public_key_b64: "b2xk" },
+    { key_id: "key_new", public_key_b64: "bmV3" },
+  ];
+  const ambiguous = select(rows);
+  assert.equal(ambiguous.status, 4);
+  assert.match(ambiguous.stderr, /multiple active D1 keys/);
+
+  const selected = select(rows, { TETHER_SIGNING_KEY_ID: "key_new" });
+  assert.equal(selected.status, 0, selected.stderr);
+  assert.equal(selected.stdout.trim(), "key_new\tbmV3");
+});
+
+test("existing signer pre-bind precedes deploy and fresh signer setup is verified", () => {
+  const script = readFileSync(new URL("../deploy.sh", import.meta.url), "utf8");
+  const deploy = script.indexOf('info "Deploying worker..."');
+  const prebind = script.indexOf('echo -n "$KEY_ID" | wrangler secret put SIGNING_KEY_ID');
+  const freshBind = script.lastIndexOf('echo -n "$KEY_ID" | wrangler secret put SIGNING_KEY_ID');
+  const verify = script.indexOf('${WORKER_URL}/admin/signer');
+  assert.ok(prebind !== -1 && prebind < deploy);
+  assert.ok(freshBind > deploy);
+  assert.ok(verify > freshBind);
+});

infra/license-worker/test/emit-non-ascii-activation.mjs

@@ -0,0 +1,63 @@
+import worker from "../worker.js";
+
+function b64(buffer) {
+  return Buffer.from(buffer).toString("base64");
+}
+
+const pair = await crypto.subtle.generateKey({ name: "Ed25519" }, true, ["sign", "verify"]);
+const privatePkcs8 = await crypto.subtle.exportKey("pkcs8", pair.privateKey);
+const publicRaw = await crypto.subtle.exportKey("raw", pair.publicKey);
+const expiresAt = new Date(Date.now() + 86400000).toISOString();
+const DB = {
+  prepare(sql) {
+    return {
+      bind() { return this; },
+      async first() {
+        if (sql.includes("FROM activation_codes")) {
+          return { license_id: "lic_unicode_live", expires_at: expiresAt, used: 0 };
+        }
+        if (sql.includes("SELECT license_json")) {
+          return { license_json: JSON.stringify({
+            license_version: 2,
+            license_id: "lic_unicode_live",
+            customer_id: "客户 É",
+            tier: "pro",
+            issued_at: new Date().toISOString(),
+            expires_at: expiresAt,
+            max_seats: 1,
+            hardware_binding: null,
+          }) };
+        }
+        if (sql.includes("FROM master_keys")) {
+          return { public_key_b64: b64(publicRaw) };
+        }
+        return null;
+      },
+      async run() { return { success: true, meta: { changes: 1 } }; },
+    };
+  },
+};
+const env = {
+  DB,
+  PRIVATE_KEY: b64(privatePkcs8),
+  SIGNING_KEY_ID: "key_unicode_live",
+};
+const response = await worker.fetch(new Request(
+  "https://worker.test/v1/activation/REFLEX-AAAA-BBBB-CCCC",
+  {
+    method: "POST",
+    headers: { "content-type": "application/json" },
+    body: JSON.stringify({
+      hardware_binding: {
+        gpu_uuid: "GPU-一",
+        gpu_name: "Éclair GPU",
+        cpu_count: 8,
+      },
+    }),
+  },
+), env);
+if (response.status !== 200) throw new Error(await response.text());
+process.stdout.write(JSON.stringify({
+  public_key_b64: b64(publicRaw),
+  response: await response.json(),
+}));