Repository navigation
P0: Run production Compose image/database smoke and close configuration audit #29
Description
Goal
Run a disposable production-like Compose image/database smoke for the final RC1 candidate after PR #95 disposition and close the original production-configuration audit with runtime evidence.
Current state — 2026-09-05
Current main remains:
b45170409543684a75579a13d7bdd6d8f8ffe8e8
Do not execute the #29 runtime proof against that SHA. PR #95 is unmerged but now ready for an explicit merge decision on exact reviewed head:
431d7da80c9bac56aa16423ab61a235a176b6fd8
Exact-head evidence:
- Codex review completed with no major issues;
- Ashbi Local CI passed isolated install, Prisma generation, TypeScript, ESLint, focused auth/navigation/security tests, and production Next build;
- GitGuardian passed with no secrets;
- all inline recovery review threads are resolved;
- PostgreSQL's documented OID assignment rules support the
FirstNormalObjectId(16384) system-schema contamination boundary; - GitHub-hosted
CI Build / Buildfailed before executable steps (steps:null), matching the known hosted-runner infrastructure incident rather than an observed application/test failure.
Note: Ashbi Local CI does not directly execute tests/backup-restore.test.ts; its assertions were separately checked against the exact branch text, while Local CI proves compile/lint/build compatibility.
PR #95 changes only recovery-focused tooling/docs/tests and hardens backup/restore correctness across upload-target reconciliation, archive/path safety, pristine-database checks, public-schema/ACL checks, prepared-transaction detection, connection isolation, restored attachment-file completeness, and high-OID user-created objects inside system-named schemas.
Candidate-binding rule
- PR fix: reconcile restored upload target during recovery drill #95 requires a separate explicit merge approval.
- After it merges, refetch the resulting
mainSHA. - That resulting SHA—not
b451704…and not the PR-head SHA—is the repository candidate for P0: Run production Compose image/database smoke and close configuration audit #29. - Build the candidate image from that exact SHA and capture its immutable digest.
- Bind every runtime evidence record to both the resulting
mainSHA and image digest.
Original finding
The historical production Compose configuration allowed weak PostgreSQL defaults, published PostgreSQL on the host, and lacked a valid DB healthcheck while the app depended on DB health.
Code-side state already fixed
- PostgreSQL credentials fail closed when required values are absent.
- PostgreSQL is not host-published.
- Database healthcheck uses configured DB identity.
- App waits for DB health, runs
prisma migrate deploy, reconciles uploads, then starts. - Readiness covers PostgreSQL, uploads read/write, and auth-email configuration.
- Verification-gated auth can be exercised in a disposable loopback smoke without claiming external-email evidence.
- Backup/restore/lifecycle tooling has explicit destructive-action guards.
Required runtime proof
Use an approved Docker-capable non-production/disposable host or isolated environment. For the post-#95 exact main candidate:
- prove required production configuration fails closed when secrets/DB identity are missing;
- build the exact candidate image and record its immutable digest;
- start the image with PostgreSQL 16 and freshly created disposable volumes/database;
- prove PostgreSQL is reachable only from the Compose network, not a host 5432 listener;
- prove migrations complete cleanly against the fresh DB;
- pass
/api/healthand/api/health/ready; - run authenticated disposable lifecycle smoke, including export/deletion and DB cleanup;
- create a disposable backup and record checksum/manifest evidence;
- restore into a separately named pristine disposable DB and empty upload directory using the hardened restore drill;
- prove restored attachment rows have corresponding regular files, migrations are current, upload reconciliation passes, and contaminated restore targets fail closed;
- record timestamps, exact SHA/digest, migration boundary and stage outcomes without secrets/private content;
- remove all disposable containers, networks, volumes, DBs, restored uploads, outbox residue and temporary artifacts created by the drill.
Hard stop conditions
Stop without attempting repair or production access if any of these are true:
- host/environment cannot be proven disposable and non-production;
- any existing customer/production DB, volume, container or upload path would be touched;
- exact checked-out SHA differs from the post-fix: reconcile restored upload target during recovery drill #95 frozen candidate;
- image digest cannot be recorded;
- restore target is not a newly created dedicated disposable database;
- restore upload directory is not explicitly disposable and empty;
- any command would require DNS, public deployment, external email, production credentials or customer data.
Recommended execution boundary — requires explicit action-time approval
- Target: isolated non-production Docker host/environment only.
- Customer impact: none.
- Incremental cost: $0 CAD if an existing approved host is used.
- No DNS, public deployment, customer data, external email, production DB or production volume access.
- The destructive restore/deletion smoke is authorized only for the explicitly named disposable resources in the approved execution request.
Acceptance criteria
- Original weak DB defaults are removed.
- PostgreSQL host publication is removed.
- DB healthcheck and app dependency are valid.
- Production startup runs migrations and upload reconciliation.
- Lifecycle/backup/restore tooling exists with fail-closed guards.
- PR fix: reconcile restored upload target during recovery drill #95 exact head completed intended Codex/local/security validation and all review threads are resolved.
- PR fix: reconcile restored upload target during recovery drill #95 disposition is complete and final candidate is rebound to resulting
mainSHA. - Exact candidate image + PostgreSQL start together in an isolated environment.
- Health/readiness and authenticated lifecycle pass.
- Backup/restore proof passes on disposable data.
- Exact-SHA/image evidence is attached with cleanup confirmation.
Approval boundary
Merging PR #95 is one explicit approval. Starting containers/volumes on an external host and running destructive disposable restore/deletion is another explicit action-time approval. Neither authorizes production deployment.
Source of truth
docker-compose.prod.yml; docker-compose.vps.yml; OPERATIONS.md; PR #95 reviewed head 431d7da80c9bac56aa16423ab61a235a176b6fd8; resulting post-merge main SHA once approved; roadmap #71.
Activity
camster91 commentedon Aug 28, 2026
PR #55 implements the remaining configuration hardening: required production credentials/secrets fail closed, PostgreSQL is no longer published on a host port, and the healthcheck uses configured database identity. Contract tests pass locally. Keeping this open until the built image and database can be smoke-tested together in an environment with Docker.
camster91 commentedon Sep 1, 2026
PR #55 exact SHA
ac9c90d2c34a5d052b1e3959fd5ee6f1b75f255enow passes the read-only dual-architecture image build: https://github.com/camster91/synapse/actions/runs/33498965202. This proves the image builds for amd64 and arm64, but it does not prove the Compose database/health/authenticated runtime acceptance criteria. The issue remains open until the built image and PostgreSQL are started together in an isolated container environment and the required smoke tests pass.[-][audit][devops] 🔴 critical: Prod docker-compose hardcodes syn-dev default DB credentials; lacks healthcheck[/-][+]P0: Run production Compose image/database smoke and close configuration audit[/+]camster91 commentedon Sep 4, 2026
Preparation update for #29: the original Compose security finding remains code-fixed on
main(required DB credentials, no PostgreSQL host port, DB healthcheck). PR #83 now also updates the disposable authenticated lifecycle so verification-gated registration will not break the runtime smoke.On PR #83 (
93e2efbf1cb733848fe81d41e7e2a0d344d415ad):local-sinkemail transport exists only for disposable loopback runs and requires exactAUTH_EMAIL_LOCAL_SINK=disposable-lifecycle-onlyopt-in plus loopback auth URL;Still missing before #29 can close: an actual Docker-capable isolated run of the exact candidate image with PostgreSQL,
/api/health+/api/health/ready, lifecycle smoke, and recorded non-sensitive evidence. No host, container, database, or production environment has been changed by this preparation.camster91 commentedon Sep 4, 2026
Post-merge RC1 status on 2026-09-04:
The relevant Compose/auth lifecycle preparation is now on
mainatb16ca80a28e96ad22a6d2cae3acb4f920466712d.Current code-side state:
prisma migrate deployand upload reconciliation;#29 remains open because the required disposable production-like Compose run has not been executed and retained as exact-SHA evidence on an approved Docker-capable host. No host/deployment/runtime mutation was performed in this update.
camster91 commentedon Sep 4, 2026
Final merged-code checkpoint (2026-09-04): current
mainisa09a91ac1ff90da23f3c12c397858ec0a460cd03. The production Compose security/configuration fixes, verification-gated auth compatibility, lifecycle safeguards, migration-aware CI contract, support tooling, operations evidence gates, legal inventory, and release-package validator are all merged.#29 still requires real runtime evidence before closure: start the exact candidate image and PostgreSQL together in a disposable isolated Docker environment, apply migrations, pass liveness/readiness and the authenticated lifecycle, prove backup + disposable restore, retain privacy-minimal exact-SHA evidence, and clean up all disposable resources. No Docker host was accessed in this session.
camster91 commentedon Sep 4, 2026
Codex/desktop handoff — disposable RC1 Compose proof
Do not execute external/container mutations until Cameron explicitly approves this exact #29 action.
Objective
Validate Synapse RC1 candidate
0c05e7037b5d07029401aad2d6515ef6c243a08cin an isolated non-production Docker environment and produce privacy-minimal runtime evidence for issue #29 without deploying publicly or touching production/customer data.Repository
https://github.com/camster91/synapseExact candidate
0c05e7037b5d07029401aad2d6515ef6c243a08cSource files / contracts
docker-compose.prod.ymldocker-compose.vps.ymlDockerfileOPERATIONS.mdOPERATIONS-RC1.mdscripts/authenticated-lifecycle-smoke.tsscripts/backup.shscripts/restore-drill.shscripts/operations-preflight.tsscripts/validate-operations-evidence.tsScope
5432listener.prisma migrate deploy, upload reconciliation,/api/health, and/api/health/ready.Acceptance criteria
Stop conditions
Stop without continuing if:
Rollback / cleanup
This exercise must create only disposable resources. Cleanup is the rollback: remove the exact Compose project, volumes, networks, synthetic DBs, outbox files, restore targets, and temporary backup artifacts created by this run. Do not remove pre-existing resources.
Evidence output
Return a concise report with: exact SHA, image identity/digest, environment class (
isolated non-production), each stage PASS/FAIL, migration head, timestamps/durations, cleanup confirmation, and any blocker. Redact all secret/private values.camster91 commentedon Sep 4, 2026
Execution update — #29 was actually launched against the frozen candidate
0c05e7037b5d07029401aad2d6515ef6c243a08cusing a temporary branch-only workflow (ops/rc1-runtime-proof) that targets the existing[self-hosted, Linux, X64, ashbi-vps]runner.Workflow run: https://github.com/camster91/synapse/actions/runs/33885206182
Workflow commit:
874104bd4d30a73b6716dff6f472ac9ae0c08e5dSafety design of the launched proof:
mainis unchanged;container_nameentries are removed only in a temporary local copy;Current blocker: the job is queued and has not executed any steps. This is not an application failure. The same
ashbi-vpsself-hosted runner path is also currently leaving a recenthuman-healthverification workflow queued, which indicates the runner is offline/unavailable or otherwise not accepting work account-wide.#29 remains open. No container, database, volume, production service, DNS, customer data or external email has been touched because the runner has not started the job.
camster91 commentedon Sep 4, 2026
Runner blocker update — 2026-09-04
The disposable RC1 proof is still queued as GitHub Actions run
33885206182with zero executed steps. Cross-repo comparison now confirms Human Health run33879553019is also queued on the same[self-hosted, Linux, X64, ashbi-vps]label, so this is not currently evidence of a Synapse workflow/runtime failure.The central runner/account incident is tracked in
camster91/ashbi-platform#399, which now contains a read-only SSH diagnostic to discover the exact runner process/systemd unit without guessing a service name. No runner restart/re-registration, billing change, permission change, production service change, or VPS mutation has been authorized or performed.#29 remains execution launched / awaiting runner availability. When
ashbi-vpsresumes, run33885206182is already prepared to test exact candidate0c05e7037b5d07029401aad2d6515ef6c243a08cwith disposable containers/DB/volumes, loopback-only app exposure, backup/restore, privacy-minimal evidence, and unconditional cleanup.camster91 commentedon Sep 4, 2026
Correction to the runner blocker classification — 2026-09-04
Further repository evidence found after the previous comment:
camster91/magic-type-quest/docs/ASHBI-CI.mddocuments the knownashbi-vps-magic-type-questrunner as repository-scoped to Magic Type Quest. It uses/opt/actions-runner, service usermagicci, and systemd unitactions.runner.camster91-magic-type-quest.ashbi-vps-magic-type-quest.service.That runner is not eligible to accept Synapse jobs merely because it also has the
ashbi-vpslabel. Therefore queued run33885206182should be classified as no proven eligible Synapse self-hosted runner, not as proof that the VPS/runner service is offline.The disposable proof workflow remains technically prepared and still checks exact candidate
0c05e7037b5d07029401aad2d6515ef6c243a08c, but it cannot execute until one of these is explicitly approved and configured:Do not broaden or repurpose the privileged Magic Type Quest runner for Synapse by default. Runner registration/permission changes are separate approval-gated infrastructure mutations.
No VPS service, runner registration, production system, or customer data was changed.
camster91 commentedon Sep 4, 2026
Decision-ready self-hosted runner option — 2026-09-04 (proposal only; not executed)
Current GitHub and repository documentation support a least-privilege way to unblock this proof without weakening the existing Magic Type Quest runner:
Proposed Synapse-specific runner
camster91/synapseonly.ashbi-vps-synapse.ashbi-vps,synapsein addition to GitHub's defaultself-hosted,Linux,X64.synapseci(do not reusemagicci)./opt/actions-runner-synapse(do not reuse/opt/actions-runner, which contains the Magic Type Quest runner registration/workspace).dockergroup must be treated as root-equivalent privileged access and the runner must remain private-repo/repository scoped.GitHub's current official docs confirm repository-level runners are supported; adding one requires repository-owner/admin access plus a time-limited registration token. On Linux, after registration,
svc.shcan install/start/status the runner as a systemd service.Sources:
Existing Ashbi pattern for comparison:
camster91/magic-type-quest/docs/ASHBI-CI.mddocuments a separate repository-scoped runner with its own service user and systemd unit. That registration must remain isolated.Installation safety requirements
/optdirectories, disk/memory, Docker health, and existing runner units first..credentials,.runner, tokens, or_workcontents.https://github.com/camster91/synapsewith the Synapse-specific name/labels/work directory.Listening for Jobs, then allow the prepared disposable workflow to run.Rollback/removal
If registration or service startup is unhealthy, stop/uninstall only the newly created Synapse runner service and remove the Synapse registration using a short-lived GitHub removal token. Do not alter the Magic Type Quest runner or production application services.
Current approval boundary
Creating the user/directory, adding Docker-group access, generating a runner registration token, registering the runner, or installing/starting its service are privileged VPS/GitHub mutations and require explicit approval for this exact setup. None have been performed.
camster91 commentedon Sep 4, 2026
Exact disposable-run preflight — candidate
0c05e7037b5d07029401aad2d6515ef6c243a08cUse this checklist at action time. It is preparation only; it does not authorize running against any existing VPS/production environment.
Required isolation
0c05e7037b5d07029401aad2d6515ef6c243a08c.Disposable Compose contract
docker-compose.prod.ymlrequiresDATABASE_URL,NEXTAUTH_URL,NEXTAUTH_SECRET,SYNAPSE_ENCRYPTION_KEY,POSTGRES_USER,POSTGRES_PASSWORD, andPOSTGRES_DB; PostgreSQL has no host5432publication; app startup isprisma migrate deploy→ uploads reconciliation → server.For #29 only, use the loopback local email sink:
EMAIL_PROVIDER=local-sinkAUTH_EMAIL_LOCAL_SINK=disposable-lifecycle-onlyNEXTAUTH_URL=http://127.0.0.1:3000AUTH_PUBLIC_URLis set, it must also be loopback.Never use that sink on a beta/public hostname; it does not satisfy #60.
Lifecycle proof
Run against the same disposable DB/HTTP target with:
LIFECYCLE_SMOKE_BASE_URL=http://127.0.0.1:3000LIFECYCLE_SMOKE_CONFIRM=create-and-delete-disposable-accountDATABASE_URL=<same disposable database>Do not set
LIFECYCLE_SMOKE_ALLOW_REMOTE=truefor this #29 exercise.Required outcome: registration/onboarding/export/wrong-password deletion/correct deletion pass and the DB assertion reports zero remaining disposable account/auth rows.
Backup proof
SYNAPSE_BACKUP_DIRmust be an absolute private directory and cannot be/or$HOME;SYNAPSE_UPLOAD_DIRmust name the disposable uploads directory. Backup must producedatabase.dump,uploads.tar.gz,manifest.txt, andchecksums.sha256.This repository backup is not encrypted/off-site by itself; do not claim #61 from this exercise.
Restore proof
Restore only into a newly created empty disposable DB and empty absolute uploads directory with:
RESTORE_CONFIRM=EMPTY_DISPOSABLE_DATABASESYNAPSE_BACKUP_PATH=<one exact backup directory>RESTORE_DATABASE_URL=<empty disposable restore DB>RESTORE_UPLOAD_DIR=<empty disposable absolute directory>The script verifies checksums/archive paths, restores DB/uploads, checks Prisma migration status, and reconciles uploads.
Required evidence
Record only:
Never record passwords, connection strings, raw tokens, generated account email/password, secret values, or file contents.
Cleanup gate
Before declaring #29 complete, remove only the explicitly named disposable containers, networks, volumes, source/restore DBs, backup directory, restore uploads directory, and any temporary files. Confirm no listener/volume/DB from the exercise remains.
camster91 commentedon Sep 5, 2026
Execution handoff remains valid conceptually but is temporarily superseded by PR #95 recovery correctness work. Do not run the disposable Docker proof until #95 is reviewed and explicitly merged, then refresh the exact candidate SHA and use the merged recovery scripts. The intended topology remains: loopback app, Compose-internal PostgreSQL only, lifecycle smoke against the same disposable DB, backup of the same disposable uploads/database, restore into a separately named empty disposable DB/uploads target, privacy-minimal evidence, and complete cleanup.
camster91 commentedon Sep 5, 2026
#29 recovery blocker update — final review found four additional fail-closed defects on PR #95: archive link scanning can be bypassed by
grep -q+pipefail; hard-linked upload files can produce backups the restore gate rejects; the empty-database preflight does not cover all user-defined object classes; and backup/restore root symlink checks can be bypassed by trailing separators or linked parent components. #29 remains held. These are being corrected on the same focused PR before it is marked ready. Do not execute the external Docker/restore proof against currentmainor an intermediate #95 head.camster91 commentedon Sep 5, 2026
#29 execution hold — current recovery prerequisite is PR #95 exact head
729094c06e8085310e3df839a5e44b025b873820.Do not run the disposable backup/restore proof against current
mainyet. The old recovery tooling can reconcile the wrong upload target and lacked several fail-closed archive/database guards. PR #95 now contains the focused recovery corrections and remains draft pending exact-head Codex + Ashbi Local CI reconciliation.After #95 is accepted and merged, #29 must bind to the resulting new
mainSHA and immutable image identity before any external Docker execution. No Docker host, production data, or external environment is authorized by this comment.camster91 commentedon Sep 5, 2026
#29 prerequisite update — PR #95 is now frozen at exact head
57ece7fc80bbc3ac125131ffc03fc66598b9d7f7. This head has passed Ashbi Local CI and GitGuardian. Fresh Codex review is still running after the final public-schema and concurrent-DDL restore guards. Keep #29 execution on hold until #95 is fully reviewed and merged; then bind the disposable Docker proof to the resulting newmainSHA/image.camster91 commentedon Sep 5, 2026
Recovery prerequisite update — 2026-09-05
PR #95 is now ready for review on exact head
f8de3ee3180a455634a3261f82020675795e85fe.Exact-head evidence:
CI Build / Build: failed before executing any steps (steps: null), consistent with the existing hosted-runner incident rather than observed application failure;The #29 runtime proof must not run against current
mainyet. Next sequence:mainSHA;This update does not authorize merge, Docker execution, production access, deployment, or destructive runtime work.
camster91 commentedon Sep 6, 2026
RC1 recovery prerequisite update: current
mainremainsb45170409543684a75579a13d7bdd6d8f8ffe8e8. PR #95 is now ready for merge review at exact head1f0877757c79e16ebdc0b91b68f1ecf8059449c3. On this exact head, the manual Codex review completed with no remaining major issues, Ashbi Local CI passed isolated install/Prisma/TypeScript/ESLint/focused tests/production build, GitGuardian passed, and all inline recovery findings are resolved. HostedCI Build / Buildstill fails before executing steps under the known runner incident. Do not run #29 against current main: merge #95 only with separate owner approval, establish the resulting newmainSHA/image, then bind the disposable Docker/backup/restore proof to that exact artifact.